When it's almost your 2,000th birthday, you're lucky to still be standing, let alone to be so popular. There's a number of reasons for the Colosseum's damaged appearance on top of the effects of time, from fire and earthquakes to opportunistic builders capitalizing on spolia (repurposing old stone for new structures).
The Colosseum did manage to stay in its prime for almost two centuries before disaster really struck. In 217 AD, the Colosseum's wooden upper level was badly damaged by fire, and in 443 AD it sustained more damage from a major earthquake.
On top of that, as Christianity rose, the popularity of gladiatorial battles declined and after 435 AD there's no more mention of them in the Colosseum. The arena still remained in use for animal hunts, but its heyday had long since passed. Another earthquake in 1349 AD saw more damage done, with the outer south side collapsing.