Currently, it does seem that the number of competitors drops off at the junior and senior level. The age of the competitors may also be a factor in the large number of intermediate and novice skaters. The junior and senior level skaters tend to be older and may be involved with other activities that take away from skating. The Intermediate and Novice levels have always been large for as long as I can recall. This is probably due to the fact that the skaters are remaining at these levels while they work on obtaining consistant double axels and triple jumps.
Competitions definitely keep getting stiffer.
In our area, one's not competitive in Intermediate without at least a clean double axel. That's definitely not a standard, though.