Some excited states cannot emit light for reasons of symmetry. Light is by nature an asymmetric thing. So only a change in the atom that is sufficiently asymmetric can produce light. In this case it cannot find any state of lower energy and different symmetry, and light emission is impossible. Only one out of every 20 million atoms in air is xenon.
Xenon is so rare, because it is the heaviest noble gas.
Other noble gases are krypton, and argon. Neon and helium are lighter still, and in fact so light that they escape from the atmosphere into outer space and are rare on Earth, although not in the universe. Radon is a radioactive noble gas, whose nucleus falls apart in a few minutes.