Absorption is due to interactions between photons and electrons. The outermost electrons in atoms are often lost to ionization. Innermost electrons are too tightly bound to the nucleus to be excited by anything short of X-rays. Most orbital electrons are paired and too tightly bound to be affected by visible light. Only unpaired electrons are loosely enough bound to interact with visible light photons.
Individual atoms emit and absorb characteristic wavelengths of light.
In their most common ionization states, only the transition metals have unpaired electrons. These are thus common color-formers in minerals. We can liken electron excitation to a pinball game. The paddles supply energy, the way a photon does to an electron, and the various targets on the board correspond to the excitation levels of the electron. Materials that have no excitation levels in the range of energies corresponding to visible light won't interact with visible light and will be colorless.