It might be easier to turn this around and ask why matter and radiation interacted in the first place.
Neutral atoms interact only with photons of highly specific energies.
Free electrons, on the other hand, interact very strongly with photons. Before recombination, there were plenty of free electrons around to scatter photons. At the densities of the early universe, both in matter and in radiation, the constant interaction of matter and photons kept them in thermal equilibrium with one another. A photon could not travel very far before encountering another free electron and being scattered again. Once the free electrons combined with the nuclei to create atoms, very few particles remained to scatter photons.