This spectrum is predicted by theoretical explanations for the origin of fluctuations in the early universe precisely because it does not pick out any later time as special. There is nothing in general relativity to say that this cannot happen, but any theory in which it does happen carries the burden of explaining when it happens. But the galaxy clustering pattern we observe is not consistent with a simple extrapolation of the scale-invariant spectrum to smaller scales. If radiation is included, two varieties of fluctuations can be distinguished. In adiabatic fluctuations, the densities of matter and radiation vary in synchrony, while in isothermal fluctuations only the matter density varies, while the radiation density is constant.
Adiabatic fluctuations arise rather naturally in models of the early universe.
Thus it seems plausible to assume that the primordial fluctuations were adiabatic. In the standard CDM model we assume that the dark matter is composed of particles with initially low velocity dispersions. In the HDM model the dark matter is assumed to be composed of neutrinos with masses of a few tens of eV . While the temperature of the universe exceeds the neutrino rest mass, the neutrinos have relativistic velocities and effectively smear out fluctuations on scales smaller than the horizon. The CDM models can also produce filaments, walls, and voids which resemble those observed, and many other predictions can be reconciled with the observations if galaxies are assumed to form at high peaks of the density field.