In most cases, only the 10-15 seconds of shaking that originate from the part of the fault nearest you will be very strong. Each cycle of shaking stresses buildings and adds to the damage. Because this is cumulative, the most damage happens at the very end of an earthquake. Farther away, the higher frequencies will have died away and you will feel a rolling motion. The amount of damage to a building does not depend solely on how hard it is shaken.
Different structures respond differently to the various frequencies.
In general, smaller buildings such as houses respond more to the higher frequencies, so proximity to the fault is a very important factor. Larger structures such as high-rises and bridges are more responsive to lower frequencies and will be more noticeably affected by the largest earthquakes. Just as sound carries differently in water and in air, seismic waves travel at different speeds in different types of rock. Passing from rock to soil, the waves slow down but get bigger. A soft, loose soil will shake more intensely than hard rock at the same distance from the same earthquake.