Fuel economy, performance and emissions begin to deteriorate and get progressively worse until the dirty filter is replaced.
A slightly dirty filter actually cleans more efficiently than a brand new filter. That's because the debris trapped by the filter element helps screen out smaller particles that try to get through. But eventually every filter reaches the point where it causes enough of a pressure drop to restrict airflow.
Removing the filter and holding it up to a light will show you how dirty it is. If it's really caked with dirt, it obviously needs to be replaced. Trying to shake or blow the dirt out is a waste of time because too much of it will be embedded in the filter fibers.