Well, the sky won't fall, but your home comforts will definitely suffer, and not just due to poor air quality. While the end result of using a clean air filter is purer indoor air quality, your air filter’s primary role is protecting the inner components of your heating and cooling system.
Air conditioners are made of highly sensitive parts that draw in air, cool it or heat it, and push it back out into your home. If those parts were to become clogged with dust, pet hair, and other airborne particles that your AC filters usually protect it from, your HVAC equipment will be unable to work properly. A dirty air filter will be unable to filter out contaminants and will lead to the particles clogging up your air conditioning system.