It is important that people understand that asteroids and comets can be dangerous, but that the danger is small and, most importantly, predictable. Large asteroids don't appear out of nowhere to hit the Earth. They orbit the sun in well-defined orbits, just like the Earth and other planets do. If we can see an asteroid, we can calculate where it is going for hundreds of years into the future. So, as long as we make serious efforts to look for asteroids, surprise impacts by large asteroids will not occur.
Comets are a slightly different matter.
Comets spend most of their time far from the sun, and plunge into the inner solar system for a few months before returning to the outer reaches of the solar system. For this reason, it is usually hard to spot most comets very many years into the future. Comets such as Hale-Bopp drop in unexpectedly. But, again, we are in luck. The solar system is an absolutely huge place, and the Earth is a very tiny target.