The majority of the 20,000 asteroids in our data banks remain on well-behaved paths forever circuiting the sun in the main belt. In 1873, however, one was found on a trajectory that brought it in to cross the orbit of Mars. It was obvious that an asteroid-planet collision was possible. Fundamentally this was not a new idea, because comets had been known to have planet-crossing orbits for some time. But there are few comets, whereas 19th-century astronomers were finding a host of asteroids.
Comets are easy to find because they are bright, being surrounded by vast clouds of water vapour.
The recognition of dark asteroids required the development of better telescopes, and then photography led to a blossoming of the discovery rate. Astronomers' fears were confirmed when in 1898 the first earth-approaching asteroid was spotted. This is 433 Eros. Then, in 1932, the first possible Earth-impacting asteroids were found. Over subsequent decades several more were discovered, essentially by accident on celestial photographs obtained with powerful telescopes.