Therefore no one indexes every single web page. What works for you on one engine may not work well on another. Many links do not work or have other errors. Some are general, like this one for botany. Internet Directory for Botany - all links Others are more restricted, like this one for butterflies.
Providers also collect links.
Sometimes, they are included with the information. Other times they are all grouped together. It can take a lot of time to gather valid links, and periodically check that they still work. Positive feedback via email is welcome. Their work can save you lots of time and effort.