Searchine uses self made spider technology to crawl your site and analyse the documents structure and text. We will take notice of hyperlinks to other pages / documents and follow those as well. The crawler will take notice of noindex and nofollow requests as it will process robots / XML sitemaps and behave just like any other crawler.