Analyses of a site's purpose, history and policies are beyond the capabilities of a crawler program. Another drawback of automated indexing is that most search engines recognize text only. The intense interest in the Web, though, has come about because of the medium's ability to display images, whether graphics or video clips. Some research has moved forward toward finding colors or patterns within images. At the same time, the way information is structured on the Web is changing so that it often cannot be examined by Web crawlers.
Many Web pages are no longer static files that can be analyzed and indexed by such programs.
In many cases, the information displayed in a document is computed by the Web site during a search in response to the user's request. The site might assemble a map, a table and a text document from different areas of its database, a disparate collection of information that conforms to the user's query. A newspaper's Web site, for instance, might allow a reader to specify that only stories on the oil-equipment business be displayed in a personalized version of the paper. The database of stories from which this document is put together could not be searched by a Web crawler that visits the site. A growing body of research has attempted to address some of the problems involved with automated classification methods.