rapid7/metasploit-framework — pull request #1560 by sempervictus (2013-03-07 19:00:29+00:00)
Title: Stop crawler from reporting 404 pages

The crawler currently reports web pages which have a 404 response.
This means that the default test targets come out as "pages" in the
DB when in fact the're just 404 reponses. It is thus possible to
have pages reported for a website which is nothing more than a bare
HTTP server. Since we can always get a 404 response by issuing a
request to a random URI this behavior appears detrimental, or at
least pointless.

If the behavior does serve a purpose, then it may be prudent to
create a datastore conditional to switch it on if required.