rapid7/metasploit-framework — pull request #1000 by tasos-r7 (2012-11-01 20:52:32+00:00)
Title: Updated support for Web modules and analysis techniques

This request contains core classes for the new Web-related functionality (along with a few updates to existing code) such as:

## Anemone

Has been hacked to support modular path extractors and dirbusting.

### Path extractors (/lib/anemone/extractors)

Single-responsibility components which when passed an Anemone::Page return an array of paths.

### Dirbuster (/lib/anemone/extractors/dirbuster.rb /lib/anemone/extractors/dirbuster/directories)

Implemented as a path extractor, feeds the crawler a bunch of directory names.

## Auxiliary::Web (/lib/msf/core/auxiliary/web.rb)

Mixin which provides a bunch of helpers to the web modules for:
* logging vulns
* fingerprinting
* detection of server-side resources
* HTTP requests (Web::HTTP)

## Auxiliary::Web::HTTP (/msf3/lib/msf/core/auxiliary/web/http.rb)

Provides a more sensible interface for the task at hand, much more high-level than Rex::Proto::Http::Client, which it wraps into a pretty bow.
Takes into account query params and merges them with explicit get params, handles redirects, can detect custom 404 pages, etc.
Generally makes life much easier...

## Auxiliary::Web::Target (/msf3/lib/msf/core/auxiliary/web/target.rb)

Holds target info like Mdm::Service, Mdm::WebSite, along with a bunch of helper attributes (like vhost, host, port, etc.).

Most interesting attributes are:
* #forms -- Array of form elements (Auxiliary::Web::Form)
* #paths -- Array of path elements (Auxiliary::Web::Path)
* #auditable -- Merge of the above two.

## Auxiliary::Web::Form (/msf3/lib/msf/core/auxiliary/web/form.rb)

Represents a form element and holds its method, action, inputs and the original Mdm::WebForm model.
Also provides a bunch of domain relevant helpers like query parsing, encoding, permutation generation and more.

Inherits from Auxiliary::Web::Fuzzable.

## Auxiliary::Web::Path (/msf3/lib/msf/core/auxiliary/web/path.rb)

Same deal as Auxiliary::Web::Form apart from a few differences in their guts since even though their
API needs to be identical their internal representation differs wildly.

Inherits from Auxiliary::Web::Fuzzable.

## Auxiliary::Web::Fuzzable (/msf3/lib/msf/core/auxiliary/web/fuzzable.rb)

Doesn't do much, provides a way to #submit the elements and #fuzz them using their permutations.
Also provides comparison and #dup methods.

The cool stuff is that it dynamically loads and includes analysis modules (Ruby modules, not framework components) under its namespace.

## Auxiliary::Web::Analysis (/msf3/lib/msf/core/auxiliary/web/analysis/)

Namespace under which all analysis techniques reside.

### Auxiliary::Web::Analysis::Taint (/pro/msf3/lib/msf/core/auxiliary/web/analysis/taint.rb)

Provides Taint analysis support to Fuzzzable elements, basically submits their permutations and uses the fuzzer's (web module's) #find_proof to determine whether or not a response matches a vulnerability signature.

### Auxiliary::Web::Analysis::Differential (/msf3/lib/msf/core/auxiliary/web/analysis/differential.rb)

Provides Differential analysis using fault/boolean injection pairs to determine whether the back-end logic can be manipulated and uses text refinement (Rex::Text.refine) to remove irrelevant/out-of-context noise data in order to make response comparisons possible.

### Auxiliary::Web::Analysis::Timing (/pro/msf3/lib/msf/core/auxiliary/web/analysis/timing.rb)

Provides timing-attack/response-time analysis. Injects delay increments and ensures responsiveness between injections to make sure that the response times are manipulatable and not just dumb luck or a dead server.

## Changes to core libs

Added:

* ```Rex::Text.to_words``` -- Splits a ```String``` to an ```Array``` of words.
* ```Rex::Text.refine``` -- Removes noise from 2 ```Strings``` and return a refined ```String``` version.