BUG/CRITICAL: using HTTP information in tcp-request content may crash the process

During normal HTTP request processing, request buffers are realigned if
there are less than global.maxrewrite bytes available after them, in
order to leave enough room for rewriting headers after the request. This
is done in http_wait_for_request().

However, if some HTTP inspection happens during a "tcp-request content"
rule, this realignment is not performed. In theory this is not a problem
because empty buffers are always aligned and TCP inspection happens at
the beginning of a connection. But with HTTP keep-alive, it also happens
at the beginning of each subsequent request. So if a second request was
pipelined by the client before the first one had a chance to be forwarded,
the second request will not be realigned. Then, http_wait_for_request()
will not perform such a realignment either because the request was
already parsed and marked as such. The consequence of this, is that the
rewrite of a sufficient number of such pipelined, unaligned requests may
leave less room past the request been processed than the configured
reserve, which can lead to a buffer overflow if request processing appends
some data past the end of the buffer.

A number of conditions are required for the bug to be triggered :
  - HTTP keep-alive must be enabled ;
  - HTTP inspection in TCP rules must be used ;
  - some request appending rules are needed (reqadd, x-forwarded-for)
  - since empty buffers are always realigned, the client must pipeline
    enough requests so that the buffer always contains something till
    the point where there is no more room for rewriting.

While such a configuration is quite unlikely to be met (which is
confirmed by the bug's lifetime), a few people do use these features
together for very specific usages. And more importantly, writing such
a configuration and the request to attack it is trivial.

A quick workaround consists in forcing keep-alive off by adding
"option httpclose" or "option forceclose" in the frontend. Alternatively,
disabling HTTP-based TCP inspection rules enough if the application
supports it.

At first glance, this bug does not look like it could lead to remote code
execution, as the overflowing part is controlled by the configuration and
not by the user. But some deeper analysis should be performed to confirm
this. And anyway, corrupting the process' memory and crashing it is quite
trivial.

Special thanks go to Yves Lafon from the W3C who reported this bug and
deployed significant efforts to collect the relevant data needed to
understand it in less than one week.

CVE-2013-1912 was assigned to this issue.

Note that 1.4 is also affected so the fix must be backported.
MEDIUM: acl: remove flag ACL_MAY_LOOKUP which is improperly used

This flag is used on ACL matches that support being looking up patterns
in trees. At the moment, only strings and IPs support tree-based lookups,
but the flag is randomly set also on integers and binary data, and is not
even always set on strings nor IPs.

Better get rid of this mess by only relying on the matching function to
decide whether or not it supports tree-based lookups, this is safer and
easier to maintain.
MEDIUM: samples: use new flags to describe compatibility between fetches and their usages

Samples fetches were relying on two flags SMP_CAP_REQ/SMP_CAP_RES to describe
whether they were compatible with requests rules or with response rules. This
was never reliable because we need a finer granularity (eg: an HTTP request
method needs to parse an HTTP request, and is available past this point).

Some fetches are also dependant on the context (eg: "hdr" uses request or
response depending where it's involved, causing some abiguity).

In order to solve this, we need to precisely indicate in fetches what they
use, and their users will have to compare with what they have.

So now we have a bunch of bits indicating where the sample is fetched in the
processing chain, with a few variants indicating for some of them if it is
permanent or volatile (eg: an HTTP status is stored into the transaction so
it is permanent, despite being caught in the response contents).

The fetches also have a second mask indicating their validity domain. This one
is computed from a conversion table at registration time, so there is no need
for doing it by hand. This validity domain consists in a bitmask with one bit
set for each usage point in the processing chain. Some provisions were made
for upcoming controls such as connection-based TCP rules which apply on top of
the connection layer but before instantiating the session.

Then everywhere a fetch is used, the bit for the control point is checked in
the fetch's validity domain, and it becomes possible to finely ensure that a
fetch will work or not.

Note that we need these two separate bitfields because some fetches are usable
both in request and response (eg: "hdr", "payload"). So the keyword will have
a "use" field made of a combination of several SMP_USE_* values, which will be
converted into a wider list of SMP_VAL_* flags.

The knowledge of permanent vs dynamic information has disappeared for now, as
it was never used. Later we'll probably reintroduce it differently when
dealing with variables. Its only use at the moment could have been to avoid
caching a dynamic rate measurement, but nothing is cached as of now.
MINOR: log: indicate it when some unreliable sample fetches are logged

If a log-format involves some sample fetches that may not be present at
the logging instant, we can now report a warning.

Note that this is done both for log-format and for add-header and carefully
respects the original fetch keyword's capabilities.
MEDIUM: samples: move payload-based fetches and ACLs to their own file

The file acl.c is a real mess, it both contains functions to parse and
process ACLs, and some sample extraction functions which act on buffers.
Some other payload analysers were arbitrarily dispatched to proto_tcp.c.

So now we're moving all payload-based fetches and ACLs to payload.c
which is capable of extracting data from buffers and rely on everything
that is protocol-independant. That way we can safely inflate this file
and only use the other ones when some fetches are really specific (eg:
HTTP, SSL, ...).

As a result of this cleanup, the following new sample fetches became
available even if they're not really useful :

  always_false, always_true, rep_ssl_hello_type, rdp_cookie_cnt,
  req_len, req_ssl_hello_type, req_ssl_sni, req_ssl_ver, wait_end

The function 'acl_fetch_nothing' was wrong and never used anywhere so it
was removed.

The "rdp_cookie" sample fetch used to have a mandatory argument while it
was optional in ACLs, which are supposed to iterate over RDP cookies. So
we're making it optional as a fetch too, and it will return the first one.
MINOR: backend: rename sample fetch functions and declare the sample keywords

The following sample fetch functions were only usable by ACLs but are now
usable by sample fetches too :

  avg_queue, be_conn, be_id, be_sess_rate, connslots, nbsrv,
  queue, srv_conn, srv_id, srv_is_up, srv_sess_rate

The fetch functions have been renamed "smp_fetch_*".
MINOR: frontend: rename sample fetch functions and declare the sample keywords

The following sample fetch functions were only usable by ACLs but are now
usable by sample fetches too :

      fe_conn, fe_id, fe_sess_rate

The fetch functions have been renamed "smp_fetch_*".
MINOR: listener: rename sample fetch functions and declare the sample keywords

The following sample fetch functions were only usable by ACLs but are now
usable by sample fetches too :

          dst_conn, so_id,

The fetch functions have been renamed "smp_fetch_*".
MEDIUM: http: unify acl and sample fetch functions

The following sample fetch functions were only usable by ACLs but are now
usable by sample fetches too :

    cook, cook_cnt, cook_val, hdr_cnt, hdr_ip, hdr_val, http_auth,
    http_auth_group, http_first_req, method, req_proto_http, req_ver,
    resp_ver, scook, scook_cnt, scook_val, shdr, shdr_cnt, shdr_ip,
    shdr_val, status, urlp, urlp_val,

Most of them won't bring much benefit at the moment, or are even aliases of
existing ones, however they'll be needed for ACL->SMP convergence.

A new val_usr() function was added to resolve userlist names into pointers.

The http_auth_group ACL forgot to make its first argument mandatory, so
there was a check in cfgparse to report a vague error. Now that args are
correctly parsed, let's report something more precise.

All urlp* ACLs now support an optional 3rd argument like their sample
counter-part which is the optional delimiter.

The fetch functions have been renamed "smp_fetch_*".

Some args controls on the sample keywords have been relaxed so that we
can soon use them for ACLs :

  - cookie now accepts to have an optional name ; it will return the
    first matching cookie if the name is not set ;
  - same for set-cookie and hdr
MINOR: session: rename sample fetch functions and declare the sample keywords

The following sample fetch functions were only usable by ACLs but are now
usable by sample fetches too :

  sc1_bytes_in_rate, sc1_bytes_out_rate, sc1_clr_gpc0, sc1_conn_cnt,
  sc1_conn_cur, sc1_conn_rate, sc1_get_gpc0, sc1_http_err_cnt,
  sc1_http_err_rate, sc1_http_req_cnt, sc1_http_req_rate, sc1_inc_gpc0,
  sc1_kbytes_in, sc1_kbytes_out, sc1_sess_cnt, sc1_sess_rate, sc1_trackers,
  sc2_bytes_in_rate, sc2_bytes_out_rate, sc2_clr_gpc0, sc2_conn_cnt,
  sc2_conn_cur, sc2_conn_rate, sc2_get_gpc0, sc2_http_err_cnt,
  sc2_http_err_rate, sc2_http_req_cnt, sc2_http_req_rate, sc2_inc_gpc0,
  sc2_kbytes_in, sc2_kbytes_out, sc2_sess_cnt, sc2_sess_rate, sc2_trackers,
  src_bytes_in_rate, src_bytes_out_rate, src_clr_gpc0, src_conn_cnt,
  src_conn_cur, src_conn_rate, src_get_gpc0, src_http_err_cnt,
  src_http_err_rate, src_http_req_cnt, src_http_req_rate, src_inc_gpc0,
  src_kbytes_in, src_kbytes_out, src_sess_cnt, src_sess_rate,
  src_updt_conn_cnt, table_avl, table_cnt,

The fetch functions have been renamed "smp_fetch_*".
MAJOR: acl: make all ACLs reference the fetch function via a sample.

ACL fetch functions used to directly reference a fetch function. Now
that all ACL fetches have their sample fetches equivalent, we can make
ACLs reference a sample fetch keyword instead.

In order to simplify the code, a sample keyword name may be NULL if it
is the same as the ACL's, which is the most common case.

A minor change appeared, http_auth always expects one argument though
the ACL allowed it to be missing and reported as such afterwards, so
fix the ACL to match this. This is not really a bug.
MAJOR: acl: remove the arg_mask from the ACL definition and use the sample fetch's

Now that ACLs solely rely on sample fetch functions, make them use the
same arg mask. All inconsistencies have been fixed separately prior to
this patch, so this patch almost only adds a new pointer indirection
and removes all references to ARG*() in the definitions.

The parsing is still performed by the ACL code though.
MAJOR: acl: remove fetch argument validation from the ACL struct

ACL fetch being inherited from the sample fetch keyword, we don't need
anymore to specify what function to use to validate the fetch arguments.

Note that the job is still done in the ACL parsing code based on elements
from the sample fetch structs.
MINOR: http: add new direction-explicit sample fetches for headers and cookies

Since "hdr" and "cookie" were ambiguously referring to the request or response
depending on the context, we need a way to explicitly specify the direction.
By prefixing the fetches names with "req." and "res.", we can now restrict such
fetches to the appropriate direction. At the moment the fetches are explicitly
declared by later we might think about having an automatic match when "req." or
"res." appears. These explicit fetches are now used by the relevant ACLs.
MINOR: payload: add new direction-explicit sample fetches

Similarly to previous commit fixing "hdr" and "cookie" in HTTP, we have to deal
with "payload" and "payload_lv" which are request-only for ACLs and req/resp for
sample fetches depending on the context, and to a less extent with other req_*
and rep_*/rep_* fetches. So let's add explicit "req." and "res." variants and
make the ACLs rely on that instead.
CLEANUP: acl: remove ACL hooks which were never used

These hooks, which established the relation between ACL_USE_* and the location
where the ACL were used, were never used because they were superseded with the
sample capabilities. Remove them now.
MEDIUM: proxy: remove acl_requires and just keep a flag "http_needed"

Proxy's acl_requires was a copy of all bits taken from ACLs, but we'll
get rid of ACL flags and only rely on sample fetches soon. The proxy's
acl_requires was only used to allocate an HTTP context when needed, and
was even forced in HTTP mode. So better have a flag which exactly says
what it's supposed to be used for.
MINOR: sample: provide a function to report the name of a sample check point

We need to put names on places where samples are used in order to emit warnings
and errors. Let's do that now.
MAJOR: acl: convert all ACL requires to SMP use+val instead of ->requires

The ACLs now use the fetch's ->use and ->val to decide upon compatibility
between the place where they are used and where the information are fetched.
The code is capable of reporting warnings about very fine incompatibilities
between certain fetches and an exact usage location, so it is expected that
some new warnings will be emitted on some existing configurations.

Two degrees of detection are provided :
  - detecting ACLs that never match
  - detecting keywords that are ignored

All tests show that this seems to work well, though bugs are still possible.
CLEANUP: acl: remove unused references to ACL_USE_*

Now that acl->requires is not used anymore, we can remove all references
to it as well as all ACL_USE_* flags.