BUG/MEDIUM: http: adjust close mode when switching to backend

Commit 179085c ("MEDIUM: http: move Connection header processing earlier")
introduced a regression : the backend's HTTP mode is not considered anymore
when setting the session's HTTP mode, because wait_for_request() is only
called once, when the frontend receives the request (or when the frontend
is in TCP mode, when the backend receives the request).

The net effect is that in some situations when the frontend and the backend
do not work in the same mode (eg: keep-alive vs close), the backend's mode
is ignored.

This patch moves all that processing to a dedicated function, which is
called from the original place, as well as from session_set_backend()
when switching from an HTTP frontend to an HTTP backend in different
modes.

This fix must be backported to 1.5.
(cherry picked from commit 4e21ff9244aefa56bcf0793a9e07edba2c3c1960)
BUG/MINOR: config: don't propagate process binding on fatal errors.

propagate_processes() must not be called with unresolved proxies, but
nothing prevents it from being called in check_config_validity(). The
resulting effect is that an unresolved proxy can cause a recursion
loop if called in such a situation, ending with a segfault after the
fatal error report. There's no side effect beyond this.

This patch refrains from calling the function when any error was met.

This bug also affects 1.5, it should be backported.
(cherry picked from commit acbe8ab38a638a076f8cf9fe2635db0e729d6a1f)
BUG/MEDIUM: check: rule-less tcp-check must detect connect failures

When "option tcp-check" is specified without any tcp-check rules, the
documentation says that it's the same as the default check method. But
the code path is a bit different, and we used to consider that since
the end of rules was reached, the check is always successful regardless
of the connection status.

This patch reorganizes the error detection, and considers the special
case where there's no tcp-check rule as a real L4 check. It also avoids
dereferencing the rule list head as a rule by itself.

While fixing this bug, another one related to the output messages'
accuracy was noticed, it will be fixed in a separate commit and is
much less important.

This bug is also present in 1.5, so this fix must be backported.
(cherry picked from commit ef953953e7f33c6a72c432fce8d47c2d84c69512)
BUG/MINOR: tcp-check: report the correct failed step in the status

The step number was reported by checking only last_started_step, which
was not set in case of error during the initial connection phase, and
caused "step 1" to be returned with an invalid check type (typically
SEND). So now we first verify that a test was started before returning
this.

In addition to this, the indication of the test type was taken from
current_step instead of last_started_step, so the error description
was matching the next action instead of the one reported in the step
ID. Thus we could get the confusing "step 1 (send)" report below :

      tcp-check connect
      tcp-check send foo

In order to ease debugging, when the port number is known for a connect,
it is indicated in the error report.

Note that this only affects asynchronous error messages, synchronous ones
are correct.

This fix must be backported to 1.5.
(cherry picked from commit 213c6785614d0228d7e96e982e5189e1d0777059)
BUG/MINOR: config: don't propagate process binding for dynamic use_backend

A segfault was reported with the introduction of the propagate_processes()
function. It was caused when a use_backend rule was declared with a dynamic
name, using a log-format string. The backend is not resolved during the
configuration, which lead to the segfault.

The patch prevents the process binding propagation for such dynamic rules, it
should also be backported to 1.5.
(cherry picked from commit 51639696e0a112ea3612e905a5722ad912b3869f)
DOC: Address issue where documentation is excluded due to a gitignore rule.

(cherry picked from commit fbc4e77321f6e92b7af658ce8257b6231c6ebac9)
DOC: indicate that weight zero is reported as DRAIN

It's not the same state but reported as such.
(cherry picked from commit c5af3a6d15358fa8b5be6151239862efde4c36f7)
[RELEASE] Released version 1.5.5

Released version 1.5.5 with the following main changes :
    - DOC: Address issue where documentation is excluded due to a gitignore rule.
    - MEDIUM: Improve signal handling in systemd wrapper.
    - BUG/MINOR: config: don't propagate process binding for dynamic use_backend
    - MINOR: Also accept SIGHUP/SIGTERM in systemd-wrapper
    - DOC: clearly state that the "show sess" output format is not fixed
    - MINOR: stats: fix minor typo fix in stats_dump_errors_to_buffer()
    - DOC: indicate in the doc that track-sc* can wait if data are missing
    - MEDIUM: http: enable header manipulation for 101 responses
    - BUG/MEDIUM: config: propagate frontend to backend process binding again.
    - MEDIUM: config: properly propagate process binding between proxies
    - MEDIUM: config: make the frontends automatically bind to the listeners' processes
    - MEDIUM: config: compute the exact bind-process before listener's maxaccept
    - MEDIUM: config: only warn if stats are attached to multi-process bind directives
    - MEDIUM: config: report it when tcp-request rules are misplaced
    - MINOR: config: detect the case where a tcp-request content rule has no inspect-delay
    - MEDIUM: systemd-wrapper: support multiple executable versions and names
    - BUG/MEDIUM: remove debugging code from systemd-wrapper
    - BUG/MEDIUM: http: adjust close mode when switching to backend
    - BUG/MINOR: config: don't propagate process binding on fatal errors.
    - BUG/MEDIUM: check: rule-less tcp-check must detect connect failures
    - BUG/MINOR: tcp-check: report the correct failed step in the status
    - DOC: indicate that weight zero is reported as DRAIN
BUG/MEDIUM: systemd: set KillMode to 'mixed'

By default systemd will send SIGTERM to all processes in the service's
control group. In our case, this includes the wrapper, the master
process and all worker processes.

Since commit c54bdd2a the wrapper actually catches SIGTERM and survives
to see the master process getting killed by systemd and regard this as
an error, placing the unit in a failed state during "systemctl stop".

Since the wrapper now handles SIGTERM by itself, we switch the kill mode
to 'mixed', which means that systemd will deliver the initial SIGTERM to
the wrapper only, and if the actual haproxy processes don't exit after a
given amount of time (default: 90s), a SIGKILL is sent to all remaining
processes in the control group. See systemd.kill(5) for more
information.

This should also be backported to 1.5.
(cherry picked from commit 74f016985ab1fa7a6a5afa301d982e77eba9c96a)
BUG/MEDIUM: config: avoid skipping disabled proxies

Paul Taylor and Bryan Talbot found that after commit 419ead8 ("MEDIUM:
config: compute the exact bind-process before listener's maxaccept"),
a backend marked "disabled" would cause the next backend to be skipped
and if it was the last one it would cause a segfault.

The reason is that the commit above changed the "while" loop for a "for"
loop but a "continue" statement still incrementing the current proxy was
left in the code for disabled proxies, causing the next one to be skipped
as well and the last one to try to dereference NULL when seeking ->next.

The quick workaround consists in not disabling backends, or adding an
empty dummy one after a disabled section.

This fix must be backported to 1.5.
(cherry picked from commit 06cc905813f1858c0606b44d1e271bd0b6a25224)
MINOR: systemd: Check configuration before start

Adds a configuration check before starting the haproxy service.
(cherry picked from commit 91a9f1a2e1421a376a4b63f9342addd59cb2b244)
BUG/MINOR: config: do not accept more track-sc than configured

MAX_SESS_STKCTR allows one to define the number of stick counters that can
be used in parallel in track-sc* rules. The naming of this macro creates
some confusion because the value there is sometimes used as a max instead
of a count, and the config parser accepts values from 0 to MAX_SESS_STKCTR
and the processing ignores anything tracked on the last one. This means
that by default, track-sc3 is allowed and ignored.

This fix must be backported to 1.5 where the problem there only affects
TCP rules.
(cherry picked from commit e1cfc1f2b469fb86b9826ff12cdd6a24a1f3ded0)
BUG/MEDIUM: backend: fix URI hash when a query string is present

Commit 98634f0 ("MEDIUM: backend: Enhance hash-type directive with an
algorithm options") cleaned up the hashing code by using a centralized
function. A bug appeared in get_server_uh() which is the URI hashing
function. Prior to the patch, the function would stop hashing on the
question mark, or on the trailing slash of a maximum directory count.
Consecutive to the patch, this last character is included into the
hash computation. This means that :

    GET /0
    GET /0?

Are not hashed similarly. The following configuration reproduces it :

    mode http
    balance uri
    server s1 0.0.0.0:1234 redir /s1
    server s2 0.0.0.0:1234 redir /s2

Many thanks to Vedran Furac for reporting this issue. The fix must
be backported to 1.5.
(cherry picked from commit fad4ffc89337277f3d5ed32b66986730e891558a)
[RELEASE] Released version 1.5.6

Released version 1.5.6 with the following main changes :
    - BUG/MEDIUM: systemd: set KillMode to 'mixed'
    - MINOR: systemd: Check configuration before start
    - BUG/MEDIUM: config: avoid skipping disabled proxies
    - BUG/MINOR: config: do not accept more track-sc than configured
    - BUG/MEDIUM: backend: fix URI hash when a query string is present
BUG/MEDIUM: http: don't dump debug headers on MSG_ERROR

When the HTTP parser is in state HTTP_MSG_ERROR, we don't know if it was
already initialized or not. If the error happens before HTTP_MSG_RQBEFORE,
random offsets might be present and we don't want to display such random
strings in debug mode.

While it's theorically possible to randomly crash the process when running
in debug mode here, this bug was not tagged MAJOR because it would not
make sense to run in debug mode in production.

This fix must be backported to 1.5 and 1.4.
(cherry picked from commit 7d59e90473bbf3995ef11d146a7a4cf5c6d0c4b9)
BUG/MAJOR: cli: explicitly call cli_release_handler() upon error

Dmitry Sivachenko reported an embarrassing problem where haproxy
would sometimes segfault upon reload. After careful analysis and
code inspection, what happens is related to the "show sess" command
on the CLI, and it is not limited to reload operations only.

When a "show sess" is running, once the output buffer is full, the
stats applet grabs a reference to the session being dumped in order
for the current pointer to be able to advance by itself should this
session disappear while the buffer is full. The applet also uses a
release handler that is called when the applet terminates to release
such references.

The problem is that upon error, the command line parser sets the
applet state to STAT_CLI_O_END indicating it wants to terminate the
processing. Unfortunately, the release handler which is called later
to clean everything up relies on the applet's state to know what
operations were in progress, and as such it does not release the
reference. A later "show sess" or the completion of the task being
watched lead to a LIST_DEL() on the task's list which point to a
location that does not match the applet's reference list anymore
and the process dies.

One solution to this would be to add a flag to the current applet's
state mentionning it must leave, without affecting the state indicating
the current operation. It's a bit invasive but could be the long term
solution. The short term solution simply consists in calling the
release handler just before changing the state to STAT_CLI_O_END.
That way everything that must be released is released in time.

Note that the probability to encounter this issue is very low.
It requires a lot of "show sess" or "show sess all" calls, and
that one of them dies before being completed. That can happen
if "show sess" is run in scripts which truncate the output (eg:
"echo show sess|socat|head"). This could be the worst case as it
almost ensures that haproxy fills a buffer, grabs a reference and
detects the error on the socket.

There's no config-based workaround to this issue, except refraining
from issuing "show sess" on large connection counts or "show sess all".
If that's not possible to block everyone, restricting permissions on
the stats socket ensures only authorized tools can connect.

This fix must be backported to 1.5 and to 1.4 (with some changes in
1.4 since the release function does not exist so the LIST_DEL sequence
must be open-coded).

Special thanks to Dmitry for the fairly complete report.
(cherry picked from commit 44cf5450005c3bb822fce3c0971f515c470b678b)
BUG/MINOR: log: fix request flags when keep-alive is enabled

Colin Ingarfield reported some unexplainable flags in the logs.
For example, a "LR" termination state was set on a request which was forwarded
to a server, where "LR" means that the request should have been handled
internally by haproxy.

This case happens when at least client side keep-alive is enabled. Next
requests in the connection will inherit the flags from the previous request.

2 fields are impacted : "termination_state" and "Tt" in the timing events,
where a "+" can be added, when a previous request was redispatched.

This is not critical for the service itself but can confuse troubleshooting.

The fix must be backported to 1.5 and 1.4.
(cherry picked from commit a83a50bd7d9389d3f799b9e3b7c7be346be3fab7)
BUG/MEDIUM: tcp: fix outgoing polling based on proxy protocol

During a tcp connection setup in tcp_connect_server(), we check if
there are pending data to start polling for writes immediately. We
also use the same test to know if we can disable the quick ack and
merge the first data packet with the connection's ACK. This last
case is also valid for the proxy protocol.

The problem lies in the way it's done, as the "data" variable is
improperly completed with the presence of the proxy protocol, resulting
in the connection being polled for data writes if the proxy protocol is
enabled. It's not a big issue per se, except that the proxy protocol
uses the fact that we're polling for data to know if it can use MSG_MORE.

This causes no problem on HTTP/HTTPS, but with banner protocols, it
introduces a 200ms delay if the server waits for the PROXY header.

This has been caused by the connection management changes introduced in
1.5-dev12, specifically commit a1a7474 ("MEDIUM: proxy-proto: don't use
buffer flags in conn_si_send_proxy()"), so this fix must be backported
to 1.5.
(cherry picked from commit fb20e4668d0d3fbdcb97eacb9695659aa7ac1a54)
BUG/MEDIUM: regex: fix pcre_study error handling

pcre_study() may return NULL even though it succeeded. In this case error is
NULL otherwise error is not NULL. Also see man 3 pcre_study.

Previously a ACL pattern of e.g. ".*" would cause error because pcre_study did
not found anything to speed up matching and returned regex->extra = NULL and
error = NULL which in this case was a false-positive. That happend only when
PCRE_JIT was enabled for HAProxy but libpcre has been built without JIT.

Signed-off-by: Christian Ruppert <c.ruppert@babiel.com>

[wt: this needs to be backported to 1.5 as well]
(cherry picked from commit 955f4613cb457a93bb278e47a9af19bf5479d27d)
BUG/MEDIUM: tcp: don't use SO_ORIGINAL_DST on non-AF_INET sockets

There's an issue when using SO_ORIGINAL_DST to retrieve the original
destination of a connection's address before being translated by
Netfilter's DNAT/REDIRECT or the old TPROXY. SO_ORIGINAL_DST is
able to retrieve an IPv4 address when the original destination was
IPv4 mapped into IPv6. At first glance it's not a big deal, but it
is for logging and for the proxy protocol, because we then have
two different address families for the source and destination. In
this case, the proxy protocol correctly detects the issue and emits
"UNKNOWN".

In order to fix this, we perform getsockname() first, and only if
the address family is AF_INET, then we perform the getsockopt() call.

This fix must be backported to 1.5, and probably even to 1.4 and 1.3.
(cherry picked from commit 5e0d0e046a7a54c6f4349e58f0e850105038a71a)