BUG/MEDIUM: polling: fix possible CPU hogging of worker processes after receiving SIGUSR1.

When run in daemon mode (i.e. with at least one forked process) and using
the epoll poller, sending USR1 (graceful shutdown) to the worker processes
can cause some workers to start running at 100% CPU. Precondition is having
an established HTTP keep-alive connection when the signal is received.

The cloned (during fork) listening sockets do not get closed in the parent
process, thus they do not get removed from the epoll set automatically
(see man 7 epoll). This can lead to the process receiving epoll events
that it doesn't feel responsible for, resulting in an endless loop around
epoll_wait() delivering these events.

The solution is to explicitly remove these file descriptors from the epoll
set. To not degrade performance, care was taken to only do this when
neccessary, i.e. when the file descriptor was cloned during fork.

Signed-off-by: Conrad Hoffmann <conrad@soundcloud.com>
[wt: a backport to 1.4 could be studied though chances to catch the bug are low]
BUG/MEDIUM: session: don't clear CF_READ_NOEXP if analysers are not called

As more or less suspected, commit b1982e2 ("BUG/MEDIUM: http/session:
disable client-side expiration only after body") was hazardous. It
introduced a regression causing client side timeout to expire during
connection retries if it's lower than the time needed to cover the
amount of retries, so clients get a 408 when the connection to the
server fails to establish fast enough.

The reason is that the CF_READ_NOEXP flag is set after the MSG_DONE state
is reached, which protects the timeout from being re-armed, then during
the retries, process_session() clears the flag without calling the analyser
(since there's no activity for it), so the timeouts are rearmed.

Ideally, these one-shot flags should be per-analyser, and the analyser
which sets them would be responsible for clearing them, or they would
automatically be cleared when switching to another analyser. Unfortunately
this is not really possible currently.

What can be done however is to only clear them in the following situations :
  - we're going to call analysers
  - analysers have all been unsubscribed

This method seems reliable enough and approaches the ideal case well enough.

No backport is needed, this bug was introduced in 1.5-dev25.
BUG/MINOR: stats: tracking servers may incorrectly report an inherited DRAIN status

The DRAIN status is not inherited between tracked servers, so the stats
page must only use the reported server's status and not the tracked
server's status, otherwise it misleadingly indicates a DRAIN state when
a server tracks a draining server, while this is wrong.
MEDIUM: proxy: make timeout parser a bit stricter

Twice in a week I found people were surprized by a "conditional timeout" not
being respected, because they add "if <cond>" after a timeout, and since
they don't see any error nor read the doc, the expect it to work. Let's
make the timeout parser reject extra arguments to avoid these situations.
REORG/MEDIUM: server: split server state and flags in two different variables

Till now, the server's state and flags were all saved as a single bit
field. It causes some difficulties because we'd like to have an enum
for the state and separate flags.

This commit starts by splitting them in two distinct fields. The first
one is srv->state (with its counter-part srv->prev_state) which are now
enums, but which still contain bits (SRV_STF_*).

The flags now lie in their own field (srv->flags).

The function srv_is_usable() was updated to use the enum as input, since
it already used to deal only with the state.

Note that currently, the maintenance mode is still in the state for
simplicity, but it must move as well.
REORG/MEDIUM: server: move the maintenance bits out of the server state

Now we introduce srv->admin and srv->prev_admin which are bitfields
containing one bit per source of administrative status (maintenance only
for now). For the sake of backwards compatibility we implement a single
source (ADMF_FMAINT) but the code already checks any source (ADMF_MAINT)
where the STF_MAINTAIN bit was previously checked. This will later allow
us to add ADMF_IMAINT for maintenance mode inherited from tracked servers.

Along doing these changes, it appeared that some places will need to be
revisited when implementing the inherited bit, this concerns all those
modifying the ADMF_FMAINT bit (enable/disable actions on the CLI or stats
page), and the checks to report "via" on the stats page. But currently
the code is harmless.
MAJOR: server: use states instead of flags to store the server state

Servers used to have 3 flags to store a state, now they have 4 states
instead. This avoids lots of confusion for the 4 remaining undefined
states.

The encoding from the previous to the new states can be represented
this way :

  SRV_STF_RUNNING
   |  SRV_STF_GOINGDOWN
   |   |  SRV_STF_WARMINGUP
   |   |   |
   0   x   x     SRV_ST_STOPPED
   1   0   0     SRV_ST_RUNNING
   1   0   1     SRV_ST_STARTING
   1   1   x     SRV_ST_STOPPING

Note that the case where all bits were set used to exist and was randomly
dealt with. For example, the task was not stopped, the throttle value was
still updated and reported in the stats and in the http_server_state header.
It was the same if the server was stopped by the agent or for maintenance.

It's worth noting that the internal function names are still quite confusing.
REORG: checks: put the functions in the appropriate files !

Checks.c has become a total mess. A number of proxy or server maintenance
and queue management functions were put there probably because they were
used there, but that makes the code untouchable. And that's without saying
that their names does not always relate to what they really do!

So let's do a first pass by moving these ones :
  - set_backend_down()       => backend.c
  - redistribute_pending()   => queue.c:pendconn_redistribute()
  - check_for_pending()      => queue.c:pendconn_grab_from_px()
  - shutdown_sessions        => server.c:srv_shutdown_sessions()
  - shutdown_backup_sessions => server.c:srv_shutdown_backup_sessions()

All of them were moved at once.
MEDIUM: server: properly support and propagate the maintenance status

This change now involves a new flag SRV_ADMF_IMAINT to note that the
maintenance status of a server is inherited from another server. Thus,
we know at each server level in the chain if it's running, in forced
maintenance or in a maintenance status because it tracks another server,
or even in both states.

Disabling a server propagates this flag down to other servers. Enabling
a server flushes the flag down. A server becomes up again once both of
its flags are cleared.

Two new functions "srv_adm_set_maint()" and "srv_adm_set_ready()" are used to
manipulate this maintenance status. They're used by the CLI and the stats
page.

Now the stats page always says "MAINT" instead of "MAINT(via)" and it's
only the chk/down field which reports "via x/y" when the status is
inherited from another server, but it doesn't say it when a server was
forced into maintenance. The CSV output indicates "MAINT (via x/y)"
instead of only "MAINT(via)". This is the most accurate representation.

One important thing is that now entering/leaving maintenance for a
tracking server correctly follows the state of the tracked server.
MEDIUM: server: allow multi-level server tracking

Now that it is possible to know whether a server is in forced maintenance
or inherits its maintenance status from another one, it is possible to
allow server tracking at more than one level. We still provide a loop
detection however.

Note that for the stats it's a bit trickier since we have to report the
check state which corresponds to the state of the server at the end of
the chain.
CLEANUP: checks: rename the server_status_printf function

This function is poorly named since it's now used exclusively with checks
and cannot be moved to server.c. Call it check_report_srv_status() instead.
MEDIUM: checks: simplify server up/down/nolb transitions

We don't have to handle the maintenance transition here anymore so we
can simplify the functions and conditions. This also means that we don't
need the disable/enable functions but only a function to switch to each
new state.

It's worth mentionning that at this stage there are still confusions
between the server state and the checks states. For example, the health
check's state is adjusted from tracked servers changing state, while it
should not be.
MAJOR: checks: move health checks changes to set_server_check_status()

We don't want to manipulate check's statuses anymore in functions
which modify the server's state. So since any check is forced to
call set_server_check_status() exactly once to report the result
of the check, it's the best place to update the check's health.
MINOR: server: make the status reporting function support a reason

srv_adm_append_status() was renamed srv_append_status() since it's no
more dedicated to maintenance mode. It now supports a reason which if
not null is appended to the output string.
MINOR: checks: simplify health check reporting functions

check_report_srv_status() was removed in favor of check_reason_string()
combined with srv_report_status(). This way we have one function which
is dedicated to check decoding, and another one dedicated to server
status.
MINOR: server: implement srv_set_stopped()

This function was extracted from check_set_server_down(). In only
manipulates the server state and does not consider the health checks
at all, nor does it modify their status. It takes a reason message to
report in logs, however it passes NULL when recursing through the
trackers chain.

For now the function is not used, but the goal is to split health
checks status from server status and to be able to change a server's
state regardless of health checks statuses.
MINOR: server: implement srv_set_running()

This function was taken from check_set_server_up(). It does not consider
health checks at all and only sets a server up provided it's not in
maintenance. The resulting state may be either RUNNING or STARTING
depending on the presence of a slowstart or not. The state change is
propagated to tracked servers.

For now the function is not used, but the goal is to split health
checks status from server status and to be able to change a server's
state regardless of health checks statuses.
MINOR: server: implement srv_set_stopping()

This function was taken from check_set_server_drain(). It does not
consider health checks at all and only sets a server to stopping
provided it's not in maintenance and is not currently stopped. The
resulting state will be STOPPING. The state change is propagated
to tracked servers.

For now the function is not used, but the goal is to split health
checks status from server status and to be able to change a server's
state regardless of health checks statuses.
MEDIUM: checks: simplify failure notification using srv_set_stopped()

Function check_set_server_down() used to set a server down. Now it first
checks if the health check's result differs from the server's state, and
only calls srv_set_stopped() if the check reports a failure while the
server is not down. Thanks to this, the conditions that were present
around its call could be removed. The function was also renamed
check_notify_failure() to better report this change.
MEDIUM: checks: simplify success notification using srv_set_running()

Function check_set_server_up() used to set a server up. Now it first
checks if all configured checks are UP, and if all tracked servers are
UP, and only calls set_srv_running() after that. That also simplified
the conditions to call the function, and its logic. The function was
also renamed check_notify_success() to better report this change.