list_keys: consistent ESCAPE usage across DB types

Without specifying an ESCAPE character for LIKE queries, the '\'
we use for escaping is treated as a literal and improperly
matched keys with '\' in them under SQLite.

This is only needed for SQLite, as the SQLite language reference
makes no reference of a default ESCAPE character in
http://www.sqlite.org/lang_expr.html

ESCAPE is supported in MySQL and Postgres, too; and defaults to
'\'.  We specify it anyways to reduce code differences between
different databases.

Tested on MySQL 5.1.66 and Postgres 8.4.13 on Debian 6.0
and SQLite 3.7.13 on Debian 7.0
list_keys: escape in Store, allow [%\\] as prefix

If we support non-SQL DBs in the future, escaping rules could
become store-specific, so Worker/Query is not the right place for
it.

Since '%' and '\' may be escaped just like any other character,
we may also allow these characters as prefixes by properly
escaping them.

Tested on MySQL 5.1.66 and Postgres 8.4.13 on Debian 6.0
and SQLite 3.7.13 on Debian 7.0
Tell the kernel we're doing sequential reads.

[ew: squashed Dave's change to make IO::AIO optional]

Signed-off-by: Eric Wong <normalperson@yhbt.net>
Don't emit warnings if we're lacking the space free of a device.  If we don't
find space on devices with known space free, try the unknowns.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
ProcManager: only log times_out_of_qworkers for new queries

Logging times_out_of_qworkers in ProcessQueues is not accurate:
recently-idle queryworkers may not be noticed and marked idle while
ProcessQueues is looping and draining the @IdleQueryWorkers pool.

Instead, only log times_out_of_qworkers when new requests are
enqueued.
Merge remote-tracking branch 'bogomips/list_keys' into next-small

* bogomips/list_keys:
  list_keys: escape in Store, allow [%\\] as prefix
  list_keys: consistent ESCAPE usage across DB types
Merge remote-tracking branch 'bogomips/pending_queries' into next-small

* bogomips/pending_queries:
  ProcManager: only log times_out_of_qworkers for new queries
mogstored: avoid bareword on IO::AIO w/o fadvise

IO::AIO 2.4 on Debian stable lacks IO::AIO::FADV_SEQUENTIAL
constant, causing compilation to fail on the bareword.
Accessing the constant as a subroutine call (via "()") avoids
the bareword and defers the error to runtime (which is trapped
by eval).

Tested under IO::AIO 2.4 on Debian stable and IO::AIO 4.15 on
Debian testing (verified fadvise64() syscall under strace).
httpfile: correct FILE_MISSING check in digest_mgmt

This is unlikely to be an issue in fsck, fsck checks
file size/existence before digesting the file.
httpfile: correct timeouts for sidechannel digest

fsck digests are deprioritized and serialized in mogstored, so
it's nearly impossible to tell what's in the queue before our
request.  Since fsck is not latency critical, extend the timeout
for that.

We also need to account for normal seek/network latency for
non-fsck digest requests, so add node_timeout to that.

These bugs were mostly hidden since we are relying on <> to
read, which may incur watchdog timeouts.
fix "drain" handling used by MultipleHosts replpolicy

MogileFS::DeviceState was never updated for the 2.40 drain changes.
The broken-since-2.40 should_have_files sub caused
ReplicationPolicy::MultipleHosts to overreplicate files, as it
was not counting drain devices in the total disks check.

Thanks to Tim on for reporting this to the mailing list at
mogile@googlegroups.com
remove unused *::get_dbh subroutines

These subroutines are unused.
ProcManager: log socketpair errors correctly

Log the correct name of the failed function and the error string
associated with the OS errno to aid in debugging.

ML Ref:
  Date: Mon, 8 Jul 2013 17:14:40 -0700 (PDT)
  From: Tim <timchen@timchen.de>
  To: mogile@googlegroups.com
  Message-Id: <12aa5269-7275-476b-86b6-863223005595@googlegroups.com>
  Subject: MogileFS crashes
httpfile: log mogstored I/O errors when checksumming

Mogstored/SideChannelClient.pm may hit the following on I/O error:

  $self->write("ERR read $uri at $offset failed\r\n");

Be prepared to show that error to tracker watchers (and any
other possible errors mogstored may return in the future).
add naive MultipleDevice replication policy

This can be useful when MultipleHosts is too noisy when hosts differ
greatly in storage capacity.

The intended target of this policy is a low-priority backup cluster
where a single host contains the bulk of the storage with a handful
of random machines helping out.  The MultipleHosts policy can be too
noisy with log messages about running out of suggestions in this case.
store: do not auto-reconnect while holding a lock

Auto-reconnect is probably always unsafe while a holding a lock
on all networked databases.

While we do not use the builtin auto-reconnect functionality of
MySQL, any auto-reconnect implementation should be affected by
the same issues upon connection failure:
https://dev.mysql.com/doc/refman/5.6/en/auto-reconnect.html

With auto-reconnect, we could be operating under the false
assumption we have a lock after the reconnect when we do not.

For now, the easiest method of recovery is to just let the
worker die while working on the current task and have the
ProcManager restart it.
store: do not disconnect for max_handles while locked

Dropping a connection while holding an advisory lock with
MySQL or Postgres will cause a fatal error, so hold the
connection open until the next time the dbh is requested
without holding a lock.
allow startup without job_master (and dependent workers)

And when running without a job_master, do not spawn job_master-dependent
workers (delete, fsck, replicate) as those workers will never get work.

Running a queryworker+monitor in a remote datacenter makes sense with
the MogileFS::Network plugins since the "create_close" size verification
is faster and more reliable if the queryworker is in the same datacenter
as the client, even if the master DB is in a remote datacenter.

Being in a remote datacenter, (master)DB-intensive operations from
delete, fsck and replicate workers can encounter high latency and an
unreliable link, so admins may disable those workers in this situation.

However, disabling delete, fsck, and replicate workers individually
still allows the job_master to fill the initial queues (which is never
processed) and prevent other trackers from processing items for 1000
seconds.

Future commits may allow job_master to ignore certain queues if there
are zero workers for that queue, but for now, stopping job_master
entirely should be sufficient for most users with trackers in a
different datacenter than the DB.

P.S. It also makes sense to disable reaper in remote datacenters, too,
but reaper does not rely on job_master.
increase receive buffers for large state events

The monitor may send large state events for large installations
with many hosts, devices, domains, or classes.  The 1K default
is too small and leads to excessive syscalls and string operations.

This increases startup performance for a mock instance with 10K domains
and 10K non-default classes.

Using the parent_ping function and "No simple reply" warning as an
informal benchmark, this change reduces the loop time from 12 to 10
loops.
monitor: do not repeat join() for the debug statement

This join() takes about 20ms on my mock instance with 10K domains
and 10K classes, so it has some impact on startup performance.