Fix return status for illegal vbucket for touch/gat

Change-Id: Ic91b052a6990ca2571ee92fe0d2a1388f73e5a6e
Reviewed-on: http://review.membase.org/5102
Tested-by: Dustin Sallings <dustin@spy.net>
Reviewed-by: Dustin Sallings <dustin@spy.net>
Getl expiration time works [backport]

Fixed the getl expiration time issue by adding the functionality to
actually have a structure passed to ep-engine that contains the
expiration time. Fixes MB-3298.

cherry-pick from: 7ef00e44d439a6134f0cb88f6f286f6f45c60980

Change-Id: Ifae10a81d343227c75bb2bd63307a47c65b0da62
Reviewed-on: http://review.membase.org/5754
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.membase.org/5972
Reviewed-by: Michael Wiederhold <mike@membase.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
Tested-by: Steve Yen <steve.yen@gmail.com>
Added definition for getl [backport]

I originally put this definition into memcached, but since getl
is an engine specific command I am moving it into ep-engine.

cherry-picked from: 614b7e9ceca5e4baf62767fbe22c8d92e1abab3e

Change-Id: I6544bf846aaf829202ecedce796c711081fc6f47
Reviewed-on: http://review.membase.org/5757
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.membase.org/5973
Reviewed-by: Michael Wiederhold <mike@membase.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
Tested-by: Steve Yen <steve.yen@gmail.com>
MB-3465 Fix to the slow vbucket chunk deletion on 1.6.5.3

We supports a separate table per vbucket from 1.6.5.3. However,
there was a bug in the vbucket chunk deletion, which each chunk
deletion task scans all the tables instead of its corresponding
single table. This consequently slows down the performance of
each chunk deletion significantly.

Change-Id: Iff809ad6cc90ee0fb6011ab51476a911c0e89c15
Reviewed-on: http://review.membase.org/5985
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Bin Cui <bin.cui@gmail.com>
MB-3373 dbconvert doesn't compile on SunOS

getopt.h on Solaris defines the name member as "char*" and not "const
char*". This cause a compile error when you try to assign it to a
constant string. To aviod compile errors let's create a macro to cast
it to a char pointer.

Change-Id: I6a6418c09261881ab8032f80025d2e54c8ebb3ec
Reviewed-on: http://review.membase.org/5990
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
MB-3751 Fix relative time conversion in touch/gat/gatq tests

Improve the tests to lower bound to expiration in touch/gat/gatq tests
in addition to testing an absolute start time instead of only relative
times

Change-Id: Ife027452a197468505173995da9d4aa19ac372ca
Reviewed-on: http://review.membase.org/5991
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
MB-3465 Provide specific SQL stmts even for single table strategy

Even for a single table strategy, we should provide a way of getting
the list of specific SQL statements that are performed on all the
items for a given vbucket.

Change-Id: I7dede19e412ee3335e84208b9f21bbf7360fe5c1
Reviewed-on: http://review.membase.org/5988
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Bin Cui <bin.cui@gmail.com>
Include ongoing mutations in vbucket transfer. MB-3736

This addresses a few basic problems that led to data loss.  They are a
bit intertwined, so presenting them as a single changeset makes sense.

1. We always ignored new mutations when in transfer mode.
2. We hung up as soon as we saw the last mutation.
3. We activated the remote end before we were complete.

Regarding #1, there was no distinction between a "dump" and a
"transfer" tap stream.  The former is intended to end approximately at
the state where it began.  It explicitly does not include any new
mutations.  The transfer is similar, but need to stop when no new data
will come in.

We ensure no new data will come in by going into a dead state, and
then ending after we've been in a dead state *and* have completed a
first-level queue transfer from the incoming mutations.  At this
point, there can be no new incoming data and there's nowhere for it to
hide.

The downside of this is that we're not quite able to expedite the
queue transfer at the moment, so this can create a block of up to a
second at the end of any transfer.  I suspect this will be not too
difficult to fix, and enough information to do so is included in this
change, but the goal of this patch is correctness above all.

Change-Id: I2a9c66922a94d804aa179a9302dd2eafb891ff82
Reviewed-on: http://review.membase.org/5989
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Expedite tap population when connections are closing.

This prevents us from waiting a full second when a connection is
shutting down to hit its final pass through populateEvents.

Change-Id: Iae7636db7694da8cd5e875b0c7f418e4d41076d7
Reviewed-on: http://review.membase.org/6049
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-3707 Extend the logging for NULL cookies

Change-Id: Id42495e080f65d6de4e8bfb82e05a9afb7c500a6
Reviewed-on: http://review.membase.org/6071
Reviewed-by: Matt Ingenthron <matt@northscale.com>
Tested-by: Matt Ingenthron <matt@northscale.com>
MB-3707 Test for NULL before calling notify_io_complete

Change-Id: I0606273fe17000aad4ced27719bfcf7f3935b8ab
Reviewed-on: http://review.membase.org/6072
Reviewed-by: Matt Ingenthron <matt@northscale.com>
Tested-by: Matt Ingenthron <matt@northscale.com>
Send more TAP ACK requests when finalizing transfer.

Change-Id: I18f5506c75d011e58c2a2171c5a4681ac49f79c9
Reviewed-on: http://review.membase.org/6094
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Dustin Sallings <dustin@spy.net>
Don't send duplicate ACK sequence numbers.

We would occasionally send the same sequence number twice and request
an ACK on it which would lead the ACK processor to only ACK the first
one causing us to timeout waiting for the second.

Change-Id: I3c02d34799af38fee43636e95ac36b3eb7c1ce04
Reviewed-on: http://review.membase.org/6095
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Dustin Sallings <dustin@spy.net>
Don't shortcut disconnects on vbucket takeovers.

This code is a *little* out of place, but definitely not what we want
for takeover since we want the extra cycle before finishing.

Change-Id: I0d3ab3fdd703173d3e1a76ce4c6038b1e286b8f0
Reviewed-on: http://review.membase.org/6096
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Dustin Sallings <dustin@spy.net>
Fix tap pending_disk_backfill stat.

The name of the stat and the implementation are entirely the opposite
of what we present.

Change-Id: I24ac894c941e6d61ed598862c013e0531dcaf888
Reviewed-on: http://review.membase.org/6184
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Added some tap stats for new internal data stuff.

Change-Id: I0070fb6ca2ea173ee32231878b5cc206407e8921
Reviewed-on: http://review.membase.org/6185
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Frequently notify tap connections during their shutdown.

Change-Id: Ic15c9e900a47f278ea692348756eddb851e0c020
Reviewed-on: http://review.membase.org/6198
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Dustin Sallings <dustin@spy.net>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Schedule backfill tasks when TAP_CLOSE_BACKFILL event is received

In the replica chain (i.e., A->B->C), when B receives all the
backfilled items (TAP_CLOSE_BACKFILL opaque message) for subscribed
vbuckets from A, B can schedule backfill tasks for those vbuckets
that will be replicated to C.

Change-Id: I73f5be80d2577e997cb28c61ee11bc9e7e007b31
Reviewed-on: http://review.couchbase.org/6922
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Bin Cui <bin.cui@gmail.com>
Synchronize accesses to TAP setVBucketFilter() and waitForBackfill()

Change-Id: I929ed9ef1902c0fa1f08e061cd03dfbabfaeac73
Reviewed-on: http://review.couchbase.org/6923
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Bin Cui <bin.cui@gmail.com>
Allow time-bound based checkpoint creation for TAP consumer.

This change allows time-bound based checkpoint creation for TAP
consumer that does not enable checkpoint synchronization.

Change-Id: Ibb8e38d92a06ca89654d775859fd0473cc0d00c8
Reviewed-on: http://review.couchbase.org/6924
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Bin Cui <bin.cui@gmail.com>