Fix scan_coordinator tests to adapt with queryport API changes

Change-Id: I37dcef8d246d53282e5fff3140f14e95d7571788
Reviewed-on: http://review.couchbase.org/44001
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
indexer: Fix incorrect refcount decrement for snapshots

The snapshot.Close() should always decrement the refcount
But the real snapshot handle dealloc should be performed only when
refcount becomes zero.

Change-Id: Iad4a50311239fac72851943231021bd024e4be1f
Reviewed-on: http://review.couchbase.org/44021
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
Index Definition Life Cycle
- Use projector idempotent API for boostrap
- Update index topology (deployment plan) upon index creation/deletion
- Stream Manager listens to index topology for finding out new index
- Stream Manager notifies projector on index deployment changes using projector API
- Clean up event manager implementation
- Fix bugs on timer, topology
- Add/Improve stream_test

Change-Id: I23eb1919dd2f1212671d33a97980e8eb58a7306b
Reviewed-on: http://review.couchbase.org/43803
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Tested-by: John Liang <johliang00@gmail.com>
Reviewed-by: John Liang <johliang00@gmail.com>
Allocate globally unique Index Instance Id from gometa

Change-Id: I9063183b7b5cf64489492bdf1ab1e0db1f8a0f43
Reviewed-on: http://review.couchbase.org/43807
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: John Liang <johliang00@gmail.com>
Tested-by: John Liang <johliang00@gmail.com>
If coordinator receives StreamEnd, it calls projector RestartVbucket on the
specific vbucket, using the last stability timestamp received.

Change-Id: Icec4b38072c5e264113f6e5d4a51532a7c4c37f4
Reviewed-on: http://review.couchbase.org/43886
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: John Liang <johliang00@gmail.com>
Tested-by: John Liang <johliang00@gmail.com>
Stabiltiy Timestamp Persistence
Persist statbility in gometa such that it can be used for coordinator
failover as well as bootstrap.  Timer will initialize based on
stability timestamp stored in gometa.

Change-Id: I025f589edbdb7a84f2144f1cf0bc4f3d6c022cc8
Reviewed-on: http://review.couchbase.org/43938
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: John Liang <johliang00@gmail.com>
Tested-by: John Liang <johliang00@gmail.com>
Keep track of the active timestamp returned by projector
Update timestamp on SanpshotMarker for type 1 and 2

Change-Id: Id972f5b4142a471e7902c3208a6be680089df64c
Reviewed-on: http://review.couchbase.org/43952
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: John Liang <johliang00@gmail.com>
Tested-by: John Liang <johliang00@gmail.com>
Push stability timestamp to indexer only if the timestamp is fully formed

Change-Id: I9d3131d6b7c0f3285aff47a65abf2f1d1639d328
Reviewed-on: http://review.couchbase.org/44029
Reviewed-by: John Liang <johliang00@gmail.com>
Tested-by: John Liang <johliang00@gmail.com>
Add StreamMonitor to monitor "liveness" of vbuckets.  Restart Vbuckets if it is not active (StreamEnd).
Reorganize unit test

Change-Id: I801ede06a0bf006ef17eb618bc2fd4c21ad75ef9
Reviewed-on: http://review.couchbase.org/44084
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: John Liang <johliang00@gmail.com>
Tested-by: John Liang <johliang00@gmail.com>
fix build issue
- fix indexer/cluster_manager_agent.go due to index manager API changes

Change-Id: I7b81f5be877adf57a54243c4e210de97364a33c3
Reviewed-on: http://review.couchbase.org/44148
Reviewed-by: John Liang <johliang00@gmail.com>
Tested-by: John Liang <johliang00@gmail.com>
adminport: url-prefix checking and log-prefix.

Change-Id: I1f921365920b0dbf48e76f77962ea23317ac0ac5
Reviewed-on: http://review.couchbase.org/44164
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
common: prefix and configuration.

- adjust url-prefix while fetching Cluster_info
- log-prefix for INFO, WARN.
- added back `adminport.listenAddr` config, parameter will be overriden by
  command line switch.

Change-Id: I38c948761acbe645a9eb217b8b3203d0decd8b2c
Reviewed-on: http://review.couchbase.org/44165
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
protobuf: format protobuf timestamps for logging.

Change-Id: Ifa7ce45749d4840fd589580ee2d1c4ea1518d516
Reviewed-on: http://review.couchbase.org/44166
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
queryport: fix log messages and lint errors.

Change-Id: I927adedf0a0f34fbdf4a7aec8fd28a65a405a467
Reviewed-on: http://review.couchbase.org/44167
Reviewed-by: Manik Taneja <manik@couchbase.com>
Tested-by: Manik Taneja <manik@couchbase.com>
N1QL: Export Indexprovider.

Change-Id: Ib6b605f0dc4779015a54b385b64c17b7753e452b
Reviewed-on: http://review.couchbase.org/44230
Reviewed-by: Manik Taneja <manik@couchbase.com>
Tested-by: Manik Taneja <manik@couchbase.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
indexer: Introduce index_snapshot abstraction

Added a new index level snapshot abstraction that be used
for implementing point-in-time scannable index snapshot.
IndexSnapshot supports Clone() and Destory() APIs. Under the
covers this APIs implement reference counting of shared resources
to avoid unnecessary copying.

IndexSnapshot can be generated whenever we decide to create an
scannable snapshot for index from a stability timestamp. It can
be used in scan coordinator to implement independent operation
without holding any mutable index data structures.

Change-Id: Ib261305ef75fd80f8df096588f5ed11b06946be7
Reviewed-on: http://review.couchbase.org/44074
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
indexer: Use index snapshot abstractions for scanner

Storage manager is refactored to generate IndexSnapshot
objects whenever a new stability timestamp is processed.
The scan coordinator will no receive IndexSnapshot
corresponding to the requested atleast-timestamp and
scan operates on IndexSnapshot instead of mutable
index partition map and index instance map data structures.

Change-Id: Ib47c110678e0e3056ae74f991d8771038fd5147f
Reviewed-on: http://review.couchbase.org/44075
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
indexer: Implement refcounting for slice close/destroy

When an index needs to be destroyed, all the slice
resources (forestdb handles, physical files) need to
be deallocated. But, there could be a number of consumers
using slice resources while a drop index needs to be processed.

Added refcounting methods IncrRef(), DecrRef() that can be used
by consumers. Destructive functions Close() and Destory() will
become a lazy operation and will be performed only when refcount
becomes zero. ie, all the consumers have finished using the resource.

Change-Id: I1d3b10b96d2698099b8e4ef7420397bb3ba263d7
Reviewed-on: http://review.couchbase.org/44076
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
scanner: Implement scan deadline timeout

Implemented a configurable timeout of 300s for
index scan operation. If the scan takes longer,
the scan coordinator will abort the scan and send
an error, ErrScanTimedOut to the client.

Change-Id: I4ae03cf3cab0f1defbc5636bdb8cf52d6f1a830c
Reviewed-on: http://review.couchbase.org/44077
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
indexer: Use system config to obtain necessary constants

Cleaned up indexer by removing all hardcoded configurable
values such as ports, num vbuckets, etc. Indexer will now
use only config values passed through system config with
keys having prefix `indexer.`.

Change-Id: I4aa78d5ef62b971eee87e59b5c93717051052dfd
Reviewed-on: http://review.couchbase.org/44081
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>