Handle Missing secondary keys.

- Opaque is computed such that it gets incremented few times every second.

CBIDXT-116: CBIDXT-133:
  - Implemented missing for composite keys, provided first-item in the key
    is not empty.

Change-Id: I4d3c3c597b5bf741c35fe47f868c7d043062acfc
Reviewed-on: http://review.couchbase.org/42103
Reviewed-by: John Liang <johliang00@gmail.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
collatejson: TypeMissing base on special literal.

Change-Id: Ifc95ae14633f7e35cf7629c585bbbba07d7f05e6
Reviewed-on: http://review.couchbase.org/42104
Reviewed-by: John Liang <johliang00@gmail.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
CBIDXT-100: CloseStream implemented.

- fix example app for dataport.
- changed application `opaque` type to uint16.
- feed's opaque value will increment dozen times a second, thus keeping it
  unique for a feed call and will not overflow for a long time.
- tools/multibuckets/multibuckets.go is passing.

Change-Id: I0d54d5bcc7c31fce3a371b446fc238410d1a333a
Reviewed-on: http://review.couchbase.org/42105
Reviewed-by: John Liang <johliang00@gmail.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
CBIDXT-110: configuration

- merged common/endpoint.go with common/router.go
- remove c.MaxVbuckets in common/*.go
- removed endpoint settings and replaced it with endpointType
  (In case different configuration is expected for same endpoint,
   across different topic, it will have be added at later point)
- script to generate config documentation.

Added config parameters for,
- adminport, dataport, queryport, projector and feed.

Change-Id: I1549a40533d2cb569e252291a1201f4e5d7e88a9
Reviewed-on: http://review.couchbase.org/42234
Reviewed-by: John Liang <johliang00@gmail.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
Projector client is now a separate sub-package.

projector client includes retry logic, with optional backoff algorithm
for "connection-refused" cases.

Change-Id: I68824fea2685f9bb6837cce21ea4f7c0a75a9f33
Reviewed-on: http://review.couchbase.org/42235
Reviewed-by: John Liang <johliang00@gmail.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
Use IndexManager Election port as part of the watcherId.  This is to make watcherId uinqiue in clusterRun.

Change-Id: I0bb9e628fb3da1e9cb96c609ce8f06e86b866bde
Reviewed-on: http://review.couchbase.org/42454
Reviewed-by: John Liang <johliang00@gmail.com>
Tested-by: John Liang <johliang00@gmail.com>
Improvements to Indexer Recovery

- Fix a few issues with timekeeper in enable/disable flush during
merging the catchup stream
- also handle the case when stream has been merged partially (ie. for
a subset of buckets) and there is another failure
- batch support in removeindex in kv_sender
- stringify streamId, stream state, index state in logging

Change-Id: Iacd437ac298c385c7f13b4972f246e28f438c4e5
Reviewed-on: http://review.couchbase.org/42281
Reviewed-by: John Liang <johliang00@gmail.com>
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Tested-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Integrate Indexer with new Projector Client

- client is now a separate sub-pkg in projector
- dataport and projector client APIs now need a config object to be passed

Change-Id: I5ab0353284726939d75b3628877f1d55725b3ff9
Reviewed-on: http://review.couchbase.org/42466
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Tested-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
CBIDXT-213 UPR Negotiation and Data Rollback

Change-Id: I9565610e49123d5cd979b6a6befd4d62a17ad26f
Reviewed-on: http://review.couchbase.org/42350
Reviewed-by: John Liang <johliang00@gmail.com>
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Tested-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Indexer/Index Manager Integration for Create/Drop DDL

Initial integration for Indexer and Index Manager to handle Create/Drop
DDL. Currently Index Manager is not deciding topology. So for now
DDL goes to all indexers.

Change-Id: I108f5b1cacb29447b901cb27123dbc93057e3a0e
Reviewed-on: http://review.couchbase.org/42446
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Tested-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
CBIDXT-109: Avoid magic numbers.

magic numbers are converted to configuration parameters.

Change-Id: I0af2e6c52b5ca8dc91b1ca65833f794ee6a230b8
Reviewed-on: http://review.couchbase.org/42457
Reviewed-by: John Liang <johliang00@gmail.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
CBIDXT-110: Method receiver to override config.

to update default-configuration with json configuration obtained via file /
command line etc ...

    jsonconfig, _ := NewConfig([]byte(`{ "timeout": 10 }`))
    config := SystemConfig.Clone().Override(jsonconfig)

Change-Id: I9b5eed672975f6c038745d3d6912ddc001279b12
Reviewed-on: http://review.couchbase.org/42458
Reviewed-by: John Liang <johliang00@gmail.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
IsEmpty() method receiver for protobuf.TsVbuuid.

TopicResponse:rollbackTimestamps slice shall contain items only for buckets
will rollback.

Change-Id: I662109f8375825fe7ed991c8756200747a664170
Reviewed-on: http://review.couchbase.org/42460
Reviewed-by: John Liang <johliang00@gmail.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
CBIDXT-218: bug fix.

Gocouchbase : connection pool overflow.

- getBucket() method is removed from Projector{} and Feed{} instance.
- Projector will immediately close the bucket instance once bucket
  information is obtained.
- Feed will call feeder:CloseFeed() for Shutdown() and DelBuckets()
  in other cases, it will close the bucket instance once the bucket
  information is obtained.

Change-Id: Ia75a7b88c66809374ccb6c4e42920cba7c3536ca
Reviewed-on: http://review.couchbase.org/42461
Reviewed-by: John Liang <johliang00@gmail.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
MB-12315: generate StreamEnd.

When a bucket crashes at go-couchbase, the channel is closed, there by KVData
should publish StreamEnd to downstream.

Change-Id: I713bb41676e95c7c16ad2edd0f8046468b0131bf
Reviewed-on: http://review.couchbase.org/42467
Reviewed-by: John Liang <johliang00@gmail.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
Missing literal, a unique string value, is used.

Change-Id: I5f85a5a1763e3ea78da399bbe60e5d5b48a61da5
Reviewed-on: http://review.couchbase.org/42515
Reviewed-by: John Liang <johliang00@gmail.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
value.Bytes() changed to value.MarshalJSON().

Change-Id: I32b972579ee153d4851aacd1488f5223cb410184
Reviewed-on: http://review.couchbase.org/42558
Reviewed-by: Chris Hillery <ceej@couchbase.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
Make Index Manager Optional

Till the Indexer/Index Manager integration is complete, making the
Index Manager optional to make things easier.

There is a flag enable_manager which can be set to true while
running Indexer to make it use Index Manager.
By default, enable_manager is set to false.

Change-Id: I250ccf9bc633ba2fa22cf5c28ccf15368ed94ce4
Reviewed-on: http://review.couchbase.org/42509
Reviewed-by: John Liang <johliang00@gmail.com>
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Tested-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
CMakeLists.txt

Change-Id: Iac9a85788dfab1617d8a8cf0ae1fca7d9189b74e