dcp_feed: Fix failoverlog.Latest()

The Latest() method was returning oldest failover log entry
instead of latest entry.

Change-Id: Icf582db4eaa37bc4665c010dba1f431c144a03e7
Reviewed-on: http://review.couchbase.org/43664
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
Match ns_server assigned ports to projector

ns_server assigns ports to projector decrementing by 1
indexer should follow same logic

Change-Id: I9deaff798bed8f88a4e04d2a0eb04935c954f34f
Reviewed-on: http://review.couchbase.org/43679
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
Disambigute messages from indexing DCP consumer

Change-Id: I48df7ce464c6ebe736514c401b6138490e56dc00
Reviewed-on: http://review.couchbase.org/43686
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Sriram Melkote <siri@couchbase.com>
Reviewed-by: Sriram Melkote <siri@couchbase.com>
indexer: Use cluster address to discover about kv nodes

Currently indexer uses local projector address to discover about
kv nodes. This makes it impossible to run index server only node
in a cluster setup. This changes makes it possible for ns_server
to provide local cluster address to the indexer and indexer can
discover kv addresses by contacting cluster endpoint.

Change-Id: Ief9071f831ad3b5f59ec3fe2aabd9c7b5420796a
Reviewed-on: http://review.couchbase.org/43681
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
dcp/cb: Add thisNode field in node services

Change-Id: I7ea28d5e43457505909d814be525511bb4bc106e
Reviewed-on: http://review.couchbase.org/43671
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
common: Implement generic retry helper

Implemented a generic retry helper than can be used to call a
function repeatedly with backoff interval until the function
returns non-error value.

Example:

    fn := func(r int, lastErr error) error {
        if r > 0 {
            fmt.Printf("Error occured (%v) .. Retrying(%d)\n", lastErr, r)
        }
        if r == 8 {
            return nil
        }
        return errors.New("failed")
    }

    rh := common.NewRetryHelper(10, time.Second*2, 2, fn)
    err := rh.Run()

Change-Id: I7e9e9b03007207a62a9f4fdcd792de41f5034083
Reviewed-on: http://review.couchbase.org/43710
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
common: Implement ClusterInfoCache helper

ClusterInfoCache can be used to obtain information about node services,
node-bucket membership, node-vbucket memberships, current node, etc.

Change-Id: If7aac1bd17c6427dec6a64e48ecf9a7f641c959f
Reviewed-on: http://review.couchbase.org/43706
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
CBIDXT-278 Send error response to scan requests during rollback

When a rollback happens, if any scan requests are waiting for a matching
atleast-timestamp snapshot to be available, fail those scan requests with
ErrIndexRollback error.

Change-Id: Ic46cc13787107ee4a5e0c33d8a2a0d3e28541704
Reviewed-on: http://review.couchbase.org/43781
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
CBIDXT-279 scanner: Add support for inclusion option in range scans

Implemented forestdb snapshot scanner to perform the following
operations:

include_low=off:
 - Read and discard all equal keys=lowkey

include_high=on:
 - Read and send all equal keys=highkey

Change-Id: I3e1eb31cf7e1a79ecc0e9e5dcb843c14de7c3458
Reviewed-on: http://review.couchbase.org/43641
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
indexer: Discover projectors using ClusterInfoCache

Migrated existing kv_sender code to make use of
ClusterInfoCache for discovering projectors across
the cluster.

Change-Id: I9611eb0557c3ff7a673058154c114d59e90c71aa
Reviewed-on: http://review.couchbase.org/43714
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
cosmetics: included import keyword with pkg.

Change-Id: Ia69c1cdb158d35a915c783fb7cac06d12a851867
Reviewed-on: http://review.couchbase.org/43674
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
removing dependency on n1ql from common/ package.

moving into protobuf/ sub-package since evaluation is done here.

Change-Id: I08596711b0eed3acb3c95b6657ce2b9aa6babee6
Reviewed-on: http://review.couchbase.org/43675
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
granular protobuf dependency and cleanup.

protobuf/ package is split into three sub-packages
    projector/ - for projector adminport messages
    data/ - for mutation path from projector to indexer
    query/ - for querying index.
This will reduce this risk of circular dependency.

I have cleaned up most of the un-used protobuf messages which were originally
added to capture design discussion on 2i. Now that we have manager and indexer
evolving with a code-base, these messages and associated descriptions are not
necessary.

Change-Id: If7dd4763fcef50ac355e4ba53685cf1f4aadf960
Reviewed-on: http://review.couchbase.org/43676
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
Instance-wise configuration.

- components shall not read configuration from SystemConfig.
  instead, application shall read SystemConfig, override them with
  command-line switch and configuration file and pass them to
  component instances.
- Config.Override() function accepts more than one configuration
  object to apply.
- Following constructors accept max-vbuckets and config arugments,
  - dataport.NewServer() and dataport.NewClient()
  - projector.client.NewClient()
  - projector.NewProjector()
  - queryport.NewClient() and queryport.NewServer()
  - queryport.Application(), example application.
- fixed `indexer` to pass configuration object as constructor arugment.

Change-Id: Ic24402b5db7296c5fca0fba4aed1946c5cc4a55d
Reviewed-on: http://review.couchbase.org/43677
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
log message if back-channel has stray message.

fixed cbq-scripts.

Change-Id: I74445d60807c70ce24ae99cda164e6923f575384
Reviewed-on: http://review.couchbase.org/43678
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
Moved queryport-client files to client/ subpkg.

This is to reduce the possibility of circular dependency.

Change-Id: I6eed286e253805112e9b8c95a9d7dd66f4a0281a
Reviewed-on: http://review.couchbase.org/43707
Reviewed-by: Sriram Melkote <siri@couchbase.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
Added n1ql client for queryport.

- implements datastore.Index{} interface.
- Export functions to,
    Load2iIndexes()
    Create2iPrimaryIndex()
    Create2iIndex()

Change-Id: I0ebd26beffca9416ae86d0a89b04d5178a9f4f32
Reviewed-on: http://review.couchbase.org/43753
Reviewed-by: Sriram Melkote <siri@couchbase.com>
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Tested-by: Pratap Chakravarthy <prataprc@couchbase.com>
Fix Drop For Indexes During Initial Build

Drop for Indexes during Initial Build needs to clean
up index from maintenance stream as well

Change-Id: I5d415002c988a42e4e9d80f82fd8502aa359d9a1
Reviewed-on: http://review.couchbase.org/43713
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Tested-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Bugfix - Set HWT only for first StreamBegin in Recovery

During recovery, HWT for the restarted stream needs to be set
to the point of restart, only for the first stream begin message.
Once it is done, rest of the stream begins can just update the
latest vbuuid.

Change-Id: I98a4aa93e0792267667e45dd2c7e15ea77b50ee8
Reviewed-on: http://review.couchbase.org/43759
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Tested-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Fix Setting RestartTS In Failure Cases

Stream may get connection failure before any flush or stream begin.
In such cases restart TS for stream needs to be set properly.

Change-Id: I0099bd95f85abb0e955d84a450fc66c8578582b1
Reviewed-on: http://review.couchbase.org/43797
Reviewed-by: Pratap Chakravarthy <prataprc@couchbase.com>
Reviewed-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>
Tested-by: Deepkaran Salooja <deepkaran.couchbase@gmail.com>