opened
ArihantRk
username_0
### Sync Gateway version
1.4.1
### Operating system
redHat 7.2

### Log output
2017/05/26 16:15:23 feed_dcp: rollback, name: ptxdataf9388637fd07ef33c4e9e5e81bc0f914_56d4e5fdfdbc2a79: vbucketId: 324, rollbackSeq: 5, partition: 324, opaqueValue: {"seqStart":0,"seqEnd":0,"snapStart":13,"snapEnd":13,"failOverLog":[[281061217003463,5],[112694005818259,5],[152980007582,0]]}, lastSeq: 13
2017-05-26T16:15:23.364+05:30 WARNING: Index pending worker process terminated. -- runtime.call64() at asm_amd64.s:480
panic: runtime error: slice bounds out of range

goroutine 173 [running]:
panic(0xb41960, 0xc420010130)
    /usr/local/go/1.7.4/go/src/runtime/panic.go:500 +0x1a1
github.com/couchbase/sync_gateway/db.(*DenseBlock).rollbackEntries(0xc42014b1c0, 0x144, 0x5, 0x14c20ef94c3b0000, 0x0)
    /home/couchbase/jenkins/workspace/sgw-unix-build@2/1.4.1/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/kv_dense_block.go:348 +0x1e0
github.com/couchbase/sync_gateway/db.(*DenseBlock).RollbackTo(0xc42014b1c0, 0x144, 0x5, 0x1213ee0, 0xc420600600, 0xc42014b100, 0xc4201a0840, 0xc4201a0840)
    /home/couchbase/jenkins/workspace/sgw-unix-build@2/1.4.1/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/kv_dense_block.go:274 +0x6e
github.com/couchbaselabs/sync-gateway-accel/indexwriter.(*DensePartitionStorage).RollbackTo(0xc4206e2570, 0xc4215a0144, 0x5, 0xc42055d588, 0xc420600600)
    /home/couchbase/jenkins/workspace/sgw-unix-build@2/1.4.1/enterprise/godeps/src/github.com/couchbaselabs/sync-gateway-accel/indexwriter/dense_channel_storage_writer.go:248 +0xc1
github.com/couchbaselabs/sync-gateway-accel/indexwriter.(*DenseStorageWriter).RollbackTo(0xc4215ae480, 0xc420190144, 0x5, 0x10, 0x1)
    /home/couchbase/jenkins/workspace/sgw-unix-build@2/1.4.1/enterprise/godeps/src/github.com/couchbaselabs/sync-gateway-accel/indexwriter/dense_channel_storage_writer.go:100 +0xa3
github.com/couchbaselabs/sync-gateway-accel/indexwriter.(*ChannelWriter).Rollback(0xc4202d8e70, 0xc4210d0144, 0x5, 0xc4202d8e70, 0x0)
    /home/couchbase/jenkins/workspace/sgw-unix-build@2/1.4.1/enterprise/godeps/src/github.com/couchbaselabs/sync-gateway-accel/indexwriter/channel_writer.go:85 +0x1ef

Full log info 

### Expected behavior
sg-accelerator should continue to work

### Actual behavior
sg-accelerator crashing.
sg-accelerator crashing with panic: runtime error: slice bounds out of range
issue
created
adamcfraser
username_1
Can you provide any additional steps to reproduce?  What was being done when you hit this issue?
comment
created
ArihantRk
username_0
@username_1 steps below
1. Restart one node lead to failover and added back with full rebalance
2. Step 1 done for other 2 nodes in 3 node cluster
3. After 2 step, all CBS were in running healthy state and syncgateway and sg-replicate are restarted (as per our design) and cameup normal

but could see sg-accelerator is crashing leading full outage as syncgateway is dependent .
its blocker for us. Any workaround on this.
comment
created
adamcfraser
username_1
Will investigate.  Until there's a fix, you could force a rebuild of your index bucket to avoid the rollback.

1. Stop sg/accel nodes
2. Flush the index bucket
3. Restart sg/accel nodes.
comment
created
ArihantRk
username_0
Thanks @username_1 for quick one.
comment
created
adamcfraser
username_1
I haven't been able to reproduce yet.  If you're able to reproduce this reliably, can you enable the  "Rollback" log key in the SG-accel config, and share the accel logs?
comment
created
ArihantRk
username_0
@username_1 currently we removed the sg-accelerator out of setup as it is not reliable and also backwarrd compatible issue after bringing sg-accelerator. we observed issue during normal traffic condition after mentioned steps. below logs link captured during issue time. 
https://gist.github.com/username_0/c9b44e87d183e62a1ec9ae316ee8dbfa

From the above logs, can this be verified what might have caused crash.
comment
created
adamcfraser
username_1
@username_0 Can you explain what you mean by 'backward compatible issue'?  

If you're able to provide logs with log '*' enabled on the sg_accel node, it will provide some more context on the error.
comment
created
ArihantRk
username_0
@username_1 below is my observations during sg-accelrator testing
before bringing sg-accelerator, we could see global sequence number shared by all syncgateway which used uniquly writing documents to cluster. also each documents will have **seq** attribute associated with it.
Each of client uses _local documents for verifying any changes in server through _changes(which doe not have any hashing(of sequence number) mechanism as per my understanding, please correct me if i am wrong in understanding)

with bringing global sequence is replaced with sequence number based on vbucket numbers(not sure exactly) and also syncgateway does not understand documents inserted without sg-accelerator.

we tried bringing sg--accelerator without flushing but we faced issues. 
please also let us know upgrading existing setup(without sg-accelerator) to sg-accelerator without data loss.
comment
created
adamcfraser
username_1
Thanks - will review.  I don't see the actual panic in the latest logs - is that just an excerpt?
comment
created
ArihantRk
username_0
@username_1 , Not sure this is panic or exception.  Issue is observed very frequently. SG-Accelerator is not recovering after that.

**Setup details**
3* CBS nodes
2* SyncGateway nodes
1* SgAccelerator

AutoFailover set to 60 seconds
**Issue reproduce steps below**
1. Run the traffic
2. Restart one of CBS nodes and node got failover
3. Restart SG-Accelerator node

Also issue observed SG-Accelerator restart during rebalance.
comment
created
adamcfraser
username_1
Can you clarify exactly what you mean by 'SG-Accelerator is not recovering'?  What's the behaviour you're observing?
comment
created
ArihantRk
username_0
@username_1 Means , Sync-gateway service is not starting. Could see below messages in error.log

panic: runtime error: slice bounds out of range

goroutine 167 [running]:
panic(0xb41960, 0xc420014150)
	/usr/local/go/1.7.4/go/src/runtime/panic.go:500 +0x1a1
github.com/couchbase/sync_gateway/db.(*DenseBlock).rollbackEntries(0xc420353780, 0x322, 0xb, 0x14c7f75fc31c0000, 0x0)
	/home/couchbase/jenkins/workspace/sgw-unix-build@2/1.4.1/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/kv_dense_block.go:348 +0x1e0
github.com/couchbase/sync_gateway/db.(*DenseBlock).RollbackTo(0xc420353780, 0x322, 0xb, 0x1213ee0, 0xc4203c0a00, 0xc420353700, 0xc420706480, 0xc420706480)
	/home/couchbase/jenkins/workspace/sgw-unix-build@2/1.4.1/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/kv_dense_block.go:274 +0x6e
github.com/couchbaselabs/sync-gateway-accel/indexwriter.(*DensePartitionStorage).RollbackTo(0xc42080db90, 0xc4204e0322, 0xb, 0xc420c25578, 0xc4203c0a00)
	/home/couchbase/jenkins/workspace/sgw-unix-build@2/1.4.1/enterprise/godeps/src/github.com/couchbaselabs/sync-gateway-accel/indexwriter/dense_channel_storage_writer.go:248 +0xc1
github.com/couchbaselabs/sync-gateway-accel/indexwriter.(*DenseStorageWriter).RollbackTo(0xc4204ed050, 0xc420ac0322, 0xb, 0x10, 0x1)
	/home/couchbase/jenkins/workspace/sgw-unix-build@2/1.4.1/enterprise/godeps/src/github.com/couchbaselabs/sync-gateway-accel/indexwriter/dense_channel_storage_writer.go:100 +0xa3
github.com/couchbaselabs/sync-gateway-accel/indexwriter.(*ChannelWriter).Rollback(0xc4206e01c0, 0xc4204e0322, 0xb, 0xc4206e01c0, 0x0)
	/home/couchbase/jenkins/workspace/sgw-unix-build@2/1.4.1/enterprise/godeps/src/github.com/couchbaselabs/sync-gateway-accel/indexwriter/channel_writer.go:85 +0x1ef
github.com/couchbaselabs/sync-gateway-accel/indexwriter.(*KvChangeIndexWriter).rollbackIndex(0xc4202608f0, 0xc420c25a48, 0xc418a85b70, 0x1497b00)
	/home/couchbase/jenkins/workspace/sgw-unix-build@2/1.4.1/enterprise/godeps/src/github.com/couchbaselabs/sync-gateway-accel/indexwriter/kv_change_index_writer.go:948 +0x1a8
github.com/couchbaselabs/sync-gateway-accel/indexwriter.(*KvChangeIndexWriter).indexEntries(0xc4202608f0, 0xc4210ac000, 0xc, 0x7d0, 0xc4200e3800, 0x400, 0x400, 0x120cd80, 0xc420223920, 0x0, ...)
comment
created
ajres
username_2
From the logs provided the panic is in kv_dens_block.go rollbackEntries, at the last line in the code below

```
func (d *DenseBlock) rollbackEntries(vbNo uint16, seq uint64) (numRemoved int, rollbackComplete bool) {

	// Work backwards through the block, removing entries greater than vbNo, seq
	indexPos := DB_HEADER_LEN + uint32(d.getEntryCount()-1)*INDEX_ENTRY_LEN
	entryPos := uint32(len(d.value))

	for {
		indexEntry := d.GetIndexEntry(int64(indexPos))
```
it looks like the "panic: runtime error: slice bounds out of range" error is actually coming from  d.GetIndexEntry

```
func (d *DenseBlock) GetIndexEntry(position int64) (indexEntry DenseBlockIndexEntry) {
	indexEntry = d.value[position : position+INDEX_ENTRY_LEN]
	return indexEntry
}
```
comment
created
ajres
username_2
@username_1 how would it make most sense to handle this panic in the calling rollbackEntries method?
comment
created
adamcfraser
username_1
The panic itself could be avoided by using a wrapper like [SafeSlice](https://github.com/couchbaselabs/sync-gateway-accel/blob/7e4f6b042e7a829acc777edc07ffcb8efffcfb81/indexwriter/util.go#L23), but we also need to understand why it's failing in the first place and get the root cause fixed.  Using SafeSlice would be a good starting point - we'd at least be able to log additional diagnostics about the slice at the time of failure, but we definitely need to get a repro scenario for this issue.
comment
created
ajres
username_2
Created ticket #2777 to cover the prevention of the panic and logging debug information.

The work to repro and fix the root cause remains on this ticket.
comment
created
ajres
username_2
There has been an issue fixed in 1.5, where rollback handling would trigger this panic if attempting to roll back the index for a given vbucket, when the channel didn't have any data for that vbucket. This has been fixed as part of https://github.com/couchbase/sync_gateway/pull/2785.
There isn't enough information on this ticket to exactly reproduce the error, and so confirm that the OP is hitting this scenario covered by https://github.com/couchbase/sync_gateway/pull/2785. 
It feels like there's a good chance they are, but if they get 1.5 and are still hitting the issue, re-open this ticket and attach the SG logs containing the additional diagnostics that have been added to 1.5.
comment
closed
ajres
username_2
issue