Project: usergrid
Issue USERGRID-785: Unable to run 2.0 to 2.1 data migration on large data set.
Attempted to run entity data migration in (UG 2.0 to UG 2.1 ) with a instance having 1.1 million entities in a single collection, within a single application. This causes the following exception on cassandra: {code} ERROR [ReadStage:16] 2015-07-02 23:33:29,720 SliceQueryFilter.java (line 206) Scanned over 100000 tombstones in ug_migrate_test.Graph_Source_Node_Edges; query aborted (see tombstone_failure_threshold) ERROR [ReadStage:27] 2015-07-02 23:33:39,723 SliceQueryFilter.java (line 206) Scanned over 100000 tombstones in ug_migrate_test.Graph_Source_Node_Edges; query aborted (see tombstone_failure_threshold) ERROR [ReadStage:27] 2015-07-02 23:33:39,723 CassandraDaemon.java (line 258) Exception in thread Thread[ReadStage:27,5,main] java.lang.RuntimeException: org.apache.cassandra.db.filter.TombstoneOverwhelmingException org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2016) {code} which results in the following exception in usergrid: {code} 2015-07-02 23:33:59,680 [Index migrate data formats] ERROR org.apache.usergrid.rest.MigrateResource- Unable to migrate data java.lang.RuntimeException: Unable to connect to casandra org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.advance(MultiRowColumnIterator.java:190) org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.hasNext(MultiRowColumnIterator.java:122) org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardsColumnIterator.hasNext(ShardsColumnIterator.java:65) org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupColumnIterator.advance(ShardGroupColumnIterator.java:120) org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupColumnIterator.hasNext(ShardGroupColumnIterator.java:68) org.apache.usergrid.persistence.core.rx.ObservableIterator.call(ObservableIterator.java:66) org.apache.usergrid.persistence.core.rx.ObservableIterator.call(ObservableIterator.java:38) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable.unsafeSubscribe(Observable.java:7495) rx.internal.operators.OperatorMerge$MergeSubscriber.handleNewSource(OperatorMerge.java:215) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:185) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:120) rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:55) rx.internal.operators.OperatorDoOnEach$1.onNext(OperatorDoOnEach.java:84) org.apache.usergrid.persistence.core.rx.ObservableIterator.call(ObservableIterator.java:71) org.apache.usergrid.persistence.core.rx.ObservableIterator.call(ObservableIterator.java:38) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable.unsafeSubscribe(Observable.java:7495) rx.internal.operators.OperatorMerge$MergeSubscriber.handleNewSource(OperatorMerge.java:215) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:185) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:120) rx.internal.operators.OnSubscribeFromIterable$IterableProducer.request(OnSubscribeFromIterable.java:96) rx.Subscriber.setProducer(Subscriber.java:177) rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:47) rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:33) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable.unsafeSubscribe(Observable.java:7495) rx.internal.operators.OperatorMerge$MergeSubscriber.handleNewSource(OperatorMerge.java:215) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:185) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:120) rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:55) rx.internal.operators.OperatorMerge$InnerSubscriber.emit(OperatorMerge.java:676) rx.internal.operators.OperatorMerge$InnerSubscriber.onNext(OperatorMerge.java:586) rx.internal.operators.OperatorMerge$InnerSubscriber.emit(OperatorMerge.java:676) rx.internal.operators.OperatorMerge$InnerSubscriber.onNext(OperatorMerge.java:586) rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:55) rx.internal.operators.OperatorFilter$1.onNext(OperatorFilter.java:54) rx.internal.operators.OperatorDoOnEach$1.onNext(OperatorDoOnEach.java:84) rx.internal.operators.OperatorMerge$MergeSubscriber.handleScalarSynchronousObservableWithRequestLimits(OperatorMerge.java:280) rx.internal.operators.OperatorMerge$MergeSubscriber.handleScalarSynchronousObservable(OperatorMerge.java:243) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:176) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:120) rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:55) rx.internal.operators.OperatorDoOnEach$1.onNext(OperatorDoOnEach.java:84) org.apache.usergrid.persistence.collection.impl.EntityCollectionManagerImpl$1.call(EntityCollectionManagerImpl.java:248) org.apache.usergrid.persistence.collection.impl.EntityCollectionManagerImpl$1.call(EntityCollectionManagerImpl.java:240) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable.unsafeSubscribe(Observable.java:7495) rx.internal.operators.OperatorMerge$MergeSubscriber.handleNewSource(OperatorMerge.java:215) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:185) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:120) rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:55) rx.internal.operators.OperatorDoOnEach$1.onNext(OperatorDoOnEach.java:84) rx.internal.operators.OperatorMerge$InnerSubscriber.emit(OperatorMerge.java:676) rx.internal.operators.OperatorMerge$InnerSubscriber.onNext(OperatorMerge.java:586) rx.internal.operators.OperatorFilter$1.onNext(OperatorFilter.java:54) rx.internal.operators.OnSubscribeFromIterable$IterableProducer.request(OnSubscribeFromIterable.java:96) rx.Subscriber.setProducer(Subscriber.java:177) rx.Subscriber.setProducer(Subscriber.java:171) rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:47) rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:33) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable.unsafeSubscribe(Observable.java:7495) rx.internal.operators.OperatorMerge$MergeSubscriber.handleNewSource(OperatorMerge.java:215) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:185) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:120) rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:55) rx.internal.operators.OperatorBufferWithSize$1.onCompleted(OperatorBufferWithSize.java:119) org.apache.usergrid.persistence.core.rx.ObservableIterator.call(ObservableIterator.java:75) org.apache.usergrid.persistence.core.rx.ObservableIterator.call(ObservableIterator.java:38) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable.unsafeSubscribe(Observable.java:7495) rx.internal.operators.OperatorMerge$MergeSubscriber.handleNewSource(OperatorMerge.java:215) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:185) rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:120) rx.internal.operators.OnSubscribeFromIterable$IterableProducer.request(OnSubscribeFromIterable.java:96) rx.Subscriber.setProducer(Subscriber.java:177) rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:47) rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:33) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable$1.call(Observable.java:144) rx.Observable$1.call(Observable.java:136) rx.Observable.unsafeSubscribe(Observable.java:7495) rx.internal.operators.OperatorSubscribeOn$1$1.call(OperatorSubscribeOn.java:62) rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Caused by: com.netflix.astyanax.connectionpool.exceptions.OperationTimeoutException: OperationTimeoutException: [host=10.16.4.135(10.16.4.135):9160, latency=5001(30012), attempts=6]TimedOutException() com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:171) com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65) com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28) com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151) com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119) com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:338) com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4.execute(ThriftColumnFamilyQueryImpl.java:532) org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.advance(MultiRowColumnIterator.java:187)	... 146 more Caused by: TimedOutException() org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:10480) org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:673) org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:657) com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4$1.internalExecute(ThriftColumnFamilyQueryImpl.java:538) com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4$1.internalExecute(ThriftColumnFamilyQueryImpl.java:535) com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60)	... 152 more {code}
Story points: 3