Project: springxd
Issue XD-3751: gpfdist may fail to shutdown with backlog
In a case where reactor's ringbuffer is full and thus handling backpressure by blocking `onNext`, shutdown phase where `onComplete` is send will cause a deadlock. This is shown by a thread dump during a shutdown. This will basically break further deployments for this stream in distributed mode while single node will show more errors during undeployment. {code} "pool-7-thread-1" #58 prio=5 os_prio=0 tid=0x979fe800 nid=0x54de runnable [0x986ad000] java.lang.Thread.State: TIMED_WAITING (parking) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338) reactor.jarjar.com.lmax.disruptor.SingleProducerSequencer.next(SingleProducerSequencer.java:122) reactor.jarjar.com.lmax.disruptor.SingleProducerSequencer.next(SingleProducerSequencer.java:97) reactor.jarjar.com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246) reactor.core.processor.util.RingBufferSubscriberUtils.onNext(RingBufferSubscriberUtils.java:30) reactor.core.processor.RingBufferProcessor.onNext(RingBufferProcessor.java:575) {code} {code} "main-EventThread" #19 daemon prio=5 os_prio=0 tid=0x9b93a400 nid=0x54b1 runnable [0x9aefe000] java.lang.Thread.State: TIMED_WAITING (parking) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338) reactor.jarjar.com.lmax.disruptor.SingleProducerSequencer.next(SingleProducerSequencer.java:122) reactor.jarjar.com.lmax.disruptor.SingleProducerSequencer.next(SingleProducerSequencer.java:97) reactor.jarjar.com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246) reactor.core.processor.util.RingBufferSubscriberUtils.onComplete(RingBufferSubscriberUtils.java:54) reactor.core.processor.RingBufferProcessor.onComplete(RingBufferProcessor.java:585) org.springframework.xd.greenplum.gpfdist.GPFDistMessageHandler.doStop(GPFDistMessageHandler.java:170) {code} I've been crafting workaround for this by trying to wait reactor stream/buffer to get drained by gpdb and finally as last resort, forcing processor in reactor to shutdown.
Story points: 5