Repository: pocoproject/poco

Issue #2085: Crash due to race condition in TCPServerDispatcher
Created: 2018-01-11T23:23:18Z
Expected behavior
Clean server shutdown with no crash.
Actual behavior
#0  0x00007fc7d030a0c1 in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fc7c0da4ed1 in ?? () from .so
#2  
#3  0x0000000009078380 in ?? ()
#4  0x00007fc7c2ff429f in Poco::PooledThread::run() () from bin/glnxa64/libPocoFoundation.so.48
#5  0x00007fc7c2feff86 in Poco::ThreadImpl::runnableEntry(void*) () from bin/glnxa64/libPocoFoundation.so.48
#6  0x00007fc7d0305064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007fc7d1ba462d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Steps to reproduce the problem
This is a very rare issue that has been causing sporadic crashes in our build and test environment. I believe the issue is reproducible when there is a server.stop() call concurrent with a new socket connection.
I believe the problem is that a new socket connection enqueues the TCPServerDispatcher instance without changing the reference count. That means that if you shutdown the server before the TCPServerDispatcher::run method starts the ref count would go to zero before the AutoPtr in run() has a chance to increment the ref count.
I think that incrementing the ref count before enqueuing TCPServerDispatcher (and decrementing at the end of run()) would resolve the issue and ensure that the run() method can be safely called during server shutdown.
POCO version
1.7.9
Compiler and version
gcc 6.3.0-GLIBC2.11
Operating system and version
debian 8
Other relevant information

Duplicate issue #2325: Bug: Segfault in ThreadPool.cpp / 1.9.0 / linux
Created: 2018-05-09T10:58:58Z
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51                                        
#1  0x00007ffff5683f5d in __GI_abort () at abort.c:90
#2  0x00007ffff5679f17 in __assert_fail_base (fmt=<optimized out>,                                                   assertion=assertion@entry=0x7ffff5c56f52 "mutex->__data.__owner == 0",
    file=file@entry=0x7ffff5c56f35 "../nptl/pthread_mutex_lock.c", line=line@entry=81,                               function=function@entry=0x7ffff5c570c0 <__PRETTY_FUNCTION__.8399> "__pthread_mutex_lock") at assert.c:92
#3  0x00007ffff5679fc2 in __GI___assert_fail (                                                                       assertion=assertion@entry=0x7ffff5c56f52 "mutex->__data.__owner == 0",
    file=file@entry=0x7ffff5c56f35 "../nptl/pthread_mutex_lock.c", line=line@entry=81,                               function=function@entry=0x7ffff5c570c0 <__PRETTY_FUNCTION__.8399> "__pthread_mutex_lock") at assert.c:101
#4  0x00007ffff5c4c348 in __GI___pthread_mutex_lock (mutex=<optimized out>) at ../nptl/pthread_mutex_lock.c:81
#5  0x000000000076fa5c in pthread_mutex_lock ()
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/ext/new_allocator.h:125
#6  0x0000000000e916ff in Poco::MutexImpl::lockImpl (this=0x7b54000004b8) at include/Poco/Mutex_POSIX.h:59
#7  Poco::FastMutex::lock (this=0x7b54000004b8) at include/Poco/Mutex.h:238                                      
#8  Poco::ScopedLock<Poco::FastMutex>::ScopedLock (mutex=..., this=<synthetic pointer>)
    at include/Poco/ScopedLock.h:38                                                                              
#9  Poco::PooledThread::idle (this=0x7b5400000280) at src/ThreadPool.cpp:130
#10 Poco::ThreadPool::housekeep (this=this@entry=0x2e1a2a8 <TCP::Server::server_pool>)                               at src/ThreadPool.cpp:423
#11 0x0000000000e921a8 in Poco::ThreadPool::getThread (this=0x2e1a2a8 <TCP::Server::server_pool>)                    at src/ThreadPool.cpp:458
#12 0x0000000000e92541 in Poco::ThreadPool::startWithPriority (this=<optimized out>,                                 priority=Poco::Thread::PRIO_NORMAL, target=..., name="TCPServerConnection") at src/ThreadPool.cpp:371
#13 0x0000000000e548ba in Poco::Net::TCPServerDispatcher::enqueue (this=0x7b44000b1440, socket=...)                  at src/TCPServerDispatcher.cpp:150
#14 0x0000000000e4cfe1 in Poco::Net::TCPServer::run (this=0x7b44000b3250) at src/TCPServer.cpp:148               #15 0x0000000000e8bce9 in Poco::ThreadImpl::runnableEntry (pThread=0x7b44000b3278) at src/Thread_POSIX.cpp:345
#16 0x000000000075fb3e in __tsan_thread_start_func ()                                                                at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/ext/new_allocator.h:125
#17 0x00007ffff5c497fc in start_thread (arg=0x7fff2b12e700) at pthread_create.c:465                              
#18 0x00007ffff575fb5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Unfortunately there is no simple snippet to reproduce, but reproducible on my machine. The impression I get is that poolThread already has been freed, but somehow still is in threadS list of ThreadPool. Though only a guess.

Closing comment (2018-05-09T13:17:28Z): probably duplicate of #2085. can you try with 1.9.1 branch?