Project: mesos
Issue MESOS-3201: Libev handle_async can deadlock with run_in_event_loop
Due to the arbitrary nature of the functions that are executed in handle_async, invoking them under the (A) {{watchers_mutex}} can lead to deadlocks if (B) is acquired before calling {{run_in_event_loop}} and (B) is also acquired within the arbitrary function. {code} ==82679== Thread #10: lock order "0x60774F8 before 0x60768C0" violated ==82679== ==82679== Observed (incorrect) order is: acquisition of lock at 0x60768C0 ==82679== at 0x4C32145: pthread_mutex_lock (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so) ==82679== by 0x692C9B: __gthread_mutex_lock(pthread_mutex_t*) (gthr-default.h:748) ==82679== by 0x6950BF: std::mutex::lock() (mutex:134) ==82679== by 0x696219: Synchronized synchronize(std::mutex*)::{lambda(std::mutex*)#1}::operator()(std::mutex*) const (synchronized.hpp:58) ==82679== by 0x696238: Synchronized synchronize(std::mutex*)::{lambda(std::mutex*)#1}::_FUN(std::mutex*) (synchronized.hpp:58) ==82679== by 0x6984CF: Synchronized::Synchronized(std::mutex*, void (*)(std::mutex*), void (*)(std::mutex*)) (synchronized.hpp:35) ==82679== by 0x6962DE: Synchronized synchronize(std::mutex*) (synchronized.hpp:60) ==82679== by 0x728FE1: process::handle_async(ev_loop*, ev_async*, int) (libev.cpp:48) ==82679== by 0x761384: ev_invoke_pending (ev.c:2994) ==82679== by 0x7643C4: ev_run (ev.c:3394) ==82679== by 0x728E37: ev_loop (ev.h:826) ==82679== by 0x729469: process::EventLoop::run() (libev.cpp:135) ==82679== ==82679== followed by a later acquisition of lock at 0x60774F8 ==82679== at 0x4C32145: pthread_mutex_lock (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so) ==82679== by 0x4C6F9D: __gthread_mutex_lock(pthread_mutex_t*) (gthr-default.h:748) ==82679== by 0x4C6FED: __gthread_recursive_mutex_lock(pthread_mutex_t*) (gthr-default.h:810) ==82679== by 0x4F5D3D: std::recursive_mutex::lock() (mutex:175) ==82679== by 0x516513: Synchronized synchronize(std::recursive_mutex*)::{lambda(std::recursive_mutex*)#1}::operator()(std::recursive_mutex*) const (synchronized.hpp:58) ==82679== by 0x516532: Synchronized synchronize(std::recursive_mutex*)::{lambda(std::recursive_mutex*)#1}::_FUN(std::recursive_mutex*) (synchronized.hpp:58) ==82679== by 0x52E619: Synchronized::Synchronized(std::recursive_mutex*, void (*)(std::recursive_mutex*), void (*)(std::recursive_mutex*)) (synchronized.hpp:35) ==82679== by 0x5165D4: Synchronized synchronize(std::recursive_mutex*) (synchronized.hpp:60) ==82679== by 0x6BF4E1: process::ProcessManager::use(process::UPID const&) (process.cpp:2127) ==82679== by 0x6C2B8C: process::ProcessManager::terminate(process::UPID const&, bool, process::ProcessBase*) (process.cpp:2604) ==82679== by 0x6C6C3C: process::terminate(process::UPID const&, bool) (process.cpp:3107) ==82679== by 0x692B65: process::Latch::trigger() (latch.cpp:53) {code} This was introduced in https://github.com/apache/mesos/commit/849fc4d361e40062073324153ba97e98e294fdf2
Story points: 3