Project: datamanagement
Issue DM-4407: Debug Qserv on ccqserv125..ccqserv149
It seems that some chunkQuery doesn't return on long queries like "Select count(*) From Object" The query stall and print in czar log: {code} 2015-11-24T15:07:55.324Z [0x7f06b37fe700] INFO root (core/modules/qdisp/Executive.cc:395) - Still 9 in flight. {code} If we look in the logs with next commands: {code:bash} cat /qserv/run/var/log/qserv-czar.log | egrep "Executive::add\(job\(id=" | grep LSST | cut -d'=' -f2 | cut -d' ' -f1 | sort > JOBID_ADD.txt cat /qserv/run/var/log/qserv-czar.log | egrep "markCompleted" | cut -d'(' -f 3 | cut -d',' -f1 | sort > JOBID_DONE.txt {code} And then {code} qserv@ccqserv125:/qserv$ diff JOBID_ADD.txt JOBID_DONE.txt 1826d1825 < 2639 2217d2215 < 2991 2904d2901 < 3609 3088d3084 < 3775 3152d3147 < 3832 3433d3427 < 4085 4227d4220 < 480 5478d5470 < 5926 6937d6928 < 7239 {code} 9 jobs aren't completed on czar. If we look the the chunk_id of one of this job: {code} /opt/shmux/bin/shmux -c "locate 5299" ccqserv{125..149}.in2p3.fr ... ccqserv148.in2p3.fr: /qserv/data/mysql/LSST/Object_5299.MYD ccqserv148.in2p3.fr: /qserv/data/mysql/LSST/Object_5299.MYI ccqserv148.in2p3.fr: /qserv/data/mysql/LSST/Object_5299.frm ... {code} Data is on worker node and onccqserv148 xrootd log: {code} [2015-11-24T15:07:42.990Z] [0x7ffbc8244700] INFO root (core/modules/xrdsvc/SsiSession.cc:125) - Enqueued TaskMsg for Resource(/chk/LSST/5299) in 0.000465 {code} But a thread seems to be locked: {code} #gdb on ccqserv148 xrootd process: (gdb) thread 7 [Switching to thread 7 (Thread 0x7ffbad7fa700 (LWP 192))] #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 185 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory. (gdb) bt #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffbd4c45c7c in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #2 0x00007ffbccbe8db9 in std::condition_variable::wait >(std::unique_lock &, lsst::qserv::wsched::BlendScheduler::) ( this=0xfba620, __lock=..., __p=...) at /usr/include/c++/4.9/condition_variable:98 #3 0x00007ffbccbe88f9 in lsst::qserv::wsched::BlendScheduler::getCmd (this=0xfba5c0, wait=true) at core/modules/wsched/BlendScheduler.cc:156 #4 0x00007ffbccba3b07 in lsst::qserv::util::EventThread::handleCmds (this=0xfbe890) at core/modules/util/EventThread.cc:45 #5 0x00007ffbccbab137 in std::_Mem_fn::operator()<, void>(lsst::qserv::util::EventThread*) const (this=0xfbe900, __object=0xfbe890) at /usr/include/c++/4.9/functional:569 #6 0x00007ffbccbaafff in std::_Bind_simple (lsst::qserv::util::EventThread*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0xfbe8f8) at /usr/include/c++/4.9/functional:1700 #7 0x00007ffbccbaae45 in std::_Bind_simple (lsst::qserv::util::EventThread*)>::operator()() (this=0xfbe8f8) at /usr/include/c++/4.9/functional:1688 #8 0x00007ffbccbaacfa in std::thread::_Impl (lsst::qserv::util::EventThread*)> >::_M_run() (this=0xfbe8e0) at /usr/include/c++/4.9/thread:115 #9 0x00007ffbd4c49970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007ffbd50ae0a4 in start_thread (arg=0x7ffbad7fa700) at pthread_create.c:309 #11 0x00007ffbd43b904d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 {code} Something seems to prevent return of chunk query result...
Story points: 5