Project: datamanagement
Issue DM-3522: Releasing un-acquired resources bug
Running a mix of queries: 75 low volume and 10 high volume that include near neighbor failed at some point with {quote} terminate called after throwing an instance of 'lsst::qserv::Bug' what(): ChunkResource ChunkEntry::release: Error releasing un-acquired resource {quote} Stack trace: {quote} #0 0x00007fb6b0cce5e9 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install expat-2.1.0-8.el7.x86_64 glibc-2.17-78.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.12.2-14.el7.x86_64 libcom_err-1.42.9-7.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libicu-50.1.2-11.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libstdc++-4.8.3-9.el7.x86_64 nss-softokn-freebl-3.16.2.3-9.el7.x86_64 openssl-libs-1.0.1e-42.el7_1.9.x86_64 pcre-8.32-14.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64 (gdb) where #0 0x00007fb6b0cce5e9 in raise () from /lib64/libc.so.6 #1 0x00007fb6b0ccfcf8 in abort () from /lib64/libc.so.6 #2 0x00007fb6b15d29b5 in __gnu_cxx::__verbose_terminate_handler() () from /lib64/libstdc++.so.6 #3 0x00007fb6b15d0926 in ?? () from /lib64/libstdc++.so.6 #4 0x00007fb6b15cf8e9 in ?? () from /lib64/libstdc++.so.6 #5 0x00007fb6b15d0554 in __gxx_personality_v0 () from /lib64/libstdc++.so.6 #6 0x00007fb6b1069913 in ?? () from /lib64/libgcc_s.so.1 #7 0x00007fb6b1069e47 in _Unwind_Resume () from /lib64/libgcc_s.so.1 #8 0x00007fb6ab554247 in lsst::qserv::wdb::ChunkResourceMgr::Impl::release (this=0x21d1cc0, i=...) at build/wdb/ChunkResource.cc:398 #9 0x00007fb6ab552696 in lsst::qserv::wdb::ChunkResource::~ChunkResource (this=0x7fb68a5f9b70, __in_chrg=) at build/wdb/ChunkResource.cc:131 #10 0x00007fb6ab560f0f in lsst::qserv::wdb::QueryAction::Impl::_dispatchChannel (this=0x7fb65848c4d0) at build/wdb/QueryAction.cc:392 #11 0x00007fb6ab55f5ab in lsst::qserv::wdb::QueryAction::Impl::act (this=0x7fb65848c4d0) at build/wdb/QueryAction.cc:187 #12 0x00007fb6ab562084 in lsst::qserv::wdb::QueryAction::operator() (this=0x7fb658050548) at build/wdb/QueryAction.cc:450 #13 0x00007fb6ab544f46 in lsst::qserv::wcontrol::ForemanImpl::Runner::operator() (this=0x7fb67400fa20) at build/wcontrol/Foreman.cc:302 #14 0x00007fb6ab551cf0 in std::_Bind_simple::_M_invoke<>(std::_Index_tuple<>) ( this=0x7fb67400fa20) at /usr/include/c++/4.8.2/functional:1732 #15 0x00007fb6ab551a8b in std::_Bind_simple::operator()() (this=0x7fb67400fa20) at /usr/include/c++/4.8.2/functional:1720 {quote} Tail of log file from xrootd log: {quote} 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG GroupSched (build/wsched/GroupScheduler.cc:139) - _getNextTasks(1)>->-> 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG GroupSched (build/wsched/GroupScheduler.cc:151) - Returning 1 to launch 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG GroupSched (build/wsched/GroupScheduler.cc:154) - _getNextTasks <<<<< 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG ScanSched (build/wsched/ScanScheduler.cc:172) - _getNextTasks(29)>->-> 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG ScanSched (build/wsched/ChunkDisk.cc:199) - ChunkDisk busyness: yes 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG ScanSched (build/wsched/ChunkDisk.cc:171) - ChunkDisk getNext: current= (scan=10436, cached=8360,8259,) candidate=10301 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG ScanSched (build/wsched/ChunkDisk.cc:184) - ChunkDisk denying task 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG ScanSched (build/wsched/ScanScheduler.cc:196) - _getNextTasks <<<<< 0821 19:08:58.531 [0x7fb68a6fb700] INFO root (build/xrdsvc/SsiSession.cc:120) - Enqueued TaskMsg for Resource(/chk/LSST/2732) in 0.001016 seconds 0821 19:08:58.531 [0x7fb6895f8700] DEBUG Foreman (build/wcontrol/Foreman.cc:175) - Registered runner 0x7fb66c141ab0 0821 19:08:58.531 [0x7fb6895f8700] DEBUG Foreman (build/wcontrol/Foreman.cc:209) - Started task Task: msg: session=434445 chunk=2732 db=LSST entry time= frag: q=SELECT o.deepSourceId,o.ra,o.decl,s.coord_ra,s.coord_decl,s.parent FROM LSST.Object_2732 AS o,LSST.Source_2732 AS s WHERE scisql_s2PtInBox(o.ra,o.decl,48.482655,-54.274507,48.555903,-54.196952)=1 AND scisql_s2PtInBox(s.coord_ra,s.coord_decl,48.482655,-54.274507,48.555903,-54.196952)=1 AND o.deepSourceId=s.objectId, sc= rt=r_4344458c9456ede5cbe0b5f42e1a1571d5dd73_2732_0 0821 19:08:58.531 [0x7fb6895f8700] INFO Foreman (build/wcontrol/Foreman.cc:296) - Runner running Task: msg: session=434445 chunk=2732 db=LSST entry time= frag: q=SELECT o.deepSourceId,o.ra,o.decl,s.coord_ra,s.coord_decl,s.parent FROM LSST.Object_2732 AS o,LSST.Source_2732 AS s WHERE scisql_s2PtInBox(o.ra,o.decl,48.482655,-54.274507,48.555903,-54.196952)=1 AND scisql_s2PtInBox(s.coord_ra,s.coord_decl,48.482655,-54.274507,48.555903,-54.196952)=1 AND o.deepSourceId=s.objectId, sc= rt=r_4344458c9456ede5cbe0b5f42e1a1571d5dd73_2732_0 0821 19:08:58.531 [0x7fb6895f8700] INFO Foreman (build/wdb/QueryAction.cc:177) - Exec in flight for Db = q_fd51ad249f62fb765e173d7b3cae5d94 0821 19:08:58.531 [0x7fb6895f8700] WARN Foreman (build/wdb/QueryAction.cc:109) - QueryAction overriding dbName with LSST 0821 19:08:58.718 [0x7fb6a0d8e700] INFO root (build/wdb/QueryAction.cc:261) - &&& _fillRows size=106 0821 19:08:58.718 [0x7fb6a0d8e700] INFO root (build/wdb/QueryAction.cc:261) - &&& _fillRows size=210 0821 19:08:58.718 [0x7fb6a0d8e700] INFO root (build/wdb/QueryAction.cc:261) - &&& _fillRows size=316 ...(thousands of _fillRows lines) terminate called after throwing an instance of 'lsst::qserv::Bug' what(): ChunkResource ChunkEntry::release: Error releasing un-acquired resource {quote}
Story points: 3