Project: datamanagement
Issue DM-3432: Debug problem with timeout
If I run 4 simultaneous large queries: 3 object scans and 1 source scan, Xrootd silently died on 2 machines. Below I pasted the tail of the log files ccqserv108 0808 00:22:33.824 [0x7fb739583700] WARN Foreman (build/wdb/QueryAction.cc:109) - QueryAction overriding dbName with LSST 0808 00:22:33.824 [0x7fb74a00e700] INFO root (build/xrdsvc/ChannelStream.cc:122) - returning buffer (256, (more)) 0808 00:22:33.826 [0x7fb74a00e700] INFO root (build/xrdsvc/ChannelStream.cc:122) - returning buffer (62, (last)) 0808 00:22:33.826 [0x7fb74a00e700] INFO root (build/xrdsvc/SsiSession.cc:153) - RequestFinished type=isStream 0808 00:22:34.468 [0x7fb739583700] INFO root (build/wdb/QueryAction.cc:261) - &&& _fillRows size=8 0808 00:22:34.469 [0x7fb739583700] DEBUG root (build/wdb/QueryAction.cc:288) - _transmit last=1 0808 00:22:34.469 [0x7fb739583700] DEBUG root (build/wdb/QueryAction.cc:307) - _transmitHeader 0808 00:22:34.469 [0x7fb739583700] INFO root (build/proto/ProtoHeaderWrap.cc:52) - msgBuf size=256 -> [[0]=40, [1]=13, [2]=2, [3]=0, [4]=0, ..., [251]=48, [252]=48, [253]=48, [254]=48, [255]=48] 0808 00:22:34.469 [0x7fb739583700] INFO root (build/xrdsvc/SsiSession_ReplyChannel.cc:85) - sendStream, checking stream 0 len=256 last=0 ----- ccqserv124 0808 00:22:25.329 [0x7f25a67bf700] DEBUG ScanSched (build/wsched/ChunkDisk.cc:234) - ChunkDisk registering for 2044 : SELECT MIN(ra) AS QS1_MIN,MAX(ra) AS QS2_MAX,MIN(decl) AS QS3_MIN,MAX(decl) AS QS4_MAX FROM LSST.Object_2044 AS QST_1_ p=0x7f257003d0b8 0808 00:22:25.329 [0x7f25a67bf700] INFO Foreman (build/wcontrol/Foreman.cc:296) - Runner running Task: msg: session=7 chunk=2044 db=LSST entry time=Fri Aug 7 23:52:06 2015 frag: q=SELECT MIN(ra) AS QS1_MIN,MAX(ra) AS QS2_MAX,MIN(decl) AS QS3_MIN,MAX(decl) AS QS4_MAX FROM LSST.Object_2044 AS QST_1_, sc= rt=r_7bff268d0e369dda8fa314132538a96ad_2044_0 0808 00:22:25.329 [0x7f25a67bf700] INFO Foreman (build/wdb/QueryAction.cc:177) - Exec in flight for Db = q_b762c96f418726ae3457c74c0350d0c4 0808 00:22:25.329 [0x7f25a67bf700] WARN Foreman (build/wdb/QueryAction.cc:109) - QueryAction overriding dbName with LSST 0808 00:22:25.330 [0x7f25a50b6700] INFO root (build/xrdsvc/ChannelStream.cc:122) - returning buffer (44, (last)) 0808 00:22:25.330 [0x7f25a50b6700] INFO root (build/xrdsvc/SsiSession.cc:153) - RequestFinished type=isStream 0808 00:22:26.218 [0x7f25a67bf700] INFO root (build/wdb/QueryAction.cc:261) - &&& _fillRows size=81 0808 00:22:26.218 [0x7f25a67bf700] DEBUG root (build/wdb/QueryAction.cc:288) - _transmit last=1 0808 00:22:26.218 [0x7f25a67bf700] DEBUG root (build/wdb/QueryAction.cc:307) - _transmitHeader 0808 00:22:26.218 [0x7f25a67bf700] INFO root (build/proto/ProtoHeaderWrap.cc:52) - msgBuf size=256 -> [[0]=40, [1]=13, [2]=2, [3]=0, [4]=0, ..., [251]=48, [252]=48, [253]=48, [254]=48, [255]=48] 0808 00:22:26.218 [0x7f25a67bf700] INFO root (build/xrdsvc/SsiSession_ReplyChannel.cc:85) - sendStream, checking stream 0 len=256 last=0
Story points: 4