<!-- Title: Owl shift. Thu Jul 19, 2007 -->
<!-- SciCo: Greg  -->
<!-- DAQAce: Petteri -->
<!-- MonAce: &nbsp; -->
<!-- CO: Laura  -->
<!-- OpManager: Farrukh -->
<!-- Notes: -->
<pre>TeV studies, calibrations, and L2 Upgrade studies in progress. </pre>

<!-- Date: Thu Jul 19 00:11:54.865 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245316>245316</A>
CALIBRATION: <U><B>toftac</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#toftac target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Thu Jul 19 00;28;01 comment by...Petteri -->
looks ok
<!-- Date: Thu Jul 19 00:11:55.703 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245318>245318</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] tracklist continues
<!-- Author: flanagan -->



<!-- Date: Thu Jul 19 00:13:52.566 2007 -->
<pre>Maggie calls from MCR. 
45 minutes of quiet time start now.</pre>
<!-- Author: greg -->

<!-- Date: Thu Jul 19 00:14:04.790 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245319>245319</A>
ACTIVATE: RunSet: ACE_QIE 
<!-- Author: Miguel M. -->


<!-- Date: Thu Jul 19 00:15:31.499 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245319>245319</A>
TERMINATE: done
<!-- Author: Miguel M. -->


<!-- Date: Thu Jul 19 00:15:59.703 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245320>245320</A>
ACTIVATE: RunSet: CALIB_CLC_QIE 
<!-- Author: Miguel M. -->


<!-- Date: Thu Jul 19 00:16:00.637 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245319>245319</A>
CALIBRATION: <U><B>calqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#calqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Thu Jul 19 01;13;24 comment by...Petteri -->
fine too
<!-- Date: Thu Jul 19 00:17:13.885 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245320>245320</A>
TERMINATE: done
<!-- Author: Miguel M. -->


<!-- Date: Thu Jul 19 00:17:14.933 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245320>245320</A>
CALIBRATION: <U><B>clcqie</B></U> calibration <A HREF=http://www-cdfonline.fnal.gov/internal/daq/Calibrations/web.html#clcqie target=_calibresults>results</A> ready for review
<!-- Author: CDFCALIB -->
<!-- Comment: Thu Jul 19 00;30;18 comment by...Petteri -->
looks ok
<!-- Date: Thu Jul 19 00:19:41.942 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245321>245321</A>
CALIBRATION: <U><B>bscqie</B></U> calibration FAILED, please investigate
<!-- Author: CDFCALIB -->
<!-- Comment: Thu Jul 19 07;54;41 comment by...convery -->
I don't understand what is being run for the BSC calibration.  The log file reports that it is looking for a 6.1.2 executable, while I thought we were testing 6.1.4.  Also, it seems to be trying to execute DBANA, which is definitely not in my script!  Looking at the log files, it looks like the 6.1.4 version was tested successfully yesterday at 14:53(?)
<!-- Date: Thu Jul 19 00:19:42.615 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245321>245321</A>
ACTIVATE: RunSet: CALIB_QIE_FWD 
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 00:20:22.910 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245321>245321</A>
TERMINATE: done
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 00:22:27.544 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245317>245317</A>
TERMINATE: tracklist runs fine in shotsetup ( no L3 ) so I am not sure why the code crashed in the EOS test
<!-- Author: flanagan -->


<!-- Date: Thu Jul 19 00:23:35.375 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245322>245322</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] tracklist continues
<!-- Author: flanagan -->


<!-- Date: Thu Jul 19 00:25:38.097 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245316>245316</A>
TERMINATE: done
<!-- Author: Miguel M. -->




<!-- Date: Thu Jul 19 00:36:34.151 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245294>245294</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=245294 target=_runSummary>245294</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->


<!-- Date: Thu Jul 19 00:49:20.640 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245324>245324</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] 
<!-- Author: flanagan -->



<!-- Date: Thu Jul 19 00:54:54.012 2007 -->
<pre>IFix crashed. We restarted it but the Voice Scheduler  
flashed on the screen for a second and disappeared. 
Petteri  restarted it and we got sounds but VOI went 
red and stayed red after 3 minutes. 
I think it is working well enough to get us  
through the night.   
 
Tell Steve in the morning.</pre>
<!-- Author: greg -->
<!-- Comment: Thu Jul 19 01;01;54 comment by...greg -->
<pre>there will be tweety bird noises all night long, 
but still we let Steve sleep!</pre>
<!-- Comment: Thu Jul 19 01;06;33 comment by...greg -->
<pre>VOI went green!</pre>
<!-- Date: Thu Jul 19 00:59:00.733 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245326>245326</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] 
<!-- Author: flanagan -->



<!-- Date: Thu Jul 19 01:03:23.618 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245327>245327</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] 
<!-- Author: flanagan -->




<!-- Date: Thu Jul 19 01:14:20.415 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245329>245329</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] 
<!-- Author: flanagan -->


<!-- Date: Thu Jul 19 01:20:38.595 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245330>245330</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] tracklist test all files to look area
<!-- Author: flanagan -->


<!-- Date: Thu Jul 19 01:20:39.886 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245330>245330</A>
TERMINATE: end to include L3
<!-- Author: flanagan -->


<!-- Date: Thu Jul 19 01:24:28.951 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245331>245331</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] track list test with all files to look area
<!-- Author: flanagan -->



<!-- Date: Thu Jul 19 01:25:40.030 2007 -->
MCR called, they are having problems with Tevatron and they need an access to fix it
<!-- Author: Petteri :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245321' target=_top>245321</a>) -->
<!-- Comment: Thu Jul 19 01;44;13 comment by...greg -->
<pre>A4 turbopump tripped.
</pre>

<!-- Date: Thu Jul 19 02:06:19.569 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245331>245331</A>
TERMINATE: so ends the tracklist test ( everyting went smoothly- even got some events which appear to have tracks)
<!-- Author: flanagan -->



<!-- Date: Thu Jul 19 02:16:36.492 2007 -->
<pre>Talked with MCR - guesstimate is shot setup  
in two hours 
</pre>
<!-- Author: greg -->

<!-- Date: Thu Jul 19 02:20:50.964 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245332>245332</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] testing XFT ? Node Algo - want to 
run parasitically
<!-- Author: flanagan -->


<!-- Date: Thu Jul 19 02:33:11.055 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245333>245333</A>
TERMINATE: l2ctrla failed
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 02:37:26.935 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245334>245334</A>
TERMINATE: l2ctrla fails in config
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 02:39:32.648 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245335>245335</A>
 Activated at 2007.07.19 02:38:23
<!-- Author: RunControl -->


<!-- Date: Thu Jul 19 02:39:33.302 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245335>245335</A>
ACTIVATE: RunSet: ACE_CALORIMETER_MINIMAL  Trigger: SUMET5 [7,792,632] 
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 02:41:41.257 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245335>245335</A>
 Terminated at 2007.07.19 02:39:39
<!-- Author: RunControl -->


<!-- Date: Thu Jul 19 02:41:42.358 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245335>245335</A>
TERMINATE: quiet time, L1A is 71.1Hz
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 03:00:27.807 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245338>245338</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] 
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 03:04:01.733 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245338>245338</A>
TERMINATE: some library stuff in l2ctrla doesn't like level 2 torture.. so will run without it
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 03:06:38.643 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245339>245339</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: PHYSICS_4_02 [8,916,694] 
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 03:16:08.780 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245339>245339</A>
TERMINATE: try different trigger table
<!-- Author: Petteri -->



<!-- Date: Thu Jul 19 03:19:05.569 2007 -->
<pre>Petteri notice ScalerMon was not updating. 
We restarted DaqMon and saw this message: 
 
b0control20:103:~> daqmon 
Attempting connection to <tcp:b0server04:5101> RTserver. 
Connected to <tcp:b0server04:5101> RTserver. 
Other error unpacking message array error while swapping msg message type table corrupt 
com.smartsockets.TipcAssertionException: error while swapping msg message type table corrupt 
        at com.smartsockets.TipcMsgImpl.swapMsgField(TipcMsgImpl.java:4198) 
        at com.smartsockets.TipcMsgImpl.swapField(TipcMsgImpl.java:4137) 
        at com.smartsockets.TipcMsgImpl.checkField(TipcMsgImpl.java:3931) 
        at com.smartsockets.TipcMsgImpl.nextMsgArray(TipcMsgImpl.java:1760) 
        at daqmsg.DaqMsgUnpack.onClassArray(DaqMsgUnpack.java:242) 
        at daqmsg.DaqMsg.process(DaqMsg.java:308) 
        at daqmsg.DaqMsg.unpack(DaqMsg.java:214) 
        at rc.vxmon.VxMonMaster$VxButton.process(VxMonMaster.java:605) 
        at com.smartsockets.TipcSrvImpl.process(TipcSrvImpl.java:901) 
        at com.smartsockets.TipcConnClientImpl.mainLoop(TipcConnClientImpl.java:1216) 
        at com.smartsockets.TipcSrvImpl.mainLoop(TipcSrvImpl.java:745) 
        at resmgr.ReplyMonitor.run(ReplyMonitor.java:44) 
 
Paging DAQ</pre>
<!-- Author: greg -->
<!-- Comment: Thu Jul 19 03;25;15 comment by...greg -->
<pre>However, ScalerMon is now updating.

We started ScalerMon with "scaler &" on b0control12
and saw what looks like the logfile spewing on screen.</pre>
<!-- Comment: Thu Jul 19 04;01;51 comment by...greg -->
<pre>Mousumi called in.  I told her we (I) probably 
overreacted.  She'll look at the java errors 
more tomorrow.
</pre>
<!-- Comment: Thu Jul 19 04;04;26 comment by...greg -->
<pre>PS - Before Mousumi called we were able to make 
daqmon work properly with "killall java" and 
starting from a new Xterminal.

</pre>
<!-- Comment: Thu Jul 19 06;41;58 comment by...greg  -->
<pre>Donatella called in.  She said they recently 
had a simialr java error for Error Handler.
It will be fixed.
</pre>
<!-- Comment: Thu Jul 19 06;48;31 comment by...Donatella T. -->
It looks like we had another problem with unpacking corrupted smartsockets messages in DaqMsg. Need to look deeper onto this.

a side comment: when you start ScalerMon from the command line, the logfile is spawn on the screen, just like RC does.
<!-- Date: Thu Jul 19 03:19:42.173 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245340>245340</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: PHYSICS_4_02 [8,916,694]
<!-- Author: Petteri -->




<!-- Date: Thu Jul 19 03:31:26.693 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-5891&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-5892  width='300' align='top'></a></td><td>Wondering if the triggertype in L2TORTURE is correct..</td></tr></table>
<!-- Author: Petteri -->
<!-- Comment: Thu Jul 19 03;38;48 comment by...Petteri -->
L2 TORTURE has the physics Trigger: PHYSICS_4_02 [8,916,694] as a default. On the white board there is a different one L2_TORTURE [18, 898, 633].
<!-- Comment: Thu Jul 19 06;42;56 comment by...greg -->
<pre>Petteri saved the cofig. to match the white board.
</pre>
<!-- Date: Thu Jul 19 03:31:32.023 2007 -->
<pre>It turns out scalermon WAS updating. 
Somehow the L2 torture run configuration  
now has the PHYSICS table.</pre>
<!-- Author: greg -->
<!-- Comment: Thu Jul 19 03;59;27 comment by...greg -->
<pre>It looks like someone may have changed L2 torture
and saved it with the physics table.
We were able to change  the configuration to 
the correct table and it remained even after
re-loading.

The java errors were probaly unrelated (I hope)

Anyway, all seems well now.
</pre>
<!-- Date: Thu Jul 19 03:37:23.873 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245341>245341</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [10,924,698] 
<!-- Author: Petteri -->




<!-- Date: Thu Jul 19 03:42:00.307 2007 -->
AAA_SHOTSETUP seems to be working fine.
<!-- Author: Petteri :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245341' target=_top>245341</a>) -->

<!-- Date: Thu Jul 19 03:44:01.208 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245341>245341</A>
TERMINATE: looking more into L2TORTURE
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 03:52:15.851 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245342>245342</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [18,898,633] 
<!-- Author: Petteri -->



<!-- Date: Thu Jul 19 03:55:22.463 2007 -->
<pre>We changed the trigger table to L2_TORTURE[18,898,633]. No single event went through. Error log says: 
2 Decision Timeout.[RXPT] 
(MLE) b0felix.fnal.gov:TEDCpp:Jul 19, 2007 03:51:19 AM->{ 3:51:19, 7/19/2007 }: L2-Pulsar-Linux: Halted while  in error state 
(RC)   3:51:20 Halt -> HALTED 
(MLE) b0felix.fnal.gov:TEDCpp:Jul 19, 2007 03:51:21 AM->{ 3:51:21, 7/19/2007 }: L2-Pulsar-Linux: Control node is unable to activate decision node #0 at address 192.168.0.11 
(RC)   3:51:22 Recover -> RECOVERED 
(RC)   3:51:25 Run -> ACTIVE 
(MLE) b0control21.fnal.gov:Thread-25:Jul 19, 2007 03:51:33 AM->  Level 2 Decision Timeout 
 
(CSL)  3:51:35 ********* L3 trigger rate has dropped to 0 Hz! Events are not being logged ! 
(TS)   3:51:37 ********* L2 trigger rate has dropped to 0 Hz! Events are not being logged ! 
(MLE) b0felix.fnal.gov:TEDCpp:Jul 19, 2007 03:52:15 AM->{ 3:52:15, 7/19/2007 }: L2-Pulsar-Linux: Halted while  in error state 
(RC)   3:52:15 Halt -> HALTED 
(MLE) b0felix.fnal.gov:TEDCpp:Jul 19, 2007 03:52:16 AM->{ 3:52:16, 7/19/2007 }: L2-Pulsar-Linux: Control node is unable to activate decision node #0 at address 192.168.0.11 
(RC)   3:52:17 Recover -> RECOVERED 
(RC)   3:52:20 Run -> ACTIVE 
(MLE) b0control21.fnal.gov:Thread-25:Jul 19, 2007 03:52:30 AM->  Level 2 Decision Timeout</pre>
<!-- Author: Petteri :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245342' target=_top>245342</a>) -->
<!-- Comment: Thu Jul 19 04;03;29 comment by...laura -->
The control node is not able to contact the main PC (192.168.0.11) which is instead alive...We had the same problem running with the parasitic one...something wrong in the library in L2_TORTURE configuration?!?!
<!-- Date: Thu Jul 19 03:56:28.606 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245342>245342</A>
TERMINATE: no events going through..
<!-- Author: Petteri -->



<!-- Date: Thu Jul 19 03:59:31.924 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245343>245343</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] 
<!-- Author: Petteri -->





<!-- Date: Thu Jul 19 04:11:07.364 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245343>245343</A>
TERMINATE: run cosmics for a while
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 04:17:21.881 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245344>245344</A>
 Activated at 2007.07.19 04:17:18
<!-- Author: RunControl -->


<!-- Date: Thu Jul 19 04:17:46.252 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245344>245344</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL_L2UP  Trigger: COSMICS [15,893,674] 
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 04:18:22.106 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245344>245344</A>
 Terminated at 2007.07.19 04:17:54
<!-- Author: RunControl -->


<!-- Date: Thu Jul 19 04:21:33.798 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245345>245345</A>
 Activated at 2007.07.19 04:21:07
<!-- Author: RunControl -->


<!-- Date: Thu Jul 19 04:22:44.983 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245345>245345</A>
 Terminated at 2007.07.19 04:22:25
<!-- Author: RunControl -->


<!-- Date: Thu Jul 19 04:22:45.265 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245345>245345</A>
TERMINATE: no good, no events going through
<!-- Author: Petteri -->
<!-- Comment: Thu Jul 19 04;23;16 comment by...Petteri -->
<pre>Comments from errorlog:

b0felix.fnal.gov:TEDCpp:Jul 19, 2007 04:21:02 AM->{ 4:21:2, 7/19/2007 }: L2-Pulsar-Linux: Control node is unable to contact decision node process in node  #0 at address 192.168.0.10; Decision node may be in trouble.  -->  
 Additional Information: 

 Attention !!!!.  L2PULSAR_NODELOST !!!!
 
Control node is unable to contact decision node process.
Decision node is sick. Page Level 2 for help.
[MLE]
(RC)   4:21:06 Config -> READY
(RC)   4:21:10 Activate -> ACTIVE
(CSL)  4:21:19 ********* L3 trigger rate has dropped to 0 Hz! Events are not being logged !
L2 Decision Timeout.[RXPT]
(MLW) b0control21.fnal.gov:Thread-25:Jul 19, 2007 04:21:20 AM->Received error message: errmon Not a state HaltRecoverRun L2 Decision Timeout.[RXPT](MLE) b0control21.fnal.gov:Thread-25:Jul 19, 2007 04:21:20 AM->Requested Halt-Recover-Run issued (errmon)
(MLE) b0felix.fnal.gov:TEDCpp:Jul 19, 2007 04:21:26 AM->{ 4:21:26, 7/19/2007 }: L2-Pulsar-Linux: Halted while  in error state
(RC)   4:21:26 Halt -> HALTED
(MLE) b0felix.fnal.gov:TEDCpp:Jul 19, 2007 04:21:28 AM->{ 4:21:28, 7/19/2007 }: L2-Pulsar-Linux: Control node is unable to activate decision node #0 at address 192.168.0.11
(MLE) b0felix.fnal.gov:TEDCpp:Jul 19, 2007 04:21:28 AM->{ 4:21:28, 7/19/2007 }: L2-Pulsar-Linux: Control node is unable to activate decision node #0 at address 192.168.0.10
(RC)   4:21:29 Recover -> RECOVERED
(RC)   4:21:31 Run -> ACTIVE
(MLE) b0control21.fnal.gov:Thread-25:Jul 19, 2007 04:21:40 AM->  Level 2 Decision Timeout

(MLE) b0felix.fnal.gov:TEDCpp:Jul 19, 2007 04:21:45 AM->{ 4:21:45, 7/19/2007 }: L2-Pulsar-Linux: Halted while  in error state
(RC)   4:21:46 Halt -> HALTED
(MLE) b0felix.fnal.gov:TEDCpp:Jul 19, 2007 04:21:47 AM->{ 4:21:47, 7/19/2007 }: L2-Pulsar-Linux: Control node is unable to activate decision node #0 at address 192.168.0.11
(MLE) b0felix.fnal.gov:TEDCpp:Jul 19, 2007 04:21:47 AM->{ 4:21:47, 7/19/2007 }: L2-Pulsar-Linux: Control node is unable to activate decision node #0 at address 192.168.0.10
(RC)   4:21:48 Recover -> RECOVERED
(RC)   4:21:50 Run -> ACTIVE
(TS)   4:21:57 ********* L2 trigger rate has dropped to 0 Hz! Events are not being logged !
(MLE) b0control21.fnal.gov:Thread-25:Jul 19, 2007 04:21:56 AM->  Level 2 Decision Timeout
</pre>

<!-- Comment: Thu Jul 19 05;38;48 comment by...laura -->
<pre>Fetching library libl2algo_COSMICS_v-15.so.1.0.893
issuing: /root/run/official/l2pulsar_node/scripts/fetch.tcsh libl2algo_COSMICS_v-15.so.1.0.893 /cdf/trigger/libl2algo_COSMICS_v-15.so.1.0.893
rsync rsync://192.168.0.1/L2-CAL/libl2algo_COSMICS_v-15.so.1.0.893 /cdf/trigger/libl2algo_COSMICS_v-15.so.1.0.893
libl2algo_COSMICS_v-15.so.1.0.893
           0   0%    0.00kB/s    0:00:00^M     5572421 100%   69.92MB/s    0:00:00 (xfer#1, to-check=0/1)

sent 112 bytes  received 5573229 bytes  11146682.00 bytes/sec
total size is 5572421  speedup is 1.00
Loading library from: /cdf/trigger/libl2algo_COSMICS_v-15.so.1.0.893
L1Container created called
RecesContainer created called
SvtContainer created
TrackContainer created
MuonContainer created
isolationContainer created
Loaded symbols from shared library
about to create decsion object ...
created decsion object ...
decsion object : set numtrigs to 17
about to create data object ...
L2Error object created
created data object ...
_L2prescales[0] : 1
_L2prescales[1] : 1
_L2prescales[2] : 1
_L2prescales[3] : 1
_L2prescales[4] : 1
_L2prescales[5] : 1
_L2prescales[6] : 1
_L2prescales[7] : 1
_L2prescales[8] : 1
_L2prescales[9] : 1
_L2prescales[10] : 1
_L2prescales[11] : 1
_L2prescales[12] : 1
_L2prescales[13] : 1
_L2prescales[14] : 1
_L2prescales[15] : 1
_L2prescales[16] : 1219499995
Caught seg fault!


dying gracefully ...

unmapping filar
</pre>
<!-- Date: Thu Jul 19 04:27:17.893 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245346>245346</A>
 Activated at 2007.07.19 04:26:00
<!-- Author: RunControl -->


<!-- Date: Thu Jul 19 04:27:18.543 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245346>245346</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,893,674] 
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 04:27:19.562 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245346>245346</A>
 Terminated at 2007.07.19 04:26:26
<!-- Author: RunControl -->


<!-- Date: Thu Jul 19 04:27:20.571 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245346>245346</A>
TERMINATE: same thing, no events going through
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 04:31:31.426 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245347>245347</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] 
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 04:36:03.694 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245347>245347</A>
TERMINATE: try once more cosmics
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 04:40:17.452 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245348>245348</A>
 Activated at 2007.07.19 04:39:28
<!-- Author: RunControl -->


<!-- Date: Thu Jul 19 04:40:18.445 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245348>245348</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,893,674] 
<!-- Author: Petteri -->



<!-- Date: Thu Jul 19 04:46:05.446 2007 -->
<pre>Short review:  
 
We tried L2TORTURE[18,898,633], with and without L2UP. The parasitic PC craches when running with it and need to be rebooted. In both cases we didn't get any events through (we got L2 Decision TO). 
 
We tried COSMICS[15,893,674], with and without L2UP. Basicly same thing happened here as with L2TORTURE and no events going through. 
 
AAA_SHOTSETUP works fine.</pre>
<!-- Author: Petteri -->

<!-- Date: Thu Jul 19 04:52:22.320 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245348>245348</A>
 Terminated at 2007.07.19 04:51:38
<!-- Author: RunControl -->


<!-- Date: Thu Jul 19 04:52:23.214 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245348>245348</A>
TERMINATE: not working, back to shotsetup
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 04:58:28.728 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245349>245349</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_02 [10,924,698] 
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 04:58:29.792 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245349>245349</A>
TERMINATE: L2 torture again
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 05:02:34.791 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245350>245350</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [18,898,633] 
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 05:11:30.033 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245351>245351</A>
 Activated at 2007.07.19 05:10:55
<!-- Author: RunControl -->


<!-- Date: Thu Jul 19 05:11:31.602 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245351>245351</A>
ACTIVATE: RunSet: ACE_COSMIC_NOTRACKS  Trigger: COSMICS_NOTRACKS [9,897,639] 
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 05:24:00.546 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245351>245351</A>
 Terminated at 2007.07.19 05:23:22
<!-- Author: RunControl -->


<!-- Date: Thu Jul 19 05:24:01.945 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245351>245351</A>
TERMINATE: no events coming through L2
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 05:26:50.142 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245352>245352</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] 
<!-- Author: Petteri -->





<!-- Date: Thu Jul 19 05:53:20.963 2007 -->
<pre>TeV people still working, we are just hanging out. 
 
Probably won't see beam on our shift. 
 
</pre>
<!-- Author: greg -->

<!-- Date: Thu Jul 19 06:08:59.161 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245352>245352</A>
TERMINATE: trying once more L2TORTURE
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 06:11:48.553 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245353>245353</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [18,898,633] 
<!-- Author: Petteri -->


<!-- Date: Thu Jul 19 06:17:25.670 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245354>245354</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] 
<!-- Author: Petteri -->





<!-- Date: Thu Jul 19 06:46:02.570 2007 -->
<pre>Peeked into the MCR log - they are still trying to 
recover cryo in A4.</pre>
<!-- Author: greg -->
<!-- Comment: Thu Jul 19 07;28;48 comment by...greg -->
<pre>Another look:

They are gearing up and getting ready to go!

</pre>

<!-- Date: Thu Jul 19 07:38:55.636 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=245355>245355</A>
ACTIVATE: RunSet: AAA_SHOTSETUP_L2UP  Trigger: PHYSICS_4_02 [10,924,698] 
<!-- Author: Petteri -->



<!-- Date: Thu Jul 19 07:46:41.368 2007 -->
<font color=red>Shift Summary: </font><pre>No beam all night.  TeV troubles with cryo in  A4 
 
Successful L2 tracklist data to parasitic pc performed. 
 
 
</pre> 
<!-- Author: Greg Feild -->


<!-- Date: Thu Jul 19 07:55:35.094 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:00</TD>
<TD ALIGN=CENTER>::</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->