<!-- Title: Eve shift. Thu Jul  7, 2005 -->
<!-- SciCo: Dan Amidei -->
<!-- DAQAce: Michael Morello/Georgios Choudalakis -->
<!-- MonAce: Svenja Richter -->
<!-- CO: Yangheng Zheng -->
<!-- OpManager: Aron Soha -->
<!-- Notes: -->
<pre>Low luminosity at tail of store #4252. 
Turn system over to L3 and SVT work. 
Will have two test runs to check results. 
</pre>


<!-- Date: Thu Jul 7 16:22:04.599 2005 -->
<pre>We are taking data including svt09.     
Today configuration is:     
- AMSRW with RW disabled   
- AM++ readout sorted by roadID 
- normal RW enabled  
- killonbit18 enabled for SVT and not for svt09  
- "slow abort" i.e. wait for svt09 is active    
</pre>
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200660' target=_top>200660</a>) -->


<!-- Date: Thu Jul 7 16:25:28.737 2005 -->
<pre>Procmon error: DownTime error--> 
 
07-Jul-2005 16:21:07 Status: CHECK FAILED 
07-Jul-2005 16:21:09 Status: CHECKING 
07-Jul-2005 16:21:11 Status: CHECK FAILED 
Thu Jul 7 16:21:14 CDT 2005   procAction  down   check   b0dap85 
Checking Downtime Logger 
Found 0 Downtime Logger threads running (>=1 expected) 
Downtime Logger is not running correctly 
 
 
I followed the procedure without success. I'm going to page the DAQ expert. 
N.B. The data taking is finished, Alberto and Markus are doing the tests respectively on SVT and EVB. 
</pre>
<!-- Author: MIchael  -->
<!-- Comment: Thu Jul  7 16;53;42 comment by...Michael  -->
I'm sorry! The data taking is continuing. The guys are doing the tests.
<!-- Comment: Thu Jul  7 20;12;18 comment by...Michael  -->
Again! Jane fixed it.
<!-- Date: Thu Jul 7 16:30:40.339 2005 -->
<pre>Used start of SVT tests to include L3 subfarm 11 and 16 and exclude 2, 8 and 14.</pre>
<!-- Author: markus -->


<!-- Date: Thu Jul 7 16:38:29.479 2005 -->
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!!  
 Hardware EVB has detected a problem with data quality in  
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE).  
 Datataking is paused: events are not being processed.  
 - HRR should help.  
[MLE]
<!-- Author: MIchael  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200660' target=_top>200660</a>) -->


<!-- Date: Thu Jul 7 16:50:39.575 2005 -->
<pre>Strange error from Event Builder 
 
Error logger--> 
43'51" b0l3pcom1.fnal.gov:main:Jul 7, 2005 4:43:48 PM->Host b0eb11.fnal.gov, task tRec_0 
SCPU-P0-E-TracerEventId: Event 196608, crate 0, channel 9 has either bad Tracer ID or bad markers around Tracer word. 
 Tracer word event ID is 0, should be 0. 
Tracer word = 0x00000000 (I think). 
Channel data = 0x0000002a 0x000000a3 0x53564444 0x00080008 0x00000000 0x00000009 0x00000003 0x00000001 ... 
... 0x000068c4 0x0002ffff 0x00000000 0xaa000000 0x55aaaaaa 0xa3555555 0x00000000 0x80800000 -->   
 Additional Information:  
 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!!  
 Hardware EVB has detected a problem with data quality in  
 SCPU b0eb11 ( FER crate : NOT_AVAILABLE).  
 Datataking is paused: events are not being processed.  
 - HRR should help.  
[MLE] 
(MLE) b0l3pcom1.fnal.gov:main:Jul 7, 2005 4:43:48 PM->Host b0eb11.fnal.gov, task tRec_0 
SCPU-P0-E-VrbHeader: Dump of header words for event 196608 from VRB in slot 10: 
0x00000000 0x00002bd0 0x000e004e 0x091409e0 0x085c08a4 0x01a001b0 0x01a4020c 0x000000a8 
(MLE) b0svx05:Messenger:Jul 7, 2005 4:44:20 PM->Silicon Resonator: Sl 16 Ch 4 -> e564: Bytes: 00074 
e560 18:e580 20:e5a0 
 
(RC)   16:44:22 Halt -> HALTED 
(RC)   16:44:29 Recover -> RECOVERED 
(RC)   16:44:32 Run -> ACTIVE 
 
</pre>
<!-- Author: MIchael  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200660' target=_top>200660</a>) -->
<!-- Comment: Thu Jul  7 16;51;02 comment by...Michael  -->
Again! Auto HRR.

<!-- Comment: Thu Jul  7 16;52;27 comment by...MIchael  -->
In all cases the data taking continued. It doesn't go automatically in HALTED state.


<!-- Comment: Thu Jul  7 16;56;11 comment by...MIchael -->
Again! Auto HRR.


<!-- Comment: Thu Jul  7 17;03;42 comment by...Michael  -->
Again!Manual HRR.
<!-- Date: Thu Jul 7 16:57:20.481 2005 -->
DownTimeCopy was giving errors related to the update of the rundb package.  For now I started a version which uses the old rundb v2_3k.  Will try to resolve this with the appropriate experts.
<!-- Author: Jane -->


<!-- Date: Thu Jul 7 17:00:14.038 2005 -->
Flying wire scan.
<!-- Author: MIchael :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200660' target=_top>200660</a>) -->



<!-- Date: Thu Jul 7 17:08:37.961 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11512&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11512' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11513&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11513' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11514&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11514' width='300'></a></center>
<!-- Author: Svenja -->
<!-- Comment: Thu Jul  7 17;09;02 comment by...Svenja -->
hourly plots 16:05-17:05

<!-- Date: Thu Jul 7 17:14:01.664 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200660">200660</A>
 Terminated at 2005.07.07 17:13:43
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 17:15:17.066 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200660">200660</A>
TERMINATE: Emd the run to do EVBuilder Clean Up
<!-- Author: Michael & Georgios -->


<!-- Date: Thu Jul 7 17:15:18.797 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200660">200660</A>
TERMINATE: End the run to do EVBuilder Clean Up
<!-- Author: Michael & Georgios -->


<!-- Date: Thu Jul 7 17:23:21.151 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200661">200661</A>
 Activated at 2005.07.07 17:23:17
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 17:23:42.841 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200661">200661</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_04 [1,670,576] 
<!-- Author: Michael & Georgios -->



<!-- Date: Thu Jul 7 17:37:15.491 2005 -->
<pre>The SCPU b0eb11 detected an error in a few events which look like a  byte shift. The event builder reported the error, but continued running. b0eb11 receives the data from SVT09 which could be the problem. 
 
A few minutes later we had to clean up the EVB. The problem could have been due to a scramnet communication problem due to the fact that to many L3 converter nodes where powered down at the same time. I don't know for sure.</pre>
<!-- Author: markus -->
<!-- Comment: Thu Jul  7 18;04;28 comment by...Alberto -->
Since the first error was coming from crate svt09, I power cyled it and reseat the tracer in during EVB cleanup.
<!-- Date: Thu Jul 7 17:42:49.590 2005 -->
<pre>Time Out Error. I ended the run and I did EVB Cleanup. Now it looks fine. 
 
Error Logger: 
12'25" b0l3pcom1.fnal.gov:main:Jul 7, 2005 5:12:24 PM->Host b0eb10.fnal.gov, task tsmDisp 
SM-P0-E-ScpuTimeout: SCPU ack timeout for Recover operation. -->   
 Additional Information:  
 
 Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!.  
 We STRONGLY RECOMMEND the following steps:  
 - Try Halt-Recover-Run. 
 - If it fails, do End (Abort) -> Reset -> Cleanup EVB. 
 - Start run again. </pre>
<!-- Author: Michael  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200660' target=_top>200660</a>) -->


<!-- Date: Thu Jul 7 18:00:05.501 2005 -->
Flying wire scan.
<!-- Author: Michael  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200661' target=_top>200661</a>) -->


<!-- Date: Thu Jul 7 18:07:07.947 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200661">200661</A>
 Terminated at 2005.07.07 18:06:36
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 18:07:44.491 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200661">200661</A>
TERMINATE: End the run 
<!-- Author: Michael & Georgios -->



<!-- Date: Thu Jul 7 18:09:15.929 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11515&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11515' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11516&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11516' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11517&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11517' width='300'></a></center>
<!-- Author: Svenja -->
<!-- Comment: Thu Jul  7 18;09;40 comment by...Svenja -->
hourly plots 17:05-18:05


<!-- Date: Thu Jul 7 18:10:21.675 2005 -->
The run summary web-page was not loading, so I restarted the TOMCAT 
to fix it. The experts already konw this problem since previous shift.
<!-- Author: Michael -->


<!-- Date: Thu Jul 7 18:11:05.677 2005 -->
<pre>Done with L3 subfarms 2, 8 and 14. They are back in the run. Excluded 5 and 12 for debugging - the last two.</pre>
<!-- Author: markus -->
<!-- Comment: Thu Jul  7 18;18;37 comment by...markus -->
<pre>14 needs one more interation. The ethernet cards are ok now, but the atm card is not.</pre>
<!-- Date: Thu Jul 7 18:18:36.235 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200662">200662</A>
 Activated at 2005.07.07 18:18:30
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 18:18:37.859 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200662">200662</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_04 [1,670,576] 
<!-- Author: Michael & Georgios -->




<!-- Date: Thu Jul 7 18:22:27.305 2005 -->
We are taking data including svt09.       
The current configuration is:       
- AMSRW with RW enabled     
- AM++ readout sorted by roadID  
- normal RW bypassed   
- killonbit18 enabled for SVT and not for svt09    
- "slow abort" i.e. wait for svt09 is active  
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200662' target=_top>200662</a>) -->
<!-- Comment: Thu Jul  7 18;23;59 comment by...Alberto -->
<pre>same as above with better formatting:

We are taking data including svt09.
The current configuration is: 
- AMSRW with RW enabled 
- AM++ readout sorted by roadID 
- normal RW bypassed 
- killonbit18 enabled for SVT and not for svt09 
- "slow abort" i.e. wait for svt09 is active</pre>
<!-- Date: Thu Jul 7 18:22:49.730 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200662">200662</A>
 Terminated at 2005.07.07 18:22:16
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 18:23:49.743 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200662">200662</A>
TERMINATE: End run 
<!-- Author: Michael & Georgios -->



<!-- Date: Thu Jul 7 18:29:36.991 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200663">200663</A>
 Activated at 2005.07.07 18:28:49
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 18:29:37.226 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200663">200663</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_04 [1,670,576] 
<!-- Author: Michael & Georgios -->



<!-- Date: Thu Jul 7 18:29:56.175 2005 -->
When I do partition the l2ctrl crate hangs up. I have to do manually patition of l2ctrl. 
<!-- Author: Michael  -->


<!-- Date: Thu Jul 7 19:00:04.924 2005 -->
Flying wire scan.
<!-- Author: Georgios :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200663' target=_top>200663</a>) -->
<!-- Comment: Thu Jul  7 19;59;50 comment by...Georgios -->
Again
<!-- Comment: Thu Jul  7 22;00;27 comment by...Georgios -->
again
<!-- Date: Thu Jul 7 19:07:24.964 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11518&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11518' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11519&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11519' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11520&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11520' width='300'></a></center>
<!-- Author: Svenja -->
<!-- Comment: Thu Jul  7 19;08;00 comment by...Svenja -->
hourly plots 18:05-19:05


<!-- Date: Thu Jul 7 19:09:22.278 2005 -->
L2 Decision timeout error. HRR fixed it.
<!-- Author: Georgios :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200663' target=_top>200663</a>) -->
<!-- Comment: Thu Jul  7 19;49;45 comment by...Georgios -->
Again



<!-- Comment: Thu Jul  7 20;28;09 comment by...Georgios -->
again
<!-- Comment: Thu Jul  7 20;32;31 comment by...Alberto -->
The last L2DTO happened while I was reading SPYBUFFERS in svt09 and it may have been caused by this.
<!-- Comment: Thu Jul  7 22;12;43 comment by...Georgios -->
again
<!-- Date: Thu Jul 7 20:03:27.200 2005 -->
<pre>ProcMon and RtMon Error. page DAQ</pre>
<!-- Author: Amidei -->


<!-- Date: Thu Jul 7 20:11:43.629 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11521&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11521' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11522&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11522' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11523&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11523' width='300'></a></center>
<!-- Author: Svenja -->
<!-- Comment: Thu Jul  7 20;12;17 comment by...Svenja -->
hourly plots 19:05-20:05



<!-- Date: Thu Jul 7 20:28:58.943 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200663">200663</A>
 Terminated at 2005.07.07 20:28:34
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 20:30:16.035 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200663">200663</A>
TERMINATE: end the run svt test
<!-- Author: Michael & Georgios -->



<!-- Date: Thu Jul 7 20:38:39.009 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200664">200664</A>
 Activated at 2005.07.07 20:37:53
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 20:38:40.534 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200664">200664</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_04 [1,670,576] 
<!-- Author: Michael & Georgios -->



<!-- Date: Thu Jul 7 21:01:37.286 2005 -->
<pre>CT: 2005.07.07 20:58:36 
58'36" b0dap84.fnal.gov:ConsumerErrorRe:Jul 7, 2005 8:58:36 PM->Runtime Error 1, Event 17443, RunNum 200664: SvxMon Halt Recover Run: Stuck Cellid in 5 events in Silicon/S/B1/W5/L2/C0-4 . -->   
 Additional Information:  
 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Stuck Cellid S/B1/W5/L2/C0-4 .  
 AUTO HRR will be issued   
[MLE] 
(MLW) b0dap73.fnal.gov:Thread-26:Jul 7, 2005 8:58:36 PM->Received error message: errmon Active TurboHRR HRR suggested for:b0dap84.fnal.gov:ConsumerErrorRe:Jul 7, 2005 8:58:36 PM->Runtime Error 1, Event 17443, RunNum 200664: SvxMon Halt Recover Run: Stuck Cellid in 5 events in Silicon/S/B1/W5/L2/C0-4 . 
(MLE) b0dap73.fnal.gov:Thread-26:Jul 7, 2005 8:58:36 PM->Requested Halt-Recover-Run issued (errmon) 
(MLE) b0svx07:Messenger:Jul 7, 2005 8:58:38 PM->Silicon Resonator: Sl 08 Ch 6 -> f411: Bytes: 00116 
f440 13:f520 15:f500 16:f540 18:f300 20:f320 
 
(RC)   20:58:39 Halt -> HALTED 
(RC)   20:58:46 Recover -> RECOVERED 
(RC)   20:58:48 Run -> ACTIVE</pre>
<!-- Author: Georgios :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200664' target=_top>200664</a>) -->


<!-- Date: Thu Jul 7 21:03:58.989 2005 -->
<pre>Veronica, Chris, and Rong are here to try Trigger Table test.</pre>
<!-- Author: Sci_Co -->

<!-- Date: Thu Jul 7 21:05:16.631 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200664">200664</A>
 Terminated at 2005.07.07 21:04:01
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 21:05:17.943 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200664">200664</A>
TERMINATE: end run for testing trigger table
<!-- Author: Michael x2080 -->


<!-- Date: Thu Jul 7 21:13:25.430 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200665">200665</A>
 Activated at 2005.07.07 21:13:01
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 21:13:26.691 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200665">200665</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_TEST_3_04 [3,672,578] 
<!-- Author: Michael x2080 -->


<!-- Date: Thu Jul 7 21:13:27.479 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200665">200665</A>
 Terminated at 2005.07.07 21:13:17
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 21:14:14.894 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200665">200665</A>
TERMINATE: end the run
<!-- Author: Michael x2080 -->


<!-- Date: Thu Jul 7 21:20:09.340 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200666">200666</A>
TERMINATE: end the run L2ctrl problem
<!-- Author: Michael x2080 -->


<!-- Date: Thu Jul 7 21:25:29.537 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200667">200667</A>
 Activated at 2005.07.07 21:24:50
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 21:25:30.833 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200667">200667</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_TEST_3_04 [3,672,578]  trigger table test
<!-- Author: Michael x2080 -->


<!-- Date: Thu Jul 7 21:41:42.463 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200667">200667</A>
 Terminated at 2005.07.07 21:40:24
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 21:41:43.347 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200667">200667</A>
TERMINATE: End the run
<!-- Author: Michael x2080 -->


<!-- Date: Thu Jul 7 21:46:43.047 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200668">200668</A>
 Activated at 2005.07.07 21:46:20
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 21:46:44.229 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200668">200668</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_TEST_3_04 [3,672,578] 
<!-- Author: Michael x2080 -->



<!-- Date: Thu Jul 7 21:47:50.658 2005 -->
<pre>CT: 2005.07.07 21:46:52 
46'52" b0dap84.fnal.gov:ConsumerErrorRe:Jul 7, 2005 9:46:52 PM->Runtime Error 1, Event 17668, RunNum 200668: SvxMon Halt Recover Run: Pipeline out of synch in 117 silicon readout chips . -->   
 Additional Information:  
 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Pipeline out of synch in 117 silicon readout chips .  
 AUTO HRR will be issued   
[MLE] 
(MLW) b0dap73.fnal.gov:Thread-26:Jul 7, 2005 9:46:52 PM->Received error message: errmon Active TurboHRR HRR suggested for:b0dap84.fnal.gov:ConsumerErrorRe:Jul 7, 2005 9:46:52 PM->Runtime Error 1, Event 17668, RunNum 200668: SvxMon Halt Recover Run: Pipeline out of synch in 117 silicon readout chips . 
(MLE) b0dap73.fnal.gov:Thread-26:Jul 7, 2005 9:46:52 PM->Requested Halt-Recover-Run issued (errmon) 
(MLE) b0svx03:Messenger:Jul 7, 2005 9:46:53 PM->Silicon Resonator: Sl 12 Ch 6 -> e351: Bytes: 00144 
e360 18:e380 20:e3a0 
 
(MLE) b0svx00:Messenger:Jul 7, 2005 9:46:53 PM->Silicon Resonator: Sl 16 Ch 9 -> e074: Bytes: 00148 
e080 12:e040 
 
(MLE) b0svx07:Messenger:Jul 7, 2005 9:46:53 PM->Silicon Resonator: Sl 08 Ch 6 -> f411: Bytes: 00094 
f440 13:f520 15:f500 16:f540 18:f300 20:f320 
 
(RC)   21:46:55 Halt -> HALTED 
(RC)   21:47:02 Recover -> RECOVERED 
(RC)   21:47:05 Run -> ACTIVE</pre>
<!-- Author: Michael  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200668' target=_top>200668</a>) -->


<!-- Date: Thu Jul 7 21:54:38.354 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11524&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11524' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11525&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11525' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11526&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11526' width='300'></a></center>
<!-- Author: Svenja -->
<!-- Comment: Thu Jul  7 21;55;03 comment by...Svenja -->
hourly plots 20:05-21:05



<!-- Date: Thu Jul 7 22:09:44.013 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11527&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11527' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11528&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11528' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11529&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11529' width='300'></a></center>
<!-- Author: Svenja -->
<!-- Comment: Thu Jul  7 22;10;06 comment by...Svenja -->
hourly plots 21:05-22:05


<!-- Date: Thu Jul 7 22:16:27.876 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200668">200668</A>
 Terminated at 2005.07.07 22:16:20
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 22:17:00.153 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200668">200668</A>
TERMINATE: end the run
<!-- Author: Michael x2080 -->
<!-- Comment: Thu Jul  7 22;43;23 comment by...Veronica&Kirsten -->
We have done the EOS test of table containing CLCM triggers and Mt trigger. This table contains also new MET+BJET and z->bb test triggers. Test was successful. Some details: No change on clc trigger, factor of 4 reduction on Mt rate, big reduction on z->bb and factor of 2 reduction on L2 MET. Thanks Chris!
<!-- Comment: Thu Jul  7 23;54;09 comment by...Ted -->
yes, we should see bigger difference at higher lumonisity, esp. with CLC multiplicity triggers. Thanks Chris! Veronica will have the final physics tables waiting for you when you come back next Tuesday...
<!-- Date: Thu Jul 7 22:22:38.829 2005 -->
Busy Time Out: b0svx06 and after Scanner Manager error.--> Event Builder Cleanup.
<!-- Author: Michael  -->

<!-- Date: Thu Jul 7 22:28:12.254 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200669">200669</A>
 Activated at 2005.07.07 22:26:48
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 22:28:13.101 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200669">200669</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_04 [1,670,576] 
<!-- Author: Michael x2080 -->



<!-- Date: Thu Jul 7 22:57:27.816 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200669">200669</A>
 Terminated at 2005.07.07 22:56:23
<!-- Author: RunControl -->


<!-- Date: Thu Jul 7 22:57:28.712 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200669">200669</A>
TERMINATE: end the run. 
<!-- Author: Michael x2080 -->



<!-- Date: Thu Jul 7 23:04:05.868 2005 -->
<pre>Bring down HVs 
Call MCR tell them OK to drop store. </pre>
<!-- Author: Amidei -->


<!-- Date: Thu Jul 7 23:09:36.781 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11530&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11530' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11531&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11531' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-11532&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-11532' width='300'></a></center>
<!-- Author: Svenja -->
<!-- Comment: Thu Jul  7 23;38;54 comment by...Svenja -->
hourly plots 22:05-23:08 (3 minutes more to include end of store)
<!-- Date: Thu Jul 7 23:15:06.215 2005 -->
<pre>MCR calls. Begin quiet time.</pre>
<!-- Author: Amidei -->

<!-- Date: Thu Jul 7 23:20:22.700 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200672">200672</A>
ACTIVATE: RunSet: ACE_QIE again
<!-- Author: Michael x2080 -->
<!-- Comment: Thu Jul  7 23;43;58 comment by...markus  -->
<pre>b0pcal11 failed the calibration - Michael tried to reset the crate which failed as well.</pre>
<!-- Date: Thu Jul 7 23:40:06.985 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200674">200674</A>
ACTIVATE: RunSet: CALIB_CLC_QIE 
<!-- Author: Michael x2080 -->


<!-- Date: Thu Jul 7 23:41:12.190 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200674">200674</A>
TERMINATE: end CLC Calib
<!-- Author: Michael x2080 -->


<!-- Date: Thu Jul 7 23:43:26.527 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200675">200675</A>
ACTIVATE: RunSet: CALIB_QIE_BSC 
<!-- Author: Michael x2080 -->



<!-- Date: Thu Jul 7 23:46:06.954 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200675">200675</A>
TERMINATE: end BSC QIE calibration

<!-- Author: Michael x2080 -->



<!-- Date: Thu Jul 7 23:55:46.177 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200659>
200659</A> 
x30FD3</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_04 [1,670,576]</TD>                                       
<TD ALIGN=CENTER>05:07:50</TD>                                          
<TD ALIGN=CENTER>16:08:12</TD>                                          
<TD ALIGN=CENTER>10:32:36</TD>                                          
<TD ALIGN=RIGHT>    849,379,519</TD>                                    
<TD ALIGN=RIGHT>     9,480,884</TD>                                     
<TD ALIGN=RIGHT>   2,659,385</TD>                                       
<TD ALIGN=RIGHT> 1128.724</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200660>
200660</A> 
x30FD4</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_04 [1,670,576]</TD>                                       
<TD ALIGN=CENTER>16:14:28</TD>                                          
<TD ALIGN=CENTER>17:13:43</TD>                                          
<TD ALIGN=CENTER>00:47:14</TD>                                          
<TD ALIGN=RIGHT>     40,686,000</TD>                                    
<TD ALIGN=RIGHT>       458,730</TD>                                     
<TD ALIGN=RIGHT>     144,333</TD>                                       
<TD ALIGN=RIGHT>   63.904</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200661>
200661</A> 
x30FD5</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_04 [1,670,576]</TD>                                       
<TD ALIGN=CENTER>17:23:17</TD>                                          
<TD ALIGN=CENTER>18:06:36</TD>                                          
<TD ALIGN=CENTER>00:42:16</TD>                                          
<TD ALIGN=RIGHT>     34,875,276</TD>                                    
<TD ALIGN=RIGHT>       399,903</TD>                                     
<TD ALIGN=RIGHT>     124,257</TD>                                       
<TD ALIGN=RIGHT>   54.567</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200662>
200662</A> 
x30FD6</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_04 [1,670,576]</TD>                                       
<TD ALIGN=CENTER>18:18:30</TD>                                          
<TD ALIGN=CENTER>18:22:16</TD>                                          
<TD ALIGN=CENTER>00:02:35</TD>                                          
<TD ALIGN=RIGHT>        666,216</TD>                                    
<TD ALIGN=RIGHT>        12,167</TD>                                     
<TD ALIGN=RIGHT>       3,977</TD>                                       
<TD ALIGN=RIGHT>    3.247</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>0</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200663>
200663</A> 
x30FD7</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_04 [1,670,576]</TD>                                       
<TD ALIGN=CENTER>18:28:49</TD>                                          
<TD ALIGN=CENTER>20:28:34</TD>                                          
<TD ALIGN=CENTER>01:57:43</TD>                                          
<TD ALIGN=RIGHT>    106,159,913</TD>                                    
<TD ALIGN=RIGHT>     1,202,838</TD>                                     
<TD ALIGN=RIGHT>     360,828</TD>                                       
<TD ALIGN=RIGHT>  142.829</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200664>
200664</A> 
x30FD8</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_04 [1,670,576]</TD>                                       
<TD ALIGN=CENTER>20:37:53</TD>                                          
<TD ALIGN=CENTER>21:04:01</TD>                                          
<TD ALIGN=CENTER>00:25:35</TD>                                          
<TD ALIGN=RIGHT>     16,272,517</TD>                                    
<TD ALIGN=RIGHT>       186,617</TD>                                     
<TD ALIGN=RIGHT>      63,481</TD>                                       
<TD ALIGN=RIGHT>   29.357</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200667>
200667</A> 
x30FDB</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_TEST_3_04 [3,672,578]</TD>                                  
<TD ALIGN=CENTER>21:24:50</TD>                                          
<TD ALIGN=CENTER>21:40:24</TD>                                          
<TD ALIGN=CENTER>00:15:24</TD>                                          
<TD ALIGN=RIGHT>      6,618,974</TD>                                    
<TD ALIGN=RIGHT>        63,421</TD>                                     
<TD ALIGN=RIGHT>      21,898</TD>                                       
<TD ALIGN=RIGHT>   17.212</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200668>
200668</A> 
x30FDC</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_TEST_3_04 [3,672,578]</TD>                                  
<TD ALIGN=CENTER>21:46:20</TD>                                          
<TD ALIGN=CENTER>22:16:20</TD>                                          
<TD ALIGN=CENTER>00:28:05</TD>                                          
<TD ALIGN=RIGHT>     17,328,354</TD>                                    
<TD ALIGN=RIGHT>       211,608</TD>                                     
<TD ALIGN=RIGHT>      69,022</TD>                                       
<TD ALIGN=RIGHT>   30.976</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=200669>
200669</A> 
x30FDD</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_04 [1,670,576]</TD>                                       
<TD ALIGN=CENTER>22:26:48</TD>                                          
<TD ALIGN=CENTER>22:56:23</TD>                                          
<TD ALIGN=CENTER>00:28:55</TD>                                          
<TD ALIGN=RIGHT>     18,087,538</TD>                                    
<TD ALIGN=RIGHT>       215,339</TD>                                     
<TD ALIGN=RIGHT>      69,685</TD>                                       
<TD ALIGN=RIGHT>   30.924</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:00</TD>                                                       
<TD ALIGN=CENTER>15:40:26</TD>                                          
<TD ALIGN=RIGHT>   1,090,074,307</TD>                                   
<TD ALIGN=RIGHT>     12,231,507</TD>                                    
<TD ALIGN=RIGHT>     3,516,866</TD>                                     
<TD ALIGN=RIGHT> 1501.740</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Fri Jul 8 00:04:16.784 2005 -->
<font color=red>Shift Summary: </font><pre>Start shift al low L (~20E30) by ending physics run,  
turn system over to L3/SVT work.  
 
Special runs for testing new SVT hardware 
(but also good for physics): 
 
200660 - 200663 
 
A brief physics only run 200664 
 
Then some special runs to test changes to Trigger Table 
 
200665-200668 
 
Then another brief physics run: 200669 
 
Notes: 
 
There is a problem with DB/Procmon/Tomcat that occassionally 
hangs monitoring and the run summary page. If this happens page DAQ (Jane Nachtman). She has a special script to restart this stuff. If you use standard restart it will cause a different problem. Jane + Frank C. say this one needs Badgett. 
 
CMX trigger inhibit DOES NOT WORK on the HV ramp. (Works OK on a trip though). When ramping CMX use HV monitor page, not trigger inhibit, to know that CMX is up. 
 
Store was dropped at 11pm. D0 will have an access. ~1.5 hours quiet, then go to shot setup. Plan: do calibrations during quiet,  
also Markus working on L3/EB. 
 
Cal calibrations having trouble with plug crate.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   200660-200669
Delivered Luminosity   507.7nb-1  
Acquired Luminosity    383.7nb-1  
Efficiency             75.6
   
</td></tr></table></pre>
<!-- Author: Amidei -->