<!-- Title: Day shift. Sun Feb 13, 2005 -->
<!-- SciCo: carla -->
<!-- DAQAce: Saverio Da Ronco -->
<!-- MonAce: Oriol Salto -->
<!-- CO: Lucio Cerrito -->
<!-- OpManager: Bob Wagner -->
<!-- Notes: -->
<pre>Taking data. 
B0Lum = 37.05 E30 
 
Plug HV recovered.</pre>


<!-- Date: Sun Feb 13 08:27:53.679 2005 -->
<pre>(MLE) b0l2de00:SpyAlpha:8:21:40 AM-> 
Bad data: resetting FIFOs 
 
CT: 2005.02.13 08:26:23 
26'23" b0l3pcom1.fnal.gov:main:8:26:23 AM->Host b0eb11.fnal.gov, task tRec_0 
SCPU-P0-E-TracerEventId: Event 2952092, crate 1, channel 0 has either bad Tracer ID or bad markers around Tracer word. 
 Tracer word event ID is 0, should be 12. 
Tracer word = 0x00000001 (I think). 
Channel data = 0x0000022f 0x000c0031 0x434f5444 0x00140001 0x00000002 0x0000020c 0x00000003 0x0000000f ... 
... 0x00000000 0x00000000 0x00000000 0x00000000 0x00080000 0x00000000 0x00000001 0x00000000 --> 
 Additional Information: 
 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data quality in 
 SCPU b0eb11 (forwarded by FER crate CCAL_00). 
 Datataking is paused: events are not being processed. 
 
 AUTO HRR will be issued. 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed) 
 in 30 secs. [MLE] 
 
 
 
auto HRR fixed it</pre>
<!-- Author: Saverio :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193855' target=_top>193855</a>) -->


<!-- Date: Sun Feb 13 08:33:19.335 2005 -->
<pre>CT: 2005.02.13 08:32:03 
32'3" b0l3pcom1.fnal.gov:main:8:32:03 AM->Host b0eb11.fnal.gov, task tRec_0 
SCPU-P0-E-TracerEventId: Event 3046697, crate 35, channel 2 has either bad Tracer ID or bad markers around Tracer word. 
 Tracer word event ID is 0, should be 9. 
Tracer word = 0x00000023 (I think). 
Channel data = 0x00000223 0x00090033 0x434f5444 0x00140003 0x00000002 0x00000200 0x00000003 0x0000000f ... 
... 0x00000000 0x00000000 0x00000000 0x00000000 0x02008020 0x00010055 0x00000023 0x00000000 --> 
 Additional Information: 
 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data quality in 
 SCPU b0eb11 (forwarded by FER crate PCAL_02). 
 Datataking is paused: events are not being processed. 
 
 
 
autoHRR solves.</pre>
<!-- Author: Saverio :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193855' target=_top>193855</a>) -->


<!-- Date: Sun Feb 13 08:38:23.726 2005 -->
<pre>Done TO for crate 
b0l2de00 
autoHRR fix</pre>
<!-- Author: Saverio :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193855' target=_top>193855</a>) -->


<!-- Date: Sun Feb 13 08:41:20.642 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-2373&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-2373' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-2374&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-2374' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-2375&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-2375' width='300'></a></center>
<!-- Author: Oriol -->
<!-- Comment: Sun Feb 13 08;42;34 comment by...Oriol -->
Hourly plots <b>0730</b> - <b>0830</b>


<!-- Date: Sun Feb 13 08:44:40.865 2005 -->
Got a <b>CPR alarm</b> that disappeared in few seconds.
<!-- Author: Oriol :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193855' target=_top>193855</a>) -->

<!-- Date: Sun Feb 13 08:52:53.217 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193855">193855</a>
 Terminated at 2005.02.13 08:51:16
<!-- Author: RunControl -->


<!-- Date: Sun Feb 13 08:52:54.226 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193855">193855</a>
TERMINATE: reset EVB
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 08:59:56.557 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193856">193856</a>
 Activated at 2005.02.13 08:59:15
<!-- Author: RunControl -->


<!-- Date: Sun Feb 13 08:59:57.971 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193856">193856</a>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_00 [2,566,536] 
<!-- Author: Saverio (x2080) -->



<!-- Date: Sun Feb 13 09:16:54.408 2005 -->
<pre>In the previous run (193855), we got this error sequence that force us to end the run clean up EVB and start with a new run. 
 
We got an EVB problem (SCPU_TRACER_EVENT_ID Error !!!) related to b0eb16 (forwarded by FER crate PCAL_09). auto HRR was issued,but aftere that we got again an TRACER_EVENT problem related with b0eb11 again autoHRR. problem not fixed. 
We got a orange window that suggest to try an HRR and if this not solves the problem rest the run and clean up EVB. 
We HRR, we got busy TOP for b0tsi03 crate. 
We end -reset run 
clean up EVB and strarted a new run</pre>
<!-- Author: Saverio -->


<!-- Date: Sun Feb 13 09:20:20.862 2005 -->
<pre>got 5 L2 timeouts decision in a row. No particual error messagges in the Error Display. 
After that Busy TO  caused by EVB TraceEventId in b0eb11. 
the sixth HRR seems to solved. 
 
</pre>
<!-- Author: Saverio :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193856' target=_top>193856</a>) -->


<!-- Date: Sun Feb 13 09:22:33.361 2005 -->
I got two SVX AVDD pink cells: <b>B2W5L1</b> and <b>B2W4L4</b>.<br> 
The history of both is very flat, just small steps. Unmarked them and now they are greeen.<br>
<!-- Author: Oriol :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193856' target=_top>193856</a>) -->


<!-- Date: Sun Feb 13 09:30:58.236 2005 -->
<pre>Got a lot of EVB errors messagges in row. 
All was related with b0eb11. 
They started eith SCPU_TRACER_EVENT_ID error then SCPU_BAD_VRB_BYTE_COUNT errors and finally BUSY TIMEOUT. 
A lot of HRR fixed the problem.</pre>
<!-- Author: Saverio :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193856' target=_top>193856</a>) -->


<!-- Date: Sun Feb 13 09:36:40.818 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-2381&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-2381' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-2379&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-2379' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-2378&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-2378' width='300'></a></center>
<!-- Author: Oriol -->
<!-- Comment: Sun Feb 13 09;37;47 comment by...Oriol -->
Hourly plots: <b>0830 - 0930</b>


<!-- Date: Sun Feb 13 09:51:56.486 2005 -->
b0eb11 continues to give problems: 
SCPU_TRACER_EVENT_ID errorfollowed by  BUSY TIMEOUT.  
This time just ONE HRR solved.
<!-- Author: Saverio :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193856' target=_top>193856</a>) -->


<!-- Date: Sun Feb 13 09:58:18.880 2005 -->
<pre>b0eb16: 
SCPU_TRACER_EVENT_ID error followed by BUSY TIMEOUT: 
HRR fix</pre>
<!-- Author: Saverio :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193856' target=_top>193856</a>) -->


<!-- Date: Sun Feb 13 09:59:49.929 2005 -->
<pre>D0 calls: they will insert the roman pots. 
and wandering about the luminosity.</pre>
<!-- Author: carla -->

<!-- Date: Sun Feb 13 10:08:53.777 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193856">193856</a>
 Terminated at 2005.02.13 10:07:49
<!-- Author: RunControl -->


<!-- Date: Sun Feb 13 10:09:43.524 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193856">193856</a>
TERMINATE: Tevatron  Abort
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 10;20;02 comment by...carla -->
<pre>Paged silicon - no response from LRP.
call Rong.</pre>
<!-- Date: Sun Feb 13 10:13:58.177 2005 -->
<pre>From MCR e-log: 
10:08:13-   The store just died due to D0 going unsecure, D0 door #1. - amber 
-- Sun Feb 13 10:11:33 comment by...amber --  The Tev quenched house D1, after the lowbetas tripped. </pre>
<!-- Author: carla -->


<!-- Date: Sun Feb 13 10:15:49.646 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-2383&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-2383  width='300' align='top'></a></td><td><b>BLM summary plots</b> for the Tevatron abort.</td></tr></table>
<!-- Author: Oriol -->


<!-- Date: Sun Feb 13 10:17:05.753 2005 -->
<pre>Post quench check-out: 
COLDSTAR run conguration check was succesfully. 
</pre>
<!-- Author: Saverio -->


<!-- Date: Sun Feb 13 10:17:26.627 2005 -->
<pre>Ming called: the version of the code downloaded yesterday in the CLC crate has higher thresholds, they are coming in to redownload. 
This could explain the lower luminosity.</pre>
<!-- Author: carla -->



<!-- Date: Sun Feb 13 10:21:07.434 2005 -->
<pre>Stack at 31.15 E10.</pre>
<!-- Author: carla -->


<!-- Date: Sun Feb 13 10:22:39.003 2005 -->
Got three pink SVX AVDD cells: <b>B2W5L4</b> and <b>B2W4L4</b> are the ones that I unmarked before, the history are still very flat. I just unmarked them and now they are green.<br> 
And <b>B0W5L1</b>, it dropped dramatically when the quench and then  went back to its normal value. I unmarked it and now it is green.<br>
<!-- Author: Oriol -->

<!-- Date: Sun Feb 13 10:28:56.107 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193857">193857</a>
ACTIVATE: RunSet: ACE_QIE 
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 10;46;45 comment by...Lucio -->
Calibration okay
<!-- Date: Sun Feb 13 10:29:52.007 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193857">193857</a>
TERMINATE: RunSet: ACE_QIE done
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 10:32:51.551 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193858">193858</a>
ACTIVATE: RunSet: CALIB_CLC_QIE 
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 10:32:52.991 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193858">193858</a>
TERMINATE: RunSet: CALIB_CLC_QIE done
<!-- Author: Saverio (x2080) -->



<!-- Date: Sun Feb 13 10:33:45.188 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-2384&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-2384' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-2385&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-2385' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-2386&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-2386' width='300'></a></center>
<!-- Author: Oriol -->
<!-- Comment: Sun Feb 13 10;44;58 comment by...Oriol -->
Hourly plots: <b>0930 - 1030</b>.<br>
The quench was at 10:06. All the losses are below the critical 20Hz.
<!-- Date: Sun Feb 13 10:34:26.421 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193860">193860</a>
ACTIVATE: RunSet: CALIB_QIE_FWD 
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 10;53;13 comment by...Lucio -->
Calibration okay
<!-- Date: Sun Feb 13 10:36:34.606 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193860">193860</a>
TERMINATE: RunSet: CALIB_QIE_FWD done
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 10:44:37.533 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193861">193861</a>
ACTIVATE: RunSet: BOTHPLUGS_NORMAL 
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 10:44:38.921 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193861">193861</a>
TERMINATE: HV was in standby..... sorry
<!-- Author: Saverio (x2080) -->



<!-- Date: Sun Feb 13 10:45:38.989 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193862">193862</a>
ACTIVATE: RunSet: BOTHPLUGS_NORMAL 
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 11;16;52 comment by...Lucio -->
Calibration okay
<!-- Date: Sun Feb 13 10:51:33.916 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193862">193862</a>
TERMINATE: RunSet: BOTHPLUGS_NORMAL done
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 10:53:01.321 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193863">193863</a>
ACTIVATE: RunSet: BOTHPLUGS_MM_C 
<!-- Author: Saverio (x2080) -->



<!-- Date: Sun Feb 13 10:55:19.179 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193863">193863</a>
TERMINATE: RunSet: BOTHPLUGS_MM_C done

<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 10:56:17.931 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193864">193864</a>
ACTIVATE: RunSet: BOTHPLUGS_MM 
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 10:58:15.191 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193864">193864</a>
TERMINATE: RunSet: BOTHPLUGS_MM done
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 10:59:41.001 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193865">193865</a>
ACTIVATE: RunSet: CALIB_RP_SMQIE 
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 11;20;30 comment by...Lucio -->
Calibration Okay
<!-- Date: Sun Feb 13 11:00:23.760 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193865">193865</a>
TERMINATE: RunSet: CALIB_RP_SMQIE done
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 11:02:13.210 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193866">193866</a>
ACTIVATE: RunSet: ACE_TOF_QIE_NEW 
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 11;40;40 comment by...Lucio -->
Calibration Okay
<!-- Date: Sun Feb 13 11:02:30.748 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193866">193866</a>
TERMINATE: RunSet: ACE_TOF_QIE_NEW done
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 11:05:28.790 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193867">193867</a>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 11;54;56 comment by...Lucio -->
Calibration Okay
<!-- Date: Sun Feb 13 11:06:13.653 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193867">193867</a>
TERMINATE: RunSet: SMQIECALIB done
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 11:08:28.441 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193868">193868</a>
ACTIVATE: RunSet: ACE_LED_ALL 
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 11;35;43 comment by...Lucio -->
Calibration Okay. 
Three new bad channels flagged were reported in DBANA, so I have emailed isamu@hep.upenn.edu as per instructions
<!-- Date: Sun Feb 13 11:09:15.774 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193868">193868</a>
TERMINATE: RunSet: ACE_LED_ALL done
<!-- Author: Saverio (x2080) -->



<!-- Date: Sun Feb 13 11:09:48.605 2005 -->
Got one pink cell: SVX AVDD <b>B2W5L1</b>.<br> 
Very flat history. There is no beam. I umarked it and now it is green.<br>
<!-- Author: Oriol -->

<!-- Date: Sun Feb 13 11:10:55.190 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193869">193869</a>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 12;36;43 comment by...Lucio -->
Calibration void, it returns all 999999
<!-- Date: Sun Feb 13 11:10:56.993 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193869">193869</a>
TERMINATE: RunSet: COTCALIB_ACE done
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 11:16:04.375 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193870">193870</a>
ACTIVATE: RunSet: XEF_ALL 
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 12;40;07 comment by...Lucio -->
see figure at 12:39
<!-- Date: Sun Feb 13 11:18:38.901 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193870">193870</a>
TERMINATE: RunSet: XEF_ALL done
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 11:20:30.949 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193871">193871</a>
ACTIVATE: RunSet: CALIB_MUON_ALL1 
<!-- Author: Saverio (x2080) -->



<!-- Date: Sun Feb 13 11:21:00.288 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193871">193871</a>
TERMINATE: RunSet: CALIB_MUON_ALL1 done
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 11:22:35.932 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193872">193872</a>
ACTIVATE: RunSet: TOF_TAC_CALIB 
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 11:32:39.765 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193872">193872</a>
TERMINATE: RunSet: TOF_TAC_CALIB done
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 11;36;54 comment by...Saverio -->
The endof  run was not good:
needed to reset during endind procedure because sevb crate.



<!-- Date: Sun Feb 13 11:53:20.576 2005 -->
<pre>MCR calls to ask if we want to make use of 1hr access. After consulting Ops Manager we decline.</pre>
<!-- Author: carla -->



<!-- Date: Sun Feb 13 11:55:32.408 2005 -->
<pre>ACE_CALORIMETER_MINIMAL done 
I didn't got any "Enty popup" wen the run started or ended... 
Why? Anyawy the L1 rate was 51.6Hz (OK)</pre>
<!-- Author: Saverio -->

<!-- Date: Sun Feb 13 12:03:23.890 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193873">193873</a>
TERMINATE: exclude subfarm 5
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 13;23;11 comment by...Jeff Miles -->
This was the same failure mode as occurred on <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=127&frame=2&anchor=02:55:23.192'>
Saturday owl shift</a>, where a converter node and the Scanner Manager both go down simultaneously, and the converter must be rebooted (as opposed to just excluded) to allow SCRAMNet to function properly.  We will certainly look at this.
<!-- Date: Sun Feb 13 12:11:02.302 2005 -->
Yesterday one of the CLC ADMEM board (b0clc00, slot#4) somehow 
had problem with the Xilinx program that it was using. We tried 
to re-download the Xilinx program again. The problem with the 
board went away after the download. We then perform LED test and 
it seem to indicate that it was working fine. 
 
However during the last colliding store it was observed that the 
CDF luminosity was not as "high" as before (only higher than DZero's by about 5%, and not 10% at the begining of the store). 
We later found that the version of the Xilinx program that we 
downloaded yesterday had a higher threshold than we were using 
before. So after the store was lost, we re-download the right 
version of the Xilinx program. 
 
We will keep our eyes on the new store to see if things are 
back to normal.
<!-- Author: Sasha/Ming -->


<!-- Date: Sun Feb 13 12:11:20.965 2005 -->
Run 193846 was marked bad for L2 trigger.  However, this run is bad only for the CMX trigger.  We mark these as bad for CMX.  If it is bad for L2T, then nobody will use it.  If it is bad for CMX there are many analyses that can use the data.
<!-- Author: jdl :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193846' target=_top>193846</a>) -->


<!-- Date: Sun Feb 13 12:26:31.713 2005 -->
<pre>I tried to take a COSMIC run, but during partion L3 reported an error:  
Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely 
 to have died. 
   Do not panic! Level3 will try to repair itself. We recommend: 
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS 
     - exclude all affected subfarms from your run configuration 
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!) 
     - start another run and page L3 farm pager at closest day/evening shift 
     - if >2 subfarms affected page L3 farm IMMEDIATELY 
 
The guilty subfarm (c05) was then excluded. 
I reset and tried again: 
Same error but this time ther was also this error message: 
(MLE) b0l3pcom2.fnal.gov:main:12:05:01 PM->Error on L3 node b0l3pcom2.fnal.gov (partition 0) Sun Feb 13 12:05:01 2005 l3proxy 
  in Stop (L3ProxyPartition.cc:2693) 
  @L3_TRANS_FAIL - error stopping output node:   b0l3u01 
 
 
We are unable to partition. 
L3 paged. He call back quickly.  
c05 is down and also b0eb10 is not in the scramnet. 
L3 expert is coming to fix the problem.</pre>
<!-- Author: Saverio -->


<!-- Date: Sun Feb 13 12:34:06.081 2005 -->
Error Handler(EH) has been updated to v1_30a (declared current) with a fix for an excessive rate of sound messages  seen last night. Sorry about that.  
When posssible, please close/restart the EH GUI to pick up this last version. Thanks,  
<!-- Author: Donatella T. -->
<!-- Comment: Sun Feb 13 12;38;58 comment by...Saverio -->
Error Handler re-started now.We are using the updated version v1_30a
  



<!-- Date: Sun Feb 13 12:39:13.140 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-2387&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-2387  width='300' align='top'></a></td><td>Horizontal bands in the top right figure does not match the 
reference plots.</td></tr></table>
<!-- Author: Lucio -->
<!-- Comment: Sun Feb 13 14;00;03 comment by...Larry  -->
We are happy at this stage that it is back working at all!
<!-- Date: Sun Feb 13 13:02:32.405 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-2389&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-2389  width='300' align='top'></a></td><td>The SVX AVDD cell <b>B2W5L1</b> has a very flat history but it is at the limit of the caution state:<br> 
Caution: 479.0 < - > 496.0<br> 
Current: 496.0<br> 
So from time to time it turns pink and when I unmark it, it turns yellow.<br></td></tr></table>
<!-- Author: Oriol -->
<!-- Comment: Sun Feb 13 13;24;17 comment by...Rong-Shyang -->
I believe it's okay to leave it like that for a while as long as it stay at that value. We will try to work on it later. thanks.
<!-- Date: Sun Feb 13 13:20:11.753 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193874">193874</a>
TERMINATE: svt06 problem
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 13;40;21 comment by...Saverio -->
svt06 turned red during config.
Shepherd was needed.
<!-- Date: Sun Feb 13 13:33:20.496 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193877">193877</a>
 Activated at 2005.02.13 13:33:07
<!-- Author: RunControl -->


<!-- Date: Sun Feb 13 13:33:48.992 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193877">193877</a>
ACTIVATE: RunSet: ACE_COSMIC_NOTRACKS  Trigger: COSMICS_NOTRACKS [9,467,538] 
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 13:34:54.076 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193877">193877</a>
 Terminated at 2005.02.13 13:34:14
<!-- Author: RunControl -->
<!-- Comment: Sun Feb 13 13;39;34 comment by...Saverio -->
EVB problem. Orange window ask for clean up EVB. and re-start a new run.
I did it


<!-- Date: Sun Feb 13 13:41:01.129 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193878">193878</a>
 Activated at 2005.02.13 13:40:45
<!-- Author: RunControl -->


<!-- Date: Sun Feb 13 13:41:25.844 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193878">193878</a>
ACTIVATE: RunSet: ACE_COSMIC_NOTRACKS  Trigger: COSMICS_NOTRACKS [9,467,538] 
<!-- Author: Saverio (x2080) -->



<!-- Date: Sun Feb 13 14:12:58.805 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193878">193878</a>
 Terminated at 2005.02.13 14:12:42
<!-- Author: RunControl -->


<!-- Date: Sun Feb 13 14:13:15.272 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193878">193878</a>
TERMINATE: end run to ake calib
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 14:26:29.113 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193879">193879</a>
ACTIVATE: RunSet: COTCALIB_ACE 
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 14;27;23 comment by...Saverio -->
sevb was hanging during partition transition.
wE stop and re-start sevb. All is fine.
<!-- Comment: Sun Feb 13 14;36;45 comment by...Lucio -->
COT calibration keeps on reurning the following:

COT Calibration 14:25:40 on 2005-02-13
aveStart:                        999999.00   
total numBad:                   999999     - Too Many Bad channels
   numBad  cells:                  0      
   numBad cables:                  0      
   numBad slots:                   0      

aveDrift:                     999999.000    
max number drifted in crate:    -1       
total numDrifted:           999999 
   number of drifted  cells        0      
   number of drifted cables        0     
   number of drifted  slots        0 
<!-- Date: Sun Feb 13 14:26:51.704 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193879">193879</a>
TERMINATE: RunSet: COTCALIB_ACE done
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 14;38;53 comment by...Lucio -->
Calibration Okay
<!-- Comment: Sun Feb 13 14;39;49 comment by...Lucio -->
sorry, the CLC QIE below is okay, not the COT 
<!-- Date: Sun Feb 13 14:29:18.200 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193880">193880</a>
ACTIVATE: RunSet: CALIB_CLC_QIE 
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 14;40;06 comment by...Lucio -->
Calibration Okay
<!-- Date: Sun Feb 13 14:30:32.495 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193880">193880</a>
TERMINATE: RunSet: CALIB_CLC_QIE done
<!-- Author: Saverio (x2080) -->


<!-- Date: Sun Feb 13 14:35:04.501 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193881">193881</a>
ACTIVATE: RunSet: TOF_TAC_CALIB 
<!-- Author: Saverio (x2080) -->






<!-- Date: Sun Feb 13 14:44:51.414 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193881">193881</a>
TERMINATE: RunSet: TOF_TAC_CALIB done

<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 14;56;21 comment by...Saverio -->
As before the calibration was not ended clearly. We need to abort the run. sevb crashed and we needed to restart it.

<!-- Comment: Mon Feb 14 10;34;51 comment by...Luca -->
<pre>The first crash looks related with a problem during data taking.
According to the log files, only 6800 events were taken. We need
to investigate this. The second crash looks rrelated with the
consumer not being stopped properly the  first time.

</pre>
<!-- Date: Sun Feb 13 15:00:32.848 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193882">193882</a>
 Activated at 2005.02.13 15:00:20
<!-- Author: RunControl -->


<!-- Date: Sun Feb 13 15:01:09.286 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193882">193882</a>
ACTIVATE: RunSet: ACE_COSMIC_NOTRACKS  Trigger: COSMICS_NOTRACKS [9,467,538] 
<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 15;03;29 comment by...Saverio -->
<pre>During the first attempt to start the run we got the following error message during partition:
53'40" b0l3pcom1.fnal.gov:main:2:53:40 PM->Host b0eb10.fnal.gov, task tsmReg
SM-P0-E-ScpuDropout: Lost LAN connection to SCPU b0eb23 during registration. -->
 Additional Information:

 Attention!!!. SM_SCPU_DROPOUT Error !!!
 Hardware EVB problem: SCPU b0eb23 was rebooted or hung.
 - Check that L3 is in run configuration
 - End run and Reset
 - Clean up EVB
 - Check SM/SCPU from EVB/L3 Control Pane
[MLE]
(RC)   14:53:56 Reset -> START
(MLE) b0l3pcom2.fnal.gov:main:2:54:07 PM->Error on L3 node b0l3pcom2.fnal.gov (partition 0) Sun Feb 13 14:54:07 2005 l3proxy
  in Stop (L3ProxyPartition.cc:2693)
  @L3_TRANS_FAIL - error stopping output node:   b0l3u01


We followed the instruction: reset the run, clean up EVB and now all is ok.
The same problem arose few hours ago when we started the COSMICS run after Jeff fixed the problem.</pre>


<!-- Date: Sun Feb 13 15:15:16.992 2005 -->
<pre>No chance we'll see beam in the rest of the shift.. 
 
Totals 
Date:	2005.02.13 
Shift:	day 
Delivered luminosity:	260.9 nb-1 
Acquired luminosity:	211.8 nb-1 
Efficiency:	81.2</pre>
<!-- Author: carla -->


<!-- Date: Sun Feb 13 15:20:01.237 2005 -->
Switching consumers to cosmic run mode
<!-- Author: Lucio -->

<!-- Date: Sun Feb 13 15:37:37.637 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193882">193882</a>
 Terminated at 2005.02.13 15:37:12
<!-- Author: RunControl -->


<!-- Date: Sun Feb 13 15:37:38.826 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193882">193882</a>
TERMINATE: RunSet: ACE_COSMIC_NOTRACKS  Trigger: COSMICS_NOTRACKS [9,467,538] done

<!-- Author: Saverio (x2080) -->
<!-- Comment: Sun Feb 13 15;48;29 comment by...Saverio -->
This is a trigger table test. this will take 15-20 minutes.


<!-- Date: Sun Feb 13 15:55:55.285 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193855>
193855</A> 
x2F53F</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_00 [2,566,536]</TD>                                       
<TD ALIGN=CENTER>03:05:54</TD>                                          
<TD ALIGN=CENTER>08:51:16</TD>                                          
<TD ALIGN=CENTER>03:17:13</TD>                                          
<TD ALIGN=RIGHT>    250,994,013</TD>                                    
<TD ALIGN=RIGHT>     3,333,805</TD>                                     
<TD ALIGN=RIGHT>     861,809</TD>                                       
<TD ALIGN=RIGHT>  524.261</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193856>
193856</A> 
x2F540</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_00 [2,566,536]</TD>                                       
<TD ALIGN=CENTER>08:59:15</TD>                                          
<TD ALIGN=CENTER>10:07:49</TD>                                          
<TD ALIGN=CENTER>00:58:19</TD>                                          
<TD ALIGN=RIGHT>     76,928,823</TD>                                    
<TD ALIGN=RIGHT>       873,695</TD>                                     
<TD ALIGN=RIGHT>     233,407</TD>                                       
<TD ALIGN=RIGHT>  118.327</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>15:55:02</TD>                                                       
<TD ALIGN=CENTER>04:15:32</TD>                                          
<TD ALIGN=RIGHT>     327,922,836</TD>                                   
<TD ALIGN=RIGHT>      4,207,500</TD>                                    
<TD ALIGN=RIGHT>     1,095,216</TD>                                     
<TD ALIGN=RIGHT>  642.588</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sun Feb 13 15:56:35.938 2005 -->
<pre>Got this error message after ~400 events: 
CT: 2005.02.13 15:48:37 
48'37" b0dap84.fnal.gov:ConsumerErrorRe:3:48:37 PM-&gt;DAQErrMon:DataMissingError: Error 4 Event 0 Run 193883 Bank COTD_StorableBank missing from 
 event record, nErrors 4  --&gt; 
 Additional Information: 
 
 Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!! 
 
 DAQErrMon:DataMissingError: 
 Error 4 Event 0 Run 193883 Bank COTD_StorableBank missing from event record, nErrors 4 
 
 - We suggest issuing an HRR. 
 
 - if the problem persists for more than a few minutes, 
   starting a new Run may be necessary. 
 
 
The error appear 4 times in a row. 4 HRR did not solve the problem. 
Need to end the run</pre>
<!-- Author: Saverio -->
<!-- Comment: Sun Feb 13 18;08;51 comment by...ryo -->
Did you make sure CO to switch the runmode of ConsumerMonitor?
At that point, All ConsumerMonitor configuration have to be PHYSICS.
<!-- Date: Sun Feb 13 15:57:38.428 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=193884">193884</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_TEST_HIGHLUM_3_00 [5,578,541] 
<!-- Author: Saverio (x2080) -->



<!-- Date: Sun Feb 13 15:58:29.713 2005 -->
<pre>Beam to be injected in Tevatron - </pre>
<!-- Author: carla -->


<!-- Date: Sun Feb 13 16:00:17.288 2005 -->
<font color=red>Shift Summary: </font><pre>Store 3977, Lum = 37.05 E30  
Continue data taking after plug HV recovery, run 193855 
 
10:20 Store lost 
 
Take calibrations 
 
New Xilinx pgm downloaded to the CLC admem that was replaced yesterday.  
Should fix the "lower" luminosity seen in the last store.  
 
Problems with L3, expert comes in to fix it. 
 
More calibrations and cosmics. 
 
</pre> 
<!-- Author: carla -->