<!-- Title: Owl shift. Wed Jul  2, 2003 -->
<!-- SciCo: Duncan Carlsmith -->
<!-- DAQAce: Jason Galyardt -->
<!-- MonAce: Alberto Cruz -->
<!-- CO: Juan Pablo Fernandex -->
<!-- OpManager: Nathan Eddy -->
<!-- Notes: -->
<pre>Store 2739, Stack 129e10, accrt 1.9 mA/hr 
MCR says folks are working on th elinac, no est on shot setup but it may be on this shift. 
 
At shift change, the scico machine b0dap29 monitor decided to display no signal and was power cycled. Seems fine now.</pre>


<!-- Date: Wed Jul 2 00:21:34.804 2003 -->
Error (88) >>   SCPU b0eb14 (forwarded by FER crate COT_18). <br> 
Error (79) >>  SCPU-P0-E-TracerEventId: Event 799797, crate 67, channel 8 has either bad Tracer ID or bad markers around Tracer word. <br> 
Error (76) >>  (MLE) b0cot18:Messenger:12:04:18 AM->Runtime Error 1, Event 799797: See nonsequential event number (old 4, new 15) 
 in L2 message<br><br> 
Shepherded b0cot18 successfully.  Bring on the cosmics!
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165517' target=_top>165517</a>) -->


<!-- Date: Wed Jul 2 00:25:51.794 2003 -->
Big Brother:  b0dau30 has a "Trouble" indicator in the messages column.  Here's the message:<br><br> 
/var/adm/SYSLOG (scanned at Wed Jul  2 00:05:24 CDT 2003) 
 Jul  2 00:01:01 5A:b0dau30 unix: NOTICE: Starting XFS recovery on filesystem: /cdf/code-IRIX-6.5-backup (dev: 0/541)<br> 
 Jul  2 00:01:01 5A:b0dau30 unix: NOTICE: Ending XFS recovery for filesystem: /cdf/code-IRIX-6.5-backup (/hw/module/2/slot/MotherBoard/node/xtalk/8/pci/0/scsi_ctlr/0/target/5/lun/0/disk/partition/0/block) 
  
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165517' target=_top>165517</a>) -->


<!-- Date: Wed Jul 2 00:27:15.224 2003 -->
PSM in alarm for 30 minutes. page dan ryan 218-9497
<!-- Author: Carlsmith :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165517' target=_top>165517</a>) -->


<!-- Date: Wed Jul 2 00:29:22.565 2003 -->
CO sees mean readout charge all red in si. 
page 218-8227
<!-- Author: Carlsmith -->


<!-- Date: Wed Jul 2 00:34:14.028 2003 -->
Si responded to pager. YMON is consistent with cosmic ray running
<!-- Author: carlsmith -->


<!-- Date: Wed Jul 2 00:35:23.443 2003 -->
<pre>The heartbeat for PSM has been purple for about 30 min.  We have paged the PSM expert.  The Global alarms page seems to update slowly (i.e. the pin wheel seems to freeze for a minute or so at a time).  I have also gotten multiple error messages: 
 
error number 2147212502(8004232a) 
A network transition is in progress 
 
In addition the HBM (icicle status) turned red for about a minute, but there did not seem to be any problems with icicle. 
The status is now back to green for HBM.</pre>
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165517' target=_top>165517</a>) -->
<!-- Comment: Wed Jul  2 01;13;01 comment by...Alberto -->
I talked to Dan Ryan and went upstairs to check the Voltman <br> PC.  There was an error message saying that the Database <br> has reached maximum size.  After clicking ok the gui disappeared and <br> I restarted Volt.  Dan said that he would have to come anyway.
<!-- Comment: Wed Jul  2 01;59;28 comment by...Dan Ryan -->
The Voltman DB for web plots got too large.  Compressed the DB and created a fresh one.  Should be quiet now.
<!-- Date: Wed Jul 2 00:38:25.189 2003 -->
<pre>Heartbeat alarm for CSW,CSX, and MSK.  Will watch status.</pre>
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165517' target=_top>165517</a>) -->
<!-- Comment: Wed Jul  2 01;15;00 comment by...Alberto -->
All green.
<!-- Date: Wed Jul 2 00:49:26.934 2003 -->
<table><tr><td><a href=/elog/notebooks/2003/Wed_Jul__2_2003_Owl.004843.ps align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-7480  align='top'></td><td> This is the mean readout charge right now    Blue "boxes" = acceptable range When 1 bin is out of range,   the whole histogram turns red.   Julia (the silicon expert) says is normal due to the fact that we are   running Cosmics    I paste (print and add into the cosmic reference plots) this plot for future  CO reference (just to avoid bothering silicon experts next time)  </a></td></tr></table>
<!-- Author: Juan Pablo -->


<!-- Date: Wed Jul 2 00:50:51.642 2003 -->
<table><tr><td><a href=/elog/notebooks/2003/Wed_Jul__2_2003_Owl.005045.ps align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-7484  align='top'></td><td>another plot for future CO reference (short-term chip occupancy) </a></td></tr></table>
<!-- Author: Juan Pablo -->


<!-- Date: Wed Jul 2 00:59:47.727 2003 -->
Error (166) >>  (MLE) b0cot18:Messenger:12:57:07 AM->Runtime Error 1, Event 155155: See nonsequential event number (old 9, new 15 
) in L2 message<br> 
Error (178) >>  SCPU-P0-E-VrbReadFailed: Error reading VRB in slot 10 for event 954955.<br><br> 
Successfully shepherded b0cot18 for the second time this hour.
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165517' target=_top>165517</a>) -->




<!-- Date: Wed Jul 2 01:33:23.525 2003 -->
I performed a SCRAMNet check in the Ace Controls for EVB & L3, and the output has several "FISHY" labels and one "BAD".  <br> 
"FISHY":<br> 
          b0l3c08: CSR1 0X0200: {Tx retry:FISHY}<BR> 
          b0l3c13: CSR1 0X0600: {Tx retry:FISHY}{Tx T/O retry:FISHY}<br> 
          b0l3c14: CSR1 0X0600: {Tx retry:FISHY}{Tx T/O retry:FISHY}<br> 
          b0l3c15: CSR1 0X0600: {Tx retry:FISHY}{Tx T/O retry:FISHY}<br> 
          b0l3c16: CSR1 0X0200: {Tx retry:FISHY}<br> 
          b0tsi00: CSR1 0X0600: {Tx retry:FISHY}{Tx T/O retry:FISHY}<br> 
"BAD":<BR> 
          b0l3c11: CSR1 0x0ae0: {Protocol err:BAD}{Carrier err:BAD}{Message err:BAD}{Tx retry:FISHY}<BR><BR> 
I performed the check again, and the only suspicious message was as follows<BR> 
b0l3c11: CSR1 0X02A0: {Protocal err:BAD}{Message err:BAD}{Tx retry:FISHY}  
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165517' target=_top>165517</a>) -->

<!-- Date: Wed Jul 2 01:56:15.626 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165517">165517</A>
 Terminated at 2003.07.02 01:56:13
<!-- Author: RunControl -->


<!-- Date: Wed Jul 2 01:56:46.355 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165517">165517</A>
TERMINATE: End run for short Tev studies.
<!-- Author: Jason x2080 -->




<!-- Date: Wed Jul 2 02:06:06.214 2003 -->
Called MCR to inform that our voltages are on standby
<!-- Author: Carlsmith -->


<!-- Date: Wed Jul 2 02:09:31.363 2003 -->
Dan Ryan report Voltman database was too big and needed its quarterly zip compression. This was the source of the PSM heartbeat alarm. Now OK.
<!-- Author: Carlsmith -->

<!-- Date: Wed Jul 2 02:20:00.929 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165518">165518</A>
 Activated at 2003.07.02 02:19:58
<!-- Author: RunControl -->


<!-- Date: Wed Jul 2 02:22:12.539 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165518">165518</A>
ACTIVATE: ACE_COSMICS_ALL in prep. for shot setup.
<!-- Author: Jason x2080 -->



<!-- Date: Wed Jul 2 02:23:14.941 2003 -->
Error (5) >>  (MLE) b0pcal05:Messenger:2:20:11 AM->Event 1: Bunchcounters in slot 15 (BC=0) and slot 21 (BC=125) disagree (L2B=0)<br> 
Done Timeout, but no auto HRR, even though auto HRRs are enabled.  Manual HRR recovered.
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165518' target=_top>165518</a>) -->

<!-- Date: Wed Jul 2 02:34:02.569 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165518">165518</A>
 Terminated at 2003.07.02 02:34:01
<!-- Author: RunControl -->


<!-- Date: Wed Jul 2 02:34:03.788 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165518">165518</A>
TERMINATE: CSL problems.
<!-- Author: Jason x2080 -->
<!-- Comment: Wed Jul  2 03;38;44 comment by...Hirokazu Kobayashi -->
<br>
Aces found that L3 became light green state around 2am.<br>
They were taking cosmics run at that time.<br>
Looking through the CSL monitoring history plots,<br>
at around 2pm the rate at which CSL was receiving events<br>
increased from 8 MB/s to over 30 MB/s.<br>
<br>
I guessed the cause was beam split from proton.<br>
Sometime it appears that the tevatron was dumping out some <br>protons at this time, and therefore the cosmic trigger <br>
that you were using was getting overloaded.<br>
Probably, the right course of action to decrease the rate<br>
would be to change trigger table(cosmics run -> shortsetup run).<br>  

- Hiro
<!-- Date: Wed Jul 2 02:41:11.158 2003 -->
MCR calls to say they will start shotsetup around 3:15
<!-- Author: carlsmith -->

<!-- Date: Wed Jul 2 03:11:59.041 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165519">165519</A>
 Activated at 2003.07.02 03:05:34
<!-- Author: RunControl -->


<!-- Date: Wed Jul 2 03:12:00.639 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165519">165519</A>
ACTIVATE: Try ACE_COSMICS_ALL again
<!-- Author: Jason x2080 -->


<!-- Date: Wed Jul 2 03:12:01.662 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165519">165519</A>
 Terminated at 2003.07.02 03:06:29
<!-- Author: RunControl -->


<!-- Date: Wed Jul 2 03:12:02.932 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165519">165519</A>
TERMINATE: End run to cleanup EVB as requested.
<!-- Author: Jason x2080 -->



<!-- Date: Wed Jul 2 03:18:28.953 2003 -->
MCR call: Starting shot setup
<!-- Author: carlsmith -->


<!-- Date: Wed Jul 2 03:18:44.617 2003 -->
CSL Problems:  We noticed that the deadtime was almost 100%, and L3Display showed almost all nodes green (waiting to export data).  Given the CSL problems experienced on the Eve shift due to the b0dau30 disk crash, I thought it was a CSL problem.  Here's what I did:<br> 
--> logged into b0dau32 as 'ace' and issued 'cslcom cleanup' (RC was in the Active state; the instructions in the Eve shift log did not specify an RC state).<br> 
--> Busy TO auto HRR issued, Run Control got stuck in Halt transition<br> 
--> Tried to shepherd CSL ("Recover Special Needs Client"); failed to partition, so aborted run<br> 
--> RC stuck in Abort transition, so issued Reset.<br> 
--> paged CSL expert, he came in and looked at it; his explanation is that the pilot bunches in the Tev at the time could have caused the triggers to overload, giving high deadtime.  In the future, when this happens, we should end the run, wait for the beam to settle, then restart a cosmics run, or perhaps AAA_SHOTSETUP (where high beam losses are expected).
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165518' target=_top>165518</a>) -->
<!-- Comment: Wed Jul  2 07;57;38 comment by...Jason -->
After several attempts at running with ACE_COSMICS_ALL configuration, whatever caused the hot trigger situation went away and we were able to run ACE_COSMICS_ALL for a while before shotsetup.
<!-- Date: Wed Jul 2 03:19:17.586 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165520">165520</A>
 Activated at 2003.07.02 03:19:16
<!-- Author: RunControl -->


<!-- Date: Wed Jul 2 03:20:39.278 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165520">165520</A>
 Terminated at 2003.07.02 03:19:47
<!-- Author: RunControl -->


<!-- Date: Wed Jul 2 03:20:41.212 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165520">165520</A>
TERMINATE: dto as soon as ACE_COSMICS_ALL run started. 
<!-- Author: Jason x2080 -->
<!-- Comment: Wed Jul  2 03;38;03 comment by...Jason -->
Thought we might have further hot trigger problems that would require ending the run and waiting.
<!-- Date: Wed Jul 2 03:23:01.111 2003 -->
Tried to start ACE_COSMICS_ALL, told to cleanup EVB.  Done.
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165519' target=_top>165519</a>) -->

<!-- Date: Wed Jul 2 03:27:17.712 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165521">165521</A>
 Activated at 2003.07.02 03:26:49
<!-- Author: RunControl -->


<!-- Date: Wed Jul 2 03:27:18.140 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165521">165521</A>
ACTIVATE: try ACE_COSMICS_ALL again.
<!-- Author: Jason x2080 -->
<!-- Comment: Wed Jul  2 03;38;54 comment by...Jason -->
Hot trigger problem has disappeared.  All seems well with the world.

<!-- Date: Wed Jul 2 03:45:11.396 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165521">165521</A>
 Terminated at 2003.07.02 03:45:15
<!-- Author: RunControl -->


<!-- Date: Wed Jul 2 03:45:12.560 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165521">165521</A>
TERMINATE: Final protons being loaded.
<!-- Author: Jason x2080 -->
<!-- Comment: Wed Jul  2 03;54;03 comment by...Jason -->
My mistake; 2 pilot bunches.


<!-- Date: Wed Jul 2 04:15:43.444 2003 -->
The PSM alarm went off and 1RR12C_1 was red for a few seconds.  Everything is green now.
<!-- Author: Alberto -->


<!-- Date: Wed Jul 2 04:21:18.829 2003 -->
CEM Occupany plots for cosmics are E-W asymetric. Page 218-9500.
<!-- Author: carlsmith -->

<!-- Date: Wed Jul 2 04:22:45.483 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165522">165522</A>
ACTIVATE: AAA_SHOTSETUP
<!-- Author: Jason x2080 -->



<!-- Date: Wed Jul 2 04:23:13.880 2003 -->
Injecting final protons being loaded
<!-- Author: carlsmith -->


<!-- Date: Wed Jul 2 04:24:10.228 2003 -->
<TABLE bgcolor=#cdc9a5> 
<tr><th colspan=2>Totals</th></tr> 
<tr><td>Date:</td><td>2003.07.01</td> 
<tr><td>Shift:</td><td>eve</td> 
<tr><td>Delivered luminosity:</td><td> 32.51 nb<sup>-1</sup></td></tr> 
<tr><td>Acquired luminosity:</td><td>  6.94 nb<sup>-1</sup></td></tr> 
<tr><td>Efficiency:</td><td>21.34</td></tr> 
</TABLE>  
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165522' target=_top>165522</a>) -->


<!-- Date: Wed Jul 2 04:26:02.241 2003 -->
<TABLE bgcolor=#cdc9a5> 
<TR><TH>Date</th><th>Time</th><th>BLM</th><th colspan=2>Dose</th></tr> 
<tr><td>2003.07.02</td><td>04:20:47</td><td>W Inner BLM</td><td>6.03</td><td>RADS</td></tr> 
<tr><td>2003.07.02</td><td>04:20:47</td><td>W Outer BLM</td><td>0.01</td><td>RADS</td></tr> 
<tr><td>2003.07.02</td><td>04:20:47</td><td>E Inner BLM</td><td>10.52</td><td>RADS</td></tr> 
<tr><td>2003.07.02</td><td>04:20:47</td><td>E Outer BLM</td><td>15.07</td><td>RADS</td></tr> 
</TABLE> 
<strong>Integrated dosage</strong>
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165522' target=_top>165522</a>) -->
<!-- Comment: Wed Jul  2 04;26;35 comment by...Alberto -->
Shot setup 


<!-- Date: Wed Jul 2 04:28:40.989 2003 -->
Willis responds to page and is not concerned about EW asymmetry in Cosmics run.
<!-- Author: carlsmith -->


<!-- Date: Wed Jul 2 04:33:00.955 2003 -->
stack=140e10
<!-- Author: carlsmith -->


<!-- Date: Wed Jul 2 04:34:13.152 2003 -->
Paged ops manager 314-4862 per white board instructions
<!-- Author: carlsmith -->


<!-- Date: Wed Jul 2 05:26:23.139 2003 -->
Pbars loaded
<!-- Author: carlsmith :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165522' target=_top>165522</a>) -->

<!-- Date: Wed Jul 2 05:28:58.082 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165522">165522</A>
TERMINATE: Flattop.
<!-- Author: Jason x2080 -->



<!-- Date: Wed Jul 2 05:48:09.606 2003 -->
Store 2740 - Scraping complete, d0lum 21e30, tevpr 7261e9, tevpb 520e9, lostp 4500 hz, lostpb <100, b0lum 2240 after clc up
<!-- Author: Carlsmith -->


<!-- Date: Wed Jul 2 05:50:40.559 2003 -->
Sorry, I meant B0lum = 22.4e30
<!-- Author: scico -->

<!-- Date: Wed Jul 2 05:51:49.517 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523">165523</A>
 Activated at 2003.07.02 05:51:09
<!-- Author: RunControl -->


<!-- Date: Wed Jul 2 05:52:32.256 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523">165523</A>
ACTIVATE: AAA_CURRENT, low lumi table PHYSICS_1_05[4,309,386].
<!-- Author: Jason x2080 -->



<!-- Date: Wed Jul 2 05:52:56.714 2003 -->
Error (29) >>  (MLE) b0cot19:Messenger:5:51:36 AM->Event 2277: Bunchcounters in slot 5 (BC=6) and slot 20 (BC=12) disagree (L2B=1)<br> 
Done Timeout -- auto HRR recovered.
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523' target=_top>165523</a>) -->
<!-- Comment: Wed Jul  2 06;01;43 comment by...Jason -->
Error (58) >>  (MLE) b0cot19:Messenger:6:00:37 AM->Event 85255: Bunchcounters in slot 5 (BC=121) and slot 20 (BC=21) disagree (L2B=1)<br>
Done Timeout -- auto HRR recovered.
<!-- Comment: Wed Jul  2 06;02;53 comment by...Jason -->
Error (80) >>  (MLE) b0cot19:Messenger:6:02:12 AM->Event 100709: Bunchcounters in slot 5 (BC=71) and slot 20 (BC=62) disagree (L2B=3)<br>
Done Timeout -- auto HRR recovered.
<!-- Comment: Wed Jul  2 06;05;41 comment by...Jason -->
Error (103) >>  (MLE) b0cot19:Messenger:6:04:47 AM->Event 129038: Bunchcounters in slot 5 (BC=65) and slot 20 (BC=133) disagree (L2B=1)<br>
Done Timeout -- auto HRR recovered.
<!-- Comment: Wed Jul  2 06;08;48 comment by...Jason -->
Error (108) >>  (MLE) b0cot17:Messenger:6:08:06 AM->Event 167274: Bunchcounters in slot 3 (BC=68) and slot 18 (BC=6) disagree (L2B=1)<BR>
Done Timeout -- auto HRR recovered.
<!-- Comment: Wed Jul  2 06;57;17 comment by...Alberto -->
Error (19) >>  (MLE) b0cot19:Messenger:6:55:32 AM->Event 690629: Bunchcounters in slot 5 (BC=53) and slot 20 (BC=74) disagree (L2B=1).  Auto HRR recovered.
<!-- Comment: Wed Jul  2 06;59;28 comment by...Jason -->
Error (26) >>  (MLE) b0cot10:Messenger:6:58:49 AM->Event 728540: Bunchcounters in slot 5 (BC=59) and slot 10 (BC=0) disagree (L2B=3)<br>
Done Timeout -- auto HRR recovered.
<!-- Comment: Wed Jul  2 07;23;16 comment by...Jason -->
Error (49) >>  (MLE) b0cot19:Messenger:7:22:17 AM->Event 1008598: Bunchcounters in slot 5 (BC=124) and slot 20 (BC=0) disagree (L2B=2)<br>
Done Timeout -- auto HRR recovered.
<!-- Comment: Wed Jul  2 07;26;41 comment by...Jason -->
Error (75) >>  (MLE) b0cot19:Messenger:7:23:25 AM->Event 1018998: Bunchcounters in slot 5 (BC=3) and slot 20 (BC=18) disagree (L2B=2)<br>
Done Timeout -- auto HRR recovered.  <br><br>
--->  This crate, b0cot19, has been giving Bunchcounter Mismatch errors much more frequently than any other crate the last couple of days. <---<br>
<!-- Comment: Wed Jul  2 07;27;40 comment by...Jason -->
<br>Error (95) >>  (MLE) b0cot19:Messenger:7:26:40 AM->Event 1054948: Bunchcounters in slot 5 (BC=106) and slot 20 (BC=15) disagree (L2B=1)<br>
Done Timeout -- auto HRR recovered.
<!-- Comment: Wed Jul  2 07;29;30 comment by...Jason -->
Error (117) >>  (MLE) b0cot19:Messenger:7:28:58 AM->Event 1079027: Bunchcounters in slot 5 (BC=112) and slot 20 (BC=56) disagree (L2B=2)<br>
Done Timeout -- auto HRR recovered.
<!-- Comment: Wed Jul  2 07;34;46 comment by...Jason -->
Error (139) >>  (MLE) b0cot19:Messenger:7:33:55 AM->Event 1134227: Bunchcounters in slot 5 (BC=127) and slot 20 (BC=86) disagree (L2B=3)<br>
Done Timeout -- auto HRR recovered.
<!-- Date: Wed Jul 2 05:57:06.936 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-7485&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-7485  width='300' align='top'></td><td>Losses During Shot.</a></td></tr></table>
<!-- Author: Alberto -->



<!-- Date: Wed Jul 2 06:02:27.642 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-7487&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-7487  width='300' align='top'></td><td>SVRAD during shot.</a></td></tr></table>
<!-- Author: Alberto -->





<!-- Date: Wed Jul 2 06:17:38.871 2003 -->
Busy Timeout caused by EVB TracerEventId error in SCPU = b0eb16.<br> 
(RC)   6:12:4 Halt -> HALTED 
Error (125) >>  Channel data = 0x000000bb 0x54350024 0x50454d44 0x000c0004 0x00000000 0x00000051 0x00000001 0x02d40031 ... 
Error (126) >>  ... 0x000002e0 0x00000000 0x00006021 0x000d0000 0xaa00aaaa 0x55aa5555 0x54550024 0x80350000[MLE]<br><br> 
Auto HRR recovered.
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523' target=_top>165523</a>) -->
<!-- Comment: Wed Jul  2 06;19;12 comment by...Jason -->
More:<br>
SCPU_TRACER_EVENT_ID Error from SCPU b0eb16 (forwarded by FER crate PCAL_03).
<!-- Date: Wed Jul 2 06:18:37.203 2003 -->
XFT TRIGMON occ vs phi says too many error, page expert. Paged 266-2886
<!-- Author: scico :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523' target=_top>165523</a>) -->



<!-- Date: Wed Jul 2 06:25:54.620 2003 -->
<table><tr><td><a href=/elog/notebooks/2003/Wed_Jul__2_2003_Owl.062549.ps align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-7495  align='top'></td><td>Message at the beginning of the run asking for paging the XFT expert. It cleared itself a minute later</a></td></tr></table>
<!-- Author: Juan Pablo -->
<!-- Comment: Wed Jul  2 09;04;03 comment by...Evelyn  -->
<pre>Rarely, if at the start of a run, one error is enough to trigger this alarm.  If the scale for the number of entries is small, then it is not a problem
</pre>
<!-- Date: Wed Jul 2 06:38:12.630 2003 -->
CO reports two isl wedges with no data. Paged SI 218-8227
<!-- Author: carlsmith -->
<!-- Comment: Wed Jul  2 06;47;54 comment by...Juan Pablo. -->
<pre>ISL0 B0 W7 and 8 (both sides) are the 
wedges with no data.</pre>
<!-- Date: Wed Jul 2 06:38:25.597 2003 -->
23'59" b0l3gate1.fnal.gov:main:6:29:13 AM->Host b0eb16.fnal.gov, task tRec_0 
SCPU-P0-E-CantResetVrb: Reset of VRB in slot 10 failed. 
VRB_BUS_ERROR: Alignment or VME bus error.<br><br> 
Pop-up window requested that I manually press the reset button of the VRB in slot 10 of b0eb16.  Done; Active state recovered.
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523' target=_top>165523</a>) -->


<!-- Date: Wed Jul 2 06:42:28.855 2003 -->
Julia respond to SVX page, says she will be in and look at the problem.
<!-- Author: carlsmith -->


<!-- Date: Wed Jul 2 06:43:12.238 2003 -->
I got a PSM Alarm and found all the 2nd floor power supplies were red and <br> there was a message box about not be able to connect with the <br> node.  Then they all went back to green.
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523' target=_top>165523</a>) -->



<!-- Date: Wed Jul 2 06:51:10.640 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-7496&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-7496' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-7497&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-7497' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-7498&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-7498' width='300'></a></center>
<!-- Author: Alberto -->
<!-- Comment: Wed Jul  2 06;51;36 comment by...Alberto -->
Hourly Acnet plots


<!-- Date: Wed Jul 2 06:53:09.490 2003 -->
Big Brother: b0dap31 has an "Attention" indicator under the disk catagory.  Here's the interesting bit:<br> 
/dev/sdc1              67G   58G  6.0G  91% /cdf/code-level3<br> 
/dev/sdd1              67G   58G  6.0G  91% /level3-localbackup<br><br> 
This indicator has been like this since the shift started, but the usage was 90% at the beginning of the shift.
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523' target=_top>165523</a>) -->




<!-- Date: Wed Jul 2 07:18:51.733 2003 -->
CES East Wire Occupancy per event is very high (0.18 when  
typical is 0.005) for Phi Wedge 6 wire # 31 
(statistics is low 1840)
<!-- Author: Juan Pablo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523' target=_top>165523</a>) -->


<!-- Date: Wed Jul 2 07:19:05.300 2003 -->
I got a heartbeat alarm from ces,ccr,and cpr, but it returned to green after a minute or two.
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523' target=_top>165523</a>) -->



<!-- Date: Wed Jul 2 07:23:17.449 2003 -->
<pre>PES V Layer (Layer 0) East Occupancy per event 
is very high for Phi octant 2 strip # 83 (the value 
is 0.9 when typically 0.05). 
The average raw ADC count per hit is very high too. 
</pre>
<!-- Author: Juan Pablo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523' target=_top>165523</a>) -->






<!-- Date: Wed Jul 2 07:39:07.638 2003 -->
(MLE) b0tsi02:Messenger:7:36:07 AM->DONE timeout 
(MLE) b0dap73.fnal.gov:Thread-570:7:32:36 AM->Done Timeout: COT_19<br><br> 
There was no other error in the log file associated with this DTO.  Odd.  
<!-- Author: Jason :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523' target=_top>165523</a>) -->
<!-- Comment: Wed Jul  2 07;53;49 comment by...Jason -->
This happened again with the same crate (COT_19).  Auto HRR recovered in both cases.
<!-- Comment: Wed Jul  2 07;54;49 comment by...Jason -->
Again -- auto HRR recovered.
<!-- Date: Wed Jul 2 07:49:28.152 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-7499&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-7499' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-7500&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-7500' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-7501&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-7501' width='300'></a></center>
<!-- Author: Alberto -->
<!-- Comment: Wed Jul  2 07;49;51 comment by...Alberto -->
Hourly ACnet plots.



<!-- Date: Wed Jul 2 07:54:38.773 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-7502&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-7502' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-7503&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-7503' width='300'></a></center>
<!-- Author: Alberto -->
<!-- Comment: Wed Jul  2 07;55;07 comment by...Alberto -->
End of Shift plots.
<!-- Date: Wed Jul 2 07:55:01.210 2003 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523>
165523</A> 
x28693</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_05 [4,309,386]</TD>                                       
<TD ALIGN=CENTER>05:51:09</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>01:50:09</TD>                                          
<TD ALIGN=RIGHT>     86,731,820</TD>                                    
<TD ALIGN=RIGHT>     1,345,678</TD>                                     
<TD ALIGN=RIGHT>     312,506</TD>                                       
<TD ALIGN=RIGHT>  135.512</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>07:55:02</TD>                                                       
<TD ALIGN=CENTER>01:50:09</TD>                                          
<TD ALIGN=RIGHT>      86,731,820</TD>                                   
<TD ALIGN=RIGHT>      1,345,678</TD>                                    
<TD ALIGN=RIGHT>       312,506</TD>                                     
<TD ALIGN=RIGHT>  135.512</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->




<!-- Date: Wed Jul 2 07:56:40.807 2003 -->
<TABLE bgcolor=#cdc9a5> 
<TR><TH>Date</th><th>Time</th><th>BLM</th><th colspan=2>Dose</th></tr> 
<tr><td>2003.07.02</td><td>07:50:47</td><td>W Inner BLM</td><td>76.02</td><td>RADS</td></tr> 
<tr><td>2003.07.02</td><td>07:50:47</td><td>W Outer BLM</td><td>0.00</td><td>RADS</td></tr> 
<tr><td>2003.07.02</td><td>07:50:47</td><td>E Inner BLM</td><td>0.00</td><td>RADS</td></tr> 
<tr><td>2003.07.02</td><td>07:50:47</td><td>E Outer BLM</td><td>34.73</td><td>RADS</td></tr> 
</TABLE> 
<strong>Integrated dosage</strong>
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=165523' target=_top>165523</a>) -->



<!-- Date: Wed Jul 2 08:01:43.988 2003 -->
<font color=red>Shift Summary: </font><pre>Owl: 
At shift change, the scico machine b0dap29 monitor decided to 
display no signal and was power cycled. Seems fine now. 
 
PSM expert Dan RYan was called to respond to heart beat alarm.  
Voltman web plot database needed compression. THanks Dan. 
 
Wed Jul 2 03:38:44 comment by...Hirokazu Kobayashi --   
Aces found that L3 became light green state around 2am. 
They were taking cosmics run at that time. 
Looking through the CSL monitoring history plots, 
at around 2pm the rate at which CSL was receiving events 
increased from 8 MB/s to over 30 MB/s. 
 
I guessed the cause was beam split from proton. 
Sometime it appears that the tevatron was dumping out some  
protons at this time, and therefore the cosmic trigger  
that you were using was getting overloaded. 
Probably, the right course of action to decrease the rate 
would be to change trigger table(cosmics run -&gt; shortsetup run). 
- Hiro  
 
5:48 store 2740 finally in tevpr 7261e9, tevpb 520e9, b0lum 22.4 
Started run 165522 low lum physics trigger table. 
  
Trigmon xft vs phi said "Too many errors, page expert", which we did. 
No response from expert but the error cleared. 
 
Two isl wedges with no data. 
Julia responds to SVX page, says she will be in and look at the problem. 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   165522
Delivered Luminosity   158  
Acquired Luminosity    136  
Efficiency             86
   
</td></tr></table></pre>
<!-- Author: Duncan Carlsmith -->