<!-- Title: Day shift. Sat Apr  9, 2005 -->
<!-- SciCo: virgil barnes -->
<!-- DAQAce: xavier portello -->
<!-- MonAce: pedro fernandez -->
<!-- CO: juerg beringer -->
<!-- OpManager: sung won lee -->
<!-- Notes: -->
<pre>smooth start.  be sure to check with Rick Snider around 13:00 that oracle server is returning to service -- CSL buffer could be dangerously full by ~16:00 - 17:00.</pre>


<!-- Date: Sat Apr 9 08:31:30.932 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-5974&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-5974  width='300' align='top'></a></td><td>SVX_AVDD_B0W2L3 tripped; Recovering according to instructions failed. Expert paged.</td></tr></table>
<!-- Author: Pedro -->
<!-- Comment: Sat Apr  9 08;52;40 comment by...Pedro -->
<pre>Recovering procedure according to expert's advice:
Turned PC on and tried to turn HV on of individual affected layers.
This was finally successful for all layers EXCEPT for B0W2L3.
Expert is on the way. Meanwhile we are continuing data taking w/o Silicon.
 </pre>
<!-- Comment: Sat Apr  9 11;10;13 comment by...Sebastian -->
<pre>Took the ladder (e023) out of the HWDB for now. Suspect readback problems, will take a look during quiet time. </pre>
<!-- Date: Sat Apr 9 08:39:50.104 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196346">196346</a>
 Terminated at 2005.04.09 08:38:58
<!-- Author: RunControl -->


<!-- Date: Sat Apr 9 08:39:52.629 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196346">196346</a>
TERMINATE: Taking out the silicion
<!-- Author: Cristobal x2080 -->


<!-- Date: Sat Apr 9 08:45:21.024 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196347">196347</a>
 Activated at 2005.04.09 08:44:39
<!-- Author: RunControl -->


<!-- Date: Sat Apr 9 08:45:22.982 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196347">196347</a>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_3_01 [2,618,559] 
<!-- Author: Cristobal x2080 -->



<!-- Date: Sat Apr 9 08:45:22.529 2005 -->
<pre>When taking out the Silicon and activating. We got a Heap Corrupt message from b0cot01. I shepherded it successfully.</pre>
<!-- Author: Xavier -->


<!-- Date: Sat Apr 9 08:48:16.707 2005 -->
<pre>L2 decision TO. HRRed.</pre>
<!-- Author: Xavier :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196347' target=_top>196347</a>) -->



<!-- Date: Sat Apr 9 09:02:58.199 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5975&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5975' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5976&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5976' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5977&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5977' width='300'></a></center>
<!-- Author: Pedro -->
<!-- Comment: Sat Apr  9 09;03;48 comment by...Pedro -->
<pre>Hourly plots 0745-0900 ok.</pre>

<!-- Date: Sat Apr 9 09:37:43.567 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196347">196347</a>
 Terminated at 2005.04.09 09:36:54
<!-- Author: RunControl -->


<!-- Date: Sat Apr 9 09:37:44.153 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196347">196347</a>
TERMINATE: b0pcal07 is not responding
<!-- Author: Cristobal x2080 -->


<!-- Date: Sat Apr 9 09:44:46.731 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196348">196348</a>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Cristobal x2080 -->



<!-- Date: Sat Apr 9 09:45:06.558 2005 -->
<pre>We got a PSH HV alarm. Crate 10 was effected. PowerCycled it according to instructions, now ok. Redo SMX calibartions.</pre>
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196348' target=_top>196348</a>) -->
<!-- Comment: Sat Apr  9 10;00;24 comment by...Pedro -->
<pre>Because expert asked: We did the calibrations with 
Plug HV STBY, CES/CPR OFF</pre>
<!-- Date: Sat Apr 9 09:46:32.565 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196348">196348</a>
TERMINATE: RunSet: SMQIECALIB done
<!-- Author: Cristobal x2080 -->


<!-- Date: Sat Apr 9 09:52:58.875 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196349">196349</a>
 Activated at 2005.04.09 09:52:43
<!-- Author: RunControl -->


<!-- Date: Sat Apr 9 09:54:41.485 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196349">196349</a>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_01 [2,618,559] 
<!-- Author: Cristobal x2080 -->


<!-- Date: Sat Apr 9 09:54:42.346 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196349">196349</a>
 Terminated at 2005.04.09 09:53:04
<!-- Author: RunControl -->


<!-- Date: Sat Apr 9 09:54:43.429 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196349">196349</a>
TERMINATE: RC problem
<!-- Author: Cristobal x2080 -->



<!-- Date: Sat Apr 9 10:04:13.343 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196350">196350</a>
TERMINATE: problems b0svx08
<!-- Author: Cristobal x2080 -->


<!-- Date: Sat Apr 9 10:11:31.877 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196351">196351</a>
 Activated at 2005.04.09 10:07:58
<!-- Author: RunControl -->


<!-- Date: Sat Apr 9 10:11:32.743 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196351">196351</a>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_01 [2,618,559] 
<!-- Author: Cristobal x2080 -->



<!-- Date: Sat Apr 9 10:29:55.059 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5978&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5978' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5979&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5979' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5980&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5980' width='300'></a></center>
<!-- Author: Pedro -->
<!-- Comment: Sat Apr  9 10;30;23 comment by...Pedro -->
<pre>Hourly plots 0900-1015 ok.</pre>


<!-- Date: Sat Apr 9 10:40:29.794 2005 -->
<pre>MAJOR PROBLEMS (ONE FOLLOWING THE OTHER). Let me make a chronological description: 
 
1- After the plug tripped (crate 10), the yellow button didn't help and we need to reboot it. This week they reminded us that each time we do this operation we need to to calibrations and that's why we did them. Willis called and we told to him that we were about to doing the calibrations. He agreed. Michael told us later that we don't need to do it! They will try to put clear instructions soon. 
 
2- When ending the run to do the calibrations and to introduce the Si (already recovered) I shift-clicked RUN and END buttons while recovering. It got stuck due to b0pcal07. I shepherded it but it was NOT SUCCESSFUL. It said: 
FER_smartInitMS: Couldn't connect to RTServer. 
This also happened yesterday when the RC was stuck close to the dropping of the store. However there were no RTerrors reported in ProcMon. I finally couldn't recover it and I need to reset (abort didn't help). The crate was still bad when calibrating but there I could reset and later shepherd and it worked. 
 
3- After including Si I tried to start a new run but I couldn't! Now there was b0xtrp00 memory exhausted problem. The recommendation was to shepherd. I did it and it failed. I tried to reset and later shepherd. This told me that it was successful but when I press Activate I received one more message reporting "memory exhausted". But something funny happened also: remember that I shift-clicked END in step 1? It was still yellow but I thought it was a simple GUI bug because I have even changed the RUN display to do the calibrations. But no! It seems it was still valid so it automatically ended the run. 
 
4- I configured again but I couldn't do it because both b0xtrp00 and b0svx08 where yellow. I send a config message to b0xtrp00 and this solved this problem! Later for b0svx08 there was the Si expert here but he didn't see anything wrong so we abort and configured again. It worked! But when configuring b0cmx00 complained! Geeze! Luckily shepherding was successful. 
 
We are taking data again! 
 
Now the million dolar question: in which downtime category falls all this? :)</pre>
<!-- Author: Xavier -->


<!-- Date: Sat Apr 9 10:41:03.085 2005 -->
<pre>L2 decision TO. HRRed</pre>
<!-- Author: Xavier :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196351' target=_top>196351</a>) -->


<!-- Date: Sat Apr 9 10:43:01.158 2005 -->
<pre>DQMon alarm for SVX and ISL. Lots of cells affected but Silicon expert says that is normal because they are the ones that we took out. He will fix it. DQMon is working fine ;)</pre>
<!-- Author: Xavier :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196351' target=_top>196351</a>) -->
<!-- Comment: Sat Apr  9 11;17;23 comment by...Sebastian -->
Yes, I asked the CO (Juerg) to restart SVXMon - to pick up 
the changes in the HWDB, things look fine now for SVX.
I'm still looking at ISL.
<!-- Comment: Sat Apr  9 11;40;38 comment by...Sebastian -->
Could not find anything wrong with the ISL yellow ladders. Let's give them more time to get data.
<!-- Date: Sat Apr 9 11:15:02.945 2005 -->
<pre>Silenced gas alarm: FTI-IMUSUP Lo/Hi=7.0/9.0, current=7.0.</pre>
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196351' target=_top>196351</a>) -->



<!-- Date: Sat Apr 9 11:39:25.893 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5981&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5981' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5982&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5982' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5983&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5983' width='300'></a></center>
<!-- Author: Pedro -->
<!-- Comment: Sat Apr  9 11;40;04 comment by...Pedro -->
<pre>Hourly plots 1015-1130 look ok.</pre>
<!-- Date: Sat Apr 9 11:39:31.133 2005 -->
<pre>Forgot to say that FCAL00 and 01 is included since the silicon inclusion. It was in the middle of the problems and I didn't realised that it is normally automatically included. There was a luminosity around 48e30 (we were told to include it at luminosities around 40e30 but as the number was just an estimation and we got no problems we just run with it. In that way we don't need to end the run to include it later on).</pre>
<!-- Author: Xavier :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196351' target=_top>196351</a>) -->



<!-- Date: Sat Apr 9 11:44:23.742 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196347">196347</a>
RUNSTATUS: <PRE>Marked Bad, explanation:
SVX Silicon had tripped, part of B0W2L3.  Expert is looking into the problem.
</pre>
<!-- Author: cdfscico -->


<!-- Date: Sat Apr 9 11:49:59.332 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196347">196347</a>
RUNSTATUS: <PRE>Marked Bad, explanation:
ISL 
L00 off due to already mentioned Silicon trip
</pre>
<!-- Author: cdfscico -->



<!-- Date: Sat Apr 9 11:50:14.984 2005 -->
<table border=0><td><tr><td bgcolor=#e0e030>The silicon cooling/interlocks pager is currently out of paging reach.  If there are any problems, please call my cell phone: 630-452-4489.  Will make another entry when pager is working again (this afternoon).</td></tr></td></table> 
 
  
<!-- Author: Eva H./Mircea C. -->

<!-- Date: Sat Apr 9 11:57:39.042 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196344">196344</a>
RUNSTATUS: <PRE>Marked Bad, explanation:
CSMX hot channels in CES strip and wire
</pre>
<!-- Author: cdfscico -->



<!-- Date: Sat Apr 9 12:23:47.257 2005 -->
Reformatter Errors are increasing, again seems to be clustered around one node: b0l3271  
<!-- Author: Marcel -->


<!-- Date: Sat Apr 9 12:35:22.559 2005 -->
<pre>CO Update doesn't find any root files for DAQErrMon, even after trying again later (as suggested in the error message). No recent root files for DAQErrMon in either  
/data1/consumer/results/DAQErrMon/ or /cdf/cdf-cf-data1/cdf_cf/results/DAQErrMon/. Root files for all the other consumer processes are found ok. 
 
Restarting DAQErrMon fixed the problem.</pre>
<!-- Author: J. Beringer :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196351' target=_top>196351</a>) -->


<!-- Date: Sat Apr 9 12:47:16.523 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-5986&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-5986  width='300' align='top'></a></td><td> </td></tr></table>
<!-- Author: Pedro -->


<!-- Date: Sat Apr 9 12:47:54.423 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5984&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5984' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5985&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5985' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5986&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5986' width='300'></a></center>
<!-- Author: Pedro -->
<!-- Comment: Sat Apr  9 12;48;34 comment by...Pedro -->
<pre>Hourly plots 1130-1245 ok.</pre>

<!-- Date: Sat Apr 9 12:49:16.916 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196351">196351</a>
 Terminated at 2005.04.09 12:48:57
<!-- Author: RunControl -->


<!-- Date: Sat Apr 9 12:49:17.625 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196351">196351</a>
TERMINATE: Problems with ISL
<!-- Author: Xavier x2080 -->



<!-- Date: Sat Apr 9 12:50:24.770 2005 -->
<pre>ISL problems still not fixed. The problems are in IB5W2 (L0-L3). We end the run to try to reset the VRB (b0svx07).</pre>
<!-- Author: Xavier :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196351' target=_top>196351</a>) -->
<!-- Comment: Sat Apr  9 13;05;21 comment by...Grumpy -->
that should be one VRB in b0svx07, it has 8 of them :-)
<!-- Comment: Sat Apr  9 13;37;13 comment by...Sebastian -->
Ok, the short story is that IB5W2L0-3 is back, long
story <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=243&frame=2&anchor=11:45:45'>
here</a>. Thanks to the crew.
<!-- Date: Sat Apr 9 12:53:39.545 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196352">196352</a>
 Activated at 2005.04.09 12:52:13
<!-- Author: RunControl -->


<!-- Date: Sat Apr 9 12:53:40.234 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196352">196352</a>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_01 [2,618,559] 
<!-- Author: Xavier x2080 -->



<!-- Date: Sat Apr 9 12:54:50.273 2005 -->
<pre>CER_SVXMON_HRR. It says that pipeline is out of synch in 152 silicon readout chips (probably due to the reset done). HRR fixed.</pre>
<!-- Author: Xavier :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196352' target=_top>196352</a>) -->



<!-- Date: Sat Apr 9 13:10:59.765 2005 -->
<pre>DQMon alarm again for ISL (lots of cells B5/W2). Talking with the expert he says that this is normal. If one looks at b0dap56 one can find what is fake and what is wrong. In this case DQMon is working too well for Si. It needs to be tuned. Silicon people will discuss and they will give a clear input to DQMon experts.</pre>
<!-- Author: Xavier :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196352' target=_top>196352</a>) -->
<!-- Comment: Sat Apr  9 13;13;25 comment by...Marcel -->
In this case (sorry Mario..) it is a DQMON screw up, because, it is still lookig at the file from run 196351...
<!-- Comment: Sat Apr  9 13;32;14 comment by...Xavier -->
Now the ISL is fine and the run number is correct. It has been automatically. I regret not having taken a look whether the run was the right one previously. If it wasn't something funny is going on because it makes no much sense for me that it has read the previous run, given an alarm and changed the run after a while automatically? Let's see if the expert can enlighten us...
<!-- Comment: Sat Apr  9 13;48;59 comment by...Oriol (DQMon) -->
<br>
It is possible that DQMon was reading the output of SVXMon of the previous run. Probably SVXMon takes a certain time until it updates its output, the file that DQMon reads. I assumed (wrongly) that this happened near instantaneously as in the case of YMon and other monitors.<br>
I will introduce a check in DQMon to be sure that it is reading the  output of the current run in the RC, and will leave the Silicon buttons in DQMon in blue (not enough statistics) until the SVXMon output file is updated.
<!-- Comment: Sat Apr  9 13;52;23 comment by...Marcel (silicon) -->
You are right, since we have some ladders that take their time at the beginning, we decided to write out the file only after 1000 events. So your proposal is exactly the right way to do it :-)

<!-- Date: Sat Apr 9 13:11:30.001 2005 -->
L2 decision TO. HRRed.
<!-- Author: Xavier :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196352' target=_top>196352</a>) -->
<!-- Comment: Sat Apr  9 13;24;58 comment by...Xavier -->
Another one.
<!-- Comment: Sat Apr  9 15;20;32 comment by...Xavier -->
Another one.
<!-- Comment: Sat Apr  9 15;41;01 comment by...Xavier -->
Another one.
<!-- Date: Sat Apr 9 13:23:57.699 2005 -->
<pre>Done TO for b0pcal10. Shepherd was successful. The error said: 
 
(MLE) b0wcal01:Messenger:Apr 9, 2005 1:18:18 PM->Event 312410: Bunchcounters in slot 5 (BC=112) and slot 19 (BC=6) 
 disagree (L2B=2)  
L2B  5:   6  59 112  15 
L2B 19:   6  59   6  15 
EHx: 00090012 00150012 002A0012 00370012 
(MLE) b0wcal01:Messenger:Apr 9, 2005 1:18:18 PM->Runtime Error 1, Event 312410: Bunch counter mismatch, mismatch c 
ount = 1 
(MLE) b0dap84.fnal.gov:ConsumerErrorRe:Apr 9, 2005 1:18:20 PM->DAQErrMon:DataSyncError: Error 2 Event 312410 Run 1 
96352 TMPD BC mismatchSlotMask 0x00080000 Block 16 Address 109  
(MLE) b0l3pcom1.fnal.gov:main:Apr 9, 2005 1:19:03 PM->Host b0eb16.fnal.gov, task tRec_0 
SCPU-P0-E-VrbReadFailed: Error reading VRB in slot 12 for event 324934. 
The VRB had no event. 
CT: 2005.04.09 13:19:05 
19'5" 1 crate/s: b0pcal10(44),  Done TO. [RXPT] 
(MLE) b0dap73.fnal.gov:Thread-2340:Apr 9, 2005 1:19:05 PM->Requested Halt-Recover-Run issued (errmon) 
(MLE) b0dap73.fnal.gov:Thread-2340:Apr 9, 2005 1:19:05 PM->Done Timeout: PCAL_10  
(MLE) b0svx00:Messenger:Apr 9, 2005 1:19:06 PM->Silicon Resonator: Sl 12 Ch 4 -> e044: Bytes: 00094 
 
(MLE) b0svx03:Messenger:Apr 9, 2005 1:19:06 PM->Silicon Resonator: Sl 10 Ch 9 -> e334: Bytes: 00046 
 
(MLE) b0svx04:Messenger:Apr 9, 2005 1:19:06 PM->Silicon Resonator: Sl 18 Ch 9 -> e494: Bytes: 00104 
 
(MLE) b0svx02:Messenger:Apr 9, 2005 1:19:06 PM->Silicon Resonator: Sl 20 Ch 9 -> e2b4: Bytes: 00130 
</pre>
<!-- Author: Xavier :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196352' target=_top>196352</a>) -->




<!-- Date: Sat Apr 9 13:34:25.581 2005 -->
I was writing the comment on DQMon and when I press with the mouse the Submit button, "plops!" the x-server has been rebooted!!! I have already restarted the things: DQMon and mozilla in the computers. Some other people reported similar things. Strange... 
<!-- Author: Xavier :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196352' target=_top>196352</a>) -->
<!-- Comment: Sat Apr  9 13;38;25 comment by...Veronica -->
During yestersday's shift it also happened...




<!-- Date: Sat Apr 9 14:15:03.635 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5987&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5987' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5988&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5988' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5990&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5990' width='300'></a></center>
<!-- Author: Pedro -->
<!-- Comment: Sat Apr  9 14;15;44 comment by...Pedro -->
<pre>Hourly plots 1245-1400 ok.</pre>


<!-- Date: Sat Apr 9 14:24:29.664 2005 -->
<pre>Another strange thing: the machine claimed to be issuing an inhibit for CSP: iFix   Central Muon upgrade and Extension Scintillator from IFix. 
The run has not been halted and nothing reported from Monitoring point of view except a short alarm (not audible because it has restored right away) in GAM (which it seems it has nothing to do with the systems reported). 
 
Anyone has any idea? I repeat that the run has never been halted. All is continuing working fine.</pre>
<!-- Author: Xavier :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196352' target=_top>196352</a>) -->
<!-- Comment: Sat Apr  9 16;17;46 comment by...Xiaojain -->
I already saw such cases several times in the last a few weeks. I guess that it is just a transient problem that did not trigger the inhibitor at all, but causing the message. 


<!-- Date: Sat Apr 9 15:16:41.106 2005 -->
Got a L2 decision TO and when going to Halt it stucks due to b0fib05. Shepherding it was successful. Also, I want to point out that we have some messages coming from the errlog that don't issue an output. These are some DAQErr. I noticed that they have been more often (5-7 minutes) just before arriving to this situation. I put the last ones here: 
 
(MLE) b0dap84.fnal.gov:ConsumerErrorRe:Apr 9, 2005 2:43:17 PM->DAQErrMon:DataSyncError: Error 4 Event 1680366 Run 196352 T 
MPD BC mismatchSlotMask 0x00100000 Block 20 Address 113 
(MLE) b0xft00:Messenger:Apr 9, 2005 2:48:30 PM->Event 1763613: Bunchcounters in slot 13 (BC=121) and slot 8 (BC=0) disagre 
e (L2B=0) 
(MLE) b0xft00:Messenger:Apr 9, 2005 2:48:30 PM->Runtime Error 2, Event 1763613: Bunch counter mismatch, mismatch count = 1 
(MLE) b0xft00:Messenger:Apr 9, 2005 2:54:29 PM->Event 1858064: Bunchcounters in slot 13 (BC=139) and slot 6 (BC=138) disag 
ree (L2B=1) 
(MLE) b0xft00:Messenger:Apr 9, 2005 2:54:29 PM->Runtime Error 3, Event 1858064: Bunch counter mismatch, mismatch count = 1 
(MLE) b0dap84.fnal.gov:ConsumerErrorRe:Apr 9, 2005 3:01:39 PM->DAQErrMon:DataSyncError: Error 5 Event 1970556 Run 196352 T 
MPD TL2D L2Buffer mismatch GlobalL2Buffer 1 L2Buffer 0 
(MLE) b0dap73.fnal.gov:Thread-2340:Apr 9, 2005 3:05:25 PM->  Level 2 Decision Timeout 
 
(MLE) b0dap73.fnal.gov:Thread-2340:Apr 9, 2005 3:05:25 PM->Requested Halt-Recover-Run issued (errmon) 
(MLE) b0svx00:Messenger:Apr 9, 2005 3:05:26 PM->Silicon Resonator: Sl 18 Ch 7 -> e092: Bytes: 00154 
 
(MLE) b0svx03:Messenger:Apr 9, 2005 3:05:26 PM->Silicon Resonator: Sl 10 Ch 9 -> e334: Bytes: 00050 
 
 
<!-- Author: Xavier :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196352' target=_top>196352</a>) -->
<!-- Comment: Sat Apr  9 15;20;01 comment by...Xavier -->
MORE READABLE ENTRY BELOW!
<!-- Date: Sat Apr 9 15:17:28.482 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5991&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5991' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5992&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5992' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-5993&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-5993' width='300'></a></center>
<!-- Author: Pedro -->
<!-- Comment: Sat Apr  9 15;17;52 comment by...Pedro -->
<pre>Hourly plots 1400-1515 ok.</pre>


<!-- Date: Sat Apr 9 15:18:53.436 2005 -->
<pre>Ups!!! Sorry! This may be more readable... 
 
Got a L2 decision TO and when going to Halt it stucks due to b0fib05. Shepherding it was successful. Also, I want to point out that we have some messages coming from the errlog that don't issue an output. These are some DAQErr. I noticed that they have been more often (5-7 minutes) just before arriving to this situation. I put the last ones here: 
 
 (MLE) b0dap84.fnal.gov:ConsumerErrorRe:Apr 9, 2005 2:43:17 PM-&gt;DAQErrMon:DataSyncError: Error 4 Event 1680366 Run 196352 T MPD BC mismatchSlotMask 0x00100000 Block 20 Address 113  
 
(MLE) b0xft00:Messenger:Apr 9, 2005 2:48:30 PM-&gt;Event 1763613: Bunchcounters in slot 13 (BC=121) and slot 8 (BC=0) disagre e (L2B=0)  
 
(MLE) b0xft00:Messenger:Apr 9, 2005 2:48:30 PM-&gt;Runtime Error 2, Event 1763613: Bunch counter mismatch, mismatch count = 1  
 
(MLE) b0xft00:Messenger:Apr 9, 2005 2:54:29 PM-&gt;Event 1858064: Bunchcounters in slot 13 (BC=139) and slot 6 (BC=138) disag ree (L2B=1)  
 
(MLE) b0xft00:Messenger:Apr 9, 2005 2:54:29 PM-&gt;Runtime Error 3, Event 1858064: Bunch counter mismatch, mismatch count = 1  
 
(MLE) b0dap84.fnal.gov:ConsumerErrorRe:Apr 9, 2005 3:01:39 PM-&gt;DAQErrMon:DataSyncError: Error 5 Event 1970556 Run 196352 T MPD TL2D L2Buffer mismatch GlobalL2Buffer 1 L2Buffer 0  
 
(MLE) b0dap73.fnal.gov:Thread-2340:Apr 9, 2005 3:05:25 PM-&gt; Level 2 Decision Timeout  
 
(MLE) b0dap73.fnal.gov:Thread-2340:Apr 9, 2005 3:05:25 PM-&gt;Requested Halt-Recover-Run issued (errmon)  
(MLE) b0svx00:Messenger:Apr 9, 2005 3:05:26 PM-&gt;Silicon Resonator: Sl 18 Ch 7 -&gt; e092: Bytes: 00154  
(MLE) b0svx03:Messenger:Apr 9, 2005 3:05:26 PM-&gt;Silicon Resonator: Sl 10 Ch 9 -&gt; e334: Bytes: 00050</pre>
<!-- Author: Xavier :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196352' target=_top>196352</a>) -->





<!-- Date: Sat Apr 9 15:45:29.219 2005 -->
<table><tr><td><a href=/elog/notebooks/2005/Sat_Apr__9_2005_Day.154504.ps align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-5995  align='top'></a></td><td>WHA PMT at wedge 22 L and eta 8 is blank. Likely this is just due to low statistics, but it was also the case for the previous run, so perhaps an expert should check.</td></tr></table>
<!-- Author: J. Beringer -->

<!-- Date: Sat Apr 9 15:55:54.547 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196346>
196346</A> 
x2FEFA</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_01 [2,618,559]</TD>                                       
<TD ALIGN=CENTER>06:45:15</TD>                                          
<TD ALIGN=CENTER>08:38:58</TD>                                          
<TD ALIGN=CENTER>01:26:49</TD>                                          
<TD ALIGN=RIGHT>    104,402,688</TD>                                    
<TD ALIGN=RIGHT>     1,438,964</TD>                                     
<TD ALIGN=RIGHT>     371,228</TD>                                       
<TD ALIGN=RIGHT>  317.429</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196347>
196347</A> 
x2FEFB</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_01 [2,618,559]</TD>                                       
<TD ALIGN=CENTER>08:44:39</TD>                                          
<TD ALIGN=CENTER>09:36:54</TD>                                          
<TD ALIGN=CENTER>00:19:50</TD>                                          
<TD ALIGN=RIGHT>     21,836,019</TD>                                    
<TD ALIGN=RIGHT>       167,622</TD>                                     
<TD ALIGN=RIGHT>      47,917</TD>                                       
<TD ALIGN=RIGHT>   63.368</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196351>
196351</A> 
x2FEFF</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_01 [2,618,559]</TD>                                       
<TD ALIGN=CENTER>10:07:58</TD>                                          
<TD ALIGN=CENTER>12:48:57</TD>                                          
<TD ALIGN=CENTER>02:33:24</TD>                                          
<TD ALIGN=RIGHT>    211,476,061</TD>                                    
<TD ALIGN=RIGHT>     2,611,107</TD>                                     
<TD ALIGN=RIGHT>     632,251</TD>                                       
<TD ALIGN=RIGHT>  399.717</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196352>
196352</A> 
x2FF00</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_3_01 [2,618,559]</TD>                                       
<TD ALIGN=CENTER>12:52:13</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>02:53:08</TD>                                          
<TD ALIGN=RIGHT>    247,424,428</TD>                                    
<TD ALIGN=RIGHT>     2,733,587</TD>                                     
<TD ALIGN=RIGHT>     706,743</TD>                                       
<TD ALIGN=RIGHT>  367.554</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>15:55:04</TD>                                                       
<TD ALIGN=CENTER>07:13:12</TD>                                          
<TD ALIGN=RIGHT>     585,139,196</TD>                                   
<TD ALIGN=RIGHT>      6,951,280</TD>                                    
<TD ALIGN=RIGHT>     1,758,139</TD>                                     
<TD ALIGN=RIGHT> 1148.069</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sat Apr 9 16:01:37.088 2005 -->
<font color=red>Shift Summary: </font><pre>Trips, alarums and excursions mar an otherwise reasonable shift 
 
Waiting to hear that hardware repair in Feynman center is complete,so that we can ship events to the DB.   
 
It would be useful to clarify (and make easily available) what is  
the buffer capacity of the CSL (in hours) as a function of luminosity. 
 
In any case, it would have been better for this hardware repair to begin just after a store is dropped, rather than just as 
a store is started. 
 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   196346, 196347, 196351, 196352
Delivered Luminosity   1194.4  
Acquired Luminosity    863.6  
Efficiency             72.3
   
</td></tr></table></pre>
<!-- Author: virgil barnes -->


<!-- Date: Sat Apr 9 16:07:48.390 2005 -->
<TABLE bgcolor=#cdc9a5> 
<TR><TH>Date</th><th>Time</th><th>BLM</th><th colspan=2>Dose</th></tr> 
<tr><td>2005.04.09</td><td>16:04:35</td><td>W Inner BLM</td><td>387.00</td><td>RADS</td></tr> 
<tr><td>2005.04.09</td><td>16:04:35</td><td>W Outer BLM</td><td>105.88</td><td>RADS</td></tr> 
<tr><td>2005.04.09</td><td>16:04:35</td><td>E Inner BLM</td><td>172.99</td><td>RADS</td></tr> 
<tr><td>2005.04.09</td><td>16:04:35</td><td>E Outer BLM</td><td>377.47</td><td>RADS</td></tr> 
</TABLE> 
<strong>Integrated dosage</strong>
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=19635' target=_top>19635</a>) -->