<!-- Title: Owl shift. Sat Jan 24, 2004 -->
<!-- SciCo: P.Wilson,B.Wicklund -->
<!-- DAQAce: S.Cabrera -->
<!-- MonAce: T.Schwarz -->
<!-- CO: B.King,Y.Miyazaki -->
<!-- OpManager: M.Convery -->
<!-- Notes: -->
<pre>Shot setup for store 3189 in progress</pre>


<!-- Date: Sat Jan 24 00:13:53.656 2004 -->
<pre>There was a network glitch around midnight. 
 
Saw the following complaint from b0dau30. 
 
Jan 23 23:55:34 4A:b0dau30 unix: WARNING: ef0: link fail - check ethernet cable 
 
We have seen this problem a couple of times - will have to keep 
a watch on it.</pre>
<!-- Author: Frank Chlebana -->

<!-- Date: Sat Jan 24 00:21:04.702 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178385">178385</A>
TERMINATE: Run AAA_SHOTSETUP terminated as requested by ops manager to test DAQ
<!-- Author: Susana x2080 -->


<!-- Date: Sat Jan 24 00:27:58.030 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178386">178386</A>
ACTIVATE: AAA_SHOTSETUP ended
<!-- Author: Susana x2080 -->


<!-- Date: Sat Jan 24 00:27:59.959 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178386">178386</A>
TERMINATE: AAA_SHOTSETUP ended.
<!-- Author: Susana x2080 -->


<!-- Date: Sat Jan 24 00:51:15.164 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178387">178387</A>
TERMINATE: Reset to change to AAA_NONSILICON configuration
<!-- Author: Susana x2080 -->


<!-- Date: Sat Jan 24 00:51:25.629 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178387">178387</A>
TERMINATE: Reset to change to AAA_NOSILICON
<!-- Author: Susana x2080 -->



<!-- Date: Sat Jan 24 00:52:23.513 2004 -->
LOSTP is 25kHz.  We are waiting for MCR to tune.
<!-- Author: convery -->


<!-- Date: Sat Jan 24 00:52:41.484 2004 -->
<pre>During the shot-setup run 178385 the L3 Local Client Controller 
was in yellow color, "idle state", and in the Level 3 Display  
showed the column of processors of the subfarm 15 in orange ("old"). The run was ended. After partinioning and configuring  
for the next shot setup test run, the L3 was ready. 
Now we are ready to activate a AAA_CURRENT. 
.</pre>
<!-- Author: Susana. -->
<!-- Comment: Sat Jan 24 03;25;36 comment by...Peter -->
Actually, we think this was subfarm 14 which had problems later in the shift.
<!-- Date: Sat Jan 24 00:56:53.478 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178388">178388</A>
 Activated at 2004.01.24 00:56:18
<!-- Author: RunControl -->


<!-- Date: Sat Jan 24 00:56:57.165 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178388">178388</A>
ACTIVATE: Starting run AAA_NOSILICON 
<!-- Author: Susana x2080 -->



<!-- Date: Sat Jan 24 00:58:44.309 2004 -->
CMX Warning Here and there.  (yellow).  Previous shifts report CMX trip corresponding to high proton losses.  Proton Losses currently between 25 and 30 kHz.  Wait to see if stabilizes when proton losses lower.
<!-- Author: Tom -->


<!-- Date: Sat Jan 24 01:03:10.972 2004 -->
<pre>Summary of the evening shift 23 Jan 2004 
======================================== 
 
 
-------------------------------------------------------- 
Jan.23, 2004	Store         Time	p	pbar 
-------------------------------------------------------- 
Previous store dumped.        21:40     
 
proton injection started      22:20     Call Ops Manager 
pbars loading    started      23:10     Page SVX Primary 
pbars loading    finished     00:14       E10      E10 
Scraping end                   
	 Lumi = E30                 E30 
MCR called us at              00:34 
CLC HV up                     00:35    (call MCR) 
         C:B0ILUM             46.80  E30 
         C:LOSTP              20K  
         C:LOSTPB              1K   
         TevPR                7768 10**9   
         TevPB                 935 10**9 
H.V. (except Si) ON           00:40 
H.V. Silicon     OFF          00:59 AAA_NOSILICON   
Physics RUN# start            178388 
	 Lumi = E30           43.96 E30 
-------------------------------------------------------- 
 
 
 
 
</pre>
<!-- Author: Wilson -->


<!-- Date: Sat Jan 24 01:13:37.172 2004 -->
B0COT03 in RED in VxWorks monitor. B0COT03 was active and green  
in the LocalClientController though. Manual HRR. Window 
message appeared telling us to shepherd B0COT03. 
B0COT03 was reset and shepherd, and recovered with sucess.
<!-- Author: Susana. -->


<!-- Date: Sat Jan 24 01:28:50.492 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1818&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1818' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1817&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1817' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1816&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1816' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1815&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1815' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1814&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1814' width='300'></a></center>
<!-- Author: tom -->
<!-- Comment: Sat Jan 24 01;29;56 comment by...Tom -->
<pre>hourly plots through shot setup.  Luminosity Turns On At ~ 00:26:40

Also put rad totals after shot. maybe of interest to someone.</pre>

<!-- Date: Sat Jan 24 01:33:35.305 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178388">178388</A>
 Terminated at 2004.01.24 01:32:38
<!-- Author: RunControl -->


<!-- Date: Sat Jan 24 01:33:36.943 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178388">178388</A>
TERMINATE: Run AAA_NOSILICON ended. Silicon is going to be integrated.
<!-- Author: Susana x2080 -->


<!-- Date: Sat Jan 24 01:40:00.702 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178389">178389</A>
 Activated at 2004.01.24 01:39:36
<!-- Author: RunControl -->


<!-- Date: Sat Jan 24 01:40:40.427 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178389">178389</A>
ACTIVATE: Staring AAA_CURRENT.
<!-- Author: Susana x2080 -->



<!-- Date: Sat Jan 24 01:41:56.834 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1819&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1819' width='300'></a></center>
<!-- Author: Rainer -->
<!-- Comment: Sat Jan 24 01;48;57 comment by...Rainer -->
Following Rick Tesarek's analysis <a href='http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=copyentry&page=61&time=07:48:15.534'> here </a>, I had a look at the position of  the E0 UV collimator. It indeed looks like that moving it -320 mils was successful again earlier, but for some reason MCR elected to not run that collimator so far in any more. One notices a correlation of the collimator position with the width of B0PBSM. 
<!-- Date: Sat Jan 24 01:43:53.197 2004 -->
TeVMon Tuning finally gets 19K Lostp, SVX HV up, 
start physics run.
<!-- Author: Scico :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178389' target=_top>178389</a>) -->



<!-- Date: Sat Jan 24 02:01:57.646 2004 -->
<pre>Total deadtime average for this run:    3.93 % 
L1 10352.6 Hz 
L2   257.1 Hz 
L3    54.6 Hz</pre>
<!-- Author: Susana -->


<!-- Date: Sat Jan 24 02:15:15.393 2004 -->
Automatic HRR was issued by RC due to TimeOut for crate b0pcal11.
<!-- Author: Susana.. -->


<!-- Date: Sat Jan 24 02:22:05.045 2004 -->
<pre>Several L2 Decision Timeout errors. Run was halted and succesfully recovered after typing in b0dap73: ~reboot_alpha.</pre>
<!-- Author: Susana. -->
<!-- Comment: Sat Jan 24 14;55;56 comment by...Tom -->
Wow, it actually worked...

<!-- Date: Sat Jan 24 02:45:42.493 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1823&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1823' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1822&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1822' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1821&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1821' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1820&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1820' width='300'></a></center>
<!-- Author: tom -->
<!-- Comment: Sat Jan 24 02;46;04 comment by...tom -->
<pre>hourly plots</pre>

<!-- Comment: Sat Jan 24 02;51;59 comment by...tom -->
<pre>not sure why proton abort gap losses summed over halo counters is so high.  last shift shows same thing for different shot.  i've been informed this isn't an issue to worry about.  unsure how to interpret this plot then.  what should the aces be looking for in terms of this?</pre>


<!-- Date: Sat Jan 24 02:53:58.034 2004 -->
<pre>All the processors of the L3 sub-farm number 14 were in 
"old" state, (orange-yellow), that means: "node monitoring dead." 
The L3 expert was paged and the L3 Display was reset and re-started. 
After that, the subfarm 14 is in grey, no color. But by looking at the convertors, this subfarm is processing data. Also there are events in the output node 14. 
 
As recommended by Nuno we will not stop the run for this reason. If we have to stop the run for other different reason, then we will clean up the L3 farms to recover the monitoring control 
on subfarm 14. 
 
</pre>
<!-- Author: Susana. -->


<!-- Date: Sat Jan 24 02:55:36.139 2004 -->
<pre>Total deadtime: 6.91 
L1  11,291.7 
L2  257,4  
L3  55.1</pre>
<!-- Author: Susana. -->

<!-- Date: Sat Jan 24 03:15:21.206 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178389">178389</A>
 Terminated at 2004.01.24 03:13:42
<!-- Author: RunControl -->


<!-- Date: Sat Jan 24 03:15:22.209 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178389">178389</A>
TERMINATE: lost control of Run Control.  Had to abort run.  resetting and starting up.
<!-- Author: Susana x2080 -->



<!-- Date: Sat Jan 24 03:44:21.561 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178390">178390</A>
 Activated at 2004.01.24 03:43:34
<!-- Author: RunControl -->


<!-- Date: Sat Jan 24 03:47:32.867 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178390">178390</A>
ACTIVATE: Starting a new run after L3 farms and EVB clanup.
<!-- Author: Susana x2080 -->



<!-- Date: Sat Jan 24 03:58:13.170 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1827&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1827' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1826&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1826' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1825&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1825' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1824&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1824' width='300'></a></center>
<!-- Author: tom -->
<!-- Comment: Sat Jan 24 03;58;43 comment by...tom -->
<pre>hourly plots</pre>


<!-- Date: Sat Jan 24 03:58:57.201 2004 -->
<pre>Detail description of what has happened: 
 
1) L3 went RED in the window "Replies and Acknowlegements". 
2) Run control performed an automatic HRR. 
3) The HRR process was stuck in the HALT during more than 3 minutes. 
4) The run 178390 was ABORTed from RC. Reset. 
5) We called L3 expert. 
6) We cleaned up the L3 farms. RC in START status. 
L3 in RED, EVB in yellow after partitioning. 
7) A message popped up from RC saying we need to clean up the EVB also. 
8) We cleaned up the EVB. 
9) We cleaned up the L3 (second time) 
10) Partition and Config succesfull. 
 
 
L3 processors: 022,221,229 are not working 
 
 
 
</pre>
<!-- Author:    Susana. -->


<!-- Date: Sat Jan 24 04:22:51.552 2004 -->
<pre>From the error file /data1/cdfdaq/errmon/log/errorfile178389.log 
I found the error that looks like the cause of having stopped the automatic HRR process: 
 
 
(MLE) b0l3pcom2.fnal.gov:main:3:03:09 AM-&gt;Error on L3 node b0l3pcom2.fnal.gov (partition 1) Sat Jan 24 03:03:09 2004 
l3proxy in Halt (L3ProxyPartition.cc:1877) 
@L3_TRANS_FAIL - setup failed:      b0l3229(Unrecognized response type for request) 
</pre>
<!-- Author: Susana. -->


<!-- Date: Sat Jan 24 04:52:25.991 2004 -->
<pre>b0cot09:Messenger:4:41:31 AM-&gt;Runtime Error 1, Event 830030: TDC in slot 7 not ready for readout 
b0cot09:Messenger:4:41:31 AM-&gt;Runtime Error 2, Event 830030: Timeout on global TDC Done --&gt;   
 
 Additional Information:  
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!!  
 - Timeout on global TDC Done       
 - More &gt; 1 crate in Done TO.       
 - Auto-SHEPHERD WILL NOT BE ISSUED.   
 
 
Ace Action: Manual HRR, run recovered. 
 
(RC)   4:43:24 Run -&gt; ACTIVE CT: 2004.01.24 04:44:32 
44'32" b0dap57.fnal.gov:resmgr.ReplyMonitor 8:4:44:33 AM-&gt; Dead Time is High  --&gt;  Additional Information:  
 
 Attention!!!. FERML_HIGH_DEADTIME !!!.  
 - Trigger/DAQ Deadtime is too high. 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
 
</pre>
<!-- Author: Susana. -->
<!-- Comment: Sat Jan 24 04;53;27 comment by...Susana. -->
<pre>We checked the deadtime and looks fine, &lt; 5 %</pre>


<!-- Date: Sat Jan 24 04:56:13.384 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1831&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1831' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1830&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1830' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1829&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1829' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1828&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1828' width='300'></a></center>
<!-- Author: tom -->
<!-- Comment: Sat Jan 24 04;56;56 comment by...tom -->
<pre>hourly plots.  note single spike in pbar abort gap loss rate.</pre>

<!-- Comment: Sat Jan 24 04;58;41 comment by...tom -->
spike is in sum halo counter.  not supposed to be a big deal.


<!-- Date: Sat Jan 24 05:01:41.034 2004 -->
b0svx01:Messenger:4:58:31 AM->Silicon Timeout:!DONE- Slots: 12:e140 
b0svx06:Messenger:4:58:31 AM->Silicon Timeout:BUSY- Slots:  08:fa00 10:fa20 12:fa40 16:f800 18:f820 20:f840 
 
Ace action: manual HRR and AUTOMATIC HRRs enabled from EH Options Menu
<!-- Author: Susana. -->


<!-- Date: Sat Jan 24 05:20:56.349 2004 -->
<pre> Attention!!!. Scanner Manager SM_SCPU_TIMEOUT Error !!!.  
 We STRONGLY RECOMMEND the following steps:  
 - Try Halt-Recover-Run. 
 - If it fails, do End (Abort) -&gt; Reset -&gt; Cleanup EVB. 
 - Start run again.  
Automatic HRR successful</pre>
<!-- Author: Susana. -->


<!-- Date: Sat Jan 24 05:49:43.603 2004 -->
<pre>b0tsi02:Messenger:5:22:33 AM-&gt;BUSY timeout 
b0dap73.fnal.gov:Thread-13847:5:22:42 AM-&gt;Requested Halt-Recover-Run issued [errmon] 
b0dap73.fnal.gov:Thread-13847:5:22:42 AM-&gt;Busy Timeout: VRB_ISL_06 b0svx06:Messenger:5:22:34 AM-&gt;Silicon Timeout:BUSY- Slots:  08:fa00 10:fa20 12:fa40 16:f800 18:f820 20:f840 
b0svx02:Messenger:5:22:34 AM-&gt;Silicon Timeout:BUSY- Slots:  16:e260 
 
Automatic HRR recovered the run .</pre>
<!-- Author: Susana. -->


<!-- Date: Sat Jan 24 06:06:18.026 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1835&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1835' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1834&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1834' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1833&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1833' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1832&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1832' width='300'></a></center>
<!-- Author: tom -->
<!-- Comment: Sat Jan 24 06;06;44 comment by...tom -->
hourly plots


<!-- Date: Sat Jan 24 06:11:49.744 2004 -->
<pre> b0dap73.fnal.gov:Thread-13847:6:10:17 AM-&gt;  Level 2 Decision Timeout. Automatic HRR.</pre>
<!-- Author: Susana. -->


<!-- Date: Sat Jan 24 06:48:57.296 2004 -->
<pre>Trigger inhibit was set for SVX, ISL and LOO. 
The deadltime was 99.9 %. 
The run was HALTED, to recover from silicon trip. 
</pre>
<!-- Author: Susana -->
<!-- Comment: Sat Jan 24 14;59;33 comment by...Rainer -->
inhibit fired 6:23am.
<!-- Date: Sat Jan 24 06:50:09.271 2004 -->
<pre>The Silicon PS Gui died.  SVX/ISL/L00 all went red and tweeted at me.  Silicon PS GUI wasn't updated.  Followed Hockers' prescription to kill all java and resart the gui.  seems to be working fine now.  DAQ Ace saw trip inhibit but it didn't show up on global alarms.  Another inhibit.  Silicon Expert Paged.</pre>
<!-- Author: tom -->


<!-- Date: Sat Jan 24 06:54:26.614 2004 -->
<pre>Expert walked me through fix.  Seems fine now.  Apparently we had a loss of IFIX Communication w/GUI simultaneosly w/unknown trips in SVX & ISL.  Restarted IMON and PS GUI for silicon.  brought silicon HV to standby from HV Summary.  Checked GUI to make sure it updated silicon to standby in device status window.  THen turned HV back on for silicon.  once back we restarted the run.  Did not recieve any more inhibits.  Will paste SVXMon plots in 10 minutes.</pre>
<!-- Author: Tom -->
<!-- Comment: Sat Jan 24 16;12;28 comment by...Andy H. -->
I suspect that what happened was a CAEN crate went crazy and
powered off all its wedges, which confused and crashed the GUI.
When you restarted the GUI (thus re-establishing communication
to iFIX), iFIX saw that some wedges were off and pulled the trigger inhibit.  This is just my guess --- not easy to diagnose this from Florida (ha-ha).
<!-- Comment: Sat Jan 24 18;50;55 comment by...Rainer -->
At least part of this story is reminiscent of a 
network
<a href=
'/cgi/elog/elog.pl?nb=2004&action=view&page=70&frame=2&anchor=00:13:53.656'>
glitch </a> which took out the PSGui earlier, without any crate acting up - frank can you check whether we had another one reported by b0dau30 around 6:15am today ? Thanks.
<!-- Date: Sat Jan 24 06:59:37.072 2004 -->
<pre>The receiver display of events coming into CSL shows  
event size and count  and activity rate all zeros 
for client name b0l3u03d</pre>
<!-- Author: Susana. -->


<!-- Date: Sat Jan 24 06:59:55.360 2004 -->
<pre>Noticed on a number of machines (only windows PC's) a popup warning like the following is coming up often: 
 
Error Number: 1914 (0000077a) 
CDF_S1.RAMP.F_CV: Connection Not established with node. 
 
where CDF_S1.RAMP.F_CV oscillates between a few different things.  It doesn't seem to effect any of the systems.   
</pre>
<!-- Author: Tom -->


<!-- Date: Sat Jan 24 07:10:56.230 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1839&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1839' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1838&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1838' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1837&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1837' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1836&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1836' width='300'></a></center>
<!-- Author: tom -->
<!-- Comment: Sat Jan 24 07;11;19 comment by...tom -->
<pre>hourly plots</pre>


<!-- Date: Sat Jan 24 07:12:22.600 2004 -->
<pre> b0dap73.fnal.gov:Thread-13847:7:11:12 AM-&gt;FrontEnd Crate Error Condition from COT_10. 
Automatic HRR.</pre>
<!-- Author: Susana. -->


<!-- Date: Sat Jan 24 07:13:07.487 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1840&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1840  width='300' align='top'></a></td><td> SVXMon histograms for expert checking...</td></tr></table>
<!-- Author: Barry King -->


<!-- Date: Sat Jan 24 07:15:22.284 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1841&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1841  width='300' align='top'></a></td><td>SVXMon histograms for expert checking...</td></tr></table>
<!-- Author: Barry King -->


<!-- Date: Sat Jan 24 07:17:28.474 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1842&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1842  width='300' align='top'></a></td><td>SVXMon histograms for expert checking...</td></tr></table>
<!-- Author: Barry King -->


<!-- Date: Sat Jan 24 07:19:43.120 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1843&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1843  width='300' align='top'></a></td><td>SVXMon histograms for expert checking...</td></tr></table>
<!-- Author: Barry King -->


<!-- Date: Sat Jan 24 07:26:28.782 2004 -->
<pre>b0dap73.fnal.gov:Thread-13847:7:24:07 AM-&gt;FrontEnd Error: COT_18  
24'7" 1 crate/s: b0cot18(66),  in error.[RXPT] 
 
Automatic HRR.</pre>
<!-- Author: Susana. -->


<!-- Date: Sat Jan 24 07:49:09.033 2004 -->
HVAC Fog Horn Alarm went off again.  Looked At Temp and seems fine.  Expert knows about this monitoring problem.
<!-- Author: Tom -->


<!-- Date: Sat Jan 24 07:52:32.302 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1844&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1844  width='300' align='top'></a></td><td>Pinky on Silicon Ladder AVDD B1W1L4.  Here's current history.  unmarked and went green again.  It has went pink twice in ~last half hour.</td></tr></table>
<!-- Author: Tom -->

<!-- Date: Sat Jan 24 07:55:54.502 2004 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178388>
178388</A> 
x2B8D4</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_2_01 [3,402,419]</TD>                                       
<TD ALIGN=CENTER>00:56:18</TD>                                          
<TD ALIGN=CENTER>01:32:38</TD>                                          
<TD ALIGN=CENTER>00:33:35</TD>                                          
<TD ALIGN=RIGHT>     30,340,770</TD>                                    
<TD ALIGN=RIGHT>       455,738</TD>                                     
<TD ALIGN=RIGHT>      86,994</TD>                                       
<TD ALIGN=RIGHT>   84.459</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178389>
178389</A> 
x2B8D5</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_2_01 [3,402,419]</TD>                                       
<TD ALIGN=CENTER>01:39:36</TD>                                          
<TD ALIGN=CENTER>03:13:42</TD>                                          
<TD ALIGN=CENTER>01:16:17</TD>                                          
<TD ALIGN=RIGHT>     53,783,168</TD>                                    
<TD ALIGN=RIGHT>     1,229,396</TD>                                     
<TD ALIGN=RIGHT>     265,010</TD>                                       
<TD ALIGN=RIGHT>  165.762</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178390>
178390</A> 
x2B8D6</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_2_01 [3,402,419]</TD>                                       
<TD ALIGN=CENTER>03:43:34</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>03:37:03</TD>                                          
<TD ALIGN=RIGHT>    154,442,557</TD>                                    
<TD ALIGN=RIGHT>     3,138,273</TD>                                     
<TD ALIGN=RIGHT>     692,490</TD>                                       
<TD ALIGN=RIGHT>  359.913</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>07:55:02</TD>                                                       
<TD ALIGN=CENTER>05:26:56</TD>                                          
<TD ALIGN=RIGHT>     238,566,495</TD>                                   
<TD ALIGN=RIGHT>      4,823,407</TD>                                    
<TD ALIGN=RIGHT>     1,044,494</TD>                                     
<TD ALIGN=RIGHT>  610.134</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sat Jan 24 07:58:15.824 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1845&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1845  width='300' align='top'></a></td><td>AVDD B1W1L4 is consistenly turning back pink after being unmarked.  Paged Silicon.  Current History Shown.</td></tr></table>
<!-- Author: Tom -->


<!-- Date: Sat Jan 24 08:06:35.618 2004 -->
<font color=red>Shift Summary: </font><pre>Begin shift in shot setup for store 3189 
 
00:00 Network glitch around midnight caused problems for b0dau30.   
 
00:35 Store 3189 in: 46.8 E30 
 
00:40 HV up except Si, Losses increasing 20kHz -&gt; 25kHz 
 
00:59 Start non-Silicon run 17388 while TeV tuning to reduce losses 
 
01:40 Start run 17389 w/Si B0LOSTP down to about 20kHz 
 
00:52 and 02:53 all processors or L3 sub-farm 14 were in "old" state.  In  
      first case we ended and restarted run and the condition cleared. 
      In the second case we paged L3.  It appears that processing is working 
      (based on convertor and ouput node info) and thus monitoring is suspect. 
      We decide to wait until next end run to clean up L3 monitoring. 
 
      Occasional Done TOs and L2 TOs.  One L2 TO required new alpha reboot 
      proceedure which was executed successfully. 
       
03:15 Get done timeout from B0COT18.  During auto-HRR, RC hangs in HALT state 
      on what looks like a communication problem with L3.  L3 is red in  
      Replies and Acknowledges.  End run using ABORT then RESET.  Called 
      L3.  Cleaned up L3 Farms, then EVB, then L3 again.  
 
03:45 Start 178390 after cleanup 
 
06:48 RC message indicates Si trip.  About one minute leter get alarms from  
      SVX/ISL/L00 and find that Silicon PS Gui has died.  Restarted GUI  
      and brought HV up.  Restart run and immediately get INHIBIT from SVX 
      and ISL HV.  Had to restart IMON and PS GUI for Silicon.  Appears to  
      have lost IFIX communication. 
 
06:59 CSL receiver display all zeros for b0l3u03d.  We are not limited by this 
      but should check during day shift. 
 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   178388, 178389, 178390
Delivered Luminosity   852.5  
Acquired Luminosity    618.4  
Efficiency             72.5
   
</td></tr></table></pre>
<!-- Author: Wilson -->


<!-- Date: Sat Jan 24 08:09:09.328 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1846&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1846  width='300' align='top'></a></td><td>downtime</td></tr></table>
<!-- Author: convery -->
<!-- Comment: Sat Jan 24 08;10;01 comment by...convery -->
The downtime labelled Runcontrol is actually L3/evb cleanup.