<!-- Title: Day shift. Tue Feb  5, 2002 -->
<!-- SciCo: Bob Wagner (ANL) -->
<!-- DAQAce: Anne-Sylvie Nicollerat -->
<!-- MonAce: Kim Giolo -->
<!-- CO: Steve Wolbers -->
<!-- OpManager: Bill Orejudos -->
<!-- Notes: -->
<br>Store 955 has been dumped and we are preparing for entry for supervised access.

<!-- Date: Tue Feb 5 08:35:42 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138436">138436</A>
 Activated at 2002.02.05 08:35:10
<!-- Author: RunControl -->



<!-- Date: Tue Feb 5 08:35:42 2002 -->
<pre>We are taking a cosmic no track run (50k events). We miss PCAL and WCAL crates. CPR HV is turned on.</pre>
<!-- Author: AS Nicollerat :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138436' target=_top>138436</a>) -->


<!-- Date: Tue Feb 5 09:06:42 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Tue_Feb__5_2002_Day.090634.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-652  align='top'></td><td>PMT Spike rate vs Run number for runs taken since the October shutdown.</a></td></tr></table>
<!-- Author: R.J. Tesarek -->

<!-- Date: Tue Feb 5 09:18:46 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138436">138436</A>
 Terminated at 2002.02.05 09:18:10
<!-- Author: RunControl -->



<!-- Date: Tue Feb 5 09:20:34 2002 -->
<pre>As the 50k events run is finished, we stop the run to give FCAL crates to Koji. He is going to need them maybe for the all day...</pre>
<!-- Author: AS. Nicollerat :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138436' target=_top>138436</a>) -->


<!-- Date: Tue Feb 5 09:34:03 2002 -->
Doing a TOF QIE calibration. 
We can't see results from DBANA
<!-- Author: AS Nicollerat :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138437' target=_top>138437</a>) -->
<!-- Comment: Tue Feb  5 10;29;32 comment by...Stefano -->
<pre>I checked the TOF calibrations and everything looks fine.
NOTE: To be able to see the last DBANA plot for 
TOFQIEPED first you must view any of the calorimeter plots, 
and also one of the previous TOFQIEPED plot.
Don't ask why ;-) .. Gervasio will update the online 
documentation asap.</pre>
<!-- Date: Tue Feb 5 09:36:06 2002 -->
<pre>Done TOF TAC Calib 
 
</pre>
<!-- Author: AS&gt; Nicollerat :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138438' target=_top>138438</a>) -->


<!-- Date: Tue Feb 5 09:39:39 2002 -->
Nat is going to power down the Xtrp crates for the day.
<!-- Author: AS Nicollerat -->


<!-- Date: Tue Feb 5 09:40:05 2002 -->
I've taken the XTRP crate to begin backplane investigations.
<!-- Author: Nathan Eddy -->


<!-- Date: Tue Feb 5 09:42:34 2002 -->
Doing a Muon calibration.
<!-- Author: AS Nicollerat :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138439' target=_top>138439</a>) -->


<!-- Date: Tue Feb 5 09:48:27 2002 -->
Carla is going to take B0L1CL04 for about 10 minutes to unmask a tower and look at it with a scope.
<!-- Author: AS Nicollerat -->


<!-- Date: Tue Feb 5 10:01:10 2002 -->
<pre>L3 farm lost heart beat. 
We clean up L3 farm to restablish proxy (Ilya suggestion) 
Now it's ok</pre>
<!-- Author: AS. Nicollerat -->


<!-- Date: Tue Feb 5 10:07:42 2002 -->
Steve Nahn took the SVX crates for at least the access.
<!-- Author: AS Nicollerat -->
<!-- Comment: Tue Feb  5 10;18;44 comment by...SCN -->
We will be doing various operations on the Fib and possibly Vrb
crates during the access, and potentially continuing through
the study periods.  Please ignore potential problems in 
vxmon and PSMon as we turn off crates and whatnot.<B>DO NOT
IGNORE COOLING OR IFIX WARNINGS</B>



<!-- Date: Tue Feb 5 10:46:33 2002 -->
<pre>Satoru took PCAL</pre>
<!-- Author: AS. Nicollerat -->

<!-- Date: Tue Feb 5 10:50:44 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138441">138441</A>
 Activated at 2002.02.05 10:50:10
<!-- Author: RunControl -->


<!-- Date: Tue Feb 5 10:52:16 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138441">138441</A>
 Terminated at 2002.02.05 10:51:17
<!-- Author: RunControl -->


<!-- Date: Tue Feb 5 10:57:28 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138442">138442</A>
 Activated at 2002.02.05 10:56:58
<!-- Author: RunControl -->



<!-- Date: Tue Feb 5 10:57:28 2002 -->
<pre>Andrew Martin is preparing an upgrade of the Trigger  
Supervisor code to enable L1Done (SVX) Timeouts. 
 
In approx 5 to 10 minutes, we'd like to reboot  
the Trigger Supervisor crate, which will reset all  
partitions. 
</pre>
<!-- Author: W.Badgett -->


<!-- Date: Tue Feb 5 11:00:28 2002 -->
To check if the Level1 worked fine (after the work of Carla), we made a cosmic no track run but without CCAL04 as it needed to be rebooted.
<!-- Author: AS Nicollerat :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138442' target=_top>138442</a>) -->

<!-- Date: Tue Feb 5 11:02:04 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138442">138442</A>
 Terminated at 2002.02.05 11:00:18
<!-- Author: RunControl -->



<!-- Date: Tue Feb 5 11:05:34 2002 -->
<pre>The run is stopped as Bill wants to do some L3 work (rebooting trigger sup. crates (tsi)) and this will affect all crates. It should last something like 10 minutes. 
</pre>
<!-- Author: AS Nicollerat :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138442' target=_top>138442</a>) -->
<!-- Comment: Tue Feb  5 12;00;12 comment by...W.Badgett -->
<pre>Level 3 was not touched...   only 
trigger supervisor</pre>
<!-- Date: Tue Feb 5 11:20:29 2002 -->
turned off the CHA LEDs to check real noisy channels.
<!-- Author: Min-Jeong -->

<!-- Date: Tue Feb 5 11:22:10 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138443">138443</A>
 Activated at 2002.02.05 11:21:27
<!-- Author: RunControl -->


<!-- Date: Tue Feb 5 11:24:33 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138443">138443</A>
 Terminated at 2002.02.05 11:24:01
<!-- Author: RunControl -->
<!-- Comment: Tue Feb  5 11;27;44 comment by...AS Nicollerat -->
<pre>We stopped the run to give the clc00 crate to Alex.
We got a done TO from Level1-CAL_04 crate. Carla is working on it. We start a new cosmic no track run.</pre>
<!-- Date: Tue Feb 5 11:26:42 2002 -->
<pre>Trigger Supervisor crate b0tsi00 has been rebooted  
with an updated version of Andrew Martin's code:  v1_6 
 
This code enables Level 1 Done (SVX) timeouts and Level 2  
multi-processor error checking. 
 
Should be checked with the SVX in the run  
as soon as possible.  (Siliconers are now monopolizing their  
system) 
 
</pre>
<!-- Author: W.Badgett,A.Martin -->


<!-- Date: Tue Feb 5 11:31:54 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138445">138445</A>
 Activated at 2002.02.05 11:30:11
<!-- Author: RunControl -->


<!-- Date: Tue Feb 5 11:34:43 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138445">138445</A>
 Terminated at 2002.02.05 11:34:05
<!-- Author: RunControl -->
<!-- Comment: Tue Feb  5 11;43;25 comment by...AS Nicollerat -->
<pre>The problem is normal ! Mike Lindgren is going to solve it in 5 minutes.</pre>
<!-- Date: Tue Feb 5 11:34:43 2002 -->
We had to stop the run because CCAL04 keeps giving Done TO: 
 
b0ccal04:Messenger:11:31:10 AM->Runtime Error 4, Event 2: SMXR in slot 3 is not responding, header 0x0x3d7e 
30f 
  
<!-- Author: AS Nicollerat :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138445' target=_top>138445</a>) -->


<!-- Date: Tue Feb 5 11:43:18 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138450">138450</A>
 Activated at 2002.02.05 11:42:38
<!-- Author: RunControl -->



<!-- Date: Tue Feb 5 11:46:26 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138450">138450</A>
 Terminated at 2002.02.05 11:45:31
<!-- Author: RunControl -->


<!-- Date: Tue Feb 5 11:51:55 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138451">138451</A>
 Activated at 2002.02.05 11:50:50
<!-- Author: RunControl -->


<!-- Date: Tue Feb 5 11:54:44 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138451">138451</A>
 Terminated at 2002.02.05 11:52:41
<!-- Author: RunControl -->
<!-- Comment: Tue Feb  5 11;57;06 comment by...AS Nicollerat -->
We stopped the run, Carla seems to be happy with L1 output (if the problem with level1_cal_04 crate comes again, we need to page her).

Ilya and Nuno are going to reboot level3 farms, it should take about 2 hours.

Tony and Hiro are restarting CSL at the same time.



<!-- Date: Tue Feb 5 12:12:59 2002 -->
<pre> 
Restart CSl to make a small change to monitoring code. 
</pre>
<!-- Author: Tony V. and Hiro K. -->


<!-- Date: Tue Feb 5 12:41:21 2002 -->
We accessed to collision hall to investigate 
PPR bad channel. 
<a href = "/cgi/elog/det-system-elog.pl?nb=plug">See here </a> for detail.
<!-- Author: Satoru / Yoshi -->


<!-- Date: Tue Feb 5 12:44:05 2002 -->
Unmasked trigger tower eta=6,phi=17, had. 
It seems ok now.
<!-- Author: carla -->


<!-- Date: Tue Feb 5 12:47:46 2002 -->
<pre>Cheng-Ju Lik is taking COT and XFT crates for about 3Hrs. 
 
We did a SMX calib.  run 138454</pre>
<!-- Author: AS. Nicollerat :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138454' target=_top>138454</a>) -->
<!-- Comment: Tue Feb  5 13;40;06 comment by...AS Nicollerat -->
SMX calibration is giving bad results in CCAL11. Karen is investigating it...
<!-- Date: Tue Feb 5 13:27:17 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138459">138459</A>
 Activated at 2002.02.05 13:26:43
<!-- Author: RunControl -->



<!-- Date: Tue Feb 5 13:33:17 2002 -->
<pre>I am done with the rebooting (and filesystem check) of the Level 3 farm. This time we actually rebooted it twice in a row, and the results were that 2 nodes (b0l3192, b0l3198) failed in the first rebooting, and none (as expected) in the 2nd. 
Level 3 is available again.</pre>
<!-- Author: Nuno -->

<!-- Date: Tue Feb 5 13:37:02 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138459">138459</A>
 Terminated at 2002.02.05 13:36:20
<!-- Author: RunControl -->
<!-- Comment: Tue Feb  5 13;38;41 comment by...AS Nicollerat -->
Run was stopped because CCAL was powercycled...
We will take another SMX calibration to see if the bad channel is now away...

<!-- Date: Tue Feb 5 13:39:32 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138459">138459</A>
 Terminated at 2002.02.05 13:36:20
<!-- Author: RunControl -->




<!-- Date: Tue Feb 5 14:01:29 2002 -->
With Bob Handler replaced BMU-W-100 and BMU-W-101 pre-amp boards. 
Oscillation went away, but causality questionable.  Also 
replaced BMU-W-286 and BMU-W-287 pre-amp boards to fix another 
oscillation that was bad last week but had gone away.  Also, 
replaced two certainly dead PAD's on BSU-W-10/11R and BSU-W-32/33R.    
<!-- Author: Camille Ginsburg -->


<!-- Date: Tue Feb 5 14:05:42 2002 -->
New SMX calibration to check if the CCAL11 bad channels are gone and it is the case. 
(!! as people are doing work on PCAL, results on PCAL crates should'nt be taken into account !!).  
<!-- Author: AS Nicollerat :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138460' target=_top>138460</a>) -->


<!-- Date: Tue Feb 5 14:14:13 2002 -->
<pre>Kevin Lannon is taking CMP_00 and CMX_00 for half an hour</pre>
<!-- Author: AS. Nicollerat -->


<!-- Date: Tue Feb 5 14:19:47 2002 -->
We cannot partition because of level3, we tried to clean it up but the same error remains, we paged them: 
 
b0l3gate2.fnal.gov:main:2:05:53 PM->Error on L3 node b0l3gate2 (partition 0) Tue Feb 5 14:05:52 2002 l3prox 
y  
  in Reset (L3ProxyPartition.cc:673) 
  @L3_TRANS_FAIL - transition actions failed:      b0l3036(Unrecognized response type for request)  
<!-- Author: AS Nicollerat :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138461' target=_top>138461</a>) -->


<!-- Date: Tue Feb 5 14:24:28 2002 -->
All consumers died earlier (not sure of the time).  I contacted Kaori and she tells me that the probably cause was the CS/L stopping and restarting, which apparently did occur earlier.   
I have restarted all the consumers.  
<!-- Author: Steve Wolbers -->


<!-- Date: Tue Feb 5 14:33:57 2002 -->
<pre>   Level3 failed Partition. It turns out that our attempts 
to restore one processor node (b0l3044) brought down another 
(b0l3036), for some reason automatic recovery script did 
not set up IP properly. We now marked this node (036) offline 
just in case. 
</pre>
<!-- Author: Ilya/Nuno -->


<!-- Date: Tue Feb 5 14:49:01 2002 -->
<pre>CPR Work: 
 
Trying to fix dead region in cpr Ephi=20, channels 9-16; 
swapped cables to see if dead region 
follows cable or electronics.     
 
SMX work: 
 
  Swapped 4 SMD cards which were giving header errors  
during data taking.   
 
Wphi=6, ccal02 slot3 SMX2 
Wphi=17, ccal05 slot5 SMX5 
Ephi=9, ccal11 slot3 SMX6 
 
and Pcal08, slot4 SMX5 
 
We also fixed a PSM monitoring problem for Wphi=12 
by replacing a picofuse at the CMU 5V junction box. 
 
 </pre>
<!-- Author: byrum, lindgren -->


<!-- Date: Tue Feb 5 14:49:08 2002 -->
<pre>Marco took the SVT crates for a couple of hours.</pre>
<!-- Author: AS Nicollerat -->

<!-- Date: Tue Feb 5 14:52:38 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138467">138467</A>
 Activated at 2002.02.05 14:51:26
<!-- Author: RunControl -->


<!-- Date: Tue Feb 5 14:54:14 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138467">138467</A>
 Terminated at 2002.02.05 14:53:10
<!-- Author: RunControl -->
<!-- Comment: Tue Feb  5 15;36;01 comment by...Ilya/Nuno -->
<pre>  Marked online the good node b0l3036. Ocatl 44 is 36!</pre>
<!-- Date: Tue Feb 5 14:54:14 2002 -->
<pre>   We have run out of RTserver licenses, so Level3 proxy 
could not be restarted. We paged an expert (I think it was 
DAQ-RC pager), and he advised to close all DAQMON windows 
and try again. This worked well. Then we reopened some DAQMON 
windows after starting the run, still everything worked. 
</pre>
<!-- Author: Ilya and Aces -->

<!-- Date: Tue Feb 5 14:59:43 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138469">138469</A>
 Activated at 2002.02.05 14:58:51
<!-- Author: RunControl -->



<!-- Date: Tue Feb 5 14:59:54 2002 -->
<pre>We finally started a cosmic no track run (50k events) without COT and Silicon crates. We set L2 auto accept since we don't have xtrp crates.</pre>
<!-- Author: AS Nicollerat :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138469' target=_top>138469</a>) -->

<!-- Date: Tue Feb 5 15:10:42 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138469">138469</A>
 Terminated at 2002.02.05 15:09:49
<!-- Author: RunControl -->


<!-- Date: Tue Feb 5 15:14:31 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138469">138469</A>
 Terminated at 2002.02.05 15:09:49
<!-- Author: RunControl -->



<!-- Date: Tue Feb 5 15:15:55 2002 -->
We turned on the shower max HV on Plug HV crate 5 
to check PPR singal. 
Alarm rings for the Plug HV crate 5, but looks no problem.
<!-- Author: Satoru / Yoshi -->
<!-- Comment: Tue Feb  5 16;36;08 comment by...Satoru / Yoshi -->
HV is now turned off again.
<!-- Date: Tue Feb 5 15:20:08 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138470">138470</A>
 Activated at 2002.02.05 15:18:25
<!-- Author: RunControl -->


<!-- Date: Tue Feb 5 15:23:25 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138470">138470</A>
 Terminated at 2002.02.05 15:22:49
<!-- Author: RunControl -->
<!-- Comment: Tue Feb  5 15;28;42 comment by...AS Nicollerat -->
<pre>we took a second cosmic no tracks run just to make sure we have enough data.</pre>


<!-- Date: Tue Feb 5 15:48:49 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138471">138471</A>
 Activated at 2002.02.05 15:48:08
<!-- Author: RunControl -->


<!-- Date: Tue Feb 5 15:53:01 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=138471">138471</A>
 Terminated at 2002.02.05 15:51:38
<!-- Author: RunControl -->


<!-- Date: Tue Feb 5 15:55:54 2002 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>15:55:03</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Tue Feb 5 15:59:45 2002 -->
<font color=red>Shift Summary: </font><pre>                Day Shift Summary Tuesday 05 Feb 2002 
  
****************** Beam Status ***************** 
Store 955 ended and supervised access until 19:30 begins. 
  
***************   History  ********************* 
08:35 Take cosmic no track run (50k) for CPR folks. 
10:57 Andrew Martin prepares upgrade of Trigger Supervisor code to enable 
      L1Done (SVX) Timeouts.  This addresses the problem of 
      data-taking-stops-but-no-timeout we have experienced recently. 
11:26 Trigger Supervisor crate b0tsi00 rebooted with above updated 
      version of code. 
14:59 Take another cosmic no track run to allow Karen Byrum to check 
      CPR repairs made today. 
16:00 Supervised access continues.  It will be followed this evening 
      by Accelerator studies.  Next shot will be Owl Thursday 07 Feb.           </pre> 
<!-- Author: Bob Wagner (ANL) -->