<!-- Title: Eve shift. Sun May 26, 2002 -->
<!-- SciCo: Steve Errede -->
<!-- DAQAce: Dongwook Jang/S.Carron -->
<!-- MonAce: Andrea Bocci -->
<!-- CO: Stefano Giagu -->
<!-- OpManager: Dave Ambrose -->
<!-- Notes: -->
<pre>Continue recovery from this morning's power glitch.  
 
Struggling to bring up DAQ - presently focussing on L1 crate problems.</pre>

<!-- Date: Sun May 26 16:24:51 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145331">145331</A>
 Activated at 2002.05.26 16:24:36
<!-- Author: RunControl -->


<!-- Date: Sun May 26 16:26:01 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145331">145331</A>
 Terminated at 2002.05.26 16:25:41
<!-- Author: RunControl -->


<!-- Date: Sun May 26 16:31:38 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145332">145332</A>
 Activated at 2002.05.26 16:31:07
<!-- Author: RunControl -->


<!-- Date: Sun May 26 16:34:49 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145332">145332</A>
 Terminated at 2002.05.26 16:34:48
<!-- Author: RunControl -->



<!-- Date: Sun May 26 16:43:26 2002 -->
We checked the CLC & Forward Detectors PC. The stand alone monitor works fine and IFIX is running there. From VNODE1 instead they are still gray. I think for the same reason of the Plug. 
CLC, BSC, RPS and MNP are OFF now.
<!-- Author: Andrea B. -->


<!-- Date: Sun May 26 16:49:08 2002 -->
Steve Hahn calls - nominally to help with DAQ. Also will try to get master clock reset & working
<!-- Author: Errede -->


<!-- Date: Sun May 26 17:06:16 2002 -->
Trying to get the DAQ going, I noticed many sync errors and 
CDF_ERROR pulled by many crates. Walking to the clock crate: 
there is no clock or BC... this is why Steve Hahn was called. <P> 
On another subject, I recovered the 2 TDC's in the htdc crate 
that have been reported as failing.  
<!-- Author: Arnd -->


<!-- Date: Sun May 26 18:04:35 2002 -->
Clock sequencing was reloded, experts say clock looks fine now.
<!-- Author: Sebastian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145333' target=_top>145333</a>) -->


<!-- Date: Sun May 26 18:05:07 2002 -->
Summary of the HV situation: 
 
1) The vnode seems have some comunication/setup problem. Experts (JC) will come this evening to put it back properly 
 
2) Systems reachable from the Control Room: 
  - COT ........ SYBY 
  - TOF ........ OFF 
  - Silicon..... STBY 
  - CMU ........ SYBY 
  - CMP ........ SYBY 
  - CMX ........ SYBY 
  - BMU ........ SYBY 
  - CES ........ SYBY 
  - CPR ........ SYBY 
  - CCR ........ SYBY 
  - CEM ........ ON 
  - CHA ........ ON 
  - WHA ........ ON 
  - GAM ........ ON 
 
3) Systems unreachable from the Control Room 
 
  - CSW, MSK, CSX, BSU, TSU..... I think some expert worked on       back them ON. 
  - PEM, PHA, PSH ..... Expert put them ON. 
  - 
<!-- Author: ACE -->
<!-- Comment: Sun May 26 18;54;38 comment by...niccolo&#39; -->
With Andrea and Stefano (thanks!) we fixed the communication problem from the plug monitor (PEM, PHA, PSH).
<!-- Comment: Sun May 26 19;11;33 comment by...Andrea B. -->
Update HV summary <A href= 
'/cgi/elog/elog.pl?nb=2002&action=copyentry&page=438&time=19:01:59'> 
here</A> 
<!-- Date: Sun May 26 18:10:20 2002 -->
Tried spike run, but problems with b0l1gl00, paging Greg 
<!-- Author: Aces -->

<!-- Date: Sun May 26 18:18:40 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145334">145334</A>
 Activated at 2002.05.26 18:18:33
<!-- Author: RunControl -->


<!-- Date: Sun May 26 18:20:55 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145334">145334</A>
 Terminated at 2002.05.26 18:19:01
<!-- Author: RunControl -->



<!-- Date: Sun May 26 18:26:10 2002 -->
Arnd and Steve H. manage to get the CDF clock reset. 
 
Arnd finds Level I FRED crate is still hosed. Called Carla at home - 
not there yet. She will call as soon as gets in.
<!-- Author: Errede -->

<!-- Date: Sun May 26 18:32:26 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145336">145336</A>
 Activated at 2002.05.26 18:32:15
<!-- Author: RunControl -->


<!-- Date: Sun May 26 18:36:33 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145336">145336</A>
 Terminated at 2002.05.26 18:35:24
<!-- Author: RunControl -->


<!-- Date: Sun May 26 18:39:44 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145337">145337</A>
 Activated at 2002.05.26 18:39:35
<!-- Author: RunControl -->


<!-- Date: Sun May 26 18:42:21 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145337">145337</A>
 Terminated at 2002.05.26 18:41:13
<!-- Author: RunControl -->



<!-- Date: Sun May 26 18:45:05 2002 -->
Arnd tried to make run control working. We include all possible crates and got an error from slot 11 in b0musc00, so we took out this slot. After that we got done time out error from b0l2de01. Now we are trying to figure it out.
<!-- Author: ace -->

<!-- Date: Sun May 26 18:47:48 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145339">145339</A>
 Activated at 2002.05.26 18:47:22
<!-- Author: RunControl -->



<!-- Date: Sun May 26 18:50:24 2002 -->
Having problems with a muon TDC (slot 11) in B0musc00 crate, muon TDC expert not available at present, so page COT TDC expert (Mike Kirby). He calls, he's coming in to help - swap out bad TDC.
<!-- Author: Errede -->
<!-- Comment: Sun May 26 19;45;00 comment by...Kirby -->
See full entry below.
<!-- Date: Sun May 26 18:52:30 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145339">145339</A>
 Terminated at 2002.05.26 18:52:14
<!-- Author: RunControl -->



<!-- Date: Sun May 26 18:59:39 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145340">145340</A>
 Activated at 2002.05.26 18:59:15
<!-- Author: RunControl -->


<!-- Date: Sun May 26 19:00:00 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145340">145340</A>
 Terminated at 2002.05.26 18:59:55
<!-- Author: RunControl -->



<!-- Date: Sun May 26 19:01:06 2002 -->
Let's summarize the things done so far: 
 
slot 11 in b0musc00 is taken out 
b0l1gl00 -> Calar's working on 
b0l2de01 -> because l1gl00's taken out 
b0svt03 is taken out -> expert(?)'s working on 
 
Mario's trying something to make sure all crates working properly.
<!-- Author: Ace -->


<!-- Date: Sun May 26 19:01:59 2002 -->
Update on HV Monitor situation. <br> 
<br> 
System back to normal operation: <br> 
COT, Silicon, CMU,CMP,CMX,BMU,CES,CPR,CCR --- on STBY <br> 
CEM, CHA, WHA,PEM, PHA, PSH,GAM BSC, RPS  --- on ON  <br> 
TOF, CLC, MNP     ----   on OFF <br> 
<br> 
System not reachable: <br> 
<br> 
CSW, MSK, CSX, CSP, BSU, TSU <br> 
Expert <A href=  
'/cgi/elog/elog.pl?nb=2002&action=copyentry&page=437&time=14:27:58'>  
was around</A> trying to fix it. <br> 
<br> 
PTM - It is gray on the global alrm. I gonna take a look to the PC <br>
<!-- Author: Andrea B. -->


<!-- Date: Sun May 26 19:03:48 2002 -->
<pre>After the site-wide power failure, the XFT finderboards failed 
cold-start.  Somehow the power glitch corrupted the flash RAMs 
on some of the finderboards, re-programming the flash RAMs 
solved the problem. 
 
Returning the XFT crates to aces.</pre>
<!-- Author: Cheng-Ju -->



<!-- Date: Sun May 26 19:28:42 2002 -->
Everything (thanks to Arnd and Mario, saved our lives) is running now exept b0svt03 and b0musc00, Mike Kirby is looking at the muon Scintilator, we are paging again the svt expert. b0l2de01 crate has been pulled out and in again, the spare one was pulled out but not placed back in (they just forgot to o it), this seems to have fixed previous problems.
<!-- Author: Dongwook and Sebastian -->


<!-- Date: Sun May 26 19:29:45 2002 -->
The Level 1 global crate is working again. Here is what happened. <P> 
The VME bus would not work at all, ie. any reads/writes to any 
board in the crate (TRACER/PreFred/Fred) would return errors. 
Neither resetting nor powercycling helped, both before and after 
the clock was working. Under Carla's instructions, I extracted 
first a couple of prefred's (up to slot 4) - still no VME. Then 
the remaining PreFred's the FRED in slot 8, and the spare FRED in 
slot ? (17?) - VME came back. After reinstalling first some PreFred's, 
then FRED in slot 8 and the remaining preFRED boards, the crate 
still looked OK. Taking a cosmic run confirmed this. I forgot 
to reinstall the Fred in slot 17, so we don't know if the 
problem disappeared mysteriously or because of the missing 
board. <P> 
For the rest of CDF, only SVT03 and the TDC in slot 11 of b0musc00 are not yet working again. 
  
<!-- Author: Arnd -->


<!-- Date: Sun May 26 19:36:31 2002 -->
<pre>We paged SVT expert - expert calls - he's coming in (from Chi-town)  
ETA ~ 1 hr.</pre>
<!-- Author: Errede -->



<!-- Date: Sun May 26 19:49:26 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Sun_May_26_2002_Eve.194917.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-3393  align='top'></td><td>BAd Channel Plot</a></td></tr></table>
<!-- Author: Sebastian -->
<!-- Comment: Sun May 26 19;50;38 comment by...Sebastian -->
After running Calorimeter QIE calibration, Many bad channels found


<!-- Comment: Sun May 26 19;55;31 comment by...Sebastian -->
Run # 145348
<!-- Date: Sun May 26 19:50:42 2002 -->
<pre>Looking at the problem in b0musc00, slot 11 (SN 251), I saw that this card was not responding during a mapping of the crate in VISIONdemo.  so after a bus reset, logged back into the crate, and the card was now visible to a cratemap.  when trying to do basic reads and writes to the control register though, it seem that the VME interface was glitchy.  I could write zeros to the control register, but when trying to write non-zero values, the register wouldn't change but stay as all zeros.  It didn't give a bus error, just wouldn't change.  This wasn't the case when trying to get TDCTest going, here the VME bus errors were rather copious.  anyway, I went down and reseated the card, and this seems to have fixed the problem.  I can read and write to the control register, and have run the full crate through TDCTest 4 times.  the card does have channel failures, but these failures are known failures and have been in the system for atleast a month.  I'll inform the Muon TDC expert about these channel failures when he returns from kart racing.  I'll return the crate to the aces, and let them reintegrate.</pre>
<!-- Author: Kirby -->



<!-- Date: Sun May 26 19:59:21 2002 -->
SM QIE calibration
<!-- Author: Sebastian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145350' target=_top>145350</a>) -->


<!-- Date: Sun May 26 20:03:53 2002 -->
We made a pre-emptive call to XTRP expert (Nathan Eddy) 
XTRP crate is actually fine. Just in case he is needed, 
both he and Eric James are at 847-372-9039 (in Indiana) 
until ~ noon tomorrow. They will be back here tomorrow afternoon.
<!-- Author: Errede -->


<!-- Date: Sun May 26 20:10:49 2002 -->
The PTM Pc on the 3rd floor was OFF. We called the expert and instructed by him at the phone we bring up the Plug Temperature system. The expert says to wait one hour or so to have the temperature stable.
<!-- Author: Andrea B. -->


<!-- Date: Sun May 26 20:12:12 2002 -->
Calorimeter LED Calibration
<!-- Author: Sebatian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145351' target=_top>145351</a>) -->


<!-- Date: Sun May 26 20:47:03 2002 -->
Xenon Calibration
<!-- Author: Sebastian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145352' target=_top>145352</a>) -->
<!-- Comment: Sun May 26 20;49;22 comment by...Sebastian  -->
When turning on the xenon flashers, cannot succeed with select all, a couple had to be turned on separately, specially wedge 23 which took several tries until it turned on.


<!-- Date: Sun May 26 20:55:13 2002 -->
Muon scintillators system (CSW, MSK, CSX, CSP, BSU, TSU) is  
reachable now.   
<!-- Author: Oleg Poukhov -->


<!-- Date: Sun May 26 20:57:33 2002 -->
MCR is turning on TeV. Stacking Pbars. Now have 33.2E10 in stack, currently stacking @ 0.36 mA/hr
<!-- Author: Errede -->


<!-- Date: Sun May 26 21:01:51 2002 -->
tested muon scinbtilator crate and it worked (M. Kirby fixed it)
<!-- Author: Dogwook -->


<!-- Date: Sun May 26 21:03:14 2002 -->
tested muon scinbtilator crate and it worked (M. Kirby fixed it)
<!-- Author: Dogwook -->


<!-- Date: Sun May 26 21:36:50 2002 -->
CLC QIE calibration
<!-- Author: Sebastian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145353' target=_top>145353</a>) -->


<!-- Date: Sun May 26 21:40:47 2002 -->
HV summary update: <br> 
<br> 
All systems are now back to normal operation. IFIX looks working fine. The Plug Temperature has no alarms. All muon scintillators system green and working.<br> 
The only problem left is the gray color of the PTM box in the global alarm page. From the PCTEMP pn the 3rd floor, the monitor and IFIX are running fine.
<!-- Author: Andrea B. -->


<!-- Date: Sun May 26 21:44:19 2002 -->
COT calibratiuon
<!-- Author: Sebastian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145354' target=_top>145354</a>) -->


<!-- Date: Sun May 26 21:48:24 2002 -->
Muon Calibration
<!-- Author: Sebastian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145355' target=_top>145355</a>) -->


<!-- Date: Sun May 26 21:52:10 2002 -->
TOF TAC Calibration
<!-- Author: Sebastian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145356' target=_top>145356</a>) -->


<!-- Date: Sun May 26 21:56:35 2002 -->
TOF QIE Calibration
<!-- Author: Sebastian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145357' target=_top>145357</a>) -->


<!-- Date: Sun May 26 22:02:33 2002 -->
BSCQIE calibration
<!-- Author: Sebastian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145358' target=_top>145358</a>) -->


<!-- Date: Sun May 26 22:11:13 2002 -->
Miniplug QIE Calibration
<!-- Author: Sebastian  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145359' target=_top>145359</a>) -->


<!-- Date: Sun May 26 22:23:33 2002 -->
<table border=0><td><tr><td bgcolor=#e0e030>Note to Owl shift personnel: 
CDF clock is currently in "free run" mode. If want to take beam data (with TevKE .ge. 150 GeV), you will need to put CDF clock in "normal" mode.  To do this, with Run_Control idled, go to B0dap53 console (the right-most terminal at ACE desk). Type in "whoami" - you should get response "cdfdaq".  
 
If so, then type in "setup clock<cr>" (<cr> = carriage return), 
 
then type in "cd $CLOCK_DIR"  
 
then type in "clockGUI b0clk00".  
 
Wait a bit, and the Salvador Dali memorial master clock program will pop up. Bring it down from the top monitor, click on the clockGui HELP button.  
 
In the "Set Sync Delay" port, type in "6<cr>" 
Then click on "Set Sync Delay", then click on "Normal" for 
Normal running of clock. 
 
Then click on the "SFMs" tab, open this up and click on the 
"Good Luck" button. You're done. 
 
Don't hesitate to call Steve Hahn to help you through this tonight/tomorrow morning. He's expecting a call from you on this.</td></tr></td></table>
<!-- Author: Errede -->


<!-- Date: Sun May 26 22:36:39 2002 -->
<table border=0><td><tr><td bgcolor=#e0e030>Note to Owl Shift: 
 
According to MCR e-log, the power glitch this morning was due to a dirty insulator, right where the lab taps off of ComEd power - the insulator had arced over.  ComEd people looked at it (from a helicopter) - they claim that it looks good now, but....
 
This could happen again - maybe tomorrow morning, about same time as it happened today (05:39), if weather conditions are similar to what they were this morning - foggy....</td></tr></td></table>
<!-- Author: Errede -->

<!-- Date: Sun May 26 22:49:21 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145361">145361</A>
 Activated at 2002.05.26 22:48:54
<!-- Author: RunControl -->


<!-- Date: Sun May 26 22:55:21 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145361">145361</A>
 Terminated at 2002.05.26 22:54:43
<!-- Author: RunControl -->



<!-- Date: Sun May 26 22:56:11 2002 -->
Performed Rick's Run wit 30k events, looked fine.
<!-- Author: Sebastian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145361' target=_top>145361</a>) -->


<!-- Date: Sun May 26 23:18:31 2002 -->
Plug laser Calibration
<!-- Author: Sebastian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145362' target=_top>145362</a>) -->

<!-- Date: Sun May 26 23:34:23 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145363">145363</A>
 Activated at 2002.05.26 23:33:53
<!-- Author: RunControl -->



<!-- Date: Sun May 26 23:34:23 2002 -->
Started Cosmic Ray Run
<!-- Author: Sebastian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=145362' target=_top>145362</a>) -->


<!-- Date: Sun May 26 23:36:37 2002 -->
Earlier svt problem solved by expert, finally all is working.
<!-- Author: Aces -->

<!-- Date: Sun May 26 23:55:21 2002 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:02</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sun May 26 23:55:46 2002 -->
Last update on HV. Now everything works fine. JC stopped by and restart the latest version of the global alarm pictures. The PTM is turned green finally. 
<!-- Author: Andrea B. -->


<!-- Date: Sun May 26 23:57:24 2002 -->
Ollie Keimschies from MCR calls - he's noticed that SVX abort clock has crazy #'s in it. We need to recover/reset this, too. 
Paged SVX expert. He calls - we find T:B0LMAA rate meter (to left of IFIX monitor) is pegged - CAMAC module driving it is somehow 
pegging it - its not the rate meter that is bad. But we can't find the problem, working over the phone. CAMAC module may have been frapped in the power glitch. So SVX expert is coming in to take a look at it.
<!-- Author: Errede -->
<!-- Comment: Mon May 27 00;20;36 comment by...Andy H. -->
Problem fixed simply by unplugging the clock input to the CAMAC
module and then plugging it back in.  Didn't expect it to be
that easy...
<!-- Date: Mon May 27 00:02:26 2002 -->
<font color=red>Shift Summary: </font><pre>We inherit job of recovering from power glitch earlier this morning. Many problems - we focus on getting DAQ operational so that we can ascertain whether or not we need to make access. Little by little we succeed in getting CDF back together again - with much help from many experts called in. See above for full details. We are able to carry out all calibrations successfully, and do some cosmic ray data taking. A miracle, it seems... 
The power glitch this morning - at 05:39 - occurred when dirty insulator arced over. It was foggy & cold. ComEd claims that the insulator looks good now... But this could happen again e.g. tomorrow morning ~ same time, if weather conditions are similar... </pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   145331-145362
Delivered Luminosity   -  
Acquired Luminosity    -  
Efficiency             -
   
</td></tr></table></pre>
<!-- Author: Errede -->