<!-- Title: Day shift. Wed Nov  7, 2007 -->
<!-- SciCo: Stephan Lammel -->
<!-- DAQAce: Carolina Deluca -->
<!-- MonAce: &nbsp; -->
<!-- CO: Khaldoun Makhoul -->
<!-- OpManager: Tom Wright -->
<!-- Notes: -->
<pre>in access</pre>

<!-- Date: Wed Nov 7 08:01:35.710 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253410>253410</A>
ACTIVATE: RunSet: TOF_TAC_CALIB 
<!-- Author: Carolina Deluca -->


<!-- Date: Wed Nov 7 08:01:36.414 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253407>253407</A>
CALIBRATION: <U><B>toftac</B></U> calibration FAILED, please investigate
<!-- Author: CDFCALIB -->
<!-- Comment: Wed Nov  7 08;04;32 comment by...carolina -->
Couldn't go to end state... sevb was unresponsive. Joachim tried to restart it from SoftEvb Proxy but didn'y work so we aborted the run and tried again. Second time was fine.

<!-- Comment: Wed Nov  7 09;33;36 comment by...Carolina -->
<pre>Donatella came in and explained me that in fact for calibrations we do use different proxys. So to check that this is working when doing calibrations one has to click on Proxy on RC and under that click on CalibConsumers.
Calibrations use different CSL than the current one so if that needs to be restarted has to check which one is. There are instructions for this in the CSL web page under "Which CSL am I using?" link.</pre>
<!-- Comment: Wed Nov  7 09;36;23 comment by...Donatella T. -->
we've seen this before.. last time was the TofTac CalibConsumer that got stuck and we restarted the Calibration CSL. I asked Caroline to keep the Proxy windows open so we can see which process hangs.
<!-- Date: Wed Nov 7 08:08:25.158 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253411>253411</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [19,986,705] 
<!-- Author: Carolina Deluca -->



<!-- Date: Wed Nov 7 08:18:01.054 2007 -->
BMU Access Report: We made an access yesterday to swap preamps on SE100, SE116 and NE49. After the access, HV tripped on 60 (around 300 V on CAT!) and sector 48 was drawing very high current on anode and too small amount of current on CAT. Also noticed was CAT on sector 12 was drawing 1/3 of current it should draw. We went in this morning again.  It turns out that the HV jumper cables were hanging (my bad!). It must have happened when we were in a hurry to get out of way of survey guys. The HV sector 60 came back to normal even before this morning's access. The sector 48 is back to normal now. The puzzling low current on CAT in sector 12 turns out to be the modification we had long time ago (it is actually having only 4 stacks and neiboring sectors share the rest. We didn't have time to swap HV board on NW177, which is out of chain now. Other than that, BMU is back to normal again.
<!-- Author: Woohyun+Varsha+Yeongdae -->


<!-- Date: Wed Nov 7 08:33:03.719 2007 -->
Investigated noise in Wedge 12 East CES yesterday, but I forgot to make an e-log entry.  The noise on the first 32 strips was no longer apparent, and there were no obvious cable faults.  Replaced the SMD board corresponding to those strips and did an electronics calibration.  No change observed.  Today we will look at the HV 
and recheck the SMD.
<!-- Author: Lindgren, Rutherford -->

<!-- Date: Wed Nov 7 09:11:56.031 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253411>253411</A>
TERMINATE: run cosmics for Laura
<!-- Author: Carolina Deluca -->


<!-- Date: Wed Nov 7 09:14:57.497 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253412>253412</A>
 Activated at 2007.11.07 09:14:38
<!-- Author: RunControl -->


<!-- Date: Wed Nov 7 09:15:03.677 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253412>253412</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,985,706] l2pulsar07 included
<!-- Author: Carolina Deluca -->


<!-- Date: Wed Nov 7 09:17:17.057 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253412>253412</A>
 Terminated at 2007.11.07 09:16:56
<!-- Author: RunControl -->


<!-- Date: Wed Nov 7 09:17:18.669 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253412>253412</A>
TERMINATE: done
<!-- Author: Carolina Deluca -->



<!-- Date: Wed Nov 7 09:18:52.916 2007 -->
Connor asked me to not start any run while they are working on EVB
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253412' target=_top>253412</a>) -->


<!-- Date: Wed Nov 7 09:25:00.279 2007 -->
<table border=0><td><tr><td bgcolor=#e0e030>HAPPY BIRTHDAY DONATELLA AND ARON!</td></tr></td></table>
<!-- Author: dee -->




<!-- Date: Wed Nov 7 09:57:26.998 2007 -->
Mike Lindgren asked for ccal12 crate, so release it from partition
<!-- Author: carolina -->


<!-- Date: Wed Nov 7 10:19:05.088 2007 -->
Got a CRYO alarm but it was automatically silenced. LLDEW is 38.42%
<!-- Author: carolina -->
<!-- Comment: Wed Nov  7 10;31;39 comment by...carolina -->
Another cryo alarm that was silenced, fine again.


<!-- Date: Wed Nov 7 10:39:34.097 2007 -->
PSM alarm on arch 12E that corresponds to ccal12, where Lindgren and Rutherford are working on
<!-- Author: carolina -->

<!-- Date: Wed Nov 7 10:59:42.990 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253426>253426</A>
TERMINATE: try ignoring reces
<!-- Author: Carolina Deluca -->


<!-- Date: Wed Nov 7 10:59:49.163 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253426>253426</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [19,986,705] no ccal12 crate in partition
<!-- Author: Carolina Deluca -->



<!-- Date: Wed Nov 7 11:01:33.966 2007 -->
JDL asked for COT crates 07, 08, 10 and 17. I also release them from partition
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253426' target=_top>253426</a>) -->

<!-- Date: Wed Nov 7 11:05:21.044 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253427>253427</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [19,986,705] no ccal12+ignoring reces. No COT07, 08, 10, 17
<!-- Author: Carolina Deluca -->


<!-- Date: Wed Nov 7 11:06:17.310 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253427>253427</A>
TERMINATE: use L2 autoaccept to avoid L2 To's we are experiencing
<!-- Author: Carolina Deluca -->


<!-- Date: Wed Nov 7 11:10:42.174 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253428>253428</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [19,986,705] no ccal12+ignoring reces. Noc COT07, 08, 10 and 17. Running with L2 autoaccept for L3 people
<!-- Author: Carolina Deluca -->



<!-- Date: Wed Nov 7 11:11:10.632 2007 -->
Done Timeout detected in crates: 
IMU_00 
Auto-HRR fixed it
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253428' target=_top>253428</a>) -->
<!-- Comment: Wed Nov  7 11;12;15 comment by...carolina -->
again

<!-- Date: Wed Nov 7 11:12:49.568 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253428>253428</A>
TERMINATE: L3 people is done with their test
<!-- Author: Carolina Deluca -->


<!-- Date: Wed Nov 7 11:17:08.130 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253429>253429</A>
 Activated at 2007.11.07 11:16:50
<!-- Author: RunControl -->


<!-- Date: Wed Nov 7 11:17:44.418 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253429>253429</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,985,706] no ccal12+ignoring reces. No COT07, 08, 10, 17 crates
<!-- Author: Carolina Deluca -->



<!-- Date: Wed Nov 7 11:46:38.937 2007 -->
Don't get the entry but SMX calib is DONE
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253433' target=_top>253433</a>) -->
<!-- Comment: Wed Nov  7 11;54;31 comment by...carolina -->
I checked the web and the plots are already there


<!-- Date: Wed Nov 7 11:55:37.913 2007 -->
It seems I don't get the entries from RunControl... Now I'm running cosmics but nothing appears here.
<!-- Author: Carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253434' target=_top>253434</a>) -->
<!-- Comment: Wed Nov  7 12;47;55 comment by...carolina -->
Stephan emailed Bill Badgett and Donatella 


<!-- Comment: Wed Nov  7 13;59;41 comment by...carolina -->
I am including ccal12 and excluding COT07, 07, 10 and 17
<!-- Date: Wed Nov 7 13:56:44.511 2007 -->
<pre>Done TO in LEVEL2_PULSAR_02: 
 
  ELog        true 
1 crate/s: b0l2pu02(151),  Done TO. [RXPT] 
(MLE) b0control21.fnal.gov:Thread-25:Nov 7, 2007 01:53:42 PM->Done Timeout: LEVEL2_PULSAR_02  
(MLW) b0control21.fnal.gov:Thread-25:Nov 7, 2007 01:53:42 PM->Received error message: errmon Not a state HaltRecoverRun 1 crate/s: b0l2pu02(151),  Done TO. [RXPT](MLE) b0control21.fnal.gov:Thread-25:Nov 7, 2007 01:53:42 PM->Requested Halt-Recover-Run issued (errmon) 
(MLE) b0felix.fnal.gov:TEDCpp:Nov 7, 2007 01:54:10 PM->{ 13:54:10, 11/7/2007 }: L2-Pulsar-Linux: Halted while  processing happily 
 
Auto-HRR did start but it got stuck before arriving to "halt" 
Run is halted</pre>
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253434' target=_top>253434</a>) -->
<!-- Comment: Wed Nov  7 13;58;22 comment by...carolina -->
Jonathan just came in, he is trying to shepherd the crate



<!-- Date: Wed Nov 7 14:00:42.569 2007 -->
Stopped cosmics run. Jonathan wants to make some tests
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253434' target=_top>253434</a>) -->
<!-- Comment: Wed Nov  7 14;01;15 comment by...carolina -->
Messages from RC are still missing....


<!-- Comment: Wed Nov  7 14;41;51 comment by...Donatella T. -->
it has something to do with the Oracle database, but I'm not sure about what to restart... We need Bill intervention.
<!-- Date: Wed Nov 7 14:07:27.760 2007 -->
Jonathan took his calibration I run cosmics again but all cot crates this time
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253437' target=_top>253437</a>) -->


<!-- Date: Wed Nov 7 14:11:23.498 2007 -->
End this run to include XFT (Gene requested)
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253438' target=_top>253438</a>) -->


<!-- Date: Wed Nov 7 14:14:18.782 2007 -->
Gene is now running cosmics with XFT
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253439' target=_top>253439</a>) -->


<!-- Date: Wed Nov 7 14:26:31.707 2007 -->
<pre>L2 TO auto-HRR fixed it 
 
L2 Decision Timeout.[RXPT] 
(MLW) b0control21.fnal.gov:Thread-25:Nov 7, 2007 02:25:19 PM->Received error message: errmon Not a state HaltRecoverRun L2 Decision Timeout.[RXPT](MLE) b0control21.fnal.gov:Thread-25:Nov 7, 2007 02:25:19 PM->Requested Halt-Recover-Run issued (errmon) 
(MLE) b0felix.fnal.gov:TEDCpp:Nov 7, 2007 02:25:45 PM->{ 14:25:45, 11/7/2007 }: L2-Pulsar-Linux: Halted while  processing happily 
(RC)   14:25:25 Halt -> HALTED 
(RC)   14:25:32 Recover -> RECOVERED 
(RC)   14:25:40 Run -> ACTIVE 
</pre>
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253439' target=_top>253439</a>) -->


<!-- Date: Wed Nov 7 14:36:03.952 2007 -->
<pre>We checked the voltage on ccal12 in an attempt to diagnose the noise problem we've been seeing during operation.  We also ran pedestals and the numbers looked a little low, but we'll leave things as they are and wait to see if anything changes.</pre>
<!-- Author: Rutherford,Lindgren -->



<!-- Date: Wed Nov 7 14:42:14.973 2007 -->
<pre>EVB error: 
FERML_TM2_HEVB2B_EVENTS_PENDING_TIMEOUT !!!  
 - AUTO HRR will be issued.  
 
b0tsi00:Messenger:Nov 7, 2007 02:39:57 PM->TM2 Timeout waiting for pending events from HEVB2b, partition 0, lastEvent 81697 
 
Just after HRR: 
 
L2 Decision Timeout.[RXPT] 
(MLW) b0control21.fnal.gov:Thread-25:Nov 7, 2007 02:41:04 PM->Received error message: errmon Not a state HaltRecoverRun L2 Decision Timeout.[RXPT](MLE) b0control21.fnal.gov:Thread-25:Nov 7, 2007 02:41:04 PM->Requested Halt-Recover-Run issued (errmon) 
(MLE) b0felix.fnal.gov:TEDCpp:Nov 7, 2007 02:41:52 PM->{ 14:41:52, 11/7/2007 }: L2-Pulsar-Linux: Halted while  processing happily 
 
Running fine again 
(RC)   14:41:31 Halt -> HALTED</pre>
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253439' target=_top>253439</a>) -->


<!-- Date: Wed Nov 7 14:46:49.376 2007 -->
Stopped run because of continuous L2 TO's that prevented us from running. Try again. Same configuration Gene was testing
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253439' target=_top>253439</a>) -->


<!-- Date: Wed Nov 7 14:50:15.717 2007 -->
<pre>Tried to run cosmics again: 
 
 Attention !!!!.  L2PULSAR_NODELOST !  
  
Control node is unable to contact decision node process. 
Decision node is sick. Page Level 2 for help. 
 
 
MLE) b0csl20.fnal.gov:csl_driver:Nov 7, 2007 02:46:57 PM->Logger partition 0 is in state: 4. 
 (INITIAL 0,CONNECTED 1,ACTIVE_IDLE 2,ACTIVE 3,COMPLETED 4,BUSY_STATE 5,ERROR_STATE 6)  
(MLE) b0svt09:Messenger:Nov 7, 2007 02:47:08 PM->FER_E_SVT, (info) hc=2478488567 mc=703417488 
(MLE) b0svt07:Messenger:Nov 7, 2007 02:47:12 PM->FER_E_SVT, (info) hc=2478488567 mc=703417488 
(MLE) b0svt03:Messenger:Nov 7, 2007 02:47:16 PM->FER_E_SVT, (info) hc=2478488567 mc=703417488 
(MLE) b0svt00:Messenger:Nov 7, 2007 02:47:16 PM->FER_E_SVT, (info) hc=2478488567 mc=703417488 
(MLE) b0svt01:Messenger:Nov 7, 2007 02:47:16 PM->FER_E_SVT, (info) hc=2478488567 mc=703417488 
(MLE) b0svt02:Messenger:Nov 7, 2007 02:47:16 PM->FER_E_SVT, (info) hc=2478488567 mc=703417488 
(MLE) b0svt04:Messenger:Nov 7, 2007 02:47:16 PM->FER_E_SVT, (info) hc=2478488567 mc=703417488 
(MLE) b0svt05:Messenger:Nov 7, 2007 02:47:17 PM->FER_E_SVT, (info) hc=2478488567 mc=703417488 
(MLE) b0svt08:Messenger:Nov 7, 2007 02:47:29 PM->FER_E_SVT, (info) hc=2478488567 mc=703417488 
(MLE) b0svt06:Messenger:Nov 7, 2007 02:47:43 PM->FER_E_SVT, (info) hc=2478488567 mc=703417488 
b0felix.fnal.gov:TEDCpp:Nov 7, 2007 02:48:36 PM->{ 14:48:36, 11/7/2007 }: L2-Pulsar-Linux: Control node is unable to contact decision node process in node  #0 at address 192.168.0.10; Decision node may be in trouble.  -->   
 Additional Information:  
 
 Attention !!!!.  L2PULSAR_NODELOST !!!! 
  
Then a L2 decision TO. Active is red, auto-HRR doesn't help.  </pre>
  
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253440' target=_top>253440</a>) -->


<!-- Date: Wed Nov 7 14:57:57.561 2007 -->
CMX wedge 14 tripped is anyone working on there?
<!-- Author: carolina -->
<!-- Comment: Wed Nov  7 15;00;16 comment by...carolina -->
I put the HV in the wedge to be OFF, then I'll turn it on again to see if it is recovered


<!-- Comment: Wed Nov  7 15;04;57 comment by...carolina -->
Fine again
<!-- Date: Wed Nov 7 15:02:11.297 2007 -->
Called Gene. He asked us to page L2 and to start a new run ignoring L2 stereo. We did it but L2 To's came back again and auto-HRR didn't work. Laura is now here
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253441' target=_top>253441</a>) -->
<!-- Comment: Wed Nov  7 15;21;19 comment by...Laura and Gene -->
The L2 PC was stucked: you can ping it, but you can't log in. We've reboot the PC and now we are running cosmic in the default configuration (ignoreXFT set).


<!-- Date: Wed Nov 7 15:38:46.511 2007 -->
Cryo alarm it was silenced
<!-- Author: carolina :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=253443' target=_top>253443</a>) -->
<!-- Comment: Wed Nov  7 15;45;23 comment by...carolina -->
Fine again


<!-- Date: Wed Nov 7 16:00:02.635 2007 -->
Earlier today, we swapped XTC cards that seemed to have problems on several TDCs today. 
<pre> 
COT07, Slot 18: TDC 534, XTC2 080 -> 159  -- Many errors 
COT08, Slot 13: TDC 519, XTC2 018 -> 166  -- Possible L2 errors 
COT08, Slot 14, TDC 562, XTC2 140 -> 004  -- Possible L2 errors 
COT10, Slot 05, TDC 541, XTC1 235 -> 104  -- Low efficiency 
</pre> 
In the above cases, the TDCs were returned to their original slots.  We also swapped a TDC where there was low XTC efficiency.  In that case it was a board that has displayed a time offset in the TDC data.  It is COT17, Slot 13. 
<pre> 
TDC 551, XTC2 054  -->  TDC 613, XTC2 122 
</pre>
<!-- Author: jdl & Martin Frank -->


<!-- Date: Wed Nov 7 16:00:27.000 2007 -->
<font color=red>Shift Summary: </font><pre>in access</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   253410-253443
Delivered Luminosity   0  
Acquired Luminosity    0  
Efficiency             100%
   
</td></tr></table></pre>
<!-- Author: Stephan -->