<!-- Title: Eve shift. Tue Nov 18, 2003 -->
<!-- SciCo: Akiya Miyamoto -->
<!-- DAQAce: Dale Stentz -->
<!-- MonAce: Takasumi Maruyama -->
<!-- CO: Marurizio Loreti -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre>- TeV studies in progress. 
- Experts are working on prematch boards and XTRP. 
- Work on SVT, XFT, PSM, Cal. monitoring PCs, and COT HV in paralle. 
  </pre>


<!-- Date: Tue Nov 18 16:16:12.013 2003 -->
XTRP test is done for a moment.  There may be another this evening.
<!-- Author: akiya -->

<!-- Date: Tue Nov 18 16:20:13.735 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174393">174393</A>
ACTIVATE: Starting Cosmics_NOTRACKS
<!-- Author: Dale x2080 -->
<!-- Comment: Tue Nov 18 16;35;44 comment by...Dale -->
<P> 
Kaori suggested that I test something that may have been a problem before.  After selecting a partition, we went to 'Proxy' in Run Control and went to 'ConsumerMonitor'.  This brought up the ConsumerMonitor Proxy Viewer.  I was told to hit 'mode' once.  This Changed the Green/Yellow flashing 'Parition_5' to a light blue (teal color).  Kaoria explained that this was for SVX which wasn't  included in COSMICS_NOTRACKS.  Thus, we don't know the effect of this (whether it helped the orginal problem or not). </P>
<!-- Comment: Tue Nov 18 16;38;32 comment by...Dale -->
(Clarify) The problem was an error message that keeped on occuring.  Also note that this was done before starting the run.
<!-- Comment: Wed Nov 19 08;18;17 comment by...SCN -->
Same as<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=view&page=961&frame=2&anchor=12:43:17.196'>seen on Monday</a>...they are working on it I am told.
<!-- Date: Tue Nov 18 16:28:33.581 2003 -->
We have error message in Global Alram Monitor. <br> 
"Error Number 1994(0000077a)" <br> 
"BOLTMAN PSM_STAFF_CV connection NOT established with node"  
<!-- Author: Taka Maruyama -->

<!-- Date: Tue Nov 18 16:30:25.491 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174393">174393</A>
TERMINATE: Stoping to do CM LED calibration
<!-- Author: Dale -->



<!-- Date: Tue Nov 18 16:37:46.848 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174394">174394</A>
ACTIVATE: CEM LED run in D-mode for EMTiming system
<!-- Author: Max x2120 -->




<!-- Date: Tue Nov 18 16:40:41.576 2003 -->
<pre>Eric James is going to be swaping (reseting) some muon trigger cards (we won't start a run while he doing that [calibration run continues]).</pre>
<!-- Author: Dale -->
<!-- Comment: Tue Nov 18 16;47;15 comment by...Dale -->
replace trigger cards with Prematch board
<!-- Date: Tue Nov 18 16:45:33.084 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-12078&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-12078  width='300' align='top'></td><td>Test of Ace ability to losses plot in e-log.</a></td></tr></table>
<!-- Author: Taka Maruyama -->



<!-- Date: Tue Nov 18 16:59:37.430 2003 -->
Finished with XFT tests.  Something I did along the way made the XFT crates unhappy and they wouldn't pass "Partition" until I rebooted them all.  Now everything seems happy.
<!-- Author: K. Lannon -->


<!-- Date: Tue Nov 18 17:01:04.878 2003 -->
Error message in BigBrother display and paged Online Computer pager
<!-- Author: akiya -->
<!-- Comment: Tue Nov 18 17;53;50 comment by...akiya -->
The status in BigBrother is all green now.
<!-- Date: Tue Nov 18 17:06:28.902 2003 -->
Kevin called.  His work on XFT completed.
<!-- Author: akiya -->


<!-- Date: Tue Nov 18 17:07:53.488 2003 -->
<pre>We attempted to run COSMIC_NOTRACKS but L3 went bad before the run.  We got it to start and then cleaned the L3 farm.</pre>
<!-- Author: Dale -->


<!-- Date: Tue Nov 18 17:17:30.481 2003 -->
ICICLE crashed.  Maruyama is trying to restart it.
<!-- Author: akiya -->
<!-- Comment: Tue Nov 18 17;24;35 comment by...Taka -->
ICICLE window has errro message for this: <br>
(1) The instruction at "0x7800cb04" referenced memory at "0x7815c012". The memory could not be read. <br>
<!-- Comment: Tue Nov 18 17;27;50 comment by...Taka Maruyama -->
Instruction said that we should simply ctrl-c and restart the 
ICICLE with double clicking. Therefore, I follow the instruction.
(Restarted it)
<!-- Comment: Tue Nov 18 17;28;36 comment by...Taka Maruyama -->
Now ICICLE looks fine.
<!-- Date: Tue Nov 18 17:18:39.124 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174397">174397</A>
ACTIVATE: going to end - the bad crate is b0tsi00 (this is the same error as the previous attemp.
<!-- Author: Dale x2080 -->


<!-- Date: Tue Nov 18 17:19:38.371 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174397">174397</A>
TERMINATE: successfuly Aborted/reset.
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 17:19:45.008 2003 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>B0dap61.fnal.gov shows trouble on its CPU  
Error message  
red Tue Nov 18 16:55:15 CST 2003 up: 4 days, 0 users, 126 procs, load=7.29red Tue Nov 18 16:55:15 CST 2003 up: 4 days, 0 users, 126 procs, load=7.29</pre></td></tr></td></table>
<!-- Author: Dale -->
<!-- Comment: Tue Nov 18 18;21;08 comment by...Dale -->
<pre>Problem has gone away on b0dap61 (after about 45 min).  The expert seem to think it was not a problem and to ingore it at this time of day.</pre>
<!-- Date: Tue Nov 18 17:26:13.979 2003 -->
<b>PSM Changes</b><br> 
In an attempt to troubleshoot the mysterious dataline 2 "flipped bit" problem, I have done the following:<br> 
1. Moved dataline 2 to voltman comm port 1 (was comm port 2).  RESULT:  Squat.<br> 
2. Moved dataline 2 to comm port 10 and dataline 10 to comm port 1.  This means that dataline 2 is now running through the 8 channel rs485, and dataline 10 is running through the little black rs485 box.  RESULT:  (Apparent) Success!!!  Dataline 10 (which now monitors the 2nd floor trigger room) shows no symptoms of the old dataline 2.  And dataline 2 shows no symptoms of the old dataline 2.  <p> 
CMX_NW_1 appears to have lost its analog address of 74.  Probing dataline 2 shows that there is no address 74, but there is now an address 10--which is most likely coming from CMX_NW_1.  I've emailed Roberto about it.
<!-- Author: dan ryan -->
<!-- Comment: Tue Nov 18 17;33;16 comment by...Taka Maruyama -->
So, the expert said that we can disable the PSM alarm from CMX_NW_1. And did disable.
<!-- Date: Tue Nov 18 17:29:54.701 2003 -->
<pre>We attempted to run again and reached the 'active' run state via the L3 monitor.  However, Run Control seem to freeze (we suspect that the evb went bad).  It showed that we had not reached active. 
 
We could not abort, end, or reset (it was frozen).  We closed RunControl.  We restarted but we could not 'rejoin orphanged parition'.  (to be continued...)</pre>
<!-- Author: Dale -->


<!-- Date: Tue Nov 18 17:33:04.744 2003 -->
<pre>For the last three attemps we have had Icicle restart ask for (the Mon. Ace  has fix/restarted this) 
 
The error message: 
 
(MLE) b0dap73.fnal.gov:Thread-25:5:18:25 PM->SlowControl Icicle heart beat not updated ever!</pre>
<!-- Author: Dale -->


<!-- Date: Tue Nov 18 17:36:15.528 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174398">174398</A>
ACTIVATE: Starting COSMIC_NOTRACKS.

Eric James disabled the error line for b0trmu00.  Explaination of the problem to follow below.
<!-- Author: Dale x2080 -->
<!-- Comment: Tue Nov 18 17;45;39 comment by...Dale -->
<pre>We got the following message which indicates that the muon trigger (no b0tsi00) was the bad guy.

CT: 2003.11.18 16:58:44
Partition:5 Run number: 174396 (0x2a93c)
(RC)   16:59:18 Config -> READY
(MLE) b0tsi00:Messenger:4:59:23 PM->TS failed when trying to Run (State 2)
CT: 2003.11.18 16:59:26
59'26" 1 crate/s: b0mutr00(104),  in error.[RXPT]
(MLE) b0dap73.fnal.gov:Thread-3770:4:59:25 PM->FrontEnd Crate Error Condition from:

(MLE) b0dap73.fnal.gov:Thread-3770:4:59:25 PM->FrontEnd Error: </pre>
<!-- Comment: Tue Nov 18 17;48;53 comment by...Dale -->
<pre>Note of course this run was running fine (in other words, disabling the error line for the muon trigger worked - but there may be a more pressing problem).</pre>
<!-- Date: Tue Nov 18 17:42:45.327 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174398">174398</A>
TERMINATE: Ending run.  One of the boards in b0trmu00 (pre-match board 0) may be bad.
<!-- Author: Dale x2080 -->






<!-- Date: Tue Nov 18 18:00:01.396 2003 -->
<pre>Here is a summary of work done with consumer online monitors today. 
 
New cdfsoftb0 5.2.0int2c release was built, declared to be 'current default'. 
TrigMon, BeamMon, SiliMon does not work with this new version, so  
in the startup script, they are set to run with 5.1.0c (previous default) 
version.  Problem with BeamMon and SiliMon is at the execution time. 
They both have the same problem and seems to be associated with the 
new SVX module which is needed for SVXMon. (SVXMon people notified). 
TrigMon problem is in 'built' stage.  This is known for a while and 
being looked at by Charles, et. al. 
Main changes with the new version is: 
- ConsumerErrorReceiver fix is in (not tested yet, since silicon is not  
  included) 
- YMon has IMU fix, Olga's updates are in, but not turned on in tcl. 
- SVXMon is linked with new code, but not tested yet (no silicon data, yet). 
- Base release contains David D's new TDC DSP readout code, but not Aseet 
  CTHitSet fix, yet.  (that will be in 5.1.2 release, which will be built 
  tonight). 
In order to test monitors more, we need to: turn on HV and include COT/Silicon. 
 
We were also having problem with huge log files filling up /data1 disk coming 
from BeamMon, Stage0, and ObjectMon.  BeamMon and Stage0 problem was traced 
to frequent error message due to non-existence of COTD/Q banks produced by a  
COT tracking code. Well, we (Frank and us) are not absolutely sure, but what 
we think happening is: when COT is included, even there are no hits, COTD 
bank is there (that much is sure), but in L3 when the compression happens, 
if there are no hits in COTD bank, COTQ bank is not created(?),  
while COTD bank is dropped. When COT crates are not included, the problem 
was even worse. This to cause tracking module to dump out many 
many lines of error messages.  To prevent this to happen, for now,  
we essentially turned off Stage0 and BeamMon 
processing in tcl. Therefore, when we start Stage0/BeamMon, it will starts the 
programs but they are really not doing tracking. 
Consumer status manager will report no events are processed by those 
monitors.  Before we start taking regular physics run (and cosmic run with 
COT HV on), we must modify the tcl version running.  Kohei knows how to switch 
them.  In both cases, one needs to remove Consumer/xxxMon/ structure 
under runarea. 
ObjectMon problem is slightly different.  We have turned off TimeMonitor to 
fix the problem for a moment.  ObjectMon people are working on the problem. 
 
So, we are leaving most of the online monitors related processes running  
with the newly built release.  If shift people find problems, please contact 
online monitor people via online monitor pager (630 266-2705)  
 
 
 </pre>
<!-- Author: kohei/kaori -->
<!-- Comment: Tue Nov 18 18;21;16 comment by...Rainer -->
We are planning to run some SiliGuineaPigs later in the week to test new FIB firmware. this will produce some silicon data, enough presumably to debug that data stream to SVXMon. I don't quite understand (but ain't no offline person yet) why SiliMon has a problem and maybe we should alert SiliMon people as well. We are planning to make better use of SiliMon in the near future.
<!-- Date: Tue Nov 18 18:14:10.181 2003 -->
Got call from Roberto.  He has finished his work on SVT crate.
<!-- Author: akiya -->




<!-- Date: Tue Nov 18 18:24:37.315 2003 -->
MCR called in  and warned us that they will ramp up the Tevatron.
<!-- Author: akiya -->


<!-- Date: Tue Nov 18 18:35:13.743 2003 -->
PSM Global Alarm for 1RR18E_2.
<!-- Author: Taka Maruyama -->
<!-- Comment: Tue Nov 18 18;36;34 comment by...Taka Maruyama -->
AD channels from #0 to #2 returned values are 0!!


<!-- Date: Tue Nov 18 18:44:32.800 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-12080&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-12080  width='300' align='top'></td><td>Main Injector has beam.</a></td></tr></table>
<!-- Author: Taka Maruyama -->


<!-- Date: Tue Nov 18 18:55:07.481 2003 -->
<pre>Got L3 timeouts: 
 
(MLE) b0l3pcom1.fnal.gov:main:6:52:06 PM->Error on L3 node b0l3pcom1.fnal.gov (partition 5) Tue Nov 18 18:52:06 2003 l3proxy  
  in Initialize (L3ProxyPartition.cc:1013) 
  @L3_TRANS_FAIL - initialization timeout (will clean up):     b0l3c09  b0l3c10  b0l3c11  b0l3c12  b0l3c13  b0l3c14  b0l3c15  b0l3c16 
 
We are cleaning up L3 farm.</pre>
<!-- Author: Dale -->
<!-- Comment: Tue Nov 18 18;56;50 comment by...Dale -->
<pre>got this error:

 Attention !!!. Event Builder EVB_PROXY_SM_TIMEOUT Error !!!. 
 Event Builder proxy lost connection to Scanner Manager. 
 We RECOMMEND the following steps:
 - End of run, Reset
 - Clean up EVB 

we are cleaning up evb.</pre>
<!-- Comment: Tue Nov 18 19;13;49 comment by...Dale -->
<pre>during clean up we saw that 

b0eb14 couldn't log in (it didn't know why).  We tried EVB cleanup again but same result.  

We press reset button on the SCPU for b0eb14.  Cleaned up EVB again.</pre>
<!-- Date: Tue Nov 18 18:55:24.958 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-12081&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-12081  width='300' align='top'></td><td>B0Q5 was once 0 and increased.</a></td></tr></table>
<!-- Author: Taka Maruyama -->



<!-- Date: Tue Nov 18 18:57:13.168 2003 -->
TOF Global Alarm (Error Number 1914(0000077a). <br>  
<!-- Author: Taka Maruyama -->
<!-- Comment: Tue Nov 18 18;58;27 comment by...Taka Maruyama -->
All TOF HVs status are red.  


<!-- Comment: Tue Nov 18 19;05;16 comment by...Taka -->
TOF HV is off. but status shows red.
<!-- Comment: Tue Nov 18 19;05;55 comment by...Taka -->
Expert was paged and he will come here and investigate the reason. 
<!-- Date: Tue Nov 18 19:03:14.737 2003 -->
ICICLE has same problem as before. I simply terminate the  
previous one and start again.
<!-- Author: Taka Maruyama -->




<!-- Date: Tue Nov 18 19:11:21.948 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-12082&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-12082  width='300' align='top'></td><td>Losses look small.</a></td></tr></table>
<!-- Author: Taka Maruyama -->


<!-- Date: Tue Nov 18 19:17:41.599 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174407">174407</A>
ACTIVATE: Starting COSMICS_NOTRACKS.  <P>
We have b0svt** crates in but have taken out the HADRON_TIMEING00 at Eric James request. 
(Thus the default settings for 'cosmics no tracks' without the hadron cal. crate) 
</P>
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 19:21:01.285 2003 -->
Global Alarm for PSM for 1RR18G_1. AD Channel from 0 to 2 is 0. 
  
<!-- Author: Taka Maruyama -->
<!-- Comment: Tue Nov 18 19;23;59 comment by...Dale and Taka -->
Dale confirmed that this is also due to the experts' for Hadronic calorimeter workings.
<!-- Date: Tue Nov 18 19:22:33.656 2003 -->
Global Alarm for TOF showed once red and after blue. 
We guess this is due to expert's working.
<!-- Author: Taka Maruyama -->



<!-- Date: Tue Nov 18 19:27:03.382 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-12083&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-12083  width='300' align='top'></td><td>Tevratron beam has small energy now. </a></td></tr></table>
<!-- Author: Taka  Maruyama -->

<!-- Date: Tue Nov 18 19:27:47.587 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174407">174407</A>
TERMINATE: ending COSMICS_NOTRACKS	
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 19:33:44.320 2003 -->
<pre>TOF HV PC crashed. 
PC rebooted, iFix client & SMACS software restarted. 
Standard HV setup configuration loaded. 
Everything fine now.</pre>
<!-- Author: Stefano -->

<!-- Date: Tue Nov 18 19:51:23.755 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174409">174409</A>
ACTIVATE: Starting COSMICS_NOTRACKS

<P>
We have included all VRB crates and have excluded the hadron_timeing crate.
We selected "All files to tape" as per Max and others instructions (we hope that this is a realtively long run!).
Also note that we have de-selected the ReadList option for trigger debug/tests.
</P>
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 19:54:06.177 2003 -->
<pre>L3 display went red again and we had to clean up L3 farm again. 
 
We got "consumer monitor missing" alarm - aka, "ConsumerMonitor Partition_5 is not running!" during our first atemp to include the VRB crates.  Of course we got this about a 100 times before we were able to bring up the the ConsumerMonitor Proxy Viewer and select mode.  This gave us the alarm message once and the additional (on purpose) error:  
 
Using Silicon with Rate Limiting disabled!</pre>
<!-- Author: Dale -->

<!-- Date: Tue Nov 18 19:55:57.487 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174409">174409</A>
TERMINATE: ending COSMICS_NOTRACKS to take L2 torture
<!-- Author: Dale x2080 -->
<!-- Comment: Tue Nov 18 20;06;54 comment by...Dale -->
<P>
<I> No we didn't go crazy.</I> - We corrently saw that L2 is of course <U> not </U> an option via the white board.  We are going to consult Steve to find out if we could do L2 for a short while (the trigger guys need soming with a much faster rate - cosmics is getting it done).

</P>  

<!-- Comment: Tue Nov 18 20;07;36 comment by...Dale -->
<pre>cosmics NOT getting it done</pre>


<!-- Comment: Tue Nov 18 20;13;53 comment by...Dale -->
<pre>Steve gave us the green light for a short (~5 min) L2 torture.  We are going to include the VRB crates again.</pre>
<!-- Date: Tue Nov 18 20:09:21.055 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-12084&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-12084  width='300' align='top'></td><td>Tevatron Energy is going up and there is small beam loss  increasing due to this. </a></td></tr></table>
<!-- Author: Taka Maruyama -->


<!-- Date: Tue Nov 18 20:09:59.801 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-12085&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-12085  width='300' align='top'></td><td>Beam losses.</a></td></tr></table>
<!-- Author: Taka -->


<!-- Date: Tue Nov 18 20:10:45.554 2003 -->
Paged Op. Manager and got permision to run short time L2 torture 
for test.
<!-- Author: akiya -->



<!-- Date: Tue Nov 18 20:17:18.119 2003 -->
<pre>L3 went bad again (during the start of paritioning).  Clean up underway.  Succesful.</pre>
<!-- Author: Dale -->


<!-- Date: Tue Nov 18 20:32:32.322 2003 -->
<pre>Lots of errors when we started the run - namely: 
 
Silicon resonance dector fired upon run start. 
 
Here is the error list: 
 
FrontEnd Crate Error Condition from: 
VRB_SVX_02 
(MLE) b0wcal05:Messenger:8:23:45 PM->Runtime Error 1, Event 1: Bunch counter mismatch, mismatch count = 1 
(RC)   20:23:50 Activate -> ACTIVE 
1 crate/s: b0svx02(112),  in error.[RXPT] 
(MLE) b0dap73.fnal.gov:Thread-6167:8:23:52 PM->FrontEnd Error: VRB_SVX_02  
L1 Done Timeout.[RXPT] 
(MLE) b0dap73.fnal.gov:Thread-6167:8:23:58 PM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00 
1 crate/s: b0svx02(112),  in error.[RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error.[RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error.[RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error.[RXPT] 
 
(MLE) b0svx02:Messenger:8:24:10 PM->SRC Fatal Error:Sl 5 Resonance Detected </pre>
<!-- Author: Dale -->
<!-- Comment: Tue Nov 18 20;37;37 comment by...Dale -->
<pre>Going to start L2 torture without the VRB crates.  We also put back the SVT_Mode line to the default for L2 torture (hw=stephen4).

Unforinately the L3 farm went down on our first attemp.  Cleaned up L3 farm.

Getting ready for 5 min L2 toture</pre>
<!-- Date: Tue Nov 18 20:35:01.305 2003 -->
We have ICICLE problem once more. 
There are 3 times in this shift. 
We may have serious problem?  
<!-- Author: Taka Maruyama -->
<!-- Comment: Tue Nov 18 20;35;53 comment by...Dale and Taka -->
Dale restarted the ICICLE. At present, we have no errors.



<!-- Date: Tue Nov 18 20:41:50.483 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174413">174413</A>
ACTIVATE: Starting L2 torture (with Steve's permission)
<P>
We selected Look Area to send all files.  Readout List is unselected.  (both come at the request of Cheng-Ju)
</P>
<!-- Author: Dale x2080 -->
<!-- Comment: Tue Nov 18 20;47;10 comment by...Dale -->
<pre>Lots of Bunch counter mismatches.

L2 triger rate droped to 0 events.  We have the run in halt as we short things out.

Errors:

(MLE) b0xft00:Messenger:8:37:29 PM->Event 1041: Bunchcounters in slot 13 (BC=0) and slot 8 (BC=80) disagree (L2B=1) 
(MLE) b0xft00:Messenger:8:37:29 PM->Runtime Error 1, Event 1041: Bunch counter mismatch, mismatch count = 1
(MLE) b0xft00:Messenger:8:37:49 PM->Event 7119: Bunchcounters in slot 13 (BC=18) and slot 8 (BC=121) disagree (L2B=0) 
(MLE) b0xft00:Messenger:8:37:49 PM->Runtime Error 2, Event 7119: Bunch counter mismatch, mismatch count = 1
(MLE) b0xft00:Messenger:8:37:50 PM->Event 7185: Bunchcounters in slot 13 (BC=106) and slot 7 (BC=24) disagree (L2B=0) 
(MLE) b0xft00:Messenger:8:37:50 PM->Runtime Error 3, Event 7185: Bunch counter mismatch, mismatch count = 1
(MLE) b0xft02:Messenger:8:37:50 PM->Event 7437: Bunchcounters in slot 13 (BC=6) and slot 5 (BC=77) disagree (L2B=0) 
(MLE) b0xft02:Messenger:8:37:50 PM->Runtime Error 1, Event 7437: Bunch counter mismatch, mismatch count = 1
(MLE) b0xft02:Messenger:8:37:52 PM->Event 7999: Bunchcounters in slot 13 (BC=30) and slot 11 (BC=124) disagree (L2B=0) 
(MLE) b0xft02:Messenger:8:37:52 PM->Runtime Error 2, Event 7999: Bunch counter mismatch, mismatch count = 1
(MLE) b0xft04:Messenger:8:37:53 PM->Event 8327: Bunchcounters in slot 13 (BC=124) and slot 10 (BC=15) disagree (L2B=0) 
(MLE) b0xft04:Messenger:8:37:54 PM->Runtime Error 1, Event 8327: Bunch counter mismatch, mismatch count = 1
(MLE) b0xft00:Messenger:8:38:01 PM->Event 10603: Bunchcounters in slot 13 (BC=127) and slot 9 (BC=65) disagree (L2B=0) 
(MLE) b0xft00:Messenger:8:38:01 PM->Runtime Error 4, Event 10603: Bunch counter mismatch, mismatch count = 1
(MLE) b0xft00:Messenger:8:38:15 PM->Event 14711: Bunchcounters in slot 13 (BC=3) and slot 10 (BC=139) disagree (L2B=0) 
(MLE) b0xft00:Messenger:8:38:15 PM->Runtime Error 5, Event 14711: Bunch counter mismatch, mismatch count = 1
(MLE) b0xft04:Messenger:8:38:23 PM->Event 16998: Bunchcounters in slot 13 (BC=3) and slot 5 (BC=18) disagree (L2B=0) 
(MLE) b0xft04:Messenger:8:38:23 PM->Runtime Error 2, Event 16998: Bunch counter mismatch, mismatch count = 1
L2 Decision Timeout.[RXPT]</pre>

<!-- Date: Tue Nov 18 20:57:34.403 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174413">174413</A>
TERMINATE: Ending run - many L2 decision timeout errors.  Investigating how to minicome into L2 crates.
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 21:00:34.394 2003 -->
<table border=0><td><tr><td bgcolor=#e0e030><h3> Debugging the Ol' SRC for the Silicon </h3> 
 
We would like to include the SRC to 'parasitically' spy on any DAQ activity such that we can maybe catch that error which was plaguing us for some time. To this end I have produced run configurations in RC for the right now two most common settings: <p> 
<li> L2TORTURE_SRCDEBUG 
<li> ACE_COSMICS_NOTRACKS_SRCDEBUG 
<p> 
please use these settings if you were intending to either take L2TORTURE or COSMICS_NOTRACKS. In the event the SRC pulls <pre> Fatal Error: SL 5 L2 w/o L1</pre> please put the DAQ in halt and page Sal Rappoccio 7227599 and wait for his arrival. inform the main pager 
as well. <p> 
if the SRC pulls <pre>Fatal Error Sl 5 Resonance Detected </pre> 
a resonant trigger condition was detected. try to HRR one time but if it happens again, revert to the respective standard setting (w/o _SRCDEBUG) and make a note in the logbook. 
<p> 
What it means: We have added all the VRB crates (b0svx0X, X=0...8) to the standard settings and made the SRC forget about the fact that the silicon is OFF and no silicon is read out. <b>PLEASE DO NOT INCLUDE FIB crates (b0fib0X, X=0...7) by hand  at this time.</b> (They would not clear config but don't try anyway). This run setting is highly unusual and only permissible in a state where the silicon is OFF.  Don't be alarmed about the message <pre> 
Using Silicon with Rate Limiting disabled! 
</pre> 
some safety features needed to be disabled to make these runs work. The message stays as a reminder. 
 
Aces should forget about the existence of these settings once regular data taking is established.  
</td></tr></td></table>
<!-- Author: Rainer -->

<!-- Date: Tue Nov 18 21:12:38.443 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174415">174415</A>
ACTIVATE: Test L2 torture run.
<!-- Author: Dale x2080 -->


<!-- Date: Tue Nov 18 21:13:25.600 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174415">174415</A>
TERMINATE: Ending L2 torture due to L2 DTO.
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 21:13:41.815 2003 -->
Steve Kuhlmann is here to start L2 Isolation test.
<!-- Author: akiya -->

<!-- Date: Tue Nov 18 21:15:42.610 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174416">174416</A>
TERMINATE: End run due to L3 proxy crashing.
<!-- Author: Dale x2080 -->


<!-- Date: Tue Nov 18 21:19:42.267 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174417">174417</A>
ACTIVATE: Starting L2 torture run
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 21:20:22.122 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12086&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12086' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-12087&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-12087' width='300'></a></center>
<!-- Author: Taka Maruyama -->
<!-- Comment: Tue Nov 18 21;20;53 comment by...Taka -->
Hourly, plot.

<!-- Date: Tue Nov 18 21:25:07.531 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174417">174417</A>
TERMINATE: Ending L2 torture run
<!-- Author: Dale x2080 -->


<!-- Date: Tue Nov 18 21:30:57.671 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174418">174418</A>
TERMINATE: End run, Level-3 proxy died during Config (L3 EXPERT, please fix this!!!!)
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 21:32:23.811 2003 -->
Global Alarm for PSM. 2RR28C_1 has error, but automatically  
recovered. 
<!-- Author: Taka Maruyama -->

<!-- Date: Tue Nov 18 21:36:34.207 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174419">174419</A>
TERMINATE: Ending run
<!-- Author: Dale x2080 -->


<!-- Date: Tue Nov 18 21:36:35.896 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174419">174419</A>
ACTIVATE: End run
<!-- Author: Dale x2080 -->


<!-- Date: Tue Nov 18 21:40:14.710 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174420">174420</A>
ACTIVATE: Start run
<!-- Author: Dale x2080 -->


<!-- Date: Tue Nov 18 21:47:58.664 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174420">174420</A>
TERMINATE: ending L2 torture.  No more L2 Torture (unless given permission via Steve Hahn).
<!-- Author: Dale x2080 -->


<!-- Date: Tue Nov 18 21:58:07.291 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174421">174421</A>
ACTIVATE: Starting ACE_COSMICS_NOTRACKS_SRCDEBUG [COSMICS_NOTRACKS with VRB* crates as directed above] <br>
We also have Readout list unchecked and b0xtrp00 is includeded in the run.
<!-- Author: Dale x2080 -->
<!-- Comment: Tue Nov 18 21;59;58 comment by...Dale -->
<pre>Of course, L3 stoped again just as we were ending the L2 torture.  Restarted.  </pre>


<!-- Date: Tue Nov 18 22:04:51.225 2003 -->
Global alarm of CEM shows yellow. <br> 
phi17R eta1,3,4,5,7,8 and phi17L eta3,4,6 <br> is  
flashing (between read and white). 
<!-- Author: Taka Maruyama -->
<!-- Comment: Tue Nov 18 22;06;00 comment by...Taka  -->
Also, HV summary of CEM shows yellow.


<!-- Comment: Tue Nov 18 23;09;43 comment by...akiya -->
Paged CEM HV expert but ACEs took about an hour to learn how to fix it through a telephone.  We need a good documentation.
<!-- Date: Tue Nov 18 22:12:42.678 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-12088&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-12088  width='300' align='top'></td><td>This is CEM PMT HV summary. </a></td></tr></table>
<!-- Author: Dale and Taka -->
<!-- Comment: Tue Nov 18 23;50;05 comment by...Taka and Dale -->

<a href=
'/cgi/elog/elog.pl?nb=2003&action=view&page=965&frame=2&anchor=23:48:04.720'>
We fixed this with the aid of expert help. </a>
<!-- Date: Tue Nov 18 22:16:58.252 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174421">174421</A>
TERMINATE: Steve Kuhlmann is working on L2 and needs the run to end to replace a board.
<!-- Author: Dale x2080 -->


<!-- Date: Tue Nov 18 22:29:59.073 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174422">174422</A>
ACTIVATE: <P> Starting ACE_COSMICS_NOTRACKS_SRCDEBUG using the same modifications as <a href=
'/cgi/elog/elog.pl?nb=2003&action=view&page=965&frame=2&anchor=21:58:07.291'>
above. </a>   Note that we have switched the tape option to 'all files'. </P>
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 22:39:05.635 2003 -->
<pre>Found a bad fiber driver on IsoClique,  after replacing 
this card there have been no errors of any kind from 
L2 Isolation.  Will continue to monitor...</pre>
<!-- Author: Steve Kuhlmann -->


<!-- Date: Tue Nov 18 22:39:18.780 2003 -->
<br><b>Tevatron Beam:</b> 
<br>Please leave a note in the e-log if coalesced beam is 
observed in the Tevatron.  
<br> 
<!-- Author: R.J. Tesarek -->

<!-- Date: Tue Nov 18 23:05:26.368 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174422">174422</A>
TERMINATE: end run to enable the pulsar readout again
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 23:11:48.658 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174423">174423</A>
ACTIVATE: continue cosmics_notracks_srcdebug
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 23:16:25.709 2003 -->
<pre>L2 DTO, XTRP (0 words), HRR ok 
Cheng-Ju and I are looking into that.</pre>
<!-- Author: Burkard :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174423 ' target=_top>174423 </a>) -->

<!-- Date: Tue Nov 18 23:26:27.309 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174423">174423</A>
TERMINATE: Ending cosmic notracks.
<!-- Author: Dale x2080 -->


<!-- Date: Tue Nov 18 23:34:17.843 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174424">174424</A>
ACTIVATE: L2 torture test run
<!-- Author: Dale x2080 -->


<!-- Date: Tue Nov 18 23:37:58.395 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174424">174424</A>
TERMINATE: End special L2 torture test run.
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 23:39:54.592 2003 -->
PSM alarm for 2RR29C. This is due to the experts' work for  
pulsar.
<!-- Author: Taka -->

<!-- Date: Tue Nov 18 23:47:36.643 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174425">174425</A>
ACTIVATE: Starting Level-2 torture run
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 23:48:04.720 2003 -->
<pre>CEM is back in green state on the HV mon screen (bad channels have gone away).  The procedure for fixing this is in priciple easy - but there really should be some web documentation to fix this.  We attempted to follow the instructions that Nikolay gave to us.  We think that we did not finish doing what he asked but the cal. systems were working so we stopped.  It seemed that copying the file and running two commands via MS-DOS and then deleting 'good' values from the cem*.out was enough to get the job done.  (We did not do the same in the other directory.) 
 
In any case, we claim victory wherther or not we were the ones to actually solved the problem.</pre>
<!-- Author: Taka and Dale -->

<!-- Date: Tue Nov 18 23:48:32.797 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174425">174425</A>
TERMINATE: End l2 torture run.
<!-- Author: Dale x2080 -->



<!-- Date: Tue Nov 18 23:54:23.843 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=174426">174426</A>
ACTIVATE: start L2 torture run
<!-- Author: Dale x2080 -->


<!-- Date: Tue Nov 18 23:56:12.691 2003 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:02</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Wed Nov 19 00:02:29.879 2003 -->
<font color=red>Shift Summary: </font><pre>Summary of the shift. 
 
[Current status] 
- TeV studies all through the shift. 
- Works on XTRP, XFT and SVT crates, PSM, and L2 Isolation have been completed. 
- Prematch boards have been replaced. 
- We are running cosmic_notrack and experts are working on L2 Trigger problem. 
 
[problems] 
- Experts are working on the L2 Trigger problem, that is L2 Trigger  
  reports time out when trigger rate is as high as L2 torture, 
  but ok for low rate of cosmic_notrack.   
 
- ICICLE crashed and ACE recovered, thrre times in this shift. 
  Is this normal ? 
 
- There was Global Alarm of CEM.  We paged CEM HV expert, it took 
  about an hour to learn how to fix it.  We need a good manual ! 
 
[Notes to remember] 
- PSM alarm of CMX_NW_1 is disabled.  
  See <a href= 
'/cgi/elog/elog.pl?nb=2003&action=view&page=965&frame=2&anchor=17:26:13.979'>e-log by Dan Ryan</a>. 
 
- Kaori has implemented new version of consumer program. 
  See <a href= 
'/cgi/elog/elog.pl?nb=2003&action=view&page=965&frame=2&anchor=18:00:01.396'>e-log by Kaori</a>  
 
- There are special configurations for SRC debuging. DAQ ACE, please <a href= '/cgi/elog/elog.pl?nb=2003&action=view&page=965&frame=2&anchor=21:00:34.394'>read message from Rainer</a>. 

- <a href='/cgi/elog/elog.pl?nb=2003&action=view&page=964&frame=2&anchor=11:56:41.552'>Irina is working on Calorimeter PCs</a> which turns several buttons in the GLobal Alarm Display to grey. 
 
- Rick Tesarek requested to leave a note if coalesced beam is seen in the Tevatron. 
</pre> 
<!-- Author: Akiya Miyamoto -->