<!-- Title: Eve shift. Thu May  5, 2005 -->
<!-- SciCo: Steve Nahn -->
<!-- DAQAce: Radu Marginean -->
<!-- MonAce: Pedro Movilla Fernandez -->
<!-- CO: Mike Albrow -->
<!-- OpManager: Bob Wagner -->
<!-- Notes: -->
<h3>Recovering from access</h3> 
There are still way too many experts in the room to my taste, so my 
goal is to get rid of them, see if we can take COSMICS and L2_TORTURE, and wait for shot setup.  Not sure when that will be.

<!-- Date: Thu May 5 16:13:22.523 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197562">197562</A>
ACTIVATE: RunSet: L2TORTURE_BAREBONE  Trigger: L2_TORTURE [16,623,539] 
<!-- Author: feild -->



<!-- Date: Thu May 5 16:13:53.587 2005 -->
Asked a fellow Steve to come over and reset our Acnet terminals as 
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&action=view&page=374&frame=2&anchor=09:05:40.928'> 
requested</a> this morning. 
 
  
<!-- Author: SCN -->


<!-- Date: Thu May 5 16:15:11.150 2005 -->
<pre>TOF heartbeat alarm due to expert's intervention. We can ignore this.</pre>
<!-- Author: Pedro -->


<!-- Date: Thu May 5 16:16:05.529 2005 -->
<pre>Spent another day trying to get the new CLC multiplicity  
veto bits into the FRED card.  We need more experts! 
We'll retrun next week with Carla. 
 
</pre>
<!-- Author: greg, jonathan -->
<!-- Comment: Thu May  5 16;17;42 comment by...greg -->
<pre>The Fred card has been returned to its original state.
</pre>


<!-- Comment: Thu May  5 16;24;09 comment by...SCN -->
Chaos incarnate<ul>
<li> CMX is tripping, paged expert
<li> TOF guys are around, are they done?
<li> Source guys are checking out their procedures
<li> L3/EVB guys are waiting for Plug Guys so they can work
harmoniously on EVB upgrade
<li> JDL is trying to make a Trigger Inhibit for Flying Wires
<li> Bill is over in the corner doing who knows what
<li> Silicon guys are playing with individual ladders
<li> Greg finished with Fred
</ul>

<!-- Comment: Thu May  5 16;53;59 comment by...SCN -->
Update: One set of TOF guys are <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&action=view&page=374&frame=2&anchor=16:05:34.388'>apparently done</a>, but another set, Bauer and De Cecco, were discovered downstairs poking at a CAEN crate with screwdrivers, which
is why <font color=grey>TOF IS GREY IN IFIX</font>.  I encouraged them to describe their work verbally to the crew and textually in an elog when they are done.



<!-- Date: Thu May 5 16:21:56.307 2005 -->
<pre>CMX alarm. Miniskirt 10 tripped twice. Paged expert.</pre>
<!-- Author: Pedro -->
<!-- Comment: Thu May  5 16;34;39 comment by...Pedro -->
<pre>Expert is working on this. We are asked to leave the HV off for miniskirt 10 west until further notice.</pre>
<!-- Date: Thu May 5 16:25:06.326 2005 -->
<pre>I have rebooted all of the CDF Experiment's front end  
VME crates (except b0tsi01 and b0tsi02) both to pick up a  
slightly new version of fer (FrontEndReadout) package and  
just as a principle, since we have noticed things like  
"Heap Corrupt" errors only occur after the crates have been  
up for a while (days). 
 
The changes in fer package version v3_78 are: 
 
  . Add T:FWDEL acnet device to AcnetMonitor -- this contains  
    the flying wires state machine pre-fly delay time 
 
  . RunControl uses FWDEL to dynamically compute the auto- 
    matic HALT pre-recover delay time 
 
  . Send VxMon (/*/status/vxmon) message immediately after  
    booting a front-end crate (helps clear up crate error  
    states faster) 
 
 
   . </pre>
<!-- Author: W.Badgett -->

<!-- Date: Thu May 5 16:27:18.647 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197566">197566</A>
ACTIVATE: RunSet: CALIB_CLC_QIE --> CLC QIE quiet time calibration.
<!-- Author: Radu X2080 -->
<!-- Comment: Thu May  5 17;14;54 comment by...mike albrow -->
<pre>DBANA checks OK</pre>
<!-- Date: Thu May 5 16:28:22.932 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197564">197564</A>
TERMINATE: Test run for plug sourcing system
<!-- Author: sedov -->



<!-- Date: Thu May 5 16:30:27.974 2005 -->
<h3> Silicon access report </h3> 
Better late than never. Details can be found <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=247&frame=2&anchor=09:05:00'> 
here </a> 
 
  
<!-- Author: The Silicon Teams -->

<!-- Date: Thu May 5 16:31:10.029 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197566">197566</A>
TERMINATE: end CLC QIE calibaration.
<!-- Author: Radu X2080 -->
<!-- Comment: Thu May  5 17;16;33 comment by...mike albrow -->
<pre>DBANA check OK</pre>
<!-- Date: Thu May 5 16:37:38.052 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197567">197567</A>
TERMINATE: test run for plug sourcing system
<!-- Author: sedov -->


<!-- Date: Thu May 5 16:47:01.704 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197569">197569</A>
ACTIVATE: RunSet: CALIB_QIE_FWD: BSC QIE Calibration. 
<!-- Author: Radu x2080 -->



<!-- Date: Thu May 5 16:47:38.530 2005 -->
<pre>We have a PSM alarm: CANWT_1 channel 0 voltage is ~3.6V (should be 5V)</pre>
<!-- Author: Pedro -->
<!-- Comment: Thu May  5 17;01;01 comment by...Pedro -->
<pre>From DAQ side there is no complain about ccal01 (e.g. we tried VME Bus scan). Anyway, PSM condition turned to green after a few seconds.</pre>
<!-- Date: Thu May 5 16:48:00.463 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197569">197569</A>
TERMINATE: End BSC QIE calibration.
<!-- Author: Radu x2080 -->
<!-- Comment: Thu May  5 17;15;49 comment by...mike albrow -->
<pre>DBANA check OK</pre>
<!-- Date: Thu May 5 16:49:38.329 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197570">197570</A>
TERMINATE: test run for plug sourcing system
<!-- Author: sedov -->




<!-- Date: Thu May 5 16:56:01.516 2005 -->
I have made a flying wire trigger inhibit.  There is a Lecroy 222 module in 2RR02D that has an 8 second delay after a $CF TCLK event.  After that delay, it generates a 4 second gate.  While that gate is true, we will inhibit.  There is nominally a 10 second delay between the $CF and the wire fly, so this should bracket 2 seconds on either side.  Once we see that this works, we can remove the klugy halt sequence in Run Control.
<!-- Author: jdl -->

<!-- Date: Thu May 5 16:59:08.770 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197572">197572</A>
TERMINATE: test run for plug source system (good run?)
<!-- Author: sedov -->




<!-- Date: Thu May 5 17:02:04.777 2005 -->
<p>Following through on a request early in the day shift to 
"reset the ACnet consoles", I rebooted CNS51PC and CNS46PC. 
When the reboots were complete, I tried to bring up the TeV 
Array Display on CNS51PC as we normally do.  However, either 
starting it from the Java "Application Index" or from the  
Tevatron "Software Applications" web page always resulted in a 
crash in JAVASW.EXE.</p> 
 
<p>I then brought it up from the Java "Applications Index" on 
CNS46PC, which worked correctly.  I've talked to Ron Moore and 
he knows about this problem.</p>
<!-- Author: Steve Hahn -->


<!-- Date: Thu May 5 17:09:00.920 2005 -->
<pre>We are ignoring frequent TOF alarms due to expert's interventions.</pre>
<!-- Author: Pedro -->

<!-- Date: Thu May 5 17:10:22.401 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197573">197573</A>
TERMINATE: test run for plu sourcing system (good run?)
<!-- Author: sedov -->


<!-- Date: Thu May 5 17:11:55.746 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197574">197574</A>
ACTIVATE: RunSet: ACE_QIE Calorimeter QIE calibration.
<!-- Author: Radu x2080 -->
<!-- Comment: Thu May  5 17;26;42 comment by...mike albrow -->
<pre>No worse than example in Fix List! It says no action necessary</pre>
<!-- Date: Thu May 5 17:12:45.530 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197574">197574</A>
TERMINATE: end Calorimeter QIE calibration.
<!-- Author: Radu x2080 -->



<!-- Date: Thu May 5 17:13:22.871 2005 -->
<pre>We have finished testing the Plug sourcing system. Resources were released.</pre>
<!-- Author: Oscar Gonzalez -->



<!-- Date: Thu May 5 17:15:40.897 2005 -->
<pre>I have investigated the "b0pcal00 unresponsive error"  
during the plug source runs.   The problem is related to  
a recent change to the fer Messenger task code, which changed  
slightly the method of receiving notification of smartsockets  
messages in order to be able to have Messenger task shutdown  
cleanly from a signal from a parallel task. 
 
The Alexei's PcalSource window program blasts b0pcal00  
with smartsockets command messages at 50Hz, unlike any  
other smartsockets application.  The new Messenger method  
seems to then block out other tasks when overwhelmed with  
incoming messages.    
 
The "unresponsive" message is completely *harmless*  
for the purposes of the PCAL source runs -- please feel  
free to ignore the message during these runs -- readout  
will continue w/o problems. 
 
I will implement a solution by tomorrow. 
</pre>
<!-- Author: W.Badgett -->



<!-- Date: Thu May 5 17:23:15.168 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197576">197576</A>
 Activated at 2005.05.05 17:22:25
<!-- Author: RunControl -->


<!-- Date: Thu May 5 17:24:07.449 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197576">197576</A>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,619,538] 
<!-- Author: Radu x2080 -->




<!-- Date: Thu May 5 17:41:57.810 2005 -->
News from Run Coordinator: MI has a leak, which should take another hour or so to further isolate.  If they can fix it, shot setup (Accumulator only) should ensue shortly afterward; if not, plans may change.  
<a href="http://www-bd.fnal.gov/cgi-mach/machlog.pl?nb=runco&action=view&page=361&anchor=170802&hilite=17:08:02">Full  report</a>
<!-- Author: SCN -->

<!-- Date: Thu May 5 17:42:43.135 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197576">197576</A>
 Terminated at 2005.05.05 17:42:28
<!-- Author: RunControl -->


<!-- Date: Thu May 5 17:43:35.776 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197576">197576</A>
TERMINATE: end cosmics Run, will switch to L2 torture.
<!-- Author: Radu x2080 -->



<!-- Date: Thu May 5 17:47:02.120 2005 -->
<pre>According to expert's advice we switched miniskirt 10 west to stdby but it tripped again. We keep CMX miniskirt 10 west of for now. Expert is coming.</pre>
<!-- Author: Pedro -->
<!-- Comment: Thu May  5 18;35;49 comment by...Dale Stentz -->
<br>I've fixed it.  The channel is up and holding.  Details are given in the <a href=
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=9&frame=2&anchor=18:25:45'>
CMX e-log</a>.  Basically the cathod was (expert speak) "craping out" but I was able to "condition" it by repeative trips.  Should be good now.  Note that this problem has not been seen before in this wedge.  I suspect that the CMP work that was done around it has nothing to do with why it decided to go bad.
<br><br>
No special instructions but feel free to page me again if the wedge starts tripping again.  
<!-- Comment: Thu May  5 18;37;25 comment by...D -->
we need auto-spell check (cathode, etc.)
<!-- Date: Thu May 5 17:47:20.473 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197577">197577</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539] 
<!-- Author: Radu x2080 -->



<!-- Date: Thu May 5 17:53:19.084 2005 -->
8 L2 decission timeout so far.
<!-- Author: Radu :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197577' target=_top>197577</a>) -->


<!-- Date: Thu May 5 17:53:46.735 2005 -->
Swapped out crate-2 for TOF HV...everything seems to be functioning properly...and in the process was able to restore 
the one channel that was out of monitoring back in. For details 
see  
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=17&frame=2&anchor=17:43:46'> 
TOF-log</a>   
<!-- Author: bauer -->


<!-- Date: Thu May 5 17:53:56.635 2005 -->
<pre>Again PSM alarm for CANWT_1 channel 0 voltage is ~3.45V (should be 5V). I am going to wait a couple of minutes before further action to see if condition turns green as was the case the last time.</pre>
<!-- Author: Pedro -->
<!-- Comment: Thu May  5 18;03;18 comment by...Pedro -->
<pre>Alarm cleared itself.</pre>
<!-- Comment: Thu May  5 18;30;39 comment by...Pedro -->
<pre>Again PSM alarm for CANWT_1 channel 0 voltage is ~3.3V (should be 5V).</pre>
<!-- Comment: Thu May  5 18;36;42 comment by...Pedro -->
<pre>...and again cleared itself.</pre>
<!-- Date: Thu May 5 18:02:33.426 2005 -->
Frequent L2 decission TO during this L2 torture run.
<!-- Author: Radu :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197577' target=_top>197577</a>) -->
<!-- Comment: Thu May  5 18;06;58 comment by...Radu -->
L2 pager has been informed, he is looking into it.

<!-- Comment: Thu May  5 18;07;04 comment by...SCN -->
L2 Pager paged, Whiteson responds and is looking into it, will call
back.

<!-- Comment: Thu May  5 18;40;07 comment by...Daniel Whiteson -->
All of these come from upstream data problems, likely ISO or cluster.  
<!-- Comment: Thu May  5 23;31;58 comment by...vadim -->
As can be seen below, the problems were due to a smxr board in ccal13. both iso and clusters were innocent.
<!-- Date: Thu May 5 18:27:03.220 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197577">197577</A>
TERMINATE: end L2 torture run, we experienced frequent L2 decission timeout errors. 
<!-- Author: Radu x2080 -->



<!-- Date: Thu May 5 18:27:26.666 2005 -->
Daniel W is still looking at Frequent L2 timeouts, but doesn't need more data, so we allow Markus to take over for EVB/L3 tests.  Markus needs 10 minutes headstart to clean up, should there be a threat of beam.  
<!-- Author: SCN -->


<!-- Date: Thu May 5 18:35:33.274 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197579">197579</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539], removed b0cot12 and EVB for EVB upgrade work. 
<!-- Author: Radu x2080 -->
<!-- Comment: Thu May  5 18;43;01 comment by...Radu -->
also removed the L3 from this run.

<!-- Comment: Thu May  5 18;45;05 comment by...SCN -->
Daniel replies, the trouble is coming from upstream of the Pulsars, like in ISOLIST or whatnot.  They will continue to look at it; specifically, he will send Vadim to the control room to consult directly.

<!-- Date: Thu May 5 19:25:27.316 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197579">197579</A>
TERMINATE: end L2 torture test.
<!-- Author: Radu x2080 -->



<!-- Date: Thu May 5 19:29:09.295 2005 -->
I sent Mike home, as MCR says there is no chance of beam for at least three to four hours.  He can get here in 15 minutes if  
we need him.<BR> 
Cell Phone: 234 3234 
Home Phone: 717 8906  
<!-- Author: SCN -->
<!-- Comment: Thu May  5 19;43;36 comment by...SCN -->
<h3>Update from MCR</h3> Dwayne called and filled us in.  The leak in the MI has been confirmed, so they have to open up that section of the MI to air and go in and fix it.  If all goes perfectly, meaning they don't have to bake out the Lambertson, we could get Beam in the <B>middle of Owl shift</B> or so.  If things don't go so well, we could be looking at <font color=red><b>a few days without beam.</b></font> <P>
We will let Markus play with the new EVB, try to get Vadim to come
fix the L2 timeout issues, and if time allows Bias the Silicon and run some Cosmics, to keep the detector warm (right now using the Silicon with the new EVB ist verboten).

<!-- Comment: Thu May  5 19;47;34 comment by...SCN -->
Send Radu away, he'll return at 10:00 (he's in the trailers) 4953 to relieve Pedro.

<!-- Comment: Thu May  5 20;56;56 comment by...SCN -->
<a href="http://www-bd.fnal.gov/cgi-mach/machlog.pl?nb=runco&action=view&page=361&anchor=193327&hilite=19:33:27">Full Story</a>
<!-- Date: Thu May 5 20:08:20.869 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197584">197584</A>
TERMINATE: L2 T0 traced to SMXR in Slot 3 of b0ccal13 freezing up
<!-- Author: Radu x2080 -->
<!-- Comment: Thu May  5 20;10;05 comment by...SCN -->
Error is <pre>
b0ccal13:Messenger:May 5, 2005 8:05:34 PM->Runtime Error 1, Event 19893: ShowerMax FE board has stopped sending data, slot 3, header 0x0x3bfe338</pre>

<!-- Date: Thu May 5 20:14:13.719 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197586">197586</A>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Radu x2080 -->


<!-- Date: Thu May 5 20:15:43.653 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197586">197586</A>
TERMINATE: SMQIECALIB
<!-- Author: Pedro x2080 -->
<!-- Comment: Thu May  5 21;22;50 comment by...Pedro -->
<pre>DBANA plots look fine</pre>
<!-- Date: Thu May 5 20:24:05.983 2005 -->
Paged SMX, Mike L. is logging into ccal13 to see what gives.
<!-- Author: SCN -->

<!-- Date: Thu May 5 20:42:11.149 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197588">197588</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539] 
<!-- Author: Pedro x2080 -->
<!-- Comment: Thu May  5 20;52;19 comment by...SCN -->
Same chronic L2 Decision Timeouts, suspect the SMXR board as the source (as before).



<!-- Date: Thu May 5 21:04:14.829 2005 -->
<pre>(Silenced) voltage alarm in Generator Alarming page (generator 1).</pre>
<!-- Author: Pedro -->
<!-- Comment: Thu May  5 21;15;02 comment by...SCN -->
This was due to Power Cycling the 400 Hz on the SE ARCH, as per Lindgren instructions.  On to SMXR calibration...
<!-- Date: Thu May 5 21:10:53.697 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197591">197591</A>
TERMINATE: L2Torture
<!-- Author: Pedro x2080 -->


<!-- Date: Thu May 5 21:11:39.590 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197592">197592</A>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Pedro x2080 -->


<!-- Date: Thu May 5 21:12:35.095 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197592">197592</A>
TERMINATE: SMQIECALIB
<!-- Author: Pedro x2080 -->
<!-- Comment: Thu May  5 21;22;22 comment by...Pedro -->
<pre>DBANA plots are fine.</pre>
<!-- Date: Thu May 5 21:21:29.521 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197593">197593</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539] After 400 Hz Cycling

<!-- Author: Pedro x2080 -->


<!-- Date: Thu May 5 21:21:29.226 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197593">197593</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539] After 400 Hz Cycling

<!-- Author: Pedro x2080 -->




<!-- Date: Thu May 5 21:29:06.369 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197593">197593</A>
TERMINATE: Same L2DTOs
<!-- Author: Pedro x2080 -->


<!-- Date: Thu May 5 21:41:26.703 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197595">197595</A>
ACTIVATE: RunSet: SMQIECALIB 
<!-- Author: Shift Crew 2080 -->



<!-- Date: Thu May 5 21:42:17.500 2005 -->
<pre>PSM  yellow condition (>>communication error: no response from PS<<) for numerous racks in the first floor and half of the arches modules. Might be related to the previous power cycling of the arches(?)</pre>
<!-- Author: Pedro -->

<!-- Date: Thu May 5 21:46:27.465 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197595">197595</A>
TERMINATE: Looks Good
<!-- Author: Shift Crew 2080 -->



<!-- Date: Thu May 5 22:24:56.336 2005 -->
<pre>SE arches were power cycled. After that, we see communication loss for west central arches 06W-17W and east central arches 06E-17E and also CASWT_{1,2} and CASWB_{1,2} and CASET_{1,2} and CASEB_{1,2}. 
 
Expert Mike is on the way. 
 
PSM alarm on 1st Floor Power Supplies:  1RR21E_2 is red on the IFIX, channel 2 should be at -5.2V reads at -1.26 V (Alarm code 5). 
We also see the following crates in purple (communication lost): 
1RR18I1, 1RR18F2, 1RR12E3, 1RR12D1, 1RR12D5, 1RR12C4. 
 
Who is the expert for this? 
 
</pre>
<!-- Author: Radu and Pedro -->


<!-- Date: Thu May 5 22:42:09.262 2005 -->
<h4>Situation</h4> 
We have power cycled the 400 Hz on the SE Arch twice, both times 
running SMXQIE calibrations afterwards, which looked acceptable.  However, the persistent L2 Decision Timeouts persist.  I called L2 Pager to disable the check in the pulsar, and now we get DONE timeouts from b0ccal13 as before.<pre> 
(MLW) b0ccal13:Messenger:May 5, 2005 10:12:20 PM->Runtime Error 1, Event 156522: ShowerMax FE board has stopped sending data, slot 3, header 0x0x3bfe338 
(MLW) b0ccal13:Messenger:May 5, 2005 10:13:14 PM->Runtime Error 3, Event 166848: ShowerMax FE board has stopped sending data, slot 5, header 0x0x3bfe52c 
(MLW) b0ccal13:Messenger:May 5, 2005 10:13:28 PM->Runtime Error 6, Event 166851: ShowerMax FE board has stopped sending data, slot 3, header 0x0x42003f 
(MLW) b0ccal13:Messenger:May 5, 2005 10:13:28 PM->Runtime Error 8, Event 166851: ShowerMax FE board has stopped sending data, slot 4, header 0x0x51003c 
(MLW) b0ccal13:Messenger:May 5, 2005 10:13:28 PM->Runtime Error 10, Event 166851: ShowerMax FE board has stopped sending data, slot 5, header 0x0x480039 
(MLW) b0ccal13:Messenger:May 5, 2005 10:19:54 PM->Runtime Error 13, Event 166852: ShowerMax FE board has stopped sending data, slot 3, header 0x0x220033 
</pre> 
In addition, we've added a few errors <pre> 
 (MLE) b0ccal13:Messenger:May 5, 2005 10:13:29 PM->VISIONread/write failed in slot 27462941 
(MLE) b0ccal13:Messenger:May 5, 2005 10:13:29 PM->VISIONread/write failed in slot 0 
(MLE) b0ccal13:Messenger:May 5, 2005 10:13:29 PM->Event 166851: Bunchcounters in slot 17 (BC=26) and slot 3 (BC=63) disagree (L2B=0) 
(MLE) b0ccal13:Messenger:May 5, 2005 10:13:29 PM->VISIONread/write failed in slot 33296000 
(MLE) b0ccal13:Messenger:May 5, 2005 10:13:29 PM->VISIONread/write failed in slot 0 
(MLE) b0ccal13:Messenger:May 5, 2005 10:13:29 PM->Event 166851: Bunchcounters in slot 17 (BC=26) and slot 4 (BC=60) disagree (L2B=0) 
(MLE) b0ccal13:Messenger:May 5, 2005 10:13:29 PM->VISIONread/write failed in slot 33296000 
(MLE) b0ccal13:Messenger:May 5, 2005 10:13:29 PM->VISIONread/write failed in slot 0 
(MLE) b0ccal13:Messenger:May 5, 2005 10:13:30 PM->VISIONread/write failed in slot 0 
(MLE) b0ccal13:Messenger:May 5, 2005 10:13:30 PM->Event 166851: Bunchcounters in slot 17 (BC=26) and slot 5 (BC=57) disagree (L2B=0) 
(MLE) b0ccal13:Messenger:May 5, 2005 10:13:30 PM->Runtime Error 12, Event 166851: Bunch counter mismatch, mismatch count = 3 
(MLE) b0ccal13:Messenger:May 5, 2005 10:19:54 PM->VISIONread/write failed in slot 27462941 
(MLE) b0ccal13:Messenger:May 5, 2005 10:19:54 PM->VISIONread/write failed in slot 0 
(MLE) b0ccal13:Messenger:May 5, 2005 10:19:54 PM->Event 166852: Bunchcounters in slot 17 (BC=137) and slot 3 (BC=51) disagree (L2B=0) 
(MLE) b0ccal13:Messenger:May 5, 2005 10:19:54 PM->Runtime Error 15, Event 166852: Bunch counter mismatch, mismatch count = 1 
</pre> 
Not sure if these are related or not, they look a lot like  
the errors from <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&action=view&page=372&frame=2&anchor=23:33:39.489'>yesterday</a>.
<!-- Author: SCN -->
<!-- Comment: Thu May  5 22;45;26 comment by...SCN -->
Plan<ul>
<li> Mike L and Karen B arriving for access (not yet granted, but likely) to look at SMXR board.
<li> Paged Cal pager about mysterious VME read/write failures, just
to check (before we go into access).
</ul>

<!-- Date: Thu May 5 23:01:25.180 2005 -->
<h1><font COLOR=GREEN>PSM IS NOW ALL GREEN</font></h1> 
because we unplugged and replugged the cable.  And now  
both Aces (and the SciCo) remember where the box is.
<!-- Author: SCN and crew -->
<!-- Comment: Fri May 27 03;56;43 comment by...Fabrizio -->
<pre>which cable?I am experiencing problems with PSM, and rebooting VOLTMAN do not help</pre>
<!-- Date: Thu May 5 23:17:42.932 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197604">197604</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539] With the EVB
<!-- Author: Pedro x2080 -->



<!-- Date: Thu May 5 23:55:16.238 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:01</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Fri May 6 00:04:36.063 2005 -->
<font color=red>Shift Summary: </font><h4>Access recovery</h4> 
Managed to get the room free of almost all experts except Markus. 
Much was done, see day shift and log for details.<BR> 
Spent a long time chasing down a L2 Decision Timeout traced to 
problems with ShowerMax in b0ccal13.  The problem showed up as a high rate 
of L2 decision timeouts in L2 torture running (40 kHz/250 Hz), 
but doesn't seem to  be a big problem at "Physics High Rates" of 
26 kHz/300 Hz.  Looks like we're going in. 
 
<h4>Shot Setup ...</h4> 
on hold due to MI leak, which is now fixed.  Most likely late in 
Owl shift (?) <BR> 
<h3>For Owl</h3> 
<ul> 
<li><I>Consumers are in COSMICS mode, and should be changed to Physics 
Mode before Physics</i> 
<li> Page L2 to re-enable "full diagnostic mode" once the SMXR situation is resolved. 
</ul> 
                                                                                                               
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   Nada
Delivered Luminosity   0  
Acquired Luminosity    0  
Efficiency             
   
</td></tr></table></pre>
<!-- Author: SCN -->

<!-- Date: Fri May 6 00:12:52.506 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197606">197606</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539] 
<!-- Author: Michael x2080 -->