<!-- Title: Day shift. Wed May 16, 2001 -->
<!-- SciCo: Pekka Sinervo -->
<!-- DAQAce: Nancy Lai -->
<!-- MonAce: Nancy Lai -->
<!-- CO: &nbsp; -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre>Continue with controlled access with SVX cabling continuing in the hall.  Shower max PROM's will be replaced in all the boards.  Specific requests are: 
<ul> 
<li> Wall calorimeter (Steve Vejcik) for at least 4 hours 
<li> West plug for laser run (Deigo Cauz/Vaia Papadimitriou) for 4 hours 
<li> COT crates for shift (Jay Dittman) 
<li> CMX00 for shift (Trevor Vickey) 
</ul> 
We'll try to run cosmics in between. </pre>


<!-- Date: Wed May 16 08:50:33 2001 -->
<pre>Level 3 fails partition transition with incomplete start  
message on one of the connverters. It happened twice  
when I was trying to use all the subfarms.  
 
I've switched to 4 subfarms and unplugged the subfarms  
which failed before (c02, c06 and c13). It works.  
 
The problem is not understood yet.</pre>
<!-- Author: Boris.  -->


<!-- Date: Wed May 16 08:50:58 2001 -->
This morning, Steve V. needs wcal03, b0l1cl04, and b0l2cl04 for testing.  We can reach him in his office (x8416).
<!-- Author: Pekka -->


<!-- Date: Wed May 16 09:02:58 2001 -->
<pre>Ending the run to take out wcal03, l1cl04, l2cl04, and add in 
imu00 which is not being used.   
 
Released ccal 04, 05, 12, and 13 for Mike L. and Karen.   
They will be using them for about an hour. 
 
Will use only 4 L3 subfarms until Boris gives us the green light 
to include others.</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114835' target=_top>114835</a>) -->
<!-- Comment: Wed May 16 10;13;42 comment by...Nancy -->
<pre>These subfarms are 0, 4, 8, and 13 (the corresponding converter
nodes are b0l3c01, 05, 09, and 14.</pre>
<!-- Date: Wed May 16 09:09:13 2001 -->
<pre>Getting bunch counter errors in imu00: 
 
(MLE) b0imu00:Messenger:9:07:38 AM->Runtime Error 2500, Event 5729: Bunch counter mismatch, mismatch count = 1 
(MLE) b0imu00:Messenger:9:07:38 AM->Event 5731: Bunchcounters in slot 11 and slot 19 disagree (at least) 
(MLE) b0imu00:Messenger:9:07:57 AM->Runtime Error 3000, Event 6769: Bunch counter mismatch, mismatch count = 1 
(MLE) b0imu00:Messenger:9:07:57 AM->Event 6771: Bunchcounters in slot 11 and slot 19 disagree (at least)                                   
</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114836' target=_top>114836</a>) -->


<!-- Date: Wed May 16 09:09:50 2001 -->
<pre>Done timeout in imu00 after 10K events.  HRR.</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114836' target=_top>114836</a>) -->
<!-- Comment: Wed May 16 09;10;44 comment by...Nancy -->
<pre>Again after 11.5 K. HRR.</pre>

<!-- Comment: Wed May 16 09;12;15 comment by...Nancy -->
<pre>And at 13K.  End the run and remove the crate.</pre>


<!-- Date: Wed May 16 09:22:21 2001 -->
<pre>Silicon alarm - racks alarm this time.  Steve N. is 
looking into it.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 09:23:19 2001 -->
<pre>Take out wcal 00-02, 04-07 for Nikolay.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 09:26:18 2001 -->
<pre>Remove cmx00 for Trevor.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 09:35:54 2001 -->
<pre>There are problems with starting new sessions of Run Control. 
The main shift RC is still working, apparently. 
 
RC experts contacted.  Arnd is investigating and Frank will come 
to the control room shortly.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 09:39:12 2001 -->
Starting cabletest testing with wcal03,l2cl04,l1cl04
<!-- Author: Steve Vejcik -->


<!-- Date: Wed May 16 09:40:10 2001 -->
<pre>Level 3 manager showing up yellow, although the run is proceeding 
and taking data.  l3m is the only item in the Replies and  
Acknowledgments window you can't get more information about by 
right-clicking.</pre>
<!-- Author: Nancy -->
<!-- Comment: Wed May 16 09;44;26 comment by...Nancy -->
<pre>This is Run 114837.</pre>
<!-- Date: Wed May 16 09:42:02 2001 -->
<pre>Arnd restarted the RT server and RC sessions can be restarted 
again.  </pre>
<!-- Author: Nancy -->



<!-- Date: Wed May 16 09:55:51 2001 -->
<pre>Remove ccal 06, 07, 14, and 15 for Mike L. / Karen. 
 
Adding ccal 04, 05, 12, and 13 because they do not need it  
anymore.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 10:03:30 2001 -->
<pre>Printing out the LED calibration plots for Vaia.  Run 114818.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 10:08:18 2001 -->
<pre>Plug HV birds.  They're going away now.</pre>
<!-- Author: Nancy -->



<!-- Date: Wed May 16 10:22:52 2001 -->
<pre>Silicon cooling alarm.  Went away right away.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 10:34:50 2001 -->
<pre>End Run 114841 to release ccal 00, 01, 08, and 09 for Mike L. / 
Karen and to add in ccal 14 and 15 since they are no longer  
working on them. 
 
This run does not END successfully.  Have to force a RESET.</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114841' target=_top>114841</a>) -->


<!-- Date: Wed May 16 10:35:46 2001 -->
Called L2 pager twice with no response.  Would like update on status of L2 decision crate.  Currently, are not reading it out.  Stephen Miller had thought he would get back to us later yesterday...
<!-- Author: Pekka -->


<!-- Date: Wed May 16 10:40:29 2001 -->
<pre>svt crates 02 and 04 fail to respond to END or PARTITION.   
The readout stack usage is abnormally large (99% of max). 
Will reboot these crates.</pre>
<!-- Author: Nancy -->
<!-- Comment: Wed May 16 10;43;31 comment by...Nancy -->
<pre>These crates do not respond at all (ping).  Paged SVT (Bill A.).</pre>


<!-- Comment: Wed May 16 10;45;30 comment by...Nancy -->
<pre>Bill A. is investigating.</pre>
<!-- Comment: Wed May 16 10;45;58 comment by...Nancy -->
<pre>I will remove these crates from run config for the time being.</pre>
<!-- Date: Wed May 16 10:45:21 2001 -->
Turned on TOF LV on the west side (10:20)
<!-- Author: gbauer -->




<!-- Date: Wed May 16 10:55:32 2001 -->
<pre>Starting Run 114849.  Get busy condition as reported by ccal  
crates.  HRR.  Doesn't work.  Another HRR.  No luck.  End run 
and try again. 
</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114849' target=_top>114849</a>) -->
<!-- Comment: Wed May 16 10;59;12 comment by...Nancy -->
<pre>This run doesn't end properly.  Have to force another reset.

Clean up EVB and L3 farm.</pre>


<!-- Date: Wed May 16 11:06:01 2001 -->
<pre>SVX cooling trip:  
Bulkhead West #1 Supply flow reads 0.6 lpm 
Bulkhead West #2 supply flow reads 0.0 lpm 
 
SVX people know.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 11:18:31 2001 -->
<pre>Steve H. is turning on the Xenon HV.   
 
Diego wants to do the plug sourcing.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 11:26:52 2001 -->
<pre>Run 114855 for Diego and plug sourcing.  Done timeout from 
pcal07 because of slot 4 shower max card: SMXREADOUT_60.</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?Nancy' target=_top>114855</a>) -->


<!-- Date: Wed May 16 11:34:03 2001 -->
Mike/Karen are giving up b0ccal00, 06, 08 and 09.  Will keep 01 and 07, and have been given b0ccal02, 03, 10 and 11.
<!-- Author: Pekka -->


<!-- Date: Wed May 16 11:34:26 2001 -->
<pre>Run 114856 with SMXREADOUT_60 taken out from pcal07.  11.5K 
events. 
 
/cdf/data08/_ATESTR.0000/ar01c0a7.0001test on fcdfsgi2 
</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114856' target=_top>114856</a>) -->


<!-- Date: Wed May 16 11:38:30 2001 -->
Steve Hahn is taking b0ccal13 to see what is wrong with Xenon box.  No time estimate.
<!-- Author: Pekka -->


<!-- Date: Wed May 16 11:52:29 2001 -->
<pre>Run 114858 started.  Have all pcal now.  SMXREADOUT_60 (slot 4) 
in pcal07 removed.</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114858' target=_top>114858</a>) -->


<!-- Date: Wed May 16 11:54:11 2001 -->
Stephen Miller phoned in an update on the L2 crate.  The alpha board that failed has been swapped out with a spare, and Steve Kuhlmann did perform some tests on it yesterday.  Monica Tecchio is also planning to work on it today.  At some point later today, we should get an update on whether to include the crate in standard running again.  
<!-- Author: Pekka -->


<!-- Date: Wed May 16 12:10:37 2001 -->
It looks like there's a hot tower in EM eta 0 (East), phi 18. 
Stopping the run so that Carla can verify it.
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114858' target=_top>114858</a>) -->
<!-- Comment: Wed May 16 12;38;08 comment by...carla -->
<pre> All triggers in this run come from hadron towers  at eta 8 and
 15, corresponding to the overlap between central and wall, for
 which the summing has been recently been turned on.  The wall
 crates are in use by Steve and not in RC, but apparently these
 towers are not masked. They are probably masked only if the
 central crates are not on. Will check with Bill Badgett.


 The plug em tower is not "hot", maybe slightly warm. </pre>



<!-- Date: Wed May 16 12:33:59 2001 -->
Started work on FER code for TOF calibrations using wcal06 and 
puls01.  I'm in the trigger room if anyone wants them back.
<!-- Author: M Jones -->


<!-- Date: Wed May 16 12:34:00 2001 -->
Run 114862 with L2 processors at Monica's request.   
5700 events.  The file is in fcdfsgi2: 
 
/cdf/data08/_ATESTR.0000/ar01c0ae.0001test
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114862' target=_top>114862</a>) -->



<!-- Date: Wed May 16 12:42:49 2001 -->
Steve Vejcik agrees to give up a few crates for TOF work till 2 PM (Matt Jones) - wcal06 and b0puls01.  Then it goes back to Steve till 4ish.  Nikolai is negotiating with Steve to power down some of the wall crates to get access into to one of the wall PMT's. 
<p> 
Jimmy Proudfoot called to see if he could get the central and/or plug calorimeters for some time this evening to test shower max readout.  Noted the potential conflict with L2 work that has to be resolved.
<!-- Author: Pekka -->



<!-- Date: Wed May 16 12:52:19 2001 -->
We have two requests for access to the calorimeter readout this evening 
<ul> 
<li> Monica Tecchio and the other L2 folks would need to have periodic runs to test L2 boards.  Monica doesn't expect to be around between 6-9 PM. 
<li> Proudfoot would like either the central and/or plug for at least one hour, preferably more.   
</ul> 
Suggestion is that we schedule Jimmy starting around 6 PM.  
<!-- Author: Pekka -->


<!-- Date: Wed May 16 13:03:29 2001 -->
Steve Vecjik reports that he will need the wall crates from about 2 PM to about 6 PM (pessimistic estimate). He has released them for the next hour.
<!-- Author: Pekka -->


<!-- Date: Wed May 16 13:15:44 2001 -->
<pre>During L2 runs for Monica.  Added back l1cl04 and l2cl04. 
Also added the L2 board into the crate readout.  Using  
INTEGRATION_SUMET_2[2,120].</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 13:20:24 2001 -->
<pre>Had to restart RC because of out-of-memory error.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 13:21:00 2001 -->
<pre>RC dies after saying  
 
*** Error updating database ***</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 13:24:30 2001 -->
Allocated ccal07, 09 and 14 to Andrew Kovalev and Isamu Nakamura to debug LED signals.  Expect that they will be done in two hours.
<!-- Author: Pekka -->


<!-- Date: Wed May 16 13:24:32 2001 -->
<pre>Modified the QIEcalib.html and SMQIEcalib.html files to reflect 
the fact that these calibrations go into the calibration database 
and do not go to fcdfsgi2.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 13:36:21 2001 -->
Nikolai reports that his work on wall is complete for the moment, although there is still a problem with one PMT that he hasn't tracked down.
<!-- Author: Pekka -->


<!-- Date: Wed May 16 13:41:14 2001 -->
<pre>wcal06 isn't responding.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 13:42:38 2001 -->
<pre>L3 died on Partition while trying to set up Monica's run. 
Clean up evb and L3 farm.</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114864' target=_top>114864</a>) -->


<!-- Date: Wed May 16 13:47:19 2001 -->
wcal06 didn't respond to ping.  I went downstairs and confirmed 
that the power was on.  Since I was down there anyway, I pushed 
the reset button.  It seems fine now.
<!-- Author: M Jones -->


<!-- Date: Wed May 16 13:47:58 2001 -->
<pre>Run 114864.  At first, run hangs at event 4.  State 14 on TS. 
Tried HRR and it looks like the run is proceeding happily. 
Ended run on purpose after 15.6K events. 
</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114864' target=_top>114864</a>) -->


<!-- Date: Wed May 16 13:57:16 2001 -->
<pre>Just realized that we have been using the L3 subfarms 00-14 
for the past couple of hours, and they are working fine.  Hmmmm 
maybe I should page L3 and tell them to stop fixing it since  
it's working now?</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114865' target=_top>114865</a>) -->


<!-- Date: Wed May 16 13:57:48 2001 -->
<pre>Finished with puls01 and wcal06.  They have been rebooted.</pre>
<!-- Author: M Jones -->


<!-- Date: Wed May 16 13:59:11 2001 -->
<pre>Is it just me, or has the control room all of a sudden gotten 
even chillier than ever?  Brrrrrr.......</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 14:00:25 2001 -->
<pre>Run 114865 (a run under Monica's configuration) ended to 
take out pcal03 for Willis.</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114865' target=_top>114865</a>) -->


<!-- Date: Wed May 16 14:09:12 2001 -->
<pre>l3c00 seems to be a bit cranky.... Boris is looking into the  
problem.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 14:09:48 2001 -->
Steve is grabbing b0l2cl04, b0l1cl04 and b0wcal03 for about half an hour.
<!-- Author: Pekka -->


<!-- Date: Wed May 16 14:20:34 2001 -->
<pre>Dan Cyr is using b0imu00 to look into the bunch counter errors 
in the crate reported earlier in the shift.</pre>
<!-- Author: Nancy -->


<!-- Date: Wed May 16 14:24:33 2001 -->
<pre>b0l3001 needed reboot again.  
It caused l3proxy stop during core management routines.  
 
</pre>
<!-- Author: Boris -->


<!-- Date: Wed May 16 14:35:15 2001 -->
<pre>ccal 00 gives a done timeout.  HRR doesn't recover it. 
Ending the run: 114869.</pre>
<!-- Author: Nancy :: (run <a href='/cgi-bin/runSummaryRun.cgi?114869' target=_top>114869</a>) -->


<!-- Date: Wed May 16 14:36:29 2001 -->
Installed the BSC-1 on both sides.
<!-- Author: K.Terashi, K.Kotelnikov -->


<!-- Date: Wed May 16 14:38:09 2001 -->
Dan Cyr requested and obtained IMU01 and MUSC00 for debugging for an hour or so.
<!-- Author: Pekka -->


<!-- Date: Wed May 16 14:38:46 2001 -->
Rick Tesarek looked at ccal01 and declared it "fine."
<!-- Author: Pekka -->


<!-- Date: Wed May 16 14:39:52 2001 -->
Steve Vejcik is done with trigger crates, and now needs all the wall crates, puls01 (will be on 1st floor).
<!-- Author: Pekka -->


<!-- Date: Wed May 16 14:45:12 2001 -->
Greg Feild and Michael Schmidt are modifying the VME chipset in TS#2 to test this fix.  Also swapping in a new FRED card to fix some minor problems.
<!-- Author: Pekka -->


<!-- Date: Wed May 16 15:07:37 2001 -->
I finally saw a repeat of the bunch counter mismatch in b0imu00 : i had taken >  200k events w/o an error, then  started a new run - i was minicommed into the crate and saw a :  
This system *IN USE* via rlogin by user: vejcik. 
shell restarted.   
 
I don't know if a vxlogin  could cause a BC mismatch - or if this was mere coincidence..... 
  
<!-- Author: dan -->


<!-- Date: Wed May 16 15:10:17 2001 -->
Cabletest work done for day. Successfully performed scripted communications to test admem,txpt,ts,DCAS.  admem diagnostic fifo 
gets filled with preset data and on HRR from ts starts pumping it through to DIRAC and thence to DCAS.  Scripting enables some configurabilitiy for diagnostic tool.   
  
<!-- Author: Steve Vejcik -->


<!-- Date: Wed May 16 15:22:40 2001 -->
<pre>More interesting imu00 problems : i tested that the fix to TSU dbanks would allow tdctest to run - it worked. However the BMU's failed to run! This was strange as yesterday it ran fine - by trial and error found out that tdc in slot 18 isnt happy with tdc test. It works with normal daq running however! In tdctest  - tracer done not set error followed by done timeout. Ill look into this tonight. 
 
</pre>
<!-- Author: dan -->
<!-- Comment: Wed May 16 15;55;42 comment by...dan -->
<pre>took 120k events w/imu00 no bc mismatches 114884	</pre>
<!-- Date: Wed May 16 15:43:02 2001 -->
<pre>replaced tdc in b0musc00 slot 8 and reprogrammed fram1 of all tdcs in musc00 
</pre>
<!-- Author: dan -->



<!-- Date: Wed May 16 15:59:30 2001 -->
<pre>returning b0musc00 + both imu crates to ace</pre>
<!-- Author: dan -->


<!-- Date: Wed May 16 16:04:42 2001 -->
When trying to do a test run with the modified FRED and TS in partition 2, find that the plug crates have been booked under Willis Sakumoto's account. Shift crew were never consulted on this and now we cannot located Willis.   
<p> 
Vaia dispatched to investigate.<font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: Pekka -->


<!-- Date: Wed May 16 16:06:34 2001 -->
<font color=red>Shift Summary: </font><pre>Finishing shift with the detector still in pieces.  New shift left with the task of testing the TS/FRED changes and coordinating the calorimeter work for the evening.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   None
Delivered Luminosity   0  pb-1
Acquired Luminosity    0  pb-1
Efficiency             100
   
</td></tr></table></pre>
<!-- Author: Pekka -->


<!-- Date: Wed May 16 16:08:00 2001 -->
<pre>The TS card for partition 2 has had ithe VME chip  
CY7C960 replaced with CY7C9g0A.  We ask that the  
shift crews please  use this partiton  and report 
ANY problems.  If this partition works well we plan  
on changing the VME chips on the  remaining boards 
in the TS crate over the next couple of  days. 
 
 
</pre><font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: Gret  -->


<!-- Date: Wed May 16 16:11:22 2001 -->
<pre>The FRED card has been  surgically  altered in order to  
properly gate the trigger  signals to the scalers using  
the new GFRED  marker.  Please report  any problems with 
FRED or apparent miscalculations in the trigger rates 
from the scaler GUI. 
</pre><font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: Greg -->