<!-- Title: Day shift. Thu May 10, 2001 -->
<!-- SciCo: J. Spalding/R. Roser -->
<!-- DAQAce: Rob Snihur -->
<!-- MonAce: Rob Snihur -->
<!-- CO: &nbsp; -->
<!-- OpManager: &nbsp; -->
<!-- Notes: -->
<pre>This will be a positive shift, even tho the access continues...</pre>


<!-- Date: Thu May 10 08:36:04 2001 -->
Ended the inherited run to give the CAL to Rick, COT to Jay.
<!-- Author: rob :: (run <a href='/cgi-bin/runSummaryRun.cgi?114413' target=_top>114413</a>) -->


<!-- Date: Thu May 10 08:42:46 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>IMU experts instruct that all HV alarms for that system should be ignored thru the weekend.</td></tr></td></table>
<!-- Author: rob -->


<!-- Date: Thu May 10 08:55:22 2001 -->
<pre>I came in to take a look on level3.  
 
Monitoring is dead on three nodes (out of 24) 
Looks like an old problem when monitoring can not  
find shared memory segment allocated by l3_node.  
The priority of this prodlem is going up slowly.  
 
Rob asked for some extra error messages. 
I'll send the request to responsible parties.   
</pre>
<!-- Author: Boris.  -->


<!-- Date: Thu May 10 09:06:14 2001 -->
Reinstalled 30-foot Co-60 source wire in Plug source driver 4W. 
The active seed is the original one, in a new "wire".   RSO Joe Leo has checked the radiation field around the driver with the source garaged, and it is less than 4 mR/hour at one foot from the protective cover, which is standard for these sources.  (The Plug sources are pointlike, the radiation field when unshielded is less than 5 mR/hr at one meter, and the inverse square law applies.)  Installation took approximately 30 minutes, and VEB received less than 2 mR dose to body.
<!-- Author: Virgil Barnes -->


<!-- Date: Thu May 10 09:09:14 2001 -->
Del Alspach has requested that the old Astro gas head in the Cot Baggie Box be replaced with a new Point Watch head. We are going to start on it now. 
 
George  
<!-- Author: George Wyatt -->


<!-- Date: Thu May 10 09:53:23 2001 -->
<pre>The DB machine is getting its IP address changed. 
After that, Nelly will do some work for 1 or more hours. 
 
I try to switch to the integration DB.  Steps: 
1) 	Exit RC 
	setup fer 
	source ~cdfdaq/int 
2) 	Start RC 
	select ResMgr in Proxy menu 
	stop ResMgr_Prd 
	start ResMgr_Int 
3)	proceed as usual 
 
Result: The trigger table COSMICS[1,112]  
	could not be found in the trigger DB.   
	The integration instance of the trigger DB  
	must not be up to date!</pre>
<!-- Author: rob -->
<!-- Comment: Thu May 10 11;27;49 comment by...rob -->
<pre>Moral of the story below: L3 needs to recompile their code;
also need to uncheck "UseLevel3Manager" in the expert options of Run Control,
when using integration DB.</pre>
<!-- Date: Thu May 10 09:55:35 2001 -->
<pre>CSL machine, b0dau32, rebooted with a new IP address. 
CSL processes restarted. 
</pre>
<!-- Author: Tony Vaiciulis -->


<!-- Date: Thu May 10 09:58:17 2001 -->
Since the above procedure failed, I did a little experimenting. 
I stopped the DBMon_Prd (success), and tried to start DBMon_Int (failure).  I guess this was a bad thing because I kept getting 
2 info windows that popped up every ~minute or so. 
I then tried to start DBMon_Prd again, but could not. 
Exit run control.
<!-- Author: rob -->
<!-- Comment: Fri May 11 10;48;30 comment by...W.Badgett -->
<pre>DBMon is a database monitoring program 
and is not required for running.</pre>
<!-- Date: Thu May 10 10:06:39 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Please page level3  
pager before distributing a new tag set. We want to  
monitor the time it will take for the whole farm  
to go throught ColdStart transition. </pre></td></tr></td></table>
<!-- Author: Boris.  -->


<!-- Date: Thu May 10 10:09:40 2001 -->
I restarted run control, got a few of the annoying double windows, but they somehow stopped coming. 
I chose an older trigger table, but L3 failed the partition transition.  Boris says they need to recompile to switch to integration DB, and he is doing this now.
<!-- Author: rob -->


<!-- Date: Thu May 10 11:19:54 2001 -->
<pre>Arnd recompiled run control to include the fix for 
failure to receive acknowledge messages on a partition transition, 
and this in now the default version. 
 
I stopped and restarted run control.  Error Display appears. 
Partition works for all clients except L3 and L3M. 
Looks like RC did not send or L3 did not receive the partition command. 
Possibly a RT server problem.  I right-click on l3 in the replies & ack window, 
and issue a partition command.  This works. 
 
We logged into b0dau30, and see there are 183 rtserver connections, 
but Bill B. says the limit is ~200. 
 
When we remove l3m from the run, the partition transition works. 
Bill says l3m must have a problem when the production DB is down. 
</pre>
<!-- Author: rob -->


<!-- Date: Thu May 10 11:21:27 2001 -->
<pre>I switched l3proxy to integration database.  
Page level 3 pager when need to switch back to production.  
</pre>
<!-- Author: Boris.  -->


<!-- Date: Thu May 10 11:23:50 2001 -->
Bird alarm for PEM & PHA & PSH: all for CAEN crate 11. 
These 3 guys also lost their heartbeat in teh Global Alarms picture.
<!-- Author: rob -->



<!-- Date: Thu May 10 11:31:56 2001 -->
<pre>Run control was updated to fix a problem that occasionally  
it did not hear the reply from the crates during the  
partition transistion. 
 
This change is in the default version of run control.</pre>
<!-- Author: Frank Chlebana -->


<!-- Date: Thu May 10 11:46:23 2001 -->
Finally started a run with an old trigger table: 
COSMIC_RUN_TABLE_1[3,97]. 
 
Had to kill DAQMON and GUIs that are started from it. 
source ~cdfdaq/int, then restart.  Now I can see the rates.
<!-- Author: rob :: (run <a href='/cgi-bin/runSummaryRun.cgi?114219' target=_top>114219</a>) -->


<!-- Date: Thu May 10 11:48:10 2001 -->
<pre>Notice that there is a error message printed in the  
front end crate 
 
b0ccal01   "ShowerMax not sending data" 
 
This message does not show up in the error handler.</pre>
<!-- Author: Frank Chlebana -->
<!-- Comment: Fri May 11 05;10;10 comment by...Arnd -->
<pre>The above is a 'warning' message, and these do not show up by
default in the error handler.
</pre>
<!-- Date: Thu May 10 11:51:11 2001 -->
<pre>Got a whole bunch of warnings of the form: 
(MLE) b0ccal07:Messenger:11:16:17 AM->ADMEM in slot 17 not properly configured   
Many different ccal crates and slots. 
Does not prevent us from running.</pre>
<!-- Author: rob :: (run <a href='/cgi-bin/runSummaryRun.cgi?114219' target=_top>114219</a>) -->


<!-- Date: Thu May 10 11:51:55 2001 -->
<pre>Plug HV work on CAEN crate 11: replaced the optical decoupler 
board. Maybe this will help to reduce the rate of crate 11 
overvoltage spikes reported to slow controls.</pre>
<!-- Author: Niccolo and Willis -->


<!-- Date: Thu May 10 12:09:04 2001 -->
<pre>Got a BUSY TO.   
Error messages (subset included below) indicate an EVB/VRB problem; 
Steve Tether's extra diagnostics appear to be working. 
HRR works just fine. 
When I try to decode the crate listed below, I get the message: 
Link number 6 is not used for VRB number 1 for SCPU 2 in partition 0. 
 
-----------following from errorfile------------- 
 
(MLE) b0l3gate2.fnal.gov:main:11:43:07 AM->Error on L3 node b0l3004 (partition 0) Thu May 10 11:43:07 2001 l3_node 
  in refoInt_reformatProc (l3_refevt.c:610) 
  @L3_REFORMAT_ERROR 
  
RAWREF Error - VRB-CLEANP - with code 20 -- START 
 Line -  945 in: src/library/rr_rawevt.c 
 Buffer position: 5776 
 Error count: 1 of 1 
 Reject rate: 0.021 percent 
  
 VRB(non-SVX) has zero lenght data, VRB problem. 
 Probabaly EVB tries to clear VRB pipeline in HRR. 
 Length 
  0x00000000 
  
 Error location code: 0x01002160 
  
  
 Crate Id?: 0x00000010                                                                                                
  
 Error during scanning MINI structure! 
 Last Unit prob. worked on (all counts from 0): 
  SCPU: 2 
  VRB : 1 
  LINK: 6 
  MINI: 0 
  
 Buffer (5756 until posi+2=5778) 
 0x0d000000  0x57ede183  0x08002016  0x15000000  0x3c7c0b00  0xe4291000 
 0xd4070600  0xd88e2f00  0x380e0000  0x00000000  0x00000000  0x00000000 
 0xaaaaaaaa  0x55555555  0x08002016  0x00000080  0x00000000  0x00000000 
 0x00000000  0x00000000  0x00000000  0x00000000  0x00000000 
  
  
 NOTE: Event rejected!                                                                                               </pre>
<!-- Author: rob :: (run <a href='/cgi-bin/runSummaryRun.cgi?114219' target=_top>114219</a>) -->
<!-- Comment: Thu May 10 12;22;58 comment by...rob -->
B. Badgett thinks the crate is not properly decoded because
the tool is trying to use the production DB.



<!-- Date: Thu May 10 12:32:59 2001 -->
Cosmic Ray run in progress.  Muon rate is high -- of order 3khz. 
Eric James was notified.
<!-- Author: Rob R. -->


<!-- Date: Thu May 10 12:44:29 2001 -->
<br><b>Calorimeter Electronics Status:</b> 
<br>Performed a routine check of the calorimeter electronics and 
detectors.  In summary, we have the following problems midway 
through the access: 
<ul> 
  <li>Light leaks in EndWall wedges 19,21 on the east side.</li> 
  <li>Noise problems in 2 PIN diode channels for the plug laser 
      system.</li> 
  <li>Excessive noise in CEM wedge 6-E.</li> 
</ul> 
More information may be found in the wedge and plug e-logs for today. 
<br>
<!-- Author: R.J. Tesarek -->


<!-- Date: Thu May 10 12:48:59 2001 -->
Margherita Vittone called and had me kill the Global Alarms picture, then restart it.  She has fixed the problem with the  PEM, PHA, & PSH heartbeats.  They are all green now.
<!-- Author: rob -->


<!-- Date: Thu May 10 13:05:27 2001 -->
Nellie Called -- database work is complete.   Also refreshing integration DB.  We can go ahead and begin using DB again
<!-- Author: RR -->


<!-- Date: Thu May 10 13:17:32 2001 -->
Pressure testing water system on north muon wall.   Pressure test failed at one of the hart unions located on the north wall itself.  In process of fixing this.
<!-- Author: RR -->


<!-- Date: Thu May 10 13:58:49 2001 -->
Ground connection from the notch to the upper and lower COT racks and is done.  
<!-- Author: fred lewis -->


<!-- Date: Thu May 10 14:09:36 2001 -->
<pre>Changed the Dirac config in the hardware database to set  
correctly the AUXSUMEN0 and AUXSUMEN1 registers: 
 
UPDATE CARDS SET AUXSUMEN1=0 WHERE CARD_TYPE='DIRAC' AND SLOT=5; 
UPDATE CARDS SET AUXSUMEN0=0 WHERE CARD_TYPE='DIRAC' AND SLOT=6; 
UPDATE CARDS SET AUXSUMEN1=0 WHERE CARD_TYPE='DIRAC' AND SLOT=10; 
UPDATE CARDS SET AUXSUMEN0=0 WHERE CARD_TYPE='DIRAC' AND SLOT=11; 
UPDATE CARDS SET AUXSUMEN1=0 WHERE CARD_TYPE='DIRAC' AND SLOT=14; 
UPDATE CARDS SET AUXSUMEN0=0 WHERE CARD_TYPE='DIRAC' AND SLOT=15; 
UPDATE CARDS SET AUXSUMEN1=0 WHERE CARD_TYPE='DIRAC' AND SLOT=19; 
UPDATE CARDS SET AUXSUMEN0=0 WHERE CARD_TYPE='DIRAC' AND SLOT=20; 
 
A run should be taken with the new settings.   This is per 
Carla's orders. 
</pre>
<!-- Author: W.Badgett -->


<!-- Date: Thu May 10 14:26:33 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>iFIX: 
When VNODE hangs, use Task Manager to stop the Workspace and 
restart it. For the procedure, please look at iFIX recovery procedure in the Web page or the hard copy in the MCS binder which is located in the two screens of Vnode1. 
 
         
 
</pre></td></tr></td></table>
<!-- Author: jc Yun -->
<!-- Comment: Thu May 10 15;14;51 comment by...rob -->
Tested the procedure: it works fine.
<!-- Date: Thu May 10 14:33:14 2001 -->
<pre>Looked into problem of abnormally high CMU cosmic 
trigger rate.  Traced problem to trigger inputs  
from west CMU wedges 10 & 11.  For the time being, 
I'm removing these wedges from trigger while I 
investigate.  With these wedges removed, CMU cosmic 
rates return to normal. 
 
I also re-enabled trigger input from west CMU wedge 
five. This wedge was noisy during 36x36 due to bad  
power supply.  Hyunsoo fixed this problem a couple  
of days ago.  Trigger rates look fine with this  
wedge re-enabled.</pre>
<!-- Author: Eric James -->


<!-- Date: Thu May 10 14:40:22 2001 -->
With the help of Jamie the old Astro Gas Head in the COT Baggie Box has been replaced with a new Point Watch head. Zeroed and calibrated.
<!-- Author: George Wyatt -->


<!-- Date: Thu May 10 14:40:42 2001 -->
We will turn off the WHA HV for work on the pm fibers. 
<br>  
The central hadron laser will be also on only for WHA channels. 
<br>
<!-- Author: fotis -->


<!-- Date: Thu May 10 14:46:24 2001 -->
<pre>switching off HV on the East Plug calorimeters. 
Opening the cow boxes and remove LED boxes.</pre>
<!-- Author: diego cauz -->


<!-- Date: Thu May 10 14:47:55 2001 -->
<pre>Try to start up a run with production DB after Eric is done with his tests. 
Note that I had to go into the Proxy menu, stop the ResMgr_Prd, and start it again. 
I also chose to stop the ResMgr_Int. 
To do these things, one may need to click on the "Mode" box 
for either of the 2 instances. 
 
The current problem is a partition transition failure for both L3 & L3M. 
Called Sarah Demers (in Kirsten's office) & she is coming to restart L3M.</pre>
<!-- Author: rob -->


<!-- Date: Thu May 10 15:00:37 2001 -->
l3Manager wasn't restarted after morning testing - I was testing and making some changes while the database was down this morning and didn't synchronize with the control room on starting the manager up again.
<!-- Author: Sarah Demers -->


<!-- Date: Thu May 10 15:04:13 2001 -->
<pre>Had the same problem with run control, even though it was supposed to fixed. 
I am talking about RC not getting a partition acknowledgement from all the crates.  I reproduced the problem with full debugging on, and saved the log file to ~cdfdaq/rcGUI_debug_part_ack.log.  Sent email to mccrory@fnal.gov.</pre>
<!-- Author: rob -->
<!-- Comment: Thu May 10 16;14;24 comment by...Elliott McCrory -->
<pre>I am aware of this issue.  I have made another fix to RC that shoud fix it (but that is what I thought this morning, too).</pre>
<!-- Date: Thu May 10 15:26:00 2001 -->
<pre>run control was recompiled to allow for changing the tdc test 
crate node name from  
 
b0diag00 b0diag01 
 
changed the link current -&gt; devel  
to               current -&gt; test 
 
</pre>
<!-- Author: Frank Chlebana -->


<!-- Date: Thu May 10 15:34:40 2001 -->
Carla requests a Rick run. 
Send her email when it's done. 
I am waiting on L3 experts to recompile to re-enable 
use of the production DB.  
<!-- Author: rob -->
<!-- Comment: Thu May 10 15;44;30 comment by...rob -->
She wants the whole calorimeter included, so I will wait until people are done working...
<!-- Date: Thu May 10 15:38:39 2001 -->
<pre>I switched from integration data base to production  
data base for event builder and level3. For level3 i had to modify and recompile  
l3proxy.cc under root since right now we are running Boris's  
private version of level3 code  
 </pre>
<!-- Author: Arkadiy -->



<!-- Date: Thu May 10 15:48:28 2001 -->
<pre>CMP Access Report 
 
Tony worked on the gas leaks in CMP NW stacks 7 and 9.  He seemed 
to feel rather confident that the leak in stack 9 has been fixed. 
Stack 7 may take more effort still. 
 
We swapped out the HV electronics package in Top stack 10 in an 
attempt to quench a ~2 MHz oscillation.  This swap was indeed  
successful and the 2 MHz oscillation is no longer present. 
 
We spent some time investigating a ~12 MHz oscillation in several 
CMP channels (mainly North side of the Top Yoke).  Carl was kind 
enough to shut down all CCUs, but this did not stop the oscillations. 
We still do not know the source of these oscillations. 
</pre>
<!-- Author: Trevor and Hyunsoo -->


<!-- Date: Thu May 10 15:54:32 2001 -->
CMP HV trip, North wall, CAEN crate 3, channels 7, 3, 2, 6. 
Reset them all.
<!-- Author: rob -->
<!-- Comment: Thu May 10 15;56;23 comment by...rob -->
Channels 2 & 6 trip again on their way back up.

<!-- Comment: Thu May 10 16;01;56 comment by...rob -->
And again.  Trevor will check that all the wires are connected.


<!-- Date: Thu May 10 16:03:02 2001 -->
<font color=red>Shift Summary: </font><pre>Production DB back running, cosmics run in progress.  Work done on many systems -- see log book for details</pre> 
<!-- Author: RR -->