<!-- Title: Day shift. Tue Jul 24, 2001 -->
<!-- SciCo: Tom Devlin -->
<!-- DAQAce: Andras Zsenei -->
<!-- MonAce: Alexei Sedov -->
<!-- CO: Heather Gerberich -->
<!-- OpManager: Doug Benjamin -->
<!-- Notes: -->
<pre>
Today's Tasks from 8AM Meeting
==============================
L1       -  Greg Field work on system, including "bit-15" problem
L2       -  Monica coming in to work on problems:
            works OK @ 100 Hz  fails @ 1000 Hz
            Also Kuhlmann wants to do ReCes work.
L3       -  working onf "private version" problems with display
            Constraint:  can debug this morning, but a working
            version of L3 must be back in system by noon for DAQ check.
Silicon  -  "Rescued" several laddars yesterday
            Lengthy discussion of "drip" & ice problems in coll. hall.
            Count of wedges up = 78  
            (?? including 13 to be  brought up today)
COT      -  Want to take all crates for ~15 minutes to do final
            test before closing.
CAL      -  Aces need to ensure that calibration runs end properly,
            even when run has abnormal end.  Otherwise subsequent
            runs are screwed up.
            Need to re-install scaffolding in pit to fix light leak.
            CAL triger wait software needs re-installation.
Muons    -  CMP bottom died.  Fuse blown on low voltage.
            they will close up this morning.
TOF      -  Installation 3/4 done.  Errors in calibrations need
            expert help.

Aces     -  Run cosmic rays with whatever fragments of system
            are available until noon.  Then start integrating
            everything back into the system.

All keys should be returned by 2PM.  Search & Secure at 5 PM.
=============================================================
</pre>


<!-- Date: Tue Jul 24 08:54:49 2001 -->
Removed Silicon Crates from Partition 1 (current Ace Partition) 
so that we can pre-test some additional wedges.
<!-- Author: SCN :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=116048' target=_top>116048</a>) -->


<!-- Date: Tue Jul 24 08:56:48 2001 -->
<pre>Replaced the TS in partition 1.  This partition 
should be completely healthy now.  We request that 
the shift crew please test this partition out today. 
In particular, this partition should be tested for 
the various calibrations.   Thanks. 
 </pre>
<!-- Author: Greg F. -->


<!-- Date: Tue Jul 24 08:59:01 2001 -->
<pre>I have taken the FRED and scaler crates for more  
"bit 15" studies.  </pre>
<!-- Author: Greg -->


<!-- Date: Tue Jul 24 09:03:33 2001 -->
Disable Ifix communication of End Plug HV System to perform some test.
<!-- Author: Ambra Gresele -->


<!-- Date: Tue Jul 24 09:09:47 2001 -->
<pre>Take CLC00 and CLC01 offline to download Xilinx program 
into CLC Admem, and test the Xilinx program. Will need the 
crate for about 1-2 hours.</pre>
<!-- Author: Ming -->


<!-- Date: Tue Jul 24 09:10:54 2001 -->
<pre>working on the plug cal crates and the b0puls01</pre>
<!-- Author: diego cauz -->


<!-- Date: Tue Jul 24 09:11:27 2001 -->
<pre>I removed the CLC and BSC PreFRED cards so I could 
get to the FRED card with the scope.  The PreFRED 
cards are stowed in the spares cabinet and are 
currently marked "offline" in the database. 
</pre>
<!-- Author: Greg -->


<!-- Date: Tue Jul 24 09:28:06 2001 -->
I replaced fer/run control with a new built, compiled from 
scratch. No problems expected, just a warning...  
<!-- Author: Arnd -->


<!-- Date: Tue Jul 24 09:38:10 2001 -->
<pre>There is a good chance that the problems with monitoring 
programs on b0dap57 on owl shift were due to things being 
accidentally configured to run the daqmon programs with the 
version of java being used to track OutOfMemory errors in 
the main run control. This version appears to use much 
more CPU time, so with several CPU intensive daqmon 
programs on b0dap57 they did not appear to process the 
smartsockets messages very promptly. 
Until the fer table file it is fixed later this morning 
if one wants to start a new copy of the Level 3 display 
then create a new window, do "setup fer", then do 
"java rc.l3mon.Display"</pre>
<!-- Author: patrick -->


<!-- Date: Tue Jul 24 09:50:49 2001 -->
<pre>Java Exception. Run Control was restarted.</pre>
<!-- Author: Alexei Sedov -->


<!-- Date: Tue Jul 24 10:04:43 2001 -->
About to swap the TOF VTT enable cable on b0wcal07.
<!-- Author: M Jones -->
<!-- Comment: Tue Jul 24 10;14;35 comment by...Andras -->
<pre>This caused an alarm in PSM (East End Wall CONEB_3).</pre>


<!-- Date: Tue Jul 24 10:20:26 2001 -->
<pre>Finished swapping TOF VTT enable cable in b0wcal07.  Power is 
back on in that crate.</pre>
<!-- Author: M Jones -->


<!-- Date: Tue Jul 24 10:26:38 2001 -->
<pre>Successfully download Xilinx program into ADMEM in slot #4 
of the CLC00 crate. Test shows that this program has the right 
threshold value.</pre>
<!-- Author: Ming -->


<!-- Date: Tue Jul 24 10:38:31 2001 -->
<pre>SVX alarm, on the Silicon GUI it says trip on SVX B4 W8.</pre>
<!-- Author: Andras -->


<!-- Date: Tue Jul 24 11:01:33 2001 -->
<pre>PS Monitoring is online again. 
</pre>
<!-- Author: Marcin -->


<!-- Date: Tue Jul 24 11:05:04 2001 -->
<pre>L2 backplane work: 
 
 Roberto and Fred work on replacing  3 pins on the magicbus 
backplane.   2 pins in slot 16 and 1 pin in slot 18. 
All fixed.</pre>
<!-- Author: F.Lewis,Roberto -->


<!-- Date: Tue Jul 24 11:14:45 2001 -->
I take all SVT crates.  I can return them whenever 
needed.  I am either in B0 2nd floor west test stand 
area, in trigger room, or at x5017. 
<p>  
<!-- Author: Ashmanskas -->


<!-- Date: Tue Jul 24 11:16:55 2001 -->
<pre>SMX work: 
  We take ccal08 to study the wire preamps in East Phi=0. 
 
  The HV to Wedge East 0 has been disconnected since Jul 12 
and all 32 wire channels associated with SMX 2 still look 
alive.   Yesterday at 4:51 pm (Jul 23), we returned this 
voltage to Standby.  We then toggled the Voltage to on and 
standby several times and wrote a pedestal file with each 
configuration.  The preamps all still look alive at  
this point and we left the HV on.     
 
  Today, at 10:00am, we discover 3 of the preamps 
are dead.   We have a protection circuit to install 
(if it makes it here before 2pm).</pre>
<!-- Author: byrum,lindgren -->


<!-- Date: Tue Jul 24 11:18:23 2001 -->
<pre>Problems w/ l2de00. There is no FER_Messenger. Bus reset didn't help. Arnd Meyer was called. He fixed it but now I have the same problem w/ clc01. Since Arnd mentioned that the problem may be connected w/ SmartSockets I called Jim Patrick.</pre>
<!-- Author: Alexei Sedov -->


<!-- Date: Tue Jul 24 11:21:47 2001 -->
<pre>PSM alarm from First Floor 1RR18D_1. I went down to check, but 
apparently no one was touching the crate and it seemed to be on. 
When I got back to the control room everything was green again.</pre>
<!-- Author: Andras -->


<!-- Date: Tue Jul 24 11:32:01 2001 -->
<pre>finished working with thw plug crates</pre>
<!-- Author: diego cauz -->


<!-- Date: Tue Jul 24 11:40:15 2001 -->
<pre>Requested XFT crates and b0cot10 for 1-2 hrs to swap 
an XTC mezzanine card and test the replacement.</pre>
<!-- Author: Jay Dittmann -->


<!-- Date: Tue Jul 24 11:44:49 2001 -->
<pre> 
The code for reloading the Level 3 tag information is now in  
place.  The problem had been a class that I thought was reloading  
was not. 
 
So, "this time fer sure", you should not have to restart the 
Run Control for new level 3 tag sets. 
 
</pre>
<!-- Author: W.Badgett -->


<!-- Date: Tue Jul 24 11:59:16 2001 -->
<pre>Level3 failed partition transition. l3proxy had troubles  
with b0l3u01. Disk hda6 is 100% full on u01. 
 
cron.daily was not run on this node for several weeks.   
run it manually. hda6 went down to 64%. 
 </pre>
<!-- Author: Boris.  -->
<!-- Comment: Tue Jul 24 12;21;27 comment by...Boris.  -->
<pre>crond daemon is not running on b0l3u01. 
Restarted. Checked for crond on all the other nodes. 
Everything is fine. Try to start a new run. </pre>


<!-- Date: Tue Jul 24 12:32:12 2001 -->
<pre>Took QIE calibration run 121435 in expert mode 
and loaded the constants. There is still some noise in the electronics as it was last night.</pre>
<!-- Author: vaia, alex :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=121435' target=_top>121435</a>) -->


<!-- Date: Tue Jul 24 12:41:43 2001 -->
<pre>Fred is released to the shiftcrew for their "Rick Run"</pre>
<!-- Author: Greg Feild -->


<!-- Date: Tue Jul 24 12:44:59 2001 -->
<pre>Finished with b0cot10, so all the COT crates are now 
available.  The XFT crates are still under expert control...  A bad XTC mezzanine card was replaced in b0cot10 slot 5.</pre>
<!-- Author: Jay Dittmann -->


<!-- Date: Tue Jul 24 12:53:48 2001 -->
<pre>Karen asked me to do showermax QIE calibration. So run 121438 was devoted to showermax.</pre>
<!-- Author: Alexei Sedov :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=121438' target=_top>121438</a>) -->


<!-- Date: Tue Jul 24 13:14:33 2001 -->
<pre>PSM Arches 15W has all its channels except 4, 6 and 9 turned 
off. Paged Mike Lindgren and he is going down to check.</pre>
<!-- Author: Andras -->


<!-- Date: Tue Jul 24 13:19:30 2001 -->
<pre>   I have taken over Boris's L3 work. I have cleaned  
up all systems by hand (were in somehwat of disarray) 
and had to reboot one output node b0l3u01. After that 
we had no problem starting the run. 
  At this time we started official version of L3 display 
from the command line, according to Jim's instructions. 
It works normally.</pre>
<!-- Author: Ilya -->


<!-- Date: Tue Jul 24 13:23:42 2001 -->
<pre>PSM 1st Floor 1RR18F_1 turned red. I went down and found 
Nathan Eddy working on it.</pre>
<!-- Author: Andras -->


<!-- Date: Tue Jul 24 13:24:23 2001 -->
<pre>TOF QiePed Consumer updated, we are using the same consumer used  
by the calorimeter folks. We also found a bug on the  
shell script launched by Run Control. Issues addressed:  
 
The consumer ends automatically after the active transition ends  
 (it is worth to have the proxy consumer display open to see when  
 the consumer finishs sending the data)  
 
The calibrations can be downloaded.  
 
The consumer is going to be compatible with the new sevb.  </pre>
<!-- Author: Ivan, Gervasio -->


<!-- Date: Tue Jul 24 13:35:01 2001 -->
<pre>   We had one successful run with L3, but I asked 
to stop in order to switch to a newer version of L3. 
This version had been in use until last owl shift, 
but when L3 Display troubles started, Boris has switched  
to the older version. </pre>
<!-- Author: Ilya -->


<!-- Date: Tue Jul 24 13:36:08 2001 -->
<pre>I am taking over the muon crates to work on software 
initialization code for the muon trigger crate.</pre>
<!-- Author: Eric James -->


<!-- Date: Tue Jul 24 13:37:32 2001 -->
<pre>Andy Foland went to pit to reboot a FIB crate.
13:49  He returns and reports that he rebooted all of them.</pre>
<!-- Author: Devlin -->


<!-- Date: Tue Jul 24 13:56:33 2001 -->
<pre>COT heartbeat alarm.</pre>
<!-- Author: Andras -->
<!-- Comment: Tue Jul 24 14;12;34 comment by...Andras -->
<pre>The expert was around. Now it is back to normal.</pre>
<!-- Date: Tue Jul 24 13:59:39 2001 -->
<pre>PSM Arches 00E, 01E, 02E turned red. Paged the expert.</pre>
<!-- Author: Andras -->
<!-- Comment: Tue Jul 24 14;17;44 comment by...Andras -->
<pre>02E is green again, but 23E, 22E are red and 18E-21E are 
magenta (communication problem). All of the alarmed ones
have a lower fan problem (values lower than 100%, however
one of the was red with a value of 99%, while there are some
greens with 98%!!!). 22E has also -5 V CMU at -25V!!!
Mike Lindgren is around and he is going down to check it
with Karen Byrum.</pre>
<!-- Comment: Tue Jul 24 14;20;21 comment by...Andras -->
<pre>Now it is back to normal.</pre>
<!-- Date: Tue Jul 24 14:02:47 2001 -->
<pre>People are emerging from the pit and turning in keys, 
pretty much on schedule.
Tesarek and Wagner call requesting an extra 5 minutes. OK.</pre>
<!-- Author: Devlin -->


<!-- Date: Tue Jul 24 14:03:36 2001 -->
<pre>Took COT crates for final cable check.</pre>
<!-- Author: ambrose -->


<!-- Date: Tue Jul 24 14:08:48 2001 -->
Take CLC crate (b0clc00) for LED gain calibration and Qietest.
<!-- Author: Alexander Sukhanov -->





<!-- Date: Tue Jul 24 14:20:53 2001 -->
<pre>COT alarm which disappears within 2 seconds.</pre>
<!-- Author: Andras -->


<!-- Date: Tue Jul 24 14:26:09 2001 -->
<pre>Another fake COT alarm.</pre>
<!-- Author: Andras -->


<!-- Date: Tue Jul 24 14:31:56 2001 -->
End Plug HV System:Ifix is update and monitor is running.
<!-- Author: Ambra Gresele -->


<!-- Date: Tue Jul 24 14:41:40 2001 -->
<pre>Done with COT crates, all repeater plug-ins  
look OK.</pre>
<!-- Author: ambrose -->


<!-- Date: Tue Jul 24 15:04:22 2001 -->
	L00 gas cooling tripped, leading to sono alarms 
and iFix horn alarms in control room.  L00 sono 
silenced. 
<P> 
	As all enables were ACTIVE at time of cooling 
trip, all 8 racks also went down on the cooling trip; 
leading to another source of foghorn iFix alarms in 
control room.  As the racks were down, iFix lost  
communication to the power supplies (which were off). 
This led to happy singing birdie alarms in the 
control room. 
<P> 
	Cryo reset L00 gas trip.  
<P> 
	L00 gas then tripped again, this time  
accompanied by two SVX cooling lines.  Silicon 
sono alarm came on and was silenced.  Racks  
naturally stayed down.   
 
	Cryo reset trips, then  
brought up rack (after required 10 minute  
waiting period).   
<P> 
	Acknowledged iFix alarms in control room 
(clearing both the birdy alarm and the sono alarms),  
then unsilenced sono alarms.
<!-- Author: ADF -->
<!-- Comment: Tue Jul 24 15;09;05 comment by...ADF -->
In this case, there was
<UL>
<LI>A cooling alarm
<LI>An iFix alarm indicating loss of communication with
power supplies
<LI> NO GUI on monitor (due to other SVX work going on)
</UL>

<P>
Cooling + GUI = Crash
<P>
This is exactly the situation in which, if there is
not a silicon expert <i>right there</i>, you (the Ace) 
should be hitting the <b>crash button</b>.


<!-- Date: Tue Jul 24 15:13:30 2001 -->
<pre>Tried to begin a "Rick" run with the calorimeter/trigger 
using partition 1, with the TS Greg installed earlier. 
A BUSY timeout occurs immediately on activate. 
Looking at the event builder statistics, it claimed it 
saw zero events, the evb had been cleaned up before this run. 
Looking at the TS statistics, they claimed the TS 
generated some events, however, the Trigger Manager claimed 
it saw zero events. 
We switch to partition 0 and the system runs WITHOUT 
any cleanup of the EVB, L3, or Trigger Manager.</pre>
<!-- Author: patrick + sedov :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=121454' target=_top>121454</a>) -->


<!-- Date: Tue Jul 24 15:41:27 2001 -->
<pre>XFT crates being released again to shift crew.</pre>
<!-- Author: Jay Dittmann -->



<!-- Date: Tue Jul 24 15:45:27 2001 -->
<pre>Run was stopped w/ 72K events. Plug crates and ccal00 were released. Triger table was changed to the L2Torture [1,143] as was requested. New run number is 121456. 
</pre>
<!-- Author: Alexei Sedov :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=121456' target=_top>121456</a>) -->


<!-- Date: Tue Jul 24 15:46:59 2001 -->
<pre>&lt;pre&gt;SMX  work   - 
 
We swapped the preamp wire cards (ch 0-31) on West Phi=15 
which were dead.   We installed a protection circuit 
between the signal cable and the input to the preamp for 
these channels.   &lt;/pre&gt;</pre>
<!-- Author: byrum,kuhlmann -->


<!-- Date: Tue Jul 24 15:53:03 2001 -->
<pre>SMX  (shower max) work continued - 
We notied that calibrations taken at noon showed  
east phi=21 failing.   Using Voltman, we observed 
the +7.5V sagging.  We verified this at the detector 
and swapped the +7.5V rabbit supply for East Phi=21. 
 
Calibrations looked ok afterwareds. 
 
We also looked at the connections for the monitoring 
of the SMD crates for east phi 0-1, and 18-23.  These  
are all daisy-chained together and phi 0-1, 23-22  were 
giving alarms while the readout to phi 18-21 was broken. 
We pushed on connectors to the readout for phi=21. 
afterwards it all worked.   </pre>
<!-- Author: byrum,lindgren -->


<!-- Date: Tue Jul 24 16:05:36 2001 -->
Report from today's fred/scaler tests: 
<p> 
Irregularities in the scaler counts were also  
seen in the FRED data in the TFRD bank. 
The timing on one of the glitchy channels 
looks pretty good on the scope as can be seen 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=tsi&action=view&page=2&frame=2&anchor=14:52:15'> 
here. </a>  
Not able to catch the board in the act of glitching. 
One  observation today was that a couple of channels did 
not count at the beginning of the run and then came alive. 
Will try to look more tomorrow.  Had to surrender the crates 
today for other tests.  
<!-- Author: Greg -->


<!-- Date: Tue Jul 24 16:14:03 2001 -->
<font color=red>Shift Summary: </font><pre>Shift Summary 
============= 
L1       -  Replaced Trigger Superviser in partition 1.  Seems to work.  
            Needs tests, particularly various calibrations. 
            Continue searching for "bit-15" problem. 
	    Greg Field still chasing problem in FRED data in TFRD bank. see above. 
L2       -  Roberto & Fred Lewis replaced various pins on magicbus backplane, 
            slot 16, slot 18.  All fixed. 
L3       -  Boris restarted crond daemon on b0l3u01. Looks OK. 
            Ilya cleaned up systems, rebooted.  After this no problem 
            starting run.   
            Started L3 display from command line.  Works OK. 
CAL      -  Cauz working on plug cal crates & b0puls01. 
	    Pulser now works OK. Still no significant signal from laser. 
            Software problems remain.  Still working on this at end of shift. 
            Ambra Gresele: End Plug HV:  IFIX updates & monitor running. 
CLC      -  Ming successfully downloaded Xilinx =&gt; ADMEM, slot #4, CLC00. 
            Test threshhold.  Value OK. 
            Alexander Sukhanov doing LED gain calib. and Quietest. 
TOF      -  M. Jones finished swapping VTT cable in b0wcal07.  Power back on. 
            Ivan & Gervasio updated QiePed consumer. 
            Found bug in shell script launched by Run Control. 
              Consumer ends automatically, 
              Calibrations can be downloaded 
              Consumer compatible with new sevb. 
RunCntrl -  Bill Badgett installed a fix to avoid having to restart 
            Run Control for new L3 tag sets. 
Monitor  -  See note by Jim Patrick above - monitor processes & L3 display. 
COT      -  Jay Dittmann replaced bad messanine care in XTC, b0cot10 slot 5. 
            Fixed the problem. 
            Ambrose on COT crates: all repeater plug-ins look OK. 
SMX      -  Byrum & Lindgren worked on shower max.   
            Discovered 3 dead preamps - still dead. 
            Swapped preamp board and a power supply, checked cable connections.  
            Installed protection circuits on new preamp.  Seems to work.   
MUON     -  Eric James working on crates. 
XFT      -  Dittmann reports XFT crates released to shift crew.  All OK. 
 
 
========================================= 
SEE note on Silicon Crash Button at 15:05 
=========================================</pre> 
<!-- Author: Devlin -->


<!-- Date: Tue Jul 24 16:15:19 2001 -->
<pre>Start new run with L2Torture [1,143]. 
</pre><font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: Koji -->