<!-- Title: Owl shift. Sat Jun 16, 2001 -->
<!-- SciCo: Jimmy Proudfoot -->
<!-- DAQAce: Heather Ray -->
<!-- MonAce: Rob Napora -->
<!-- CO: Yi Le/ Amit Lath&nbsp; -->
<!-- OpManager: Doug Benjamin -->
<!-- Notes: -->
<pre>Continue running on store 535. Inst. Lum =0.1E30. Monitor COT, losses etc.</pre>


<!-- Date: Sat Jun 16 01:01:38 2001 -->
<ul> 
<li> inherited run 117859 from previous shift.  still running (now at 270 K).   Running with RUN_117856 configuration, 36x36_1_Coupled_noInclusive[1,137] tagset.
<li> seeing several reformatter errors (input_failure)errors from L3.  these are : 
<pre> 
SCPU 2 5 5 0 2 3 5 
VRB  0 1 0 0 0 0 0 
LINK 6 0 4 6 9 4 1 
MINI 1 5 3 1 1 1 7 
</pre> 
these correspond to crates: COT16, CCAL08, CCAL04, XFT5, COT19, COT04, CCAL01. 
We were taught in our L3 training that these are caused by the FECrates, and that a lot of them result in excessive amounts of rejected events.  If many more occur we will end the run, reset, partition, and try a cold start to clear the matter up. 
</ul>  
<!-- Author: hray :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117859' target=_top>117859</a>) -->


<!-- Date: Sat Jun 16 01:23:33 2001 -->
PSM alarm noticed by Rob came without either a trigger inhibit or an audible alarm. ACE stopped run as it was a good opportunity to clean up from reformatter errors
<!-- Author: Jimmy P :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117859' target=_top>117859</a>) -->


<!-- Date: Sat Jun 16 01:40:05 2001 -->
<pre>CMP HV is on. The proton loss is ~15 kHz.  
iFix was logged in using public account on  
the slow control. Should be logged in as ace to  
control HV.</pre>
<!-- Author: hyunsoo -->


<!-- Date: Sat Jun 16 01:53:19 2001 -->
CPR trip - SE quadrant. No obvious beam-related reason. Reset successfully. Continue trying to track down why we had neither an audible alarm or a trigger inhibit
<!-- Author: Jimmy P. :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117859' target=_top>117859</a>) -->


<!-- Date: Sat Jun 16 02:17:05 2001 -->
iFIX won't restart - it is forever in the state "initialising display". Paged JC.
<!-- Author: Jimmy P. -->


<!-- Date: Sat Jun 16 02:26:09 2001 -->
<ul> 
<li>Problems with IFIX and bird-tweeting so Jimmy suggests we enable COT inhibit.  Neither the RC gui nor the TriggerInhibit Gui will work - the  COT inhibit is still set.  We email Bill Badgett to let him know. 
<li>run 117861 started with an alpha no longer processing in spot 5 right away.  We also had the b0l302 converter node (01 farms) name turn red right away, and we're getting reformatter errors.  This converter node problem was present last night, and it doesn't appear anyone has addressed it yet (judging from lack of log entries) so we'll send email to L3 as well.  
</ul>
<!-- Author: hray :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117861' target=_top>117861</a>) -->
<!-- Comment: Sat Jun 16 07;47;52 comment by...h -->
The reformatter errors from this run are for: XFT5, CCAL01, CCAL04, CCAL08, CMX01, COT04, COT16, L2CAL05.
Once we removed converter node 2 all of these problems went away. (see rest of entries for this shift)
<!-- Date: Sat Jun 16 02:35:40 2001 -->
<p>We had another PSM trip just like <a href= 
'/cgi/elog/elog.pl?nb=2001&action=view&page=397&frame=2&anchor=23:27:06'> 
the one earlier</a>.  We paged Robin and the same action was taken as earlier.  Things have come back up and the PSM is green and happy again.  This has also been verified on voltman. 
 
<p>While working on the above we also had a CPR trip.  This was reset by Jimmy. 
 
<p>An important note on both these trips:<b>No audible alarm sounded</b> for either.  Nor did the trigger inhibits appear to work.  We noticed that the 'pinwheel' on the HV/Global Alarms system was not present, so we suspected a problem with the software being in a bogus state.  In an effort to fix things without waking anyone at this early hour we logged off and back on to the PC.  Things did not come up properly--none of the readings on either HV or the Global Alarms page came to life.  I then rebooted the PC (via the switch) and things came back up ok this time.  Unfortunately, J.C. was paged and woke during this process. 
 
<p>When I came by prior to my shift earlier (~2330) I noticed that this system was coming back up to life.  Perhaps that was also done via a logout (not a reboot) and resulted in things being in an bogus state.  Whoever decided that the NT/Windows shutdown option be greyed out on this PC might want to reconsider. 
 
<p>Good news.  While I was writing this entry we got a heartbeat/communications alert on the PSM.  Why is this good news you ask?  We had an audible alarm this time!
<!-- Author: Napora -->


<!-- Date: Sat Jun 16 02:52:16 2001 -->
<p>Another PSM alarm.  Same symptoms as before verified on voltman.  We paged poor Robin again.  
<!-- Author: Napora -->
<!-- Comment: Sat Jun 16 02;58;56 comment by...hray -->
Paused run while we called Robin.  Jimmy told us to  continue the run, look at YMon, and compare plots to see if this is problem is really a valid Power issue or if it's a problem with monitoring.
<!-- Date: Sat Jun 16 02:54:38 2001 -->
<pre>TrigMon crashes quite a bit.  It's been doing this 
for the past few shifts. See CO log.</pre>
<!-- Author: Amit Lath and Yi Le :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117861' target=_top>117861</a>) -->



<!-- Date: Sat Jun 16 03:04:33 2001 -->
CMP trips twice so far while Rob is talking to Robin. These happen after the PSM alarms.  The CMU Trigger Inhibit light does come on - but goes away while the power is still ramping. (See below - Jimmy has to reset by hand due to IFIX problem)
<!-- Author: hray :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117861' target=_top>117861</a>) -->
<!-- Comment: Sat Jun 16 03;27;01 comment by...R. Erbacher -->
Note that these trips are all occuring during a time of high beam losses.
<!-- Date: Sat Jun 16 03:10:38 2001 -->
WHA wedges 24 - 30 show low occupancy (0.2% as opposed to 1%). 
Check YMON plot #43
<!-- Author: Amit Lath and Yi Le :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117861' target=_top>117861</a>) -->


<!-- Date: Sat Jun 16 03:19:55 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>IFIX problem!  CMP: when it trips we are unable to reset the HV to either standby or on from IFIX - we get an error: Value is not accessible from this plant area. Jimmy goes downstairs to do it from there.</td></tr></td></table>
<!-- Author: the shift -->
<!-- Comment: Sat Jun 16 04;04;31 comment by...Aces -->
<p>This is because we had to log the 'ace' out to reset the crates for the PSM errors.  We do not have the password to log back in and are using 'public'.  I realize security is important, but the shift crews <b>need</b> access to the various passwords we use here.  

<p>Could someone on the day shift please page the appropriate experts and get this password and be sure to pass it along to the evening and night shifts tomorrow.
<!-- Date: Sat Jun 16 03:24:19 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1138&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1138  width='300' align='top'></td><td>the image of YMon #43 Slides, look at WHA wedge 24-30</a></td></tr></table>
<!-- Author: Amit Lath & Yi Le -->
<!-- Comment: Sat Jun 16 03;37;49 comment by...rde -->
Excellent work, Watsons!  (Seriously, this is very helpful information to know that it's a real PS fault and not just a monitoring problem.)  Thank you.
<!-- Date: Sat Jun 16 03:35:00 2001 -->
<br><b>Another CONET_4 (wcal04) trip!</b> 
<p>Got paged (again!) for the PSM alarm.  Rob says the supply is down to ~2V (nominally +5analog).  Voltman agrees.  (Strange that there is apparently a "purple" state on the heartbeat window.) 
Trip 400Hz again just to see if it ramps up ok and holds voltage for a while, but I'm fairly certain it will trip again tonight.  If it does, we will likely remove crate from the run and let Rick T know in case he wants to come in and watch it.  We're likely going to have to swap the PEI supply to investigate if this continues any more.  All the other 4 supplies in the PEI look like the voltages are ok.  Shifters say that there is nothing major they can correlate with these faults other than the high beam losses (which are incidentally causing other systems like CMP to trip).  Typically it happens mid-run.
<!-- Author: R. Erbacher -->



<!-- Date: Sat Jun 16 03:44:28 2001 -->
We have had 3 beam-loss associated trips of CMP in the last 30 minutes. MCR claims that they are not doing anything actively which might cause this. 
<!-- Author: Jimmy P -->
<!-- Comment: Sat Jun 16 03;59;45 comment by...Napora -->
The CMX and CPR also tripped once during one of the bigger losses.



<!-- Date: Sat Jun 16 04:10:49 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1139&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1139  width='300' align='top'></td><td>the YMon #43 slide returns normal now</a></td></tr></table>
<!-- Author: Amit Lath &Yi Le -->


<!-- Date: Sat Jun 16 04:51:53 2001 -->
<pre>CPR channels 12-17 tripped again.  Reset ok.</pre>
<!-- Author: Napora -->


<!-- Date: Sat Jun 16 04:52:04 2001 -->
<ul> 
<li>run 117863 started after CMP trip.  All converter node names were instantly red. HRR doesn't fix.  end run and remove b0l3cl02 (farms 1), which has been giving us the input_failure errors 
<li>run 117864 started sucessfully - at 169 K events get CPR power trip 
<table border=0><td><tr><td bgcolor=#e0e030>
<li>a few months ago I asked some people what the difference is between HRR/ColdStart/reboot, etc.  I believe (please correct me if I'm wrong) that the HRR only really affect the TS and hardware.  If there is a power trip you need to end the run to do a Coldstart, which sends the configuration messages to the crates for initialization.</td></tr></td></table>  

<li>thus said....we end the run after the power trip.  RC java out of memory error.  At b0dap59 command line, after killall -9 java, it tells us there are no java processes running???
<li>run 117866 started with 36x36_inclusive[1,137], run type 36x36_CURRENT (we couldn't find the RUN_117856 configuration used before) - but with FIB and VRB crates removed
</ul>
<!-- Author: hray :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117864' target=_top>117864</a>)(run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117864' target=_top>117866</a>) -->


<!-- Date: Sat Jun 16 05:51:29 2001 -->
DTO from COT09.  Unable to VxLogin to this crate.  Start Minicom, by the time things get  connected/we give up on VxLogin attempt the crate appears to have fixed itself.  Start run 117867.  See several b0l3_GATE1 errors : duplicate load ack from SCPU b0eb**.  Check all proxies, SCPU, etc. on L3 display - everything is running and checks out okay.  Not sure why we got these errors.
<!-- Author: hray :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=117867' target=_top>117867</a>) -->


<!-- Date: Sat Jun 16 06:03:33 2001 -->
<pre>something very odd happened with the MiniCom window for B0cot09.  We were logged in and seeing events - ie  
Runtime Error 1, event 5970: big event truncated. 
 
then : 
rlogind: This system *IN USE* via rlogin by user: vejcik.   
shell restarted 
 
and we're given a b0cot09 prompt similar to the one from VxLogin. 
 
Steve - are you somewhere fiddling with the COT09 crate?</pre>
<!-- Author: hray -->


<!-- Date: Sat Jun 16 06:41:34 2001 -->
<pre>Run 117867 ended with several DTO's that HRR didn't fix.  Checked proxies -  
b0eb10 and 15 needed cleaned up.  Clean up of EVB (and then L3 and EVB to make extra sure)  
didn't fix b0eb15 - from the EVB cleanup: 
b0eb15: I can't log in.  The CPU closed the rlogin connection, don't know why. 
from the fabulous L3 ace training session last week I have the knowledge to fix this -  
go downstairs to the appropriate crate and press "the button".   
L3 would like to be paged first, so we paged them..</pre>
<!-- Author: hray -->


<!-- Date: Sat Jun 16 06:51:47 2001 -->
Pressed the magical button, a check of SCRAMNet and  SCPUs shows eb15 is happy - however now 19 is not responding.  Try another cleanup. - NO 19 does not work.  Press the reset button on that  crate and hope L3 calls back soon.  Everything now clears checks on EVB/L3 page.   Starting new run.
<!-- Author: hray -->


<!-- Date: Sat Jun 16 07:13:06 2001 -->
<p>The PSM power supply tripped again.  I can't get the window to come up in iFix, but I checked it with voltman and it is the same problem as earlier.  We paged Robin again.  She said we should take wcal04 out of the run and that Rick T. may come in to look at things. 
 
<p>About that iFix problem.  When I click the CONET_4 button in the <i>Endwall Power Supply Monitoring</i> page nothing happens.  I poked around and some of the others come up when clicked...and some of the others don't. I am going to e-mail JC and John and see if they can have someone look into it.
<!-- Author: Napora -->


<!-- Date: Sat Jun 16 07:47:08 2001 -->
CPR Tripped again.  We are about to go to standby anyways.
<!-- Author: Napora -->
<!-- Comment: Sat Jun 16 07;48;06 comment by...Napora -->
...and CMU channels 0-5 on the East


<!-- Date: Sat Jun 16 07:57:49 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1140&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1140  width='300' align='top'></td><td>SvxRad readings for the shift were zilch.</a></td></tr></table>
<!-- Author: Napora -->


<!-- Date: Sat Jun 16 07:58:43 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1141&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1141  width='300' align='top'></td><td>Luminosity and losses  for the shift.   We had a couple of spikes...</a></td></tr></table>
<!-- Author: Napora -->


<!-- Date: Sat Jun 16 07:59:53 2001 -->
<font color=red>Shift Summary: </font><pre>The long tale of problems on the owl shift are briefly summarised below: 
 
1) reformatter errors  (twice) 
2) PSM error message from CONET4 (Endwall) - several times. We concluded that the problem really is the power supply. 
3) CMP, CPR HV trips - beam loss associated as far as we could tell 
4) Unreliable operation of iFIX monitor to communicate audible alarm, set RC trigger inhibit.  
5) non-operation of control room control of HV ! 
6) Trigmon crashed a couple of times 
7) iFIX computer needed to be rebooted 
8) PSM - heartbeat missing most of night 
9) EVB errors - hard reset to crates 15 & 19 
10) several large spikes in beam losses - not associated with MCR activity 
 
 
Ended run and brought HV to standby at 7.45am 
 
 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   117859-117868
Delivered Luminosity   ~0.8nb-1  pb-1
Acquired Luminosity    ?  pb-1
Efficiency             <50%
   
</td></tr></table></pre>
<!-- Author: Jimmy P. -->