<!-- Title: Owl shift. Tue Sep 13, 2005 -->
<!-- SciCo: Carmine Pagliarone -->
<!-- DAQAce: Shawn Kwang -->
<!-- MonAce: John Freeman -->
<!-- CO: Thomas Wright -->
<!-- OpManager: Bob Wagner -->
<!-- Notes: -->
<pre>
Continue data taking: Run 204211;
Store #4385 - Luminosity 18 E30 - Stash 66 mA.

Plans are:
- At 06:00 a.m. We may do Myron mode running until 06:45;
- At 06:45 a.m. Rise down HV (except CLC) to end the store.  
</pre>


<!-- Date: Tue Sep 13 00:06:34.965 2005 -->
Collider Flying Wire Scan
<!-- Author: Shawn K -->
<!-- Comment: Tue Sep 13 03;06;28 comment by...Shawn K -->
Again
<!-- Comment: Tue Sep 13 04;06;26 comment by...Shawn K -->
Again
<!-- Comment: Tue Sep 13 05;06;27 comment by...Shawn K -->
Again
<!-- Comment: Tue Sep 13 06;06;25 comment by...Shawn K -->
Again
<!-- Date: Tue Sep 13 00:19:21.732 2005 -->
<P> 
I have not been able to reproduce the error  
reported by Veronica, et al, about the situation  
where the ScalerMonitor is restarted in the middle  
of a run, and disables the UPS (über prescale) trigger  
and scales back the FPS (fractional prescale) trigger. 
<P> 
After restarting ScalerMonitor and waiting for a while,  
nothing happens;   since all triggers are fully enabled,  
we don't expect anything to happen. 
<P> 
I think I will request a new run to test the "run-restart"  
problem. 
<P>  
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204211' target=_top>204211</a>) -->
<!-- Comment: Tue Sep 13 08;16;50 comment by...Veronica -->
Hi  Bill, well one thing on this run is that UPS was not enabled, since this feature has not worked during the weekend. Also, to see the error you need the rates to be >20kHZ (including UPS), but at the luminosity of this run that was never going to happen. 
<!-- Date: Tue Sep 13 00:35:55.730 2005 -->
Two more ProcMon errors for "FCDFSGI1 TAPE."  The instructions are to ignore it unless it persists more than four hours.
<!-- Author: Shawn K -->


<!-- Date: Tue Sep 13 00:42:30.185 2005 -->
BUSY TO for b0cot08.  Auto HRR Issued.
<!-- Author: Shawn K :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204211' target=_top>204211</a>) -->

<!-- Date: Tue Sep 13 00:44:17.136 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204211">204211</A>
 Terminated at 2005.09.13 00:43:43
<!-- Author: RunControl -->


<!-- Date: Tue Sep 13 00:44:18.284 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204211">204211</A>
TERMINATE: ending run to test scalarmonitor
<!-- Author: Shawn x2080 -->



<!-- Date: Tue Sep 13 00:44:41.085 2005 -->
Since lumi is so low, I requested a run restart in order  
to test ScalerMonitor problems...
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204211' target=_top>204211</a>) -->


<!-- Date: Tue Sep 13 00:45:31.540 2005 -->
<pre>William Badgett starts the test of the ScalarMonitor.</pre>
<!-- Author: Carmine :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204211' target=_top>204211</a>) -->

<!-- Date: Tue Sep 13 00:48:23.786 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204213">204213</A>
TERMINATE: L3 didn't config correctly
<!-- Author: Shawn x2080 -->



<!-- Date: Tue Sep 13 01:01:59.835 2005 -->
<pre>Apparently we had a L3 problem during the ScalarMonitor test. Paged the L3 Expert. </pre>
<!-- Author: Carmine :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204211' target=_top>204213</a>) -->


<!-- Date: Tue Sep 13 01:07:48.460 2005 -->
<pre>Tried to clean up L3 farm twice, but it hung: 
 
Starting l3proxy after 1 monitoring-check iterations... 
[1] 20005 
Started L3 proxy process, initialization in progress. This might take a few minutes. 
12:59:10 The previous action is not finished yet! 
01:00:58 The previous action is not finished yet! 
01:00:59 The previous action is not finished yet! 
01:01:00 The previous action is not finished yet! 
01:01:00 The previous action is not finished yet! 
---------------------------------------------- 
Timeout waiting for l3proxy to initialize. Contact experts. 
---------------------------------------------- 
 
01:05:54 END: Shut down all L3 farm programs</pre>
<!-- Author: W.Badgett, lackey for daq ace -->


<!-- Date: Tue Sep 13 01:22:54.621 2005 -->
<pre>--->Status: Success. All online nodes are up (can ssh to them). 
 
Clean up cancelled 
01:19:26 END: Shut down all L3 farm programs 
01:19:32 BEGIN: Check nodes in  L3 farm 
Node b0l3gate2 is alive 
INFORMATIONAL: Product 'rgang' (with qualifiers ''), has no current chain (or may not exist) 
Checking L3 farm nodes -- this will take a minute 
--->Status: Success. All online nodes are up (can ssh to them). 
 
01:19:35 END: Check nodes in  L3 farm 
01:19:59 BEGIN: Shut down all L3 farm programs 
Node b0l3gate2 is alive 
INFORMATIONAL: Product 'rgang' (with qualifiers ''), has no current chain (or may not exist) 
Checking L3 farm nodes -- this will take a minute 
--->Status: Success. All online nodes are up (can ssh to them). 
 
Node b0l3gate2 is alive 
l3proxy process still exists. Kill it. 
l3proxy killed. 
l3proxy-related processes killed. 
can't read "branches": no such variable 
    while executing 
"if { $k != $branches } { 
      puts "Check l3_node_list file" 
      puts "Number of farms: $k" 
      puts "Number of converters: $branches" 
      exit..." 
    (file "/home/l3proxy/checkout/level3_gchouda/control/csh/l3_all_nodes_exe" line 193) 
l3proxy CLEAN 
l3proxy process still exists. 
To start a new l3proxy, issue stop command first. 
 
01:20:16 END: Shut down all L3 farm programs 
 
Latest attempt to clean up L3</pre>
<!-- Author: W.Badgett for Shawn on the phone -->


<!-- Date: Tue Sep 13 01:24:07.239 2005 -->
<pre>Last L3 clean up attempt before partition: 
 
01:22:02 BEGIN: Shut down all L3 farm programs 
Node b0l3gate2 is alive 
INFORMATIONAL: Product 'rgang' (with qualifiers ''), has no current chain (or may not exist) 
Checking L3 farm nodes -- this will take a minute 
--->Status: Success. All online nodes are up (can ssh to them). 
 
Clean up cancelled 
01:22:27 END: Shut down all L3 farm programs 
01:22:31 BEGIN: Stop L3 proxy 
Node b0l3gate2 is alive 
l3proxy process still exists. Kill it. 
l3proxy killed. 
eventserv: no process killed 
l3_mon_client: no process killed 
l3_mon_reporter: no process killed 
java: no process killed 
l3proxy-related processes killed. 
 
01:22:35 END: Stop L3 proxy 
01:22:42 BEGIN: Start L3 proxy 
Node b0l3gate2 is alive 
l3proxy process still exists. 
To start a new l3proxy, issue stop command first. 
 
01:22:43 END: Start L3 proxy 
01:23:02 BEGIN: Shut down all L3 farm programs 
Node b0l3gate2 is alive 
INFORMATIONAL: Product 'rgang' (with qualifiers ''), has no current chain (or may not exist) 
Checking L3 farm nodes -- this will take a minute 
--->Status: Success. All online nodes are up (can ssh to them). 
 
Clean up cancelled 
01:23:13 END: Shut down all L3 farm programs 
</pre>
<!-- Author: W.Badgett for Shawn -->
<!-- Comment: Tue Sep 13 01;25;14 comment by...Concerned Colleague -->
Command "rgang"?   Perhaps Spanky is involved in mischief...


<!-- Date: Tue Sep 13 01:35:50.183 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16366&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16366' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16367&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16367' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16369&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16369' width='300'></a></center>
<!-- Author: Freeman -->
<!-- Comment: Tue Sep 13 01;36;14 comment by...Freeman -->
Hourly plots, 2330-0030


<!-- Date: Tue Sep 13 01:39:07.273 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16370&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16370' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16371&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16371' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16372&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16372' width='300'></a></center>
<!-- Author: Freeman -->
<!-- Comment: Tue Sep 13 01;39;33 comment by...Freeman -->
Hourly plots, 0030-0130


<!-- Date: Tue Sep 13 02:00:48.669 2005 -->
<pre>Paged the Silicon expert. Problem with the voltage control in ISL 
Barrel 4 - Wedge 0 - Layer 2.</pre>
<!-- Author: Carmine :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204211' target=_top>204211</a>) -->
<!-- Comment: Tue Sep 13 02;38;16 comment by...Freeman -->
<pre>Basically, the standard procedure as taped to the desk in front of the PS Gui computer didn't fix the problem. </pre>
<!-- Date: Tue Sep 13 02:08:24.090 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204214">204214</A>
 Activated at 2005.09.13 02:08:22
<!-- Author: RunControl -->


<!-- Date: Tue Sep 13 02:08:51.841 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204214">204214</A>
 Terminated at 2005.09.13 02:08:31
<!-- Author: RunControl -->


<!-- Date: Tue Sep 13 02:08:52.503 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204214">204214</A>
TERMINATE: CONFIG/END for Sebastian.
<!-- Author: Shawn x2080 -->


<!-- Date: Tue Sep 13 02:13:27.582 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204215">204215</A>
TERMINATE: PARTITION/CONFIG/ABORT/RESET just for Silicon, per Sebastian's request
<!-- Author: Shawn x2080 -->
<!-- Comment: Tue Sep 13 02;22;49 comment by...markus -->
<pre>L3 disaster:
I was paged because L3 failed to configure. The reason was that a disk filled up and the L3 proxy was not able to write to its log file anymore. While cleaning up the disk I removed EVB2 code ( thought it's only a backup) on b0l3pcom2 which includes a product "rgang" used by the ACE control panel to check the L3 farm. This disabled the option of cleaning up the L3 farm. I cleaned up by hand and decided to reboot b0l3pcom2 which was kind of slow ( maybe just a network problem ). The machine never came back from the reboot and Shawn had to press the reset button. After all this we were able to restart. Well done Shawn and thanks for your help. Want to become a L3 expert?</pre>
<!-- Date: Tue Sep 13 02:18:14.236 2005 -->
<pre>We got another ISL Alarm. PS-trip Barrel 0 - Wedge 5 - Layer 1. 
Paged the Silicon Expert. </pre>
<!-- Author: Carmine -->

<!-- Date: Tue Sep 13 02:20:13.126 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204216">204216</A>
 Activated at 2005.09.13 02:19:18
<!-- Author: RunControl -->


<!-- Date: Tue Sep 13 02:20:14.412 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204216">204216</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_07 [2,713,594] 
<!-- Author: Shawn x2080 -->




<!-- Date: Tue Sep 13 02:37:15.780 2005 -->
Comments on ISL issues: although the global alarms claim that B0 W5 L1 -and maybe others - have tripped, looking at Imon it appears that it's again ISL B4 W0 L2 which is red. Sebastian has me turn the Ps Gui off and then on again, turn off both the B0 W5 L1 and B4 W0 L2 and turn them on again, reset the trips, and finally it looks like we're OK. 
<!-- Author: Freeman -->



<!-- Date: Tue Sep 13 02:45:35.773 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16373&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16373' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16374&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16374' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16375&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16375' width='300'></a></center>
<!-- Author: Freeman -->



<!-- Date: Tue Sep 13 03:12:07.607 2005 -->
We lose heartbeat to CPR, CLC, BSC, RPS, MNP. I go downstairs to the computer, close and restart the program. CPR's voltage is at 0% to start with so I have to manually turn it on from the computer. Now it seems we're OK. 
<!-- Author: Freeman -->


<!-- Date: Tue Sep 13 03:30:12.611 2005 -->
We had a TOF inhibit.  This resulted in the run being halted.  However a few moments later the inhibit cleared and the run recovered.
<!-- Author: Shawn K :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204216' target=_top>204216</a>) -->


<!-- Date: Tue Sep 13 03:43:00.197 2005 -->
<pre>In the current run, ScalerMonitor was left running with  
the previously mentioned special version; it seems to  
have enabled all DPS types, except for UPS. 
 
I have presently restarted ScalerMonitor with some more  
tweaks to figure out why UPS is not getting enabled... 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204216' target=_top>204216</a>) -->


<!-- Date: Tue Sep 13 03:43:14.055 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16376&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16376' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16377&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16377' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16378&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16378' width='300'></a></center>
<!-- Author: Freeman -->
<!-- Comment: Tue Sep 13 03;43;37 comment by...Freeman -->
Hourly plots, 0230-0330


<!-- Date: Tue Sep 13 03:49:26.092 2005 -->
Restart ScalerMonitor again.
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204216' target=_top>204216</a>) -->
<!-- Comment: Tue Sep 13 03;51;05 comment by...WB -->
again

<!-- Comment: Tue Sep 13 03;57;08 comment by...WB -->
again

<!-- Comment: Tue Sep 13 04;00;06 comment by...WB -->
again (no effect on data!)

<!-- Comment: Tue Sep 13 04;03;45 comment by...WB -->
again (this time fer sure, rocky)

<!-- Comment: Tue Sep 13 04;06;06 comment by...WB -->
That one did it, Rocky, I found the bug!   My TriggerMasters 
will be pleased.   But now I have to make sure UPS doesn't 
keep getting enabled...



<!-- Comment: Tue Sep 13 04;10;59 comment by...WB -->
ScalerMonitor restarted again...

<!-- Comment: Tue Sep 13 04;14;38 comment by...WB -->
Restart again, but this time set UPS back to zero to see if 
it still works...

<!-- Comment: Tue Sep 13 04;16;19 comment by...WB -->
OK, it worked again and UPS trigger is reënabled, but 
ScalerMonitor is not going crazy with retries.


<!-- Comment: Tue Sep 13 08;48;44 comment by...Rocky -->
Hey Bullwinkle, did you have anything up your sleeve???
<!-- Date: Tue Sep 13 04:28:24.836 2005 -->
For the ScalerMonitor, I found the "smoking gun(s)" for  
the UPS trigger problems and they have been fixed.  About  
the FPS and the new-run problems, it's still not clear  
exactly what's going on.   I have left in extra debug output  
to help if it happens again;  also, I may request another  
DAQ run to be started...
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204216' target=_top>204216</a>) -->


<!-- Date: Tue Sep 13 04:41:17.210 2005 -->
One tugboat toot. No visual indicator of a problem. 
<!-- Author: Freeman :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204216' target=_top>204216</a>) -->


<!-- Date: Tue Sep 13 04:44:21.744 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16379&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16379' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16380&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16380' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16381&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16381' width='300'></a></center>
<!-- Author: Freeman -->
<!-- Comment: Tue Sep 13 04;44;52 comment by...Freeman -->
Hourly plots, 0330-0430


<!-- Date: Tue Sep 13 05:02:31.627 2005 -->
BUSY TO from b0cot08.  Auto HRR Issued.
<!-- Author: Shawn K :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204216' target=_top>204216</a>) -->


<!-- Date: Tue Sep 13 05:07:18.114 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204216">204216</A>
 Terminated at 2005.09.13 05:06:44
<!-- Author: RunControl -->


<!-- Date: Tue Sep 13 05:07:19.064 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204216">204216</A>
TERMINATE: ending run for scalarmon test, take two.
<!-- Author: Shawn x2080 -->


<!-- Date: Tue Sep 13 05:12:19.716 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204221">204221</A>
 Activated at 2005.09.13 05:11:43
<!-- Author: RunControl -->


<!-- Date: Tue Sep 13 05:12:21.131 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204221">204221</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_07 [2,713,594] 
<!-- Author: Shawn x2080 -->
<!-- Comment: Tue Sep 13 05;12;45 comment by...Shawn K -->
RXPT error from b0l1cl3 at begining of run, instead of the usual BUSY TO.
<!-- Date: Tue Sep 13 05:12:21.347 2005 -->
ScalerMonitor has <I>not</I> been restarted since the  
last run;   let's sit back and see what happens.
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204221' target=_top>204221</a>) -->
<!-- Comment: Tue Sep 13 05;15;43 comment by...WB -->
FPS just kicked in

<!-- Comment: Tue Sep 13 05;18;02 comment by...WB -->
The L1 LPS (lumi enables) just kicked in...


<!-- Comment: Tue Sep 13 05;22;55 comment by...WB -->
The L2 DPS have just kicked in...

<!-- Comment: Tue Sep 13 05;37;17 comment by...WB -->
L1 UPS (über prescale) kicked in at 05:26:24<BR>
More L2 DPS trigger kicked in at 05:31:29


<!-- Comment: Tue Sep 13 05;44;33 comment by...WB -->
I believe all the DPS triggers of various sorts, are all 
fully enabled now;  not clear why it works now and not earlier...


<!-- Date: Tue Sep 13 05:55:23.225 2005 -->
Called Nils we will not do Myrond mode running during the present shift.
<!-- Author: Carmine -->


<!-- Date: Tue Sep 13 05:58:42.278 2005 -->
<pre>We got another ISL PS-trip: Barrel 4 - Wedge 0 - Layer 3. 
Paged the Silicon expert.</pre>
<!-- Author: Carmine :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204221' target=_top>204221</a>) -->


<!-- Date: Tue Sep 13 06:18:56.949 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204221">204221</A>
 Terminated at 2005.09.13 06:18:24
<!-- Author: RunControl -->


<!-- Date: Tue Sep 13 06:18:57.137 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204221">204221</A>
TERMINATE: end run due to ISL problems
<!-- Author: Shawn x2080 -->


<!-- Date: Tue Sep 13 06:37:10.359 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204222">204222</A>
 Activated at 2005.09.13 06:37:03
<!-- Author: RunControl -->


<!-- Date: Tue Sep 13 06:37:42.650 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204222">204222</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_07 [2,713,594] 
<!-- Author: Shawn x2080 -->



<!-- Date: Tue Sep 13 06:48:10.283 2005 -->
<pre>Ramping down all HV except CLC for studies.</pre>
<!-- Author: Carmine -->

<!-- Date: Tue Sep 13 06:48:59.790 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204222">204222</A>
 Terminated at 2005.09.13 06:47:52
<!-- Author: RunControl -->


<!-- Date: Tue Sep 13 06:49:00.941 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204222">204222</A>
TERMINATE: ending run to ramp down HVs
<!-- Author: Shawn x2080 -->



<!-- Date: Tue Sep 13 07:02:34.664 2005 -->
Starting EVB tests to help track down the beginning of  
run error problem...   Requires crate reboots.
<!-- Author: W.Badgett -->


<!-- Date: Tue Sep 13 07:08:18.965 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16382&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16382' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16383&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16383' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16384&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16384' width='300'></a></center>
<!-- Author: Freeman -->


<!-- Date: Tue Sep 13 07:10:09.445 2005 -->
Loud alarm buzzes once, then stops
<!-- Author: Freeman -->


<!-- Date: Tue Sep 13 07:11:01.600 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16385&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16385' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16386&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16386' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16387&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16387' width='300'></a></center>
<!-- Author: Freeman -->
<!-- Comment: Tue Sep 13 07;11;28 comment by...Freeman -->
Hourly plots, 0530-0630


<!-- Date: Tue Sep 13 07:12:18.569 2005 -->
Now taking a standard L2TORTURE run after rebooting  
all front end crates, to make sure the nominal fer  
v4_09 configuration does not have any problems;  note that  
I have disabled the error line from b0l1cl03, since this  
was occluding the EVB first-event problem.  
<!-- Author: W.Badgett -->

<!-- Date: Tue Sep 13 07:18:36.569 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204224">204224</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539] 
<!-- Author: Shawn x2080 -->



<!-- Date: Tue Sep 13 07:20:09.281 2005 -->
<pre>Kinda peculiar BOR occurrence: 
 
FrontEnd Crate Error Condition from: 
LEVEL1_CAL_00 
LEVEL1_CAL_02 
LEVEL1_CAL_04 
LEVEL1_CAL_05 
LEVEL1_GLOBAL_00 
 
perhaps a problem systemic the LEVEL1_CAL system? 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=2-4224' target=_top>2-4224</a>) -->


<!-- Date: Tue Sep 13 07:25:03.019 2005 -->
<pre>Have also disabled the error line for the crates noted  
to cause problems above, plus one more, and the original one: 
 
LEVEL1_CAL_00  
LEVEL1_CAL_01    (did not give an error previous run) 
LEVEL1_CAL_02  
LEVEL1_CAL_03    (originally masked in previous run) 
LEVEL1_CAL_04  
LEVEL1_CAL_05  
LEVEL1_GLOBAL_00  
 
Let's try this L2TORTURE run again... 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204225' target=_top>204225</a>) -->

<!-- Date: Tue Sep 13 07:25:58.913 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204225">204225</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539] 
<!-- Author: Shawn x2080 -->


<!-- Date: Tue Sep 13 07:31:32.201 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204225">204225</A>
TERMINATE: Ran OK after first Busy TO
<!-- Author: Shawn x2080 -->



<!-- Date: Tue Sep 13 07:39:01.215 2005 -->
<pre>For this run, I have disabled the reset of the TAXI  
fibre link between the Tracer and the VRB;   earlier  
obvservations by Radu indicated that this was causing  
extra bytes to be presented on the link during early  
events. 
 
I have excluded COT and CCAL crates, since they are too  
painful to reboot. 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204226' target=_top>204226</a>) -->
<!-- Comment: Tue Sep 13 07;40;35 comment by...WV -->
Receive perpetual L2 decision timeouts during this 
run;  inconclusive w.r.t EVB.

<!-- Date: Tue Sep 13 07:39:19.106 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204226">204226</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539] 
<!-- Author: Shawn x2080 -->
<!-- Comment: Tue Sep 13 07;47;18 comment by...W.Badgett -->
Still getting perpetual L2 Decision Timeouts!

Argh!!!!!!!!!!!!!!!!!!!!
<!-- Comment: Tue Sep 13 07;49;01 comment by...WB -->
Oops;  that "Argh!!!!!!!!!!!!!" belong to the following 
run, 204228
<!-- Date: Tue Sep 13 07:45:13.750 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16388&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16388' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16389&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16389' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-16390&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-16390' width='300'></a></center>
<!-- Author: Freeman -->
<!-- Comment: Tue Sep 13 07;45;38 comment by...Freeman -->
Hourly plots, 0630-0730
<!-- Date: Tue Sep 13 07:45:24.360 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204228">204228</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539] 
<!-- Author: Shawn x2080 -->
<!-- Comment: Tue Sep 13 07;47;38 comment by...W.Badgett -->
 Still getting perpetual L2 Decision Timeouts! Argh!!!!!!!!!!!!!!!!!!!!


<!-- Date: Tue Sep 13 07:52:53.779 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204229">204229</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539] I forgot pulsar can't handle 
the SMXR cards being missing;  have added CCAL;   but still got a Busy T/O.
<!-- Author: Shawn x2080 -->


<!-- Date: Tue Sep 13 07:55:23.337 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>Physics able</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L1Accepts</TH>
<TH>L2Accepts</TH>
<TH>L3Accepts</TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204211 target=_blank>204211</A> x31DB3</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_3_07 [2,713,594]</FONT></TD>
<TD ALIGN=CENTER>19:20:22</TD>
<TD ALIGN=CENTER>00:43:43</TD>
<TD ALIGN=CENTER>04:57:50</TD>
<TD ALIGN=RIGHT>    136,650,509</TD>
<TD ALIGN=RIGHT>     2,612,277</TD>
<TD ALIGN=RIGHT>     790,082</TD>
<TD ALIGN=RIGHT>  363.294</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204216 target=_blank>204216</A> x31DB8</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_3_07 [2,713,594]</FONT></TD>
<TD ALIGN=CENTER>02:19:18</TD>
<TD ALIGN=CENTER>05:06:44</TD>
<TD ALIGN=CENTER>02:21:53</TD>
<TD ALIGN=RIGHT>     70,161,894</TD>
<TD ALIGN=RIGHT>     1,056,400</TD>
<TD ALIGN=RIGHT>     337,706</TD>
<TD ALIGN=RIGHT>  131.472</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204221 target=_blank>204221</A> x31DBD</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_3_07 [2,713,594]</FONT></TD>
<TD ALIGN=CENTER>05:11:43</TD>
<TD ALIGN=CENTER>06:18:24</TD>
<TD ALIGN=CENTER>00:37:10</TD>
<TD ALIGN=RIGHT>     19,345,459</TD>
<TD ALIGN=RIGHT>       240,792</TD>
<TD ALIGN=RIGHT>      83,941</TD>
<TD ALIGN=RIGHT>   32.153</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204222 target=_blank>204222</A> x31DBE</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_3_07 [2,713,594]</FONT></TD>
<TD ALIGN=CENTER>06:37:03</TD>
<TD ALIGN=CENTER>06:47:52</TD>
<TD ALIGN=CENTER>00:10:24</TD>
<TD ALIGN=RIGHT>      2,665,993</TD>
<TD ALIGN=RIGHT>        41,393</TD>
<TD ALIGN=RIGHT>      16,131</TD>
<TD ALIGN=RIGHT>    8.545</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>0</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:01</TD>
<TD ALIGN=CENTER>08:07:17</TD>
<TD ALIGN=RIGHT>     228,823,855</TD>
<TD ALIGN=RIGHT>      3,950,862</TD>
<TD ALIGN=RIGHT>     1,227,860</TD>
<TD ALIGN=RIGHT>  535.464</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->


<!-- Date: Tue Sep 13 07:59:10.597 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=204230">204230</A>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,623,539] 
<!-- Author: Shawn x2080 -->



<!-- Date: Tue Sep 13 07:59:10.879 2005 -->
<font color=red>Shift Summary: </font><pre>00:00  Continue data taking: Run 204211; 
       Store #4385 - Luminosity 18 E30 - Stash 66 mA. 
 
00:44  Run 204211 TERMINATE for ScalarMonitor tests; 
00:48  Run 204213 TERMINATE: L3 didn't config correctly; 
 
01:01  We had a L3 problem during the ScalarMonitor test;  
       What happened is that L3 failed to configure because a disk filled  
       up and the L3 proxy was not able to write to its log file anymore; 
 
02:00  ISL Problem: PS TRIP B4W0L2; 
02:18  ISL Problem: PS-TRIP B0W5L1;  
       After turning off and on again the PS GUI and 
       turning off and on again both ISL B0W5L1 and B4W0L2 we were back ok; 
02:20  Taking data again: PHYSICS_3_07 [2,713,594]; 
05:05  Run 204216 TERMINATE for L3 tests; 
05:58  ISL Problem: PS-TRIP B4W0L3; 
06:48  Ramped down all HV except CLC for studies. 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   204211 to 204229
Delivered Luminosity   16.6 nb-1  
Acquired Luminosity    	  0.0 nb-1  
Efficiency               0.1
   
</td></tr></table></pre>
<!-- Author: Carmine Pagliarone -->