<!-- Title: Day shift. Sun Jul 30, 2006 -->
<!-- SciCo: Jay Dittmann -->
<!-- DAQAce: Niharika Ranjan -->
<!-- MonAce: Hugo Beauchemin -->
<!-- CO: Mitch Soderberg -->
<!-- OpManager: Guram -->
<!-- Notes: -->
<pre>Inherited store #4862, B0Lum 77.6E10 
Physics Run 220726 in progress. 
Stack 10.13E0, RR 84.72E10 
MCR wants to keep this store for a long time since stacking is on hold</pre>


<!-- Date: Sun Jul 30 08:28:43.393 2006 -->
<pre>During the previous shift we had RTSERVER error in ProcMon.  
Some of the error messages were: 
 
Found 1 potentially stuck ports   
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       
Inode       
tcp        0  73488 b0server05.fnal.gov:5101    b0dasvx02.fnal.gov:32998    ESTABLISHED cdfdaq     
936731391    
RTserver is not running correctly  
 
b0dasvx02 is where IMON is running. Probably, IMON stopped updating when we got the first RTSERVER error around 6am.  
 
</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220726' target=_top>220726</a>) -->


<!-- Date: Sun Jul 30 08:28:52.989 2006 -->
<pre>After previous crew as restarted IMON, many pinkies appeared.  
After unmarking them, 6 persisted. We paged the silicon expert. 
One is a knew one, so it has been set to ignore (AVDD_B0_W1_L2). 
The other 5 oscillate slightly around one of their limits. We 
monitor them continously for further changes (BIAS: B2_W1_L0, 
B2_W1_L1, B3_W2_L0, B4_W10_L2 and B5_W7_L2).</pre>
<!-- Author: Hugo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220726' target=_top>220726</a>) -->


<!-- Date: Sun Jul 30 08:57:35.288 2006 -->
<table><tr><td><a href=/elog/notebooks/2006/Sun_Jul_30_2006_Day.085720.ps align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-7740  align='top'></a></td><td>PPR Wedge Occupancy plot looks low across the board.  </td></tr></table>
<!-- Author: Mitch Soderberg -->
<!-- Comment: Sun Jul 30 08;59;23 comment by...Jay D -->
<pre>Paged expert</pre>


<!-- Comment: Sun Jul 30 09;14;48 comment by...Jay D -->
Willis responds and will take a look at the plot
<!-- Comment: Sun Jul 30 09;31;47 comment by...Willis -->
The PPR looks okay to me in the YMon 2D-occupancy plots, so the
detector itself is OK. The YMon slide comparison is probably
confused.
<!-- Comment: Sun Jul 30 11;49;28 comment by...Larry N -->
rate rise with luminosity os not part of prediction, all are high, it is ik
<!-- Comment: Sun Jul 30 11;50;05 comment by...Larry N -->
oops OK
<!-- Date: Sun Jul 30 09:02:22.267 2006 -->
<pre>IFIX complains because it has been backup since July 20th.</pre>
<!-- Author: Hugo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220726' target=_top>220726</a>) -->


<!-- Date: Sun Jul 30 09:09:55.536 2006 -->
<table><tr><td><a href=/elog/notebooks/2006/Sun_Jul_30_2006_Day.090936.ps align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-7742  align='top'></a></td><td>BMU West total hit plots have some new spikes in them.  Taichi noticed this during the previous <a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=537&frame=2&anchor=01:32:37.269'> 
shift </a>.</td></tr></table>
<!-- Author: Mitch Soderberg -->
<!-- Comment: Sun Jul 30 09;21;05 comment by...Jay D -->
<pre>Page expert.  Sunny calls...says that this is a recurring problem and cycling LV from 1st floor does not work to fix stack 52.  Need an access to fix.</pre>
<!-- Comment: Sun Jul 30 10;05;27 comment by...Sunny -->
LV dial for BMU NW section was rotated, thereby LV power cycled, anyway. If it doesn't work, an access will be needed. 
<!-- Comment: Sun Jul 30 10;17;02 comment by...Sunny  -->
We just saw the BMU NW occupancy plots after the LV for BMU NW was power cycled... the hot channels are still there. Mitch will post the plots soon. 
<!-- Date: Sun Jul 30 09:13:00.832 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7743&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7743' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7744&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7744' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7745&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7745' width='300'></a></center>
<!-- Author: Hugo -->





<!-- Date: Sun Jul 30 09:47:02.864 2006 -->
<pre>Mon ACE (Hugo) noticed that the lower bar (HV summary page) has been missing since a couple of mins for TSU. Clicking on the D shows that the HV min block is white and the value below it says 0.1% 
Clicking on NW-ISU-A shows all the boxes orange in color and we get a warning saying 'Fill the Database First from Main Menu' 
 
Hugo went down to check the PC first.</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220726' target=_top>220726</a>) -->


<!-- Date: Sun Jul 30 09:51:08.732 2006 -->
<pre> 
 DAQErrMon:DataMissingError:  
 Error 720 Event 5921929 Run 220726 Bank SL1D_StorableBank missing from event record, nErrors 303  
 
 - We suggest issuing an HRR.  
  
 - if the problem persists for more than a few minutes,  
   starting a new Run may be necessary. </pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220726' target=_top>220726</a>) -->
<!-- Comment: Sun Jul 30 09;54;23 comment by...Niharika -->
<pre>MLE) b0felix.fnal.gov:TEDCpp:Jul 30, 2006 09:50:54 AM-&gt;{ 9:50:54, 7/30/2006 }: L2-Pulsar-Linux: Halted while  waiting for ClusterFinder data
(RC)   9:50:54 Halt -&gt; HALTED
(RC)   9:51:02 Recover -&gt; RECOVERED
(RC)   9:51:04 Run -&gt; ACTIVE
(TS)   9:51:15 ********* L2 trigger rate has dropped to 0 Hz! Events are not being logged !
(MLE) b0svx01:Messenger:Jul 30, 2006 09:51:53 AM-&gt;Silicon Resonator: Sl 10 Ch 9 -&gt; e134: Bytes: 00054

(MLE) b0svx03:Messenger:Jul 30, 2006 09:51:53 AM-&gt;Silicon Resonator: Sl 10 Ch 9 -&gt; e334: Bytes: 00056

(MLE) b0svx08:Messenger:Jul 30, 2006 09:51:53 AM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; f843: Bytes: 00780

(MLE) b0svx07:Messenger:Jul 30, 2006 09:51:52 AM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; fa03: Bytes: 00780

(MLE) b0svx02:Messenger:Jul 30, 2006 09:51:53 AM-&gt;Silicon Resonator: Sl 12 Ch 4 -&gt; e224: Bytes: 00088

(MLE) b0felix.fnal.gov:TEDCpp:Jul 30, 2006 09:51:54 AM-&gt;{ 9:51:54, 7/30/2006 }: L2-Pulsar-Linux: Halted while  waiting for ClusterFinder data
(RC)   9:51:54 Halt -&gt; HALTED
(RC)   9:52:01 Recover -&gt; RECOVERED
(RC)   9:52:03 Run -&gt; ACTIVE
(CSL)  9:52:19 ********* L3 trigger rate has dropped to 0 Hz! Events are not being logged !
(MLE) b0svx01:Messenger:Jul 30, 2006 09:52:33 AM-&gt;Silicon Resonator: Sl 10 Ch 9 -&gt; e134: Bytes: 00054

(MLE) b0svx03:Messenger:Jul 30, 2006 09:52:33 AM-&gt;Silicon Resonator: Sl 10 Ch 9 -&gt; e334: Bytes: 00056

(MLE) b0svx08:Messenger:Jul 30, 2006 09:52:33 AM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; f843: Bytes: 00780

(MLE) b0svx07:Messenger:Jul 30, 2006 09:52:33 AM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; fa03: Bytes: 00780

(MLE) b0svx02:Messenger:Jul 30, 2006 09:52:33 AM-&gt;Silicon Resonator: Sl 12 Ch 4 -&gt; e224: Bytes: 00088

(MLE) b0felix.fnal.gov:TEDCpp:Jul 30, 2006 09:52:34 AM-&gt;{ 9:52:34, 7/30/2006 }: L2-Pulsar-Linux: Halted while  waiting for ClusterFinder data
(RC)   9:52:35 Halt -&gt; HALTED
(RC)   9:52:42 Recover -&gt; RECOVERED
(RC)   9:52:44 Run -&gt; ACTIVE
</pre>
<!-- Comment: Sun Jul 30 10;00;35 comment by...Niharika -->
<pre>Although I issued HRR, and we got back in active state, the number of events (in run control) were not updating, starting new run</pre>
<!-- Date: Sun Jul 30 09:54:15.322 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220726>220726</A>
 Terminated at 2006.07.30 09:53:42
<!-- Author: RunControl -->


<!-- Date: Sun Jul 30 09:54:16.868 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220726>220726</A>
TERMINATE: DataMissingErrros
<!-- Author: Niharika -->
<!-- Comment: Sun Jul 30 09;57;01 comment by...Niharika -->
<pre>b0eproxy02.fnal.gov:main:Jul 30, 2006 09:55:55 AM-&gt;Error on L3 node b0eproxy02.fnal.gov (partition 0) Sun Jul 30 09:55:55 2006 l3proxy 
  in End (L3ProxyPartition.cc:3043)
  @L3_RELAY_FAILURE Level3 Relay failure detected!
The following subfarms need to be removed from the next runs:
      None. You can use all the same subfarms.
The following Level3 converter nodes are marked bad for now:
      None.
The full list of Level3 nodes giving Relay trouble is printed below

 b0l3s03p07 b0l3s03p17</pre>


<!-- Date: Sun Jul 30 09:59:23.792 2006 -->
<pre>I went down and notice few different things. I don't know the degree of their correlation: 
 
1-Got messages saying that CDF_S3.R_JS_AC_IMU_XX.X_CV have connection not established (where XX.X stand for: NW.F, NE.A, SW.F, NW.A, SE.F, NE.F and SW.A) 
 
2-On IFIX , in TSU, CC0, all boxes are orange and a message say: check database first in main menu. 
 
3-This is the one that seems to be directly related to our TSU "problem": on the computer in 1RR15C, it is mentionned: 
   
        CCU 02 is unreadable. Mail expert he will reset CPU. 
 
On IFIX, CCU 02 is full of red boxes. 
 
 
</pre>
<!-- Author: Hugo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220726' target=_top>220726</a>) -->



<!-- Date: Sun Jul 30 10:04:13.484 2006 -->
<pre>*** Node b0cot14 in Error *** 
Heap is Corrupt 
 
while trying to start new run</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727' target=_top>220727</a>) -->


<!-- Date: Sun Jul 30 10:06:08.949 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727>220727</A>
 Activated at 2006.07.30 10:05:50
<!-- Author: RunControl -->


<!-- Date: Sun Jul 30 10:06:09.316 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727>220727</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [4,816,640] 
<!-- Author: Niharika -->




<!-- Date: Sun Jul 30 10:19:15.523 2006 -->
<table><tr><td><a href=/elog/notebooks/2006/Sun_Jul_30_2006_Day.101900.ps align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-7747  align='top'></a></td><td>Plot of TSU Occupancy for current run, for expert to look at.</td></tr></table>
<!-- Author: Mitch Soderberg -->


<!-- Date: Sun Jul 30 10:20:31.488 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-7748&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-7748  width='300' align='top'></a></td><td>Hopefully a fixed version of the previous plot.</td></tr></table>
<!-- Author: Mitch Soderberg -->


<!-- Date: Sun Jul 30 10:23:53.400 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7749&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7749' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7750&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7750' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7751&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7751' width='300'></a></center>
<!-- Author: Hugo -->

<!-- Date: Sun Jul 30 10:37:43.274 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220726>220726</A>
monitors: <tt>Received lots of DAQErrorMon "Missing Data" errors which caused us to end the run.</tt>
<!-- Author: Consumer Operator( Mitch Soderberg) -->



<!-- Date: Sun Jul 30 10:45:00.089 2006 -->
<pre>Done Timeout detected in crates: 
COT_09 
 
Issued HRR</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727' target=_top>220727</a>) -->
<!-- Comment: Sun Jul 30 10;58;43 comment by...Niharika -->
<pre>We had a couple of Done Time outs for the same COT crate. I tried shepherding it, but it failed. Shepherding worked the 4th time. 
Here are some messages from the error log:

(MLE) b0cot09:Messenger:Jul 30, 2006 10:48:46 AM-&gt;TDC in slot 8, ID PROM not found (blank check)
(MLE) b0cot09:Messenger:Jul 30, 2006 10:48:46 AM-&gt;Resetting card in slot 8 failed
(MLE) b0cot09:Messenger:Jul 30, 2006 10:48:46 AM-&gt;TDC in slot 8 Toggle of event mask failed : 0xffffffff
(MLE) b0cot09:Messenger:Jul 30, 2006 10:48:46 AM-&gt;TDC in slot 8, found DspVer=255, expected 68
(MLE) b0cot09:Messenger:Jul 30, 2006 10:48:46 AM-&gt;Wrong TDC boardType: found 15, expected 9 in slot 8
(MLE) b0cot09:Messenger:Jul 30, 2006 10:48:46 AM-&gt;Resetting card in slot 8 failed
(MLE) b0cot09:Messenger:Jul 30, 2006 10:48:46 AM-&gt;Statechange failed
b0control21.fnal.gov:Thread-36:Jul 30, 2006 10:50:12 AM-&gt;Failed Shepherding client COT_09 (b0cot09) --&gt;  
 Additional Information: 

 Attention!!!. FERML_SHEPHERD_FAILED !!!. 
 - Please take the necessary actions.
 - NO MORE Auto-SHEPHERD WILL BE ISSUED 
 - until after 3 minutes of smooth running.

Also, in the FrontEndConsole for b0cot09 we noticed that there was a message regarding VME bus failure for slot 8. 
</pre>
<!-- Date: Sun Jul 30 10:46:00.210 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-7753&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-7753  width='300' align='top'></a></td><td>Here is the plot that Sunny mentioned <a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=538&frame=2&anchor=09:09:55.536'> 
previously </a> of BMU West hits after power-cycling of BMU NW.  </td></tr></table>
<!-- Author: Mitch Soderberg -->



<!-- Date: Sun Jul 30 11:04:46.869 2006 -->
<pre>Just trying the new L3 feature: 
 
---------------------------------------------------------- 
--- Report of LoadErrors and byte count errors within the last hour --- 
LoadError in b0evme12, at 2006/07/30 10:43:44, in channel 2, receiving COT_09. 
Total errors: 1.</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727' target=_top>220727</a>) -->
<!-- Comment: Sun Jul 30 12;00;50 comment by...Conor -->
Cool.
<!-- Date: Sun Jul 30 11:16:54.460 2006 -->
<pre>Done Timeout detected in crates: 
COT_09 
</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727' target=_top>220727</a>) -->
<!-- Comment: Sun Jul 30 11;18;14 comment by...Niharika -->
<pre>Auto HRR fixed it</pre>


<!-- Date: Sun Jul 30 11:33:53.347 2006 -->
Power cycled North East side of muon scintillators to restore response from CCU NE-ISU-B (#02). Response is good now.
<!-- Author: Oleg Pukhov :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727' target=_top>220727</a>) -->
<!-- Comment: Sun Jul 30 11;44;43 comment by...Oleg -->
Despite the fact that CCU response was bad Occupancy plots for TSU and BSU look fine. It means that high voltage was correct, problem was in CCU response only.
<!-- Date: Sun Jul 30 11:36:16.438 2006 -->
TrigMon crushes w/ SVTSim.<br> 
Error is coming from svtsim_dbaccess.cc<br> 
TrigMon works fine w/o STVSim.<br>
<!-- Author: Junji :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727' target=_top>220727</a>) -->
<!-- Comment: Sun Jul 30 11;38;35 comment by...Jay D -->
page SVT expert
<!-- Comment: Sun Jul 30 11;58;49 comment by...Junji -->
<pre>svtsimmodule::beginRun: 
  run=220727
run=220727 m= mc=0 h= hc=0
%ERLOG-e Error in trigger table query! ABORTING EXECUTION!!! (exit(1) follows): 
         /cdf/onln/code/b0cdfsoft/cdfsoftb0/6.1.4o/svtsim/src/svtsim_dbaccess.cc 
        : 189   GlobalLibraryLogger 
         svtsim::svtsim_build_xtf_bitmask_from_l1_table() 
         30-Jul-2006 11:55:14 CDT run = 220727 event = 0
Termination Summary
</pre>
<!-- Comment: Sun Jul 30 13;17;34 comment by...Ray Culbertson -->
<pre>The line "run=220727 m= mc=0 h= hc=0"
should not be reporting blanks and zeros.
Tracing the code back, I see it comes from a query like:

select type from svt_data
where createdate=
  (select max(createdate) from svt_data
    where createdate&lt;=(
      select rc.fetchtime from runconfigurations rc
        where rc.runnumber=220727 )
    and name like 'map@%')
  and name like 'map@%';

which does work when executed stand-alone, so the DB is 
working and the data is there.  So now I would look at the
tcl or defaults which tell the code how to connect to the DB.

I'd also look for other error messages in the log.
</pre>
<!-- Comment: Sun Jul 30 13;45;39 comment by...Ray Culbertson -->
<pre>In ~cdfdaq/consumer_scripts/startSubConsumer_TrigMon_sl_614n.csh
there is an line:
setenv IOMAP_FILE /cdf/onln/data/cdfdaq/consumer/runarea/TrigMon/iomap.txt
which is usually where onotl_prd_read is defined, but it is not.
In /cdf/onln/data/cdfdaq/consumer/runarea/TrigMon/Consumer/TrigMon/runTrigMon.tcl
there are lines:
    DataDB      set  onotl_prd_read
    Database    set  onotl_prd_read
which seem to request the undefined aliases.
I am not sure this is a problem, for exmaple,
these aliases can be defined in code.  I also have no idea 
why any of this might have changed.  Also the other db acess
would have to be avoiding the problem.  Anyway, it might 
be worth a try to change "onotl_prd_read" to "cdfonprd".
in the tcl.

The next thing I would do is run the exe in debug to see 
what step is failing.  If the consumers expert can tell me how to run it standalone, I could try that.
</pre>
<!-- Comment: Sun Jul 30 13;49;38 comment by...Ray Culbertson -->
Also consumers expert can 
setenv DEBUG 10
and run.  This causes extensive database connection
debugging to print.  It should be obvious where the original
problem might be.
<!-- Comment: Sun Jul 30 14;08;58 comment by...Junji -->
Thanks Ray.<br>
We are trying by changing onotl_prd_read to cdfonprd.<br>
After this, we will try "setenv DEBUG 10".<br>
And there is a instruction for standalone test at<br>
https://www-cdfonline.fnal.gov/internal/mon/consumer/framework/start.html

<!-- Comment: Sun Jul 30 14;39;29 comment by...Junji -->
TrigMon crushes even if we change "onotl_prd_read" to "cdfonprd". And the error message is the same.
<!-- Comment: Sun Jul 30 14;54;22 comment by...Junji -->
I also tryed "setenv DEBUG 10", but I can't see any additional information.
<!-- Comment: Sun Jul 30 15;05;53 comment by...Junji -->
<b>TrigMon is running without SVTSim now.</b>

<!-- Comment: Sun Jul 30 15;35;17 comment by...Taichi -->
We are investigating waht is wrong with svtsim. Please keep running without svtsim for now.
<!-- Comment: Sun Jul 30 16;40;16 comment by...Junji and kaori -->
We were testing TrigMon and found that it started working with
SVTSim.  <b>TrigMon is now running with SVTSim</b>.  (We have seen this type of behaviour associated with database access before.
<!-- Date: Sun Jul 30 11:36:51.176 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7754&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7754' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7755&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7755' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7756&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7756' width='300'></a></center>
<!-- Author: Hugo -->




<!-- Date: Sun Jul 30 11:44:54.788 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7757&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7757' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7758&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7758' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7759&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7759' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7760&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7760' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7761&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7761' width='300'></a></center>
<!-- Author: Hugo -->
<!-- Comment: Sun Jul 30 11;45;49 comment by...Hugo -->
<pre>Evolution of the pinkies since they appeared this morning.</pre>



<!-- Date: Sun Jul 30 11:55:43.896 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220726>220726</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
IMU 
</PRE>Hot channels in BMU NW
<!-- Author: cdfscico -->





<!-- Date: Sun Jul 30 12:06:29.992 2006 -->
<pre>Trigger Inhibit is set from the components: 
 CMP Trip  		[082] Central Muon Upgrade Trip 00 
</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727' target=_top>220727</a>) -->


<!-- Date: Sun Jul 30 12:10:04.662 2006 -->
<pre>CMP Bottom A3 and D3 have tripped. After having put these two 
bars on "on", power ramped up and is now fine (green).</pre>
<!-- Author: Hugo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727' target=_top>220727</a>) -->


<!-- Date: Sun Jul 30 12:12:20.784 2006 -->
<pre>Attention !!!.  CER_DAQERRMON_DATA_MISSING_ERROR !!!  
  
 DAQErrMon:DataMissingError:  
 Error 1360 Event 2599669 Run 220727 Bank SL1D_StorableBank missing from event record, nErrors 632  
 
 - We suggest issuing an HRR.  
  
 - if the problem persists for more than a few minutes,  
   starting a new Run may be necessary. </pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727' target=_top>220727</a>) -->


<!-- Date: Sun Jul 30 12:40:43.541 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7762&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7762' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7763&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7763' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7764&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7764' width='300'></a></center>
<!-- Author: Hugo -->


<!-- Date: Sun Jul 30 12:50:25.659 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-7765&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-7765  width='300' align='top'></a></td><td>BMU ObjectMon plots (phi and z) show some disagreement with reference.  I didn't find this problem after a quick search of the e-log and "known problem list".</td></tr></table>
<!-- Author: Mitch Soderberg -->
<!-- Comment: Sun Jul 30 15;33;12 comment by...Phil -->
That's just the oscillation as revealed by the reconstruction.
<!-- Date: Sun Jul 30 12:59:49.133 2006 -->
<pre>Done TO. for crate/s:  
    b0imu00    
 
Auto HRR was issued</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727' target=_top>220727</a>) -->



<!-- Date: Sun Jul 30 13:25:23.417 2006 -->
<pre>DQMon alarm: PhysMon 
Run 220724 has been found bad 
 
PhysMon Info 
Run 220724 BAD: 
W BAD 
J GOOD 
COT BAD 
CAL GOOD 
MUON BAD</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727' target=_top>220727</a>) -->
<!-- Comment: Sun Jul 30 13;33;28 comment by...Niharika -->
<pre>Checked the PhysMon webpage:

For run 220724 it shows W munu as red with value 0.292 +/- 0.020</pre>


<!-- Comment: Sun Jul 30 13;48;10 comment by...Niharika -->
<pre>Have sent an email to the expert (Collin Wolfe) </pre>
<!-- Comment: Sun Jul 30 15;07;51 comment by...Niharika -->
<pre>Another PhysMon warning:

Run 220604 has been found bad</pre>
<!-- Date: Sun Jul 30 13:35:26.696 2006 -->
<table><tr><td><a href=/elog/notebooks/2006/Sun_Jul_30_2006_Day.133518.ps align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-7767  align='top'></a></td><td>One more BMU plot that is probably related to the previous <a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=538&frame=2&anchor=12:50:25.659'> 
ObjectMon</a> and <a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=538&frame=2&anchor=10:46:00.210'> 
YMon</a> plots.  </td></tr></table>
<!-- Author: Mitch Soderberg -->
<!-- Comment: Sun Jul 30 15;33;59 comment by...Phil -->
OK, you figured it out already.
<!-- Date: Sun Jul 30 13:50:22.404 2006 -->
<pre>Done TO. for crate/s:  
    b0cot09    
 
Again. Auto HRR was issued</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727' target=_top>220727</a>) -->
<!-- Comment: Sun Jul 30 14;00;53 comment by...Niharika -->
<pre>Again..
5 times in a row..</pre>
<!-- Comment: Sun Jul 30 14;07;51 comment by...Niharika -->
<pre>After a try to sheperd the crate and recover the run I got this error again


 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done  for crate b0cot09  
 - HRR is strongly RECOMMENDED.   
 
 - Autorecoveries is OFF.   
 - Try manual HRR: if this does NOT help,  
 - please SHEPHERD the offending TDC crate  
[MLE]
(MLE) b0cot09:Messenger:Jul 30, 2006 02:04:53 PM-&gt;Runtime Error 1, Event 4967641: TDC in slot 8 not ready for readout
b0cot09:Messenger:Jul 30, 2006 02:04:54 PM-&gt;Runtime Error 2, Event 4967641: Timeout on global TDC Done --&gt;  
 Additional Information: 

 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 
 - Timeout on global TDC Done  for crate b0cot09  
 - HRR is strongly RECOMMENDED.   
 </pre>
<!-- Date: Sun Jul 30 14:00:27.347 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7768&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7768' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7769&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7769' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7770&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7770' width='300'></a></center>
<!-- Author: Hugo -->




<!-- Date: Sun Jul 30 14:11:38.272 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727>220727</A>
 Terminated at 2006.07.30 14:11:09
<!-- Author: RunControl -->


<!-- Date: Sun Jul 30 14:11:39.413 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727>220727</A>
TERMINATE: DT outs from COT_09
<!-- Author: Niharika -->
<!-- Comment: Sun Jul 30 14;13;45 comment by...Niharika -->
<pre> State       TERMINATE
  ELog        true
b0eproxy02.fnal.gov:main:Jul 30, 2006 02:13:13 PM-&gt;Error on L3 node b0eproxy02.fnal.gov (partition 0) Sun Jul 30 14:13:13 2006 l3proxy 
  in Abort (L3ProxyPartition.cc:3027)
  @L3_RELAY_FAILURE Level3 Relay failure detected!
The following subfarms need to be removed from the next runs:
      None. You can use all the same subfarms.
The following Level3 converter nodes are marked bad for now:
      None.
The full list of Level3 nodes giving Relay trouble is printed below

 b0l3s13p11</pre>

<!-- Date: Sun Jul 30 14:19:59.227 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220728>220728</A>
 Activated at 2006.07.30 14:19:26
<!-- Author: RunControl -->


<!-- Date: Sun Jul 30 14:20:00.333 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220728>220728</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [4,816,640] 
<!-- Author: Niharika -->


<!-- Date: Sun Jul 30 14:20:38.980 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220728>220728</A>
 Terminated at 2006.07.30 14:20:30
<!-- Author: RunControl -->


<!-- Date: Sun Jul 30 14:21:06.053 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220728>220728</A>
TERMINATE: Done time out from COT_09
<!-- Author: Niharika -->


<!-- Date: Sun Jul 30 14:29:55.611 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220729>220729</A>
TERMINATE: DTO COT_09
<!-- Author: Niharika -->
<!-- Comment: Sun Jul 30 14;41;43 comment by...Niharika -->
<pre>Too many done time outs for b0cot09. Eric suggested that we remove the TDC in slot 8 (but not from run control).
The SciCo talked to Eric and changed the ONLINEFALG for this card from 1 to 0 in the Hard ware DB.
TDCREADOUTMODE was changed from 0 to 2 for COT_09</pre>
<!-- Comment: Sun Jul 30 15;35;24 comment by...Niharika -->
<pre>Reasons why we didn't remove the card from run control are the following:

- Removing the card from run control (which can be done by selecting the crate, COT_09 in this case, and using the edit button ) would have changed the value of the ONLINEFLAG but not TDCREADOUTMODE. Apparently, setting the value of TDCREADOUTMODE to 2 tells the crate not to wait for all the cards.

- In case we have to re-load the run configuration for some reason then the default settings will include the faulty card and cause further done time outs. The change that our SciCo made (which can be done by experts only) in the hardware database is kind of permanent till the faulty card is changed and shouldn't hamper data taking. 

</pre>
<!-- Date: Sun Jul 30 14:38:24.332 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220730>220730</A>
 Activated at 2006.07.30 14:36:52
<!-- Author: RunControl -->





<!-- Date: Sun Jul 30 14:51:01.611 2006 -->
The bad TDC (b0cot09, slot 8) has been removed from the readout, and we've resumed data-taking without further DONE timeouts from b0cot09.  An access will be needed to swap the TDC.
<!-- Author: Jay D :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220730' target=_top>220730</a>) -->



<!-- Date: Sun Jul 30 15:02:44.010 2006 -->
<pre>Done Timeout detected in crates: 
COT_09 
 
Again</pre>
<!-- Author: Niharika :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220730' target=_top>220730</a>) -->




<!-- Date: Sun Jul 30 15:10:28.515 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-7771&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-7771  width='300' align='top'></a></td><td>Before removing board #8 from the crate COT_09, expert (Eric) told us to do a power recycling. 
 
To do that, you have to click on the "RACKS" button in the "PROCESS SYSTEMS ALARMS window (the button is below the GAS, Waters, COT, etc buttons). 
 
Then, choose a Rack by clicking on it (we choose in our case "Detector Racks West"). You will get the following picture. 
 
Now, you have to figure out where your bad crate is. Use the pictures of crates location, next to the white board. In our case, the crate COT_09was in the Bottom-West rack (named, in the picture, CONWB). 
 
Click on your rack (we clicked on CONWB). 
 
Finally, hit the button which correspond to your crate to power cycle it (the name should be written in blue, below one of the "RESET POWER SUPPLY #X" tag). 
 
Wait ~1min. 
 
Note: to close the rack window, you need to click on "rack menu"!</td></tr></table>
<!-- Author: Niharika & Hugo -->
<!-- Comment: Sun Jul 30 15;14;31 comment by...Niharika & Hugo -->
<p>Here is the webpage that also has some of the above information
<a href="http://www-cdfonline.fnal.gov/ops/mcs/powercycle.html"></a>
</p>

<!-- Comment: Sun Jul 30 15;15;00 comment by...Niharika & Hugo -->
<pre>http://www-cdfonline.fnal.gov/ops/mcs/powercycle.html</pre>


<!-- Date: Sun Jul 30 15:16:40.519 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>DQMon entry: 
 
  We checked the DQMon output for run 220589. It looks OK. The  
errors reported by PhysMon should be discussed with PhysMon  
contact persons.  
 
Mario 
</pre></td></tr></td></table>
<!-- Author: Mario :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220730' target=_top>220730</a>) -->


<!-- Date: Sun Jul 30 15:19:25.193 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7772&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7772' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7773&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7773' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-7774&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-7774' width='300'></a></center>
<!-- Author: Hugo -->





<!-- Date: Sun Jul 30 15:41:05.894 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220727>220727</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
IMU 
</PRE>Hot channels in BMU NW
<!-- Author: cdfscico -->


<!-- Date: Sun Jul 30 15:55:42.928 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=220726 target=_runSummary>220726</A> x35E36</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [4,816,640]</FONT></TD>
<TD ALIGN=CENTER>06:14:38</TD>
<TD ALIGN=CENTER>09:53:42</TD>
<TD ALIGN=CENTER>03:25:57</TD>
<TD ALIGN=RIGHT>   1,206,840</TD>   <TD ALIGN=RIGHT>  1,022.37</TD>
<TD ALIGN=RIGHT>    957.90</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=220727 target=_runSummary>220727</A> x35E37</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [4,816,640]</FONT></TD>
<TD ALIGN=CENTER>10:05:50</TD>
<TD ALIGN=CENTER>14:11:09</TD>
<TD ALIGN=CENTER>03:25:24</TD>
<TD ALIGN=RIGHT>   1,067,381</TD>   <TD ALIGN=RIGHT>    909.13</TD>
<TD ALIGN=RIGHT>    763.27</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=220728 target=_runSummary>220728</A> x35E38</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [4,816,640]</FONT></TD>
<TD ALIGN=CENTER>14:19:26</TD>
<TD ALIGN=CENTER>14:20:30</TD>
<TD ALIGN=CENTER>00:00:00</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT>      4.21</TD>
<TD ALIGN=RIGHT>      0.00</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>0</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=220729 target=_runSummary>220729</A> x35E39</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [4,816,640]</FONT></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>14:29:12</TD>
<TD ALIGN=CENTER>00:00:00</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>0</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=220730 target=_runSummary>220730</A> x35E3A</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [4,816,640]</FONT></TD>
<TD ALIGN=CENTER>14:36:52</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>01:11:47</TD>
<TD ALIGN=RIGHT>     322,557</TD>   <TD ALIGN=RIGHT>    243.47</TD>
<TD ALIGN=RIGHT>    224.66</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>15:55:01</TD>
<TD ALIGN=CENTER>08:03:09</TD>
<TD ALIGN=RIGHT>     2,596,778</TD> <TD ALIGN=RIGHT>  2,179.19</TD>
<TD ALIGN=RIGHT>  1,945.83</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sun Jul 30 15:56:09.949 2006 -->
<pre>Time Of Flight Alarm: went out by itself, almost right away.</pre>
<!-- Author: Hugo :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=220730' target=_top>220730</a>) -->


<!-- Date: Sun Jul 30 16:08:48.249 2006 -->
<font color=red>Shift Summary: </font><pre>Shift begins with Run 220726 in progress. 
Store #4862, B0Lum 77.6E10 
 
Fairly smooth datataking. 
 
Action items: 
-   Hot channels in BMU West Layer 0-3 (See YMon #97 IMU CO BMU Cells West) 
      An access will be needed to investigate this 
-   A COT TDC (b0cot09, slot 8) is hanging frequently and probably needs to be replaced.  
      Access needed.  This board is currently removed from the readout. 
-   Experts investigating problem with TrigMon (SVTMon database access problem)  
      Kaori is planning to come in. 
 
Other news: 
-   Readout problem from HV crate CCU NE-ISU-B (#02).  Oleg fixed it. 
-   Another "Heap is Corrupt" error, b0cot14 
-   Needed to end run 220726 due to lots of DAQErrorMon "Missing Data" erros 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   220726,220727,220728,220729,220730
Delivered Luminosity   1802.2 nb-1  
Acquired Luminosity    1472.8 nb-1  
Efficiency             81.7
   
</td></tr></table></pre>
<!-- Author: Jay Dittmann -->