<!-- Title: Day shift. Sun Jan 18, 2004 -->
<!-- SciCo: Yoshiaki Fujii -->
<!-- DAQAce: Taka M, Tom S. -->
<!-- MonAce: Dale S, Susana C. -->
<!-- CO: Mateo Cavalli-Sforza -->
<!-- OpManager: J.J. -->
<!-- Notes: -->
<pre>Succeeded steady data taking.</pre>


<!-- Date: Sun Jan 18 08:29:49.007 2004 -->
<pre>shell text when trying to start trigger inhibit display:  we check b0dap33 to see if any processes were running that could be easily identified as inhibit and none came up. 
 
b0dap59:139:~&gt; source ~jdl/csh/setup_tixpt 
CDFVME_COMMON: Last updated by feild on Fri Oct 31 16:03:58 CST 2003 
CDFVME_COMMON: Last updated by feild on Fri Oct 31 16:03:58 CST 2003 
b0dap59:140:~&gt; InhibitDisplay 1 
Attempting connection to &lt;tcp:b0dap33:5101&gt; RTserver. 
Connected to &lt;tcp:b0dap33:5101&gt; RTserver. 
Assumed fer home: /cdf/code-common/cdfonline/inhibitDisplay/ 
Assumed cfg file: ./Inhibit.cfg 
Partition will be : 1 
            Logfile is /cdf/people1/cdfdaq/.inhmonlog 
Adding label 
Partition false 
true 
Ignorable system 95 TEV WIRE 
Ignorable system 96 PC:CMU HV 
Ignorable system 98 PC:CMP HV 
Ignorable system 99 PC:CMX HV 
Ignorable system 100 PC:IMU HV 
Ignorable system 101 PC:IMU CCU 
Ignorable system 102 PC:CES HV 
Ignorable system 103 PC:CSP CCU 
Ignorable system 104 PC:CSP CSX 
Ignorable system 120 PC:COT HV 
Ignorable system 131 IFIX:COT LV 
Ignorable system 137 IFIX:XENON OFF 
Ignorable system 142 IFIX:CSP CSX 
Ignorable system 145 IFIX:CLC HV 
Ignorable system 146 IFIX:RACKS 
Ignorable system 148 IFIX:CSP CCU 
Ignorable system 149 IFIX:IMU CCU 
Exception in thread "main" java.lang.NoClassDefFoundError: rc/InhibitConfigMess 
        at InhMonSumDisplay.maskOutOfPartition(InhMonSumDisplay.java:850) 
        at InhMonSumDisplay.allocatePartition(InhMonSumDisplay.java:762) 
        at InhMonSumDisplay.&lt;init&gt;(InhMonSumDisplay.java:127) 
        at InhMonSumFer.init(InhMonSumFer.java:171) 
        at InhMonSumFer.&lt;init&gt;(InhMonSumFer.java:143) 
        at InhMonSumFer.main(InhMonSumFer.java:314) 
 
b0dap59:141:~&gt; ps -ax 
</pre>
<!-- Author: tom -->
<!-- Comment: Sun Jan 18 08;40;25 comment by...tom -->
<a href=
'/cgi/elog/elog.pl?nb=2004&action=view&page=52&frame=2&anchor=02:19:48.820'>
first e-log entry of this problem </a>


<!-- Comment: Sun Jan 18 09;18;52 comment by...Dale -->
<pre>Not sure if this helps but here is the last 'start-up' entry for the log file .inhmonlog:

Done with DB start
TRIP:SVX00    !
                    Partition 1
parsing configfile /cdf/code-common/cdfonline/inhibitDisplay/Inhibit.cfg
TRIP:SVX00    !
TRIP:SVX01    !
Re-selecting Partition elements: 
PartitionId: &lt;1&gt;
               Free: false
           Detached: false
           Software: false
          Permanent: true
       CreationTime: 2004-01-18 03:29:30.0
         DetachTime: 2004-01-18 03:29:24.0
         ClientName: Ian x2080
           NodeName: b0dap73.fnal.gov
           UserName: cdfdaq
          ProcessId: 28214
       OwnerAddress: /_b0dap73.fnal.gov_26a7d532
  TriggerSupervisor: TRIGGER_SUPERVISOR_1
[long list of crates]</pre>
<!-- Comment: Mon Jan 19 18;11;27 comment by...steve -->
<pre>Probably noted by Bill B. later on but this was figured 
out later-new code which substituted InhibitReadoutList 
for InhibitConfigMess was meant to occur in a
 deprecating fashion.  Apparently, it simply replaced it. 
 This broke certain code pieces.</pre>
<!-- Date: Sun Jan 18 08:52:26.485 2004 -->
<pre>Date Time BLM Dose  
2004.01.18 08:47:44 W Inner BLM 351.84 RADS  
2004.01.18 08:47:44 W Outer BLM 0.00 RADS  
2004.01.18 08:47:44 E Inner BLM 5.08 RADS  
2004.01.18 08:47:44 E Outer BLM 351.84 RADS  
Integrated dosage </pre>
<!-- Author: Dale and Susana -->


<!-- Date: Sun Jan 18 08:56:15.372 2004 -->
<pre>busy timeout b0svx06 coming from event builder b0eb17 
 
MLE) b0l3pcom1.fnal.gov:main:8:52:38 AM-&gt;Host b0eb17.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 676389 from VRB in slot 16: 
0x84128312 0xa212a112 0x0104020a 0x0301a012 0x7f03c1c1 0x00000000 0xe0627ffc 0x891200f3 
1 crate/s: b0svx06(16),  busy.[RXPT] 
(MLE) b0dap73.fnal.gov:Thread-3990:8:52:57 AM-&gt;Busy Timeout: VRB_ISL_06  
(MLE) b0tsi02:Messenger:8:52:54 AM-&gt;BUSY timeout 
(MLE) b0svx06:Messenger:8:52:57 AM-&gt;Silicon Timeout:BUSY- Slots:  08:fa00 10:fa20 12:fa40 16:f800 18:f820 20:f840 
 
(RC)   8:53:9 Halt -&gt; HALTED 
(RC)   8:53:16 Recover -&gt; RECOVERED 
(RC)   8:53:19 Run -&gt; ACTIVE</pre>
<!-- Author: tom -->


<!-- Date: Sun Jan 18 09:12:23.051 2004 -->
Called Steve Vejcik at his home (on Sunday Morning!) for trigger inhibit display problem. They were not at home, and left message to call CDF control.
<!-- Author: Yoshiaki Fujii -->
<!-- Comment: Sun Jan 18 11;04;23 comment by...Yoshiaki Fujii -->
Steve called back CDF control, and discussed with DAQ-ACE. See <a href='/cgi/elog/elog.pl?nb=2004&action=view&page=53&frame=2&anchor=10:50:40.488'>here</a>. 
<!-- Date: Sun Jan 18 09:16:33.466 2004 -->
 We sometime have BigBrother error on CPU of b0dap58. 
red Sun Jan 18 09:11:14 CST 2004 up: 1 day, 2 users, 251 procs, load=4.66  
 It is periodically disappeared and got no serious run control  
issue, however maybe worth while keep looking. 
  
<!-- Author: Taka and Tom -->



<!-- Date: Sun Jan 18 09:24:44.583 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1310&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1310' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1311&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1311' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1312&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1312' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1313&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1313' width='300'></a></center>
<!-- Author: Susana and Dale -->
<!-- Comment: Sun Jan 18 09;29;53 comment by...Susana and Dale -->
<pre>Wide spike (probably fake) in B0AAGC and B0ABSM at 8:30, tevMon did not detected it.</pre>
<!-- Comment: Sun Jan 18 09;34;42 comment by...Dale and Susana -->
Seen a couple of times before such as <a href=
'/cgi/elog/elog.pl?nb=2004&action=view&page=12&frame=2&anchor=20:35:16.753'>
here</a>.  
<br>
It is rather unreal looking and clearly does not reflect the true data.
<!-- Date: Sun Jan 18 09:25:41.882 2004 -->
<pre>@24E30: L1:13kHz, L2:220Hz, L3:50Hz. 
Looks very smooth run.</pre>
<!-- Author: Taka and Tom -->




<!-- Date: Sun Jan 18 10:09:37.206 2004 -->
<pre>New silicon Pinky: 
 
SVX AVDD  Barrel 0  Wedge 8  Layer 4 
 
Currents: 
  Nominal  : 1040.0  
  Bad @ and latest value at 1015.0 (Caution  : 1017.76) 
 
Unmarking</pre>
<!-- Author: Susana and Dale -->
<!-- Comment: Sun Jan 18 10;27;50 comment by...Yoshiaki Fujii -->
Notified to Silicon Primary.
<!-- Date: Sun Jan 18 10:18:17.094 2004 -->
L3 node 113 has gone red.  all else is fine.
<!-- Author: tom -->
<!-- Comment: Sun Jan 18 14;34;51 comment by...Guillelmo -->
It means one of the two L3 executable crashed.
<!-- Date: Sun Jan 18 10:20:32.811 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1314&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1315  width='300' align='top'></a></td><td><p>SVX AVDD  Barrel 0  Wedge 8  Layer 4 turned pink again.  History clearly shows it being ok for a long time.  We'll try an HRR and page silicon to let them know.</p></td></tr></table>
<!-- Author: Dale and Susana -->
<!-- Comment: Sun Jan 18 10;21;23 comment by...Taka -->
<pre>HRR requested from Dale.</pre>


<!-- Date: Sun Jan 18 10:23:05.514 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1316&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1316' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1317&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1317' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1318&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1318' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1319&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1319' width='300'></a></center>
<!-- Author: Susana and Dale -->
<!-- Comment: Sun Jan 18 10;25;36 comment by...Susana and Dale -->
<pre>Hourly plots: 9:00 pm -10:00 pm</pre>
<!-- Date: Sun Jan 18 10:24:11.948 2004 -->
<pre>L3 node 113 error messages" 
 
ERROR at Sun Jan 18 10:05:03 2004 partition 1 
    in exe_exchange() [l3_ana_exe.c:559] 
    info: @L3_EXE_TIMEOUT:  Filter 0 timeout 
 
ERROR at Sun Jan 18 10:05:03 2004 partition 1 
    in exe_exchange() [l3_ana_exe.c:608] 
    info: Timeout receiving shmbuf from filter 
 
ERROR at Sun Jan 18 10:05:03 2004 partition 1 
    in l3_ana_exe_analyze() [l3_ana_exe.c:1007] 
    info: Error exchanging buffer to filter 
 
ERROR at Sun Jan 18 10:05:03 2004 partition 1 
    in chain_loop() [l3_chain.c:842] 
    info: Error in analysis (continuing with bad event) 
</pre>
<!-- Author: tom -->




<!-- Date: Sun Jan 18 10:28:56.083 2004 -->
<pre>Event Builder Error.  HRR fixed.  Here's the error log message. 
 
Attention!!!. SCPU_TRACER_EVENT_ID Error !!!  
 Hardware EVB has detected a problem with data quality in  
 SCPU b0eb14 (forwarded by FER crate COT_14).  
 Datataking is paused: events are not being processed.  
 - HRR should help.  
[MLE] 
(MLE) b0l3pcom1.fnal.gov:main:10:27:06 AM-&gt;Host b0eb14.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 1933366 from VRB in slot 10: 
0x00000000 0x00005438 0x003d007d 0x0d040904 0x094c0984 0x04180598 0x07d00910 0x08f007b0 
1 crate/s: b0svx06(16),  busy.[RXPT] 
 Additional Information:  
  
 This BUSY TIMEOUT was caused by EVB TracerEventId error in  
 SCPU = b0eb14  
  
 Hardware EVB has detected a problem with data quality.  
 SCPU b0eb14 reports event ID or control sequence mismatch.  
 Datataking is paused; events are not being processed.  
 - HRR should help.  
 
(MLE) b0dap73.fnal.gov:Thread-3990:10:27:20 AM-&gt;Busy Timeout: VRB_ISL_06  
(MLE) b0svx06:Messenger:10:27:14 AM-&gt;Silicon Timeout:BUSY- Slots:  08:fa00 10:fa20 12:fa40 16:f800 18:f820 20:f840 
 
(RC)   10:27:28 Halt -&gt; HALTED 
(RC)   10:27:39 Recover -&gt; RECOVERED 
(RC)   10:27:44 Run -&gt; ACTIVE</pre>
<!-- Author: tom -->


<!-- Date: Sun Jan 18 10:43:47.447 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1321&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1321  width='300' align='top'></a></td><td>Update on SVX AVDD  Barrel 0  Wedge 8  Layer 4 
<br> 
Silicon expert has been notified. (HRRs did not seem to help)  We'll keep our eye on it.</td></tr></table>
<!-- Author: Susana and Dale -->


<!-- Date: Sun Jan 18 10:50:40.488 2004 -->
Tried Running Inhibit Display gui from b0dap72 as recommended by expert.  Same Error was given.  Didn't work.  Expert will address problem tonight or tomorrow.  Shouldn't be an issue.  Run Control still will Halt on Inhibits.
<!-- Author: tom -->
<!-- Comment: Sun Jan 18 11;06;22 comment by...Yoshiaki Fujii -->
<a href='/cgi/elog/elog.pl?nb=2004&action=view&page=53&frame=2&anchor=09:12:23.051'>Here he is</a>. 
<!-- Date: Sun Jan 18 10:58:47.260 2004 -->
New red L3 nodes: #82 and #275.
<!-- Author: Taka and Tom -->




<!-- Date: Sun Jan 18 11:17:26.535 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1322&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1322' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1323&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1323' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1324&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1324' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1325&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1325' width='300'></a></center>
<!-- Author: Dale and Susana -->
<!-- Comment: Sun Jan 18 11;20;59 comment by...Dale and Susana -->
Hourly plots: 10:00 -11:00 am. A spike (~25000 kHz) at 10:30 pm in B0PBSM. 


<!-- Date: Sun Jan 18 11:39:34.557 2004 -->
<pre> An Expert (Donatella T?) called to DAQ ACE and recommend to use  
auto-HRR. We explained that this was due to bunch of L2DTO in  
this morning. We turn on the auto-HRR followed by her recommendation.  </pre>
<!-- Author: Tom and Taka -->


<!-- Date: Sun Jan 18 11:46:37.077 2004 -->
<pre>Event Builder problem with data quality in b0eb14.  HRR issues and recovered.  Error message below: 
 
Attention!!!. SCPU_TRACER_EVENT_ID Error !!!  
 Hardware EVB has detected a problem with data quality in  
 SCPU b0eb14 (forwarded by FER crate COT_14).  
 Datataking is paused: events are not being processed.  
 - HRR should help.  
[MLE] 
(MLE) b0dap73.fnal.gov:Thread-3990:11:44:43 AM-&gt;Requested Halt-Recover-Run issued [errmon] 
(MLE) b0l3pcom1.fnal.gov:main:11:44:30 AM-&gt;Host b0eb14.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 2966302 from VRB in slot 10: 
0x00000000 0x000038c8 0x002000e0 0x0998052c 0x02e803e8 0x08c00324 0x02fc0418 0x06b40958 
(MLE) b0svx06:Messenger:11:44:31 AM-&gt;Silicon Timeout:BUSY- Slots:  08:fa00 10:fa20 12:fa40 16:f800 18:f820 20:f840 
 
(RC)   11:44:46 Halt -&gt; HALTED 
(RC)   11:44:52 Recover -&gt; RECOVERED 
(RC)   11:44:55 Run -&gt; ACTIVE</pre>
<!-- Author: tom -->


<!-- Date: Sun Jan 18 11:51:03.772 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1326&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1326  width='300' align='top'></a></td><td>x</td></tr></table>
<!-- Author: Susana and Dale -->
<!-- Comment: Sun Jan 18 11;52;27 comment by...Susana and Dale -->
The cell S_AVDD_B2W2L4, that was pink at the end of previous shift
and in care of experts is know green. 


<!-- Comment: Sun Jan 18 12;13;31 comment by...Dale and Susana -->
SVX AVDD Barrel 0 Wedge 8 Layer 4 was green for a while but has returned to pink again.
<!-- Comment: Sun Jan 18 12;16;23 comment by...Susana and Dale -->
Tried HRR and unmarking for SVX AVDD Barrel 0 Wedge 8 Layer 4 as noted below.
<!-- Date: Sun Jan 18 11:56:37.591 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1327&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1327  width='300' align='top'></a></td><td><a href= 
'/cgi/elog/elog.pl?nb=2004&action=view&page=51&frame=2&anchor=20:12:46.699'> 
pcal10, slot 17 ADMEM was replaced durning the controled access</a> 
<p> 
Thus this has been fixed (as shown in the plot) and therefore will be marked as such in 
<a href= 
'/cgi/elog/fixlist.pl?nb=fixlist&action=view&page=1&frame=2&anchor=08:52:40.816'> 
FIXLIST</a> 
</p></td></tr></table>
<!-- Author: Matteo and Dale -->


<!-- Date: Sun Jan 18 12:07:26.911 2004 -->
<pre>Manually HRR by Dale's request.</pre>
<!-- Author: Tom and Taka -->


<!-- Date: Sun Jan 18 12:10:09.640 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1335&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1335  width='300' align='top'></a></td><td>x</td></tr></table>
<!-- Author: Susana and Dale -->


<!-- Date: Sun Jan 18 12:12:28.235 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1336&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1336' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1337&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1337' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1338&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1338' width='300'></a></center>
<!-- Author: Susana -->




<!-- Date: Sun Jan 18 12:33:52.511 2004 -->
L3 node 185 is now red. 4 total red nodes now.
<!-- Author: tom -->


<!-- Date: Sun Jan 18 13:05:17.322 2004 -->
<pre>Busy Timeout.  event builder b0eb22.  HRR recovered 
 
(MLE) b0l3pcom1.fnal.gov:main:1:03:16 PM-&gt;Host b0eb22.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 4000099 from VRB in slot 16: 
0x6003610f 0x62116305 0x8c126603 0x670b684b 0x69806a0b 0x6bf37a03 0x7b197cef 0x7d167e09 
1 crate/s: b0svx06(16),  busy.[RXPT] 
(MLE) b0dap73.fnal.gov:Thread-3990:1:03:21 PM-&gt;Busy Timeout: VRB_ISL_06  
(MLE) b0dap73.fnal.gov:Thread-3990:1:03:21 PM-&gt;Requested Halt-Recover-Run issued [errmon] 
(MLE) b0svx06:Messenger:1:03:22 PM-&gt;Silicon Timeout:BUSY- Slots:  08:fa00 10:fa20 12:fa40 16:f800 18:f820 20:f840 
 
(RC)   13:3:24 Halt -&gt; HALTED 
(RC)   13:3:31 Recover -&gt; RECOVERED 
(RC)   13:3:33 Run -&gt; ACTIVE 
 
</pre>
<!-- Author: tom -->


<!-- Date: Sun Jan 18 13:14:53.676 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1339&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1339' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1340&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1340' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1341&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1341' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1342&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1342' width='300'></a></center>
<!-- Author: Dale and Susana -->
<!-- Comment: Sun Jan 18 13;17;27 comment by...Susana -->
12:00 - 13:00 hourly plots. 3 spikes in B0ABSM and B0AAGC.
<!-- Date: Sun Jan 18 13:16:17.534 2004 -->
Consumer Monitor TrgL2-ISOLIST has 7% error. Notified to Steve Kuhlmann via e-mail (at 12:20) since this is not a critical plot to page LRP.
<!-- Author: Yoshiaki and Matteo. -->



<!-- Date: Sun Jan 18 13:45:32.495 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1345&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1345' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1346&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1346' width='300'></a></center>
<!-- Author: Susana and dale -->
<!-- Comment: Sun Jan 18 13;46;58 comment by...Susana and Dale -->
<pre>Routinary check of the pink cells in IMON: only two pink cells,
known problem. There are not new pink cells.</pre>


<!-- Comment: Sun Jan 18 15;48;38 comment by...Susana and Dale -->
<pre>Thus we have only the 'known' pinkies:

SVX AVDD  Barrel 0  Wedge 8  Layer 4
ISL AVDD  Barrel 0  Wedge 2  Layer 1
</pre>
<!-- Date: Sun Jan 18 14:07:36.606 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1347&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1347' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1348&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1348' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1349&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1349' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1350&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1350' width='300'></a></center>
<!-- Author: Dale and Susana -->
<!-- Comment: Sun Jan 18 14;08;31 comment by...Dale and Susana. -->
Hourly plots: 13:00 - 14:00, everything looks normal. TevMon
has been OK.
<!-- Date: Sun Jan 18 14:08:16.946 2004 -->
event builder error (timeout).  b0eb14.  HRR recovered it. 
 
 Attention!!!. SCPU_TRACER_EVENT_ID Error !!!  
 Hardware EVB has detected a problem with data quality in  
 SCPU b0eb14 (forwarded by FER crate COT_14).  
 Datataking is paused: events are not being processed.  
 - HRR should help.  
[MLE] 
(MLE) b0l3pcom1.fnal.gov:main:2:06:48 PM->Host b0eb14.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 4822601 from VRB in slot 10: 
0x00000000 0x00004fe8 0x001400d4 0x057802ac 0x05300714 0x0cf80880 0x07380a40 0x0ad8098c 
(MLE) b0dap73.fnal.gov:Thread-3990:2:06:49 PM->Requested Halt-Recover-Run issued [errmon] 
(MLE) b0svx06:Messenger:2:06:48 PM->Silicon Timeout:BUSY- Slots:  08:fa00 10:fa20 12:fa40 16:f800 18:f820 20:f840 
 
(RC)   14:6:52 Halt -> HALTED 
(RC)   14:6:58 Recover -> RECOVERED 
(RC)   14:7:1 Run -> ACTIVE
<!-- Author: tom -->



<!-- Date: Sun Jan 18 14:17:37.292 2004 -->
5th red node for L3.  Now have nodes 
 
082 
113 
155 
185 
275
<!-- Author: tom -->


<!-- Date: Sun Jan 18 14:22:53.757 2004 -->
Consumer Error Stuck CellID S/B1/W5/L4/C0-6.  HRR recovered.  
<!-- Author: tom -->



<!-- Date: Sun Jan 18 15:10:30.780 2004 -->
<pre>Done Timeout due to CCAL_15. HRR recovered it.</pre>
<!-- Author: tom -->


<!-- Date: Sun Jan 18 15:14:34.421 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1351&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1351' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1352&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1352' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1353&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1353' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1354&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1354' width='300'></a></center>
<!-- Author: Dale and Susana -->
<!-- Comment: Sun Jan 18 15;15;16 comment by...Susana and Dale -->
Hourly plots: 14:00 - 15:00


<!-- Date: Sun Jan 18 15:38:39.702 2004 -->
<b> L2 Decison timeout summary II </b>:  (JJ's request) <p> 
 
Store# 3157  run#177792-177800 <br>   
Nomal 8 HRR recoveries loses 72s, <br> 
and 1 end the run loses 8min.<br> 
trigger teble is PHYSICS_(TEST_)2_01[2,379,426]. <p> 
 
------------------------------------------------------------ <br> 
Store# 3172  summarizes in 
<a href= 
'/cgi/elog/elog.pl?nb=2004&action=view&page=47&frame=2&anchor=11:55:03.612'> 
elsewhere.</a> <p> 
 
------------------------------------------------------------ <br> 
Store# 3175  run#178014-178030  <br> 
We had 12 run ends and restarts to recover the L2 DTOs. <br>   
We couldn't recover via HRR. <br> 
Loses around 100min / 1 day. <br> 
Trigger Table is PHYSICS_2_0[3,387,419] <p> 
 
------------------------------------------------------------ <br> 
Store# 3177  run#178063-178073   <br> 
We had 7 run ends and restarts due to many of L2 DTOs. <br> 
Lost 56min. / 17 hours. Trigger Tbale is same as store#3175. 
<p> 
 This effect looks really crutial for data taking efficiency.  
                                 
 
 
 
 
 
 
   
<!-- Author: Dale and Taka -->


<!-- Date: Sun Jan 18 15:39:35.720 2004 -->
8 red L3 nodes now 
 
082 
113 
150 
155 
167 
185 
220 
275 
  
<!-- Author: tom -->


<!-- Date: Sun Jan 18 15:40:12.201 2004 -->
L2 Trigger bit 112 - a 2-track, 3-jet object with a cross section of 35 nb - consistently has a rate of about twice the calculated rate, which puts it 20 std. dev. away from expectation. "Consistently " means here that we see this behaviour both in this very long run and in yesterday's very long run 178030.  
<!-- Author: Matteo, Bob Harr & Yoshiaki :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=1178074' target=_top>1178074</a>) -->


<!-- Date: Sun Jan 18 15:42:59.510 2004 -->
<pre>2004.01.18 15:37:27 W Inner BLM 598.36 RADS  
2004.01.18 15:37:27 W Outer BLM 0.00 RADS  
2004.01.18 15:37:27 E Inner BLM 5.08 RADS  
2004.01.18 15:37:27 E Outer BLM 506.94 RADS  
Integrated dosage  
 
Date Time Quantity  
2004.01.18 15:37:22 TEVpr 6548.00 1E09  
2004.01.18 15:37:22 TEVpb 729.06 1E09  
2004.01.18 15:38:54 C:B0ILUM 18.01 E30  
2004.01.18 15:38:54 C:B0LLUM 17.70 E30  
2004.01.18 15:38:54 C:B0PLOS 1605.08 Hz  
2004.01.18 15:38:54 C:B0ALOS 127.02 Hz  
Luminosity and losses </pre>
<!-- Author: Dale and Susana. -->



<!-- Date: Sun Jan 18 15:51:04.019 2004 -->
<pre> SVTSPYMON looks not working in VME crate, b0svt02,03,04. 
We don't know why. (actually, update of the beam position is 
also stopped.) 
 I tried to restart but seems that shepharding is needed.</pre>
<!-- Author: Taka -->


<!-- Date: Sun Jan 18 15:51:14.545 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1355&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1355' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1356&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1356' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1357&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1357' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-1358&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-1358' width='300'></a></center>
<!-- Author: Dale and Susana. -->
<!-- Comment: Sun Jan 18 15;52;04 comment by...Dale and Susana. -->
Hourly plots 15:00 -15:45. They look fine.

<!-- Date: Sun Jan 18 15:56:09.078 2004 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178073>
178073</A> 
x2B799</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_2_01 [3,387,419]</TD>                                       
<TD ALIGN=CENTER>07:41:49</TD>                                          
<TD ALIGN=CENTER>07:56:00</TD>                                          
<TD ALIGN=CENTER>00:12:54</TD>                                          
<TD ALIGN=RIGHT>      6,431,389</TD>                                    
<TD ALIGN=RIGHT>       145,153</TD>                                     
<TD ALIGN=RIGHT>      34,005</TD>                                       
<TD ALIGN=RIGHT>   20.029</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178074>
178074</A> 
x2B79A</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_2_01 [3,387,419]</TD>                                       
<TD ALIGN=CENTER>08:01:11</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>07:41:56</TD>                                          
<TD ALIGN=RIGHT>    355,500,530</TD>                                    
<TD ALIGN=RIGHT>     6,215,442</TD>                                     
<TD ALIGN=RIGHT>   1,405,080</TD>                                       
<TD ALIGN=RIGHT>  592.659</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>15:55:02</TD>                                                       
<TD ALIGN=CENTER>07:54:50</TD>                                          
<TD ALIGN=RIGHT>     361,931,919</TD>                                   
<TD ALIGN=RIGHT>      6,360,595</TD>                                    
<TD ALIGN=RIGHT>     1,439,085</TD>                                     
<TD ALIGN=RIGHT>  612.688</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sun Jan 18 16:03:26.474 2004 -->
<font color=red>Shift Summary: </font><pre>Day shift started with colliding store#3177.  
 - L=24E30, stack=69 
 - We have been facing lack of trigger inhibit display. 
 
 
What happened during shifts 
- Phoned Steve Vejcik at his home for inhibit issue, but absent. 
  Later Steve called back CDF control. Discussed with DAQ-ACE,  
  but problem was not solved. He will try again later.  
- A new Pinky Silicon appeared repeatedly. Paged Si-Primary. 
- Consumer Monitor Trg-L2-ISOLIST shows 7% error. 
  It occasionally had a few % error before. This is not a critical  
  plot to page LRP. Just sent e-mail to Steve Kuhlmann. 
- EventBuilder errors happened occasionally. HRR recovered. 
- Red nodes of L3 increasing. Now 8 at the end of day shift. 
 
 
End-of-Shift Comments 
- Current store is supposed to be kept for at least 24 hours. 
  Started at 23:00, 1/17 ... means at least until 23:00, 1/18. 
- Current L=18.1, stack is 121 
- J.J.is at home. Page him if current store is lost. 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   178074
Delivered Luminosity   603  
Acquired Luminosity    588  
Efficiency             97.4%
   
</td></tr></table></pre>
<!-- Author: Yoshiaki Fujii -->


<!-- Date: Sun Jan 18 16:07:42.435 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-1359&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-1359  width='300' align='top'></a></td><td>s</td></tr></table>
<!-- Author: s -->
<!-- Comment: Sun Jan 18 16;08;16 comment by...Dale and Susana -->
<pre>Luminosity plots for the day shift.</pre>



<!-- Date: Tue Jan 27 23:54:37.362 2004 -->
<pre>got error: 
 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 16 for event 5004674. 
VRB_BUS_ERROR: Alignment or VME bus error. 
 
hrr fixed it.</pre>
<!-- Author: ian :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178539' target=_top>178539</a>) -->
<!-- Comment: Tue Jan 27 23;55;27 comment by...ian -->
<table border=0><td><tr><td bgcolor=#e0e030>ignore!!! wrong day!</td></tr></td></table>