<!-- Title: Eve shift. Tue Jun 26, 2001 -->
<!-- SciCo: Giovanni Pauletta -->
<!-- DAQAce: Heather Ray -->
<!-- MonAce: Rob Napora -->
<!-- CO: James Kraus -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre>Store in progress. 
We still have 1.2 E30
Level 2 tests in progress
About to test new trigger table.
G.P.</pre>


<!-- Date: Tue Jun 26 16:26:07 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030>
<pre>The CDF Run2 Data Base Browser: 
 
   http://cdfdbb.fnal.gov:8520/cdfr2/databases 
 
was updated this afternoon (about 1330).  The changes this time 
were extensive.  Bugs, unfortunately, are to be expected.  The 
procedure for error reports is on the initial query page of the 
browser.  If an older version of the browser is needed, then: 
 
   http://cdfdbb.fnal.gov:8520/cdfr2old/databases 
 or 
   http://cdfdbb.fnal.gov:8520/cdfr2older/databases 
</pre></td></tr></td></table>
<!-- Author: Randolph J. Herber -->


<!-- Date: Tue Jun 26 16:35:21 2001 -->
<pre>Yesterday we swapped a power supply for Shower MAx, phi=21 
That was for ccal15, slot 3. 
There was no smqie calibration after that with the result 
that the wedge showed 0 occupamncy. 
We asked teh shift crew to calibrate that particular crate 
when there is a chance.</pre>
<!-- Author: vaia -->


<!-- Date: Tue Jun 26 16:44:58 2001 -->
<pre>Same problem as before. Now with c09. 
Subfarm 08 does not processing events.  
ATM connection is ok.  
 
This is unknown problem. We figure it out quite recently.  
 </pre>
<!-- Author: Boris.  -->


<!-- Date: Tue Jun 26 17:03:23 2001 -->
Stephen and I compared his code and run control code, and found 
(and fixed) the problem causing L2 to reject when it shouldn't.  
<!-- Author: Arnd -->


<!-- Date: Tue Jun 26 17:04:08 2001 -->
<ul> 
<li>Run 118775 ended with 11.6K events to do L2 tests 
<li><b>THE FOLLOWING RUNS WERE TAKEN TO TEST L2.  PLEASE SEE L2 LOG FOR MORE INFORMATION</b>
<li>Run 118776 L2 test with RateTest[1,150] short run.  End run to change potentially bad DCAS board which was causing CLIST to hang 
<li>Run 118777 L2 test with DCAS slot 4 crate b0l2cl05 changed to SN29.  Alpha run in non-tagging mode with RateTest[1,150] table.  L3 problem - cn09 and all subfarms in that node are pale blue but not processing events. (no errors in RC error display)  Paged Boris - see his log entry above.  DTO from CMX01.  This run had several changes in prescales and unplugging of CLIST. L3 filter 0 on subfarm 125 dies.  DTO from CMU01.  Run ended at 65K.   
<li>Run 118778 started with Alpha in tagging mode.  Run ended at 16 K due to some RC changes by Arnd.
<li>Run 118779 ended because all converter node names turn red after coldstart.  Reset and repartition
<li>Run 118780  still doing L2 tests.  L3 problems:  Filter 0 on node 50 dies.  
</ul>
<!-- Author: hray -->
<!-- Comment: Wed Jun 27 15;16;44 comment by...hray -->
In my hurry I changed a DCAS in the wrong crate.  I REALLY changed DCAS in slot 4 crate b0l2cl03 which is crate 25!! 
<!-- Date: Tue Jun 26 17:39:15 2001 -->
<p>When I came on Ace shift this evening I learned that there had been a 'burp' in the BSC. <p>First, there were some <a href= 
'/cgi/elog/elog.pl?nb=2001&action=view&page=429&frame=2&anchor=09:42:21'>alarms during the day 
shift</a> that seemed to go away on there own. Then the ace noticed that there was a sudden <a 
href='/cgi/elog/elog.pl?nb=2001&action=view&page=429&frame=2&anchor=11:34:36'> jump in the losses 
</a> around 1130 this morning. <p>It turns out that 5 of the 18 channels on the BSC had been tripped, but the alarms hadn't stayed in 
their red state. The apparent jump in the losses occurred when Koji <a href= 
'/cgi/elog/elog.pl?nb=2001&action=view&page=429&frame=2&anchor=12:29:07'>reset the tripped 
channels</a> and they began reading again. This means that the <strong>BSC was giving inaccurate, lower readings of the losses for 
some still as yet undetermined amount of time</strong>. <p>I am going to e-mail the global alarms specialists and see if we can get the 
BSC to stay in an alarm state when this happens.  
<p>In the meantime, <strong><font color='yellow'>Monitoring Aces</font></strong> need to take all BSC alarms <i>very seriously</i>.  Even if one seems to recover on its own you should page Koji at 218-4400 so he can check for these trips.
<!-- Author: Napora -->


<!-- Date: Tue Jun 26 17:42:51 2001 -->
<pre>I modified the following L2 executables to use the correct 
cut value for the Jet trigger. The scale factor is 
off by a factor of 4. This will also be the case for all new 
trigger tables until the TriggerDB code is updated. 
36x36_1_Coupled_noInclusive v-4 
36x36_1_Coupled v-17 
PHYSICS0_DECOUPLED v-7</pre>
<!-- Author: Stephen Miller -->


<!-- Date: Tue Jun 26 17:43:50 2001 -->
<pre>We had a run with L3 trashing events and a high L2 accept 
rate. We saw a rate limitation of 160 Hz going into L3.  
Boris checked the L3 rates and noted that internally the events 
are being sent to the output nodes.  
 
To do a realistic test we need to actually reject events at  
L3 (i.e. not send to the output nodes). 
 
For this we will need a L3 random filter with a rejection factor 
of 10. So we can run at a input rate of 200 Hz with an output 
rate of 20 Hz... 
 
In the earlier tests we were able to run at about a L1 rate of 
about 16 KHz when the CLIST board was removed. 
 
With the CLIST board in we saw that the clustering code was  
hanging leading to long timeout delays which limited the  
L1 rates to about 1-2 KHz. 
 
The L1 rate test trigger had a L1 rate of about 17 KHz.</pre>
<!-- Author: Frank Chlebana -->


<!-- Date: Tue Jun 26 18:04:59 2001 -->
<pre>The CLC LED Calibration instruction, which was pasted next to 
the ACE's Run Control monitor, was removed. This is because 
currently the CLC LED calibration should only be carried out 
by CLC experts. Having this instruction pasted there may confuse 
the ACEs. 
</pre>
<!-- Author: Ming -->


<!-- Date: Tue Jun 26 18:19:26 2001 -->
Run 118783 started with PHYSICS0_DECOUPLED[7,150] trigger table.  Majority of subfarms are green or darkblue (output or busy state).  Number of events pending in EVB is ~25 or 27.  Most events in CSL logger.  We paged dh group to ask all files from this run go to the look area, so this could account for the backup....I walked over to L2 and it looks as if CLIST/clustering is hanging again.  Lots o files written to look area.
<!-- Author: hray -->


<!-- Date: Tue Jun 26 18:20:50 2001 -->
<br><b>b0puls01:</b> 
<br> Installed ADMEMs into slots 20,21 of this crate.  Details 
are found in the <a href="/cgi/elog/det-system-elog.pl?nb=wedge&action=copyentry&page=63&time=18:18:45"> wedge e-log.</a>
<!-- Author: R.J. Tesarek -->


<!-- Date: Tue Jun 26 18:28:17 2001 -->
<pre>COT SL7 S6c trippped.  There was absolutely nothing interesting to see on the detailed display.  I reset it.</pre>
<!-- Author: Napora -->


<!-- Date: Tue Jun 26 18:29:36 2001 -->
I forgot this one before...before beginning run 118783 I did a SM QIE calibration for crate CCAL15 ONLY.  Vaia requested this because it was not done after yesterday's access.  This was run 118782.
<!-- Author: hray -->


<!-- Date: Tue Jun 26 18:42:50 2001 -->
<pre>Looking at Ymon at shower max. After the smqie calibration 
wedge 21 looks fine now.</pre>
<!-- Author: vaia :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=118783' target=_top>118783</a>) -->


<!-- Date: Tue Jun 26 18:48:18 2001 -->
Ended run 118783 with 100K events as requested.  There was a  COT trip at event 65732.  Halt run to reset HV.  As recovering run all subfarms for converter node 07 turned orange.  Start run 118784 with default 36x36_1_coupled_noinclusive[2,142] with a reset and partition to recover all L3  subfarms.  
<!-- Author: hray -->


<!-- Date: Tue Jun 26 18:53:29 2001 -->
<table><tr><td><a href=/elog/notebooks/2001/Tue_Jun_26_2001_Eve.185300.ps align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1318  align='top'></td><td>There is a spike in the Starttime for SL5 of the COT</a></td></tr></table>
<!-- Author: Jim Kraus -->


<!-- Date: Tue Jun 26 19:02:32 2001 -->
all L3 display still green or  dark blue (busy or output).  26 pending events in EVB.  Logger full of events in CSL.  Paged CSL and they will come to control room.
<!-- Author: hray -->


<!-- Date: Tue Jun 26 19:04:24 2001 -->
Another Flying Wire.  This time the COT gave the 'tweety-bird alarm' immediately <i>after</i> I turned the COT back on.  It stayed green and powered up normally, however. 
<!-- Author: Napora -->


<!-- Date: Tue Jun 26 20:52:55 2001 -->
<ul> 
<li>CSL expert came to control room and said everything was okay 
<li>During run 118784 got two BTO from CCAL05,06,07,13,14,15.  HRR recovered these crates.  Checked all EVB/L3 proxies, etc, and all was okay. 
<li>Tony V. comes to control room because he notices rates are low.  At his suggestion we end run at 232K events.  Tony believes Disk 3 causing backup.  He believes if we ColdStart that Disk 2 will be used instead.  Coldstart still initiates Disk 3.  We end and reset/partition.   
<li>run dead in the water with 48 events with no error messages and no L3/EVB errors.  End and do a coldstart.  run 118787 starts and gives several BTO immediately with CCAL00,05,06,07,13,14,15.  All crates ping okay.  All crates Minicom <ctrl c> okay.  Since we had done 3 HRR and still got BTO we end and reset/partition.   
<li>EVB does not pass partition.  A check shows it needs cleaned up.  (Sorry Ilya - I forgot to dumpSm before I cleaned up).  We clean up and start new run. 
<li>run 118788 started.  Disk 2 logging the data.  Now some L3 farms are light blue, pending events for EVB is 2, and rates for streams total is now 20 MB/s.  Thanks, Tony! 
</ul>
<!-- Author: hray -->


<!-- Date: Tue Jun 26 21:21:42 2001 -->
<pre>There is an unusually low number of events in the EMCluster Plot  
in ObjectMon around Phi = 5.5 rad.</pre>
<!-- Author: Jim Kraus :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=118788' target=_top>118788</a>) -->


<!-- Date: Tue Jun 26 21:56:13 2001 -->
The Pisa box for the CSX NW arch had blown a fuse 
It has been replaced.  
<!-- Author: G.P. :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=118790' target=_top>118790</a>) -->


<!-- Date: Tue Jun 26 22:32:40 2001 -->
<ul> 
<li>run 118788 ended with again several BTO's in CCAL crates.  HRR did not help.  Halted, checked all L3/EVB actions, RR, and still get BTO's.  Rick T. was here at the time.  We could find nothing wrong with the ccal crates.  We try cleaning up EVB.  I did a dumpSm for L3 debugging purposes.  Notice VRB errors on RC error -> go downstairs and reset VRB by hand for crate 21E, corresponding to CCAL.  Page L3 to let them know and for some more insight.. 
<li>Run 118789 started.  EB fails activate.  RC error says to check ScramNet - check says everything is okay.  Cleanup EVB.   
<li>try again - twice in a row EVERYTHING TIMES OUT during partition (except L3/EVB, which show no problems).  manually reset TSI00 by hand and clean up EVB 
<li>now TSI00 will not pass partition.  Vxlogin shows NO RC messages arriving at TSI00.  Paging experts. 
</ul>
<!-- Author: hray -->
<!-- Comment: Tue Jun 26 22;42;35 comment by...patrick -->
<pre>Unless there is some message indicating data corruption
in the VRB, then the crates listed with the BUSY timeout
are very rarely the cause of the problem.

In this particular case, the VRB in slot 10 of the
b0eb15 crate got in a bad state and does not respond
to VME operations:

(MLE) b0l3gate1.fnal.gov:main:9:10:56 PM->Host b0eb15.fnal.gov, task tRec_0
SCPU-P0-E-CantResetVrb: Reset of VRB in slot 10 failed.
VRB_NOT: The module at the given VME address has the wrong module ID.

To demonstrate this, log into
b0vrb04 (in the same crate as b0eb15, but has the
capability to do VISIONdemo unlike b0eb15); do
VISIONdemo, and map the crate. Slot 10 will not
respond. What has recovered this in the past is
to power cycle the crate. If this is not successful,
then the VRB must be swapped.

Whenever all the other problems with b0tsi00 etc.
are solved, the original problem must also still
be solved.</pre>

<!-- Comment: Tue Jun 26 23;15;24 comment by...hray -->
How are we as aces supposed to know that slot 10 should be responding?  I've just done VISIONdemo and slot 10 shows (EMPTY) while slots 12 and 14  have the values of 0x3.  Please let us know.


<!-- Comment: Tue Jun 26 23;21;09 comment by...patrick -->
<pre>Although the error that ended 118788 was admittedly
hard to find in the forest of messages (something we
clearly need to work on) the begin run for 118789 failed with:

Error Handler Log File
All timestamps are referred to CT
CT: 6/26/2001, 21:37'36"
Partition:0 Run number: 118789 (0x1d005)
(RC)   21:39:26 Config -> READY
(MLE) b0l3gate1.fnal.gov:main:9:40:17 PM->Host b0eb15.fnal.gov, task tCtrl_0
SCPU-P0-E-CantResetVrb: Reset of VRB in slot 10 failed.
VRB_NOT: The module at the given VME address has the wrong module ID.
(MLE) b0l3gate1.fnal.gov:Sender:9:40:40 PM->Could not start new run 118789.
(RC)   21:40:41 Activate -> ERROR

This should be an indication that there should be a module
responding in slot 10 of that crate, but was not.

</pre>


<!-- Comment: Tue Jun 26 23;51;21 comment by...h -->
ah, yes, we were worried about other things at that time, as well.  thank you.

<!-- Comment: Wed Jun 27 07;06;25 comment by...SCN -->
And a reminder-when you power down the Vrb crate to recover
this Vrb, you'll need to clean up the EVB.
<!-- Date: Tue Jun 26 23:48:37 2001 -->
<ul> 
<li>Call Greg F. We see in Vxlogin window that FER processes haven't been connected to the b0tsi00 crate.  We minicom into the crate and they reboot.  Greg is not sure why a minicom would cause FER to be reconnected.  We did NOT issue any more crate resets/etc for that crate.  This is a Jim Patrick question. 
<li>so, now everything TIMES OUT AGAIN during coldstart but FER is in TSI00.  We page Arnd.  He has us kill RT, cleanup L3 and EVB.  Again, get DTO during ColdStart.  We see all crates giving Timed out waiting for r/o list messages in RC error window.   (this is present all times crates fail during coldstart).  Again we kill RT and cleanup L3 and EVB, this time killing ALL java processes and restarting.  Again it fails.  Arnd is coming in. 
WE HAVE BEEN DEAD FOR 3 HOURS NOW.
</ul>
<!-- Author: hray -->



<!-- Date: Tue Jun 26 23:59:34 2001 -->
<font color=red>Shift Summary: </font><pre>Shift started with luminosity ar 1.2 E10 
Rate studies in progress 
DCAS board replaced 
 
After tests, took 100k data with PHYSICS0_DECOUPLED[7,150] 
(run 118783) Checked that data went to look area. 
Then went back to runnning with the default trigger table. 
 
Flying wires on the hour. 
 
About 22:00, we started having DAQ problems. 
(RT server - related ?) 
We are passing them on to the experts and the next shift. 
Good luck!</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   118776 - 118788
Delivered Luminosity   481 nb-1
Acquired Luminosity    67 nb-1
Efficiency             14%
   
</td></tr></table></pre>
<!-- Author: G.P. -->


<!-- Date: Wed Jun 27 00:01:14 2001 -->
Called Ilya about Patricks entry regarding VRB power cycle/change.  He said either he or Boris would power cycle the b0eb15 crate while we are waiting for Arnd.
<!-- Author: hray -->


<!-- Date: Wed Jun 27 00:06:36 2001 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2001&action=view&page=-1321&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2001&action=view&page=-1321  width='300' align='top'></td><td>Luminosity and losses for the evening.  Svx Rads were 0 all night.</a></td></tr></table>
<!-- Author: Napora -->