<!-- Title: Eve shift. Mon Nov 28, 2005 -->
<!-- SciCo: Marco Rescigno -->
<!-- DAQAce: Lester Pinera -->
<!-- MonAce: &nbsp; -->
<!-- CO: &nbsp; -->
<!-- OpManager: Guram -->
<!-- Notes: -->
<pre>Expert working on Event Builder and Level3</pre>


<!-- Date: Mon Nov 28 16:13:57.152 2005 -->
Expert is doing some work on central calorimeter crates, so he asked that they be removed from the cosmics run configuration.
<!-- Author: Lester -->

<!-- Date: Mon Nov 28 16:19:37.209 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207787">207787</a>
TERMINATE: end test fram download
<!-- Author: radum -->



<!-- Date: Mon Nov 28 16:34:27.266 2005 -->
the b0ccal00 crate has been powercycled in order to perform some fram download tests. Will postpone further tests until work on the  boot files will be complete. b0ccal00 will need to be rebooted when the boot files will be restored.  
<!-- Author: Radu -->
<!-- Comment: Mon Nov 28 17;37;03 comment by...Lester -->
Does this mean that I can use all ccal crates except this one when running cosmics?
<!-- Date: Mon Nov 28 17:07:43.504 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207804">207804</a>
 Activated at 2005.11.28 17:07:35
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 17:11:44.034 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207804">207804</a>
 Terminated at 2005.11.28 17:08:47
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 17:32:32.376 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207805">207805</a>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,729,599] 
<!-- Author: Lester x2080 -->



<!-- Date: Mon Nov 28 17:41:15.364 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207806">207806</a>
TERMINATE: L2 decision TO problems
<!-- Author: Lester x2080 -->


<!-- Date: Mon Nov 28 17:50:59.919 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207807">207807</a>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,729,599] 
<!-- Author: Lester x2080 -->


<!-- Date: Mon Nov 28 17:51:21.863 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207807">207807</a>
TERMINATE: L2 Decision TO persists
<!-- Author: Lester x2080 -->



<!-- Date: Mon Nov 28 17:59:06.932 2005 -->
<pre>Level3 Testing:  
 
Checked that we can run the Level3 proxy from node b0eproxy01 in the EventBilder cluster (it usually runs from b0eproxy02). This took a little bit of configuration, but now it works fine. This is good, we now have a backup node ready in case of an emergency failure of the main Level3 proxy node. The system has been returned to normal state again.</pre>
<!-- Author: Conor -->

<!-- Date: Mon Nov 28 18:01:47.032 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207809">207809</a>
 Activated at 2005.11.28 18:01:01
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 18:01:48.831 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207809">207809</a>
 Terminated at 2005.11.28 18:01:41
<!-- Author: RunControl -->



<!-- Date: Mon Nov 28 18:01:48.303 2005 -->
<pre>Have recompiled a whole passel o' packages on the new  
/cdf/onln/code/cdfprod code disk, taking code from  
the CVS repository.    Below is a list of packages on  
the new disk;  if the date says 2005.11.28 next to the package,  
then it was recompiled today.     All CDF front-end crates  
have been rebooted.  I hope I didn't screw up too much... 
Apologies for the long entry. 
 
 
ace/                 LP/DT 
acromag/             reference only 
admem/               2005.11.28 
admemdaq/            2005.11.28  
admemmod/            2005.11.28 
asdqtest/            obsolete?? 
bncclc/  
cabletest/           obsolete 
CalibDB/             2005.11.08 
cal_calib/           obsolete 
camac/               2005.11.28   
cdfdb/               2005.11.17 
cdfevb/              reference only 
cdfevb2/             reference only 
cdfr2dm.save/        LP/DT 
cdfr2dmROOT.save/    LP/DT 
cdfvme_admem/        2005.11.28 
cdfvme_common/       2005.11.28        (VxWorks-5.3) 
cdfvme_doc/          2005.11.28        put into CVS        
cdfvme_example/      2005.11.28        put into CVS 
cdfvme_tdc/          2005.11.28 
cdfvme_template/     2005.11.28        required revisions 
cdfvme_testclk/      2005.11.28        put into CVS 
cdfvme_teststand/    2005.11.28        required revisions 
cdfvme_tracer/                         does not compile  (but works) 
cdfvme_ts/           2005.11.28        put into CVS; required revisions 
cdfvme_vme/          2005.11.28 
cdf_daq/             obsolete 
cdf_frc/             obsolete 
cdf_tsi/             obsolete 
cdf_vrc/             obsolete 
cdf_vxworks/         obsolete 
ces8210/ 
clcdaq/              2005.11.17 
clique/              2005.11.28 
clock/               2005.11.28 
cmuAsd/              2005.11.08 
cosmic_readout/      obsolete 
cosmic_test/         obsolete 
cot_calibration/     obsolete? 
cratesum/            2005.11.28        (has orphaned later versions) 
csl/ 
daqmsg/              2005.11.28 
dbaccess/            obsolete 
dbbroker/ 
dcas/                2005.11.28 
dirac/               2005.11.28 
errmon/              2005.11.09 
fdaq/                2005.11.28 
fer/                 2005.11.08 
festival/ 
fision/ 
fision5500/ 
fisionwin/ 
fred/                2005.11.28 
gl2/                 2005.11.28 
hdwdb/               2005.11.08 
images/              n/a 
inhibit/             2005.11.08 
inhibitDisplay/      obsolete 
jas/                 commercial 
jdk/                 commercial 
jfc/                 commercial 
jimi/                commercial 
jpython/             commercial 
l1cal/               data only 
l2cal/               2005.11.28 
l2muon/              obsolete 
l2proc/              obsolete 
l2test/                                Ron Moore?!? 
l2trig/              obsolete 
l2tstand/            obsolete 
l3manager/ 
locos/               2005.11.28 
logbook/             n/a 
lumi/ 
matchbox/ 
merlin/ 
microdaq/            obsolete 
mtc/ 
mtsc/ 
mutrig/ 
mysql_jdbc/          commercial 
online_banks/        obsolete 
OptimizeIt/          commerical 
oracle/              commercial 
oraInventory/        commercial 
patch-SPR97010/      commercial 
pfred/               2005.11.28 
prematch/ 
proxy/               2005.11.08 
pulser/                            Ron Moore? 
r2dm/                LP/DT 
rawref/              LP/DT/MK 
reflexmem/           obsolete 
reset_crate/         2005.11.08 
resmgr/              2005.11.08 
robin/ 
rtilib/              commercial 
rtilm/               commercial 
smartsockets/        commercial 
smx/                 2005.11.28    required revisions 
speech_tools/ 
struck/              2005.11.28 
svttest.test/ 
svtvme.test/ 
svxdaq/              2005.11.28  (SG) 
svxii/               2005.11.28  (SG) 
t22-cp1-drivers/     commercial 
t22-cp1-ppc/         commercial 
tdc/ 
tdctstand/ 
tdc_db/ 
tdc_dsp/ 
testalpha/           obsolete 
testclock/           2005.11.28 
testmtc/ 
tof/ 
tor221_mv5500_1_2_1/ commercial 
tor_generic/         commercial 
tor_mv2301/          commercial 
tor_mv2304/          commercial 
tor_mv2400/          commercial 
tor_mv5500/          commercial 
tor_mv5500_new/      commercial 
tor_ppc/             commercial 
tracer/              2005.11.28 
tracerdaq/           2005.11.28 
tracermod/ 
trbroker/            obsolete 
trc/                 obsolete 
TriggerDB/           2005.11.16 
TriggerDBGui/        2005.11.09 
ts/ 
ttcp/ 
upsdb/               n/a 
vase/                obsolete 
vdb/                 obsolete 
vibe/                obsolete 
vmetro/              2005.11.28 
vrb/ 
vrbcaldaq/ 
vrbtest/             2005.11.28 
vxboot/              n/a 
vxworks/             commercial 
vxworks_generic/     commercial 
vxworks_kernel/      commercial 
vx_tools/            reference only 
xft/ 
XFTbin/ 
xftdaq/ 
xftmod/ 
XFTtestvectors/ 
xpt/ 
xtrp/ 
xtrp3TT/ 
xtrpdaq/ 
 
 
 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207809' target=_top>207809</a>) -->

<!-- Date: Mon Nov 28 18:07:40.935 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207810">207810</a>
 Activated at 2005.11.28 18:06:58
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 18:07:41.402 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207809">207809</a>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,730,598] 
<!-- Author: Lester x2080 -->


<!-- Date: Mon Nov 28 18:12:23.164 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207810">207810</a>
 Terminated at 2005.11.28 18:11:29
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 18:12:24.230 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207810">207810</a>
TERMINATE: No events flowing through
<!-- Author: Lester x2080 -->


<!-- Date: Mon Nov 28 18:15:13.975 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207813">207813</a>
 Activated at 2005.11.28 18:14:52
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 18:19:30.530 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207813">207813</a>
 Terminated at 2005.11.28 18:16:17
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 18:19:31.055 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207817">207817</a>
 Activated at 2005.11.28 18:18:57
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 18:19:33.648 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207817">207817</a>
 Terminated at 2005.11.28 18:19:28
<!-- Author: RunControl -->



<!-- Date: Mon Nov 28 18:54:31.760 2005 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>We have successfully loaded b0fib00 and b0svx00 with  
the latest CDF VxWorks kernel, and have booted it off  
the latest /cdf/onln/code/cdfonln disk, with all the  
latest software builds.  Woo-hoo!!! 
 
Tomorrow the rest.... 
</pre></td></tr></td></table>
<!-- Author: W.Badgett, Sebastian Grinstein -->

<!-- Date: Mon Nov 28 19:07:12.959 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207836">207836</a>
ACTIVATE: RunSet: ALL_COT  Trigger: PHYSICS_3_07 [2,725,597] 
<!-- Author: jameseb -->


<!-- Date: Mon Nov 28 19:15:47.974 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207843">207843</a>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,729,599] 
<!-- Author: Lester x2080 -->


<!-- Date: Mon Nov 28 19:15:48.478 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207843">207843</a>
TERMINATE: L2 Decision TO persist
<!-- Author: Lester x2080 -->


<!-- Date: Mon Nov 28 19:20:02.293 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207848">207848</a>
ACTIVATE: RunSet: ALL_COT  Trigger: PHYSICS_3_07 [2,725,597] 
<!-- Author: jameseb -->


<!-- Date: Mon Nov 28 19:56:40.732 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207856">207856</a>
TERMINATE: Ending L2Torture to change to cosmics
<!-- Author: Lester x2080 -->


<!-- Date: Mon Nov 28 20:02:42.581 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207857">207857</a>
 Activated at 2005.11.28 20:00:24
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 20:02:43.484 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207857">207857</a>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,730,598] 
<!-- Author: Lester x2080 -->


<!-- Date: Mon Nov 28 20:02:44.326 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207857">207857</a>
 Terminated at 2005.11.28 20:02:27
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 20:07:11.340 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207857">207857</a>
TERMINATE: L3 display went green
<!-- Author: Lester x2080 -->


<!-- Date: Mon Nov 28 20:07:12.468 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207858">207858</a>
 Activated at 2005.11.28 20:04:41
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 20:07:14.167 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207858">207858</a>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,730,598] 
<!-- Author: Lester x2080 -->


<!-- Date: Mon Nov 28 20:10:21.025 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207858">207858</a>
 Terminated at 2005.11.28 20:08:29
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 20:10:22.115 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207858">207858</a>
TERMINATE: Ending to cleanup L3/evb
<!-- Author: Lester x2080 -->


<!-- Date: Mon Nov 28 20:17:23.447 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207859">207859</a>
 Activated at 2005.11.28 20:15:27
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 20:17:24.704 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207859">207859</a>
 Terminated at 2005.11.28 20:16:51
<!-- Author: RunControl -->


<!-- Date: Mon Nov 28 20:18:06.044 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207859">207859</a>
TERMINATE: L3 green again.  Will page expert
<!-- Author: Lester x2080 -->



<!-- Date: Mon Nov 28 20:18:11.487 2005 -->
Ulrich has informed me that in preparation for tomorrow's power outage the svx chillers have been warmed up and this is the reason that L00,SVX and ISL may become yellow in iFix. 
<!-- Author: Lester -->


<!-- Date: Mon Nov 28 20:54:57.683 2005 -->
So we finally paged L2 about the L2 Decision TO we've been getting all night.  Vadim called back and he initially identified the problem as coming from shower max (the readout for those systems are on the ccal crates).  Vadim claims that he didn't do anything, we just started a fresh L2Torture.  We then got a bunch of Done TO from Cot02,04,09 crates--Shepherded all of them and things were running smoothly.  Guram and I then decided to end the L2 Torture and start a cosmics run.  Shortly after starting the cosmics run L3 display went green indicating that there was a problem with L3/CSL.  Strange thing is that events were still passing through L3 and the CSL looked fine (total input rate was at about ~10 MB/s). Total deadtime on scalar mon was high ~88%. I ended that run, cleaned up L3 and evb just to start fresh, but the same problem persisted so we switched back to L2 Torture.  We then paged L3 expert Alberto who insists that it is a CSL problem.  I will investigate this 
<!-- Author: Lester -->
<!-- Comment: Mon Nov 28 23;19;20 comment by...Alberto -->
<pre>L3 Page
I didn't find any error/warning in the log files of some L3 machines. The fact that all L3 output nodes were working and sending data to CSL also indicates that L3 was performing correctly. I would make sure with CSL experts that the limit of CSL performance had been reached.</pre>
<!-- Date: Mon Nov 28 21:22:22.303 2005 -->
BMU access report:  Sunny and I investigated two HV sectors (NW24 and SE144) that showed noisy stacks. A couple of stacks (NW26-27 and SE150-151) show most of noise standing out in occupancy plots, but they spread over six stacks (NW24-29 and SE150-155) when observed signals with scope before TDC -- transition cards?  Both noise patterns look like the ones caused by small HV sparking and when examined with portable HV supply, small sparkings were evident in both cases.  The culprit stacks (NW26 and SE151) were isolated by taking Xend jumper out and HV boards were replaced.  We will watch over those two other HV sectors (NE12 and NE60) which used to trip many times recently. 
<!-- Author: Woo-Hyun & Sunny -->

<!-- Date: Mon Nov 28 22:03:03.977 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207863">207863</a>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,729,599] 
<!-- Author: Lester x2080 -->


<!-- Date: Mon Nov 28 22:12:25.311 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207864">207864</a>
ACTIVATE: RunSet: L2TORTURE  Trigger: L2_TORTURE [16,729,599] 
<!-- Author: Lester x2080 -->



<!-- Date: Mon Nov 28 22:34:37.873 2005 -->
While running L2Torture I noticed that the Runsection number in RunControl was updating at an unusually fast rate (I'm not even sure if this number should be advancing during L2Torture).  If you look at run number 207860, the number of the runsections is ~60k.  We paged DAQ and he agreed that this was strange.  He suggested that we start a new RunControl session and restart CSL. I had to issue a cslcom cleanup because not all processes had stopped after issuing a cslcom stop command.  We then started a new run, but the problem with the advancing RS persists with the current run (207864). The DAQ pager does not know what's wrong, but he thinks this is not an urgent problem and it can wait until morning. 
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=207864' target=_top>207864</a>) -->


<!-- Date: Mon Nov 28 23:55:57.143 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>Physics able</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L1Accepts</TH>
<TH>L2Accepts</TH>
<TH>L3Accepts</TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:00</TD>
<TD ALIGN=CENTER>::</TD>
<TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Tue Nov 29 00:00:00.546 2005 -->
<font color=red>Shift Summary: </font><pre>Running L2_TORTURE with some mysteries going on.</pre> 
<!-- Author: Marco Rescigno  -->