<!-- Title: Day shift. Sat Sep 30, 2000 -->
<!-- Leader: Bill Carithers -->
<!-- Ace: Mike Kirby -->
<!-- Co1: Heather Gerberich -->
<!-- Co2: Hiro Matsunaga -->
<!-- Co3: Kevin Burkett(SOS) -->
<!-- Co4: Willis Sakumoto(SOS) -->
<!-- Co5: &nbsp; -->
<!-- Notes: -->
<pre>Shift plan: 
Run cosmics unless beam unexpectedly appears 
Plug crates--  Rick T. 
CLC crates (1:00-3:00)--Alexei, Ming, Sergei 
If there might be beam, call Steve Chappa in for clock test 
 
SVX rad log: 
West inner		.4811 
West outer		.5359 
East inner		17.08 
East outer		22.94</pre>


<!-- Date: Sat Sep 30 09:02:09 2000 -->
<pre><br><b>Plug Occupancy Problem:</b> 
<br>After much head scratching, found that corrupt calibration data were loaded into the CAFE cards.  It appears that this incident can be attributed to a failed "CALIB LOAD" during the 
QIE calibration taken by the evening shift crew.  Reloaded the 
CAFE cards with CORRECT data using a standalone program and all appears ok now.  For more details, see plug e-log for 9/30/00. 
<br></pre>
<!-- Author: R.J. Tesarek, M.Lindgren :: (run 99970) -->


<!-- Date: Sat Sep 30 09:19:59 2000 -->
<pre>Regarding the reformatter errors attributed on 
evening shift to b0cot14 - This crate fails the 
standalone tracer-VRB link test. Hence it 
should be removed for runs with the hardware 
event builder</pre>
<!-- Author: patrick -->


<!-- Date: Sat Sep 30 09:54:25 2000 -->
<HTML> 
<HEAD> 
   <META NAME="GENERATOR" CONTENT="Mozilla/4.05 [en] (X11; I; IRIX64 6.2 IP25) [Netscape]"> 
</HEAD> 
<BODY> 
&nbsp; 
 
<P>************* DIRAC&nbsp; BC&nbsp;mismatch problem understood (well, 
almost) ******* 
 
<P>******* people who uses fdaq tool please read this message ********* 
<BR>&nbsp; 
 
<P>last night while on owl shift &nbsp;I took a look at the bc mismatch 
problem with DIRAC crate 2, 
<BR>b0l1cl02.&nbsp; what &nbsp;I found is quite strange: DIRAC&nbsp;BC&nbsp;mismatch 
errors can be caused by reading the register or L2 buffers with fdaq tool 
! 
<BR>&nbsp; 
 
<P>First I&nbsp;took a long run with DIRAC b0l1cl02 with 20,000,.000 events, 
there was no single error. 
<BR>then I&nbsp;used the fdaq tool to read some register and L2 buffers 
on one of the DIRAC&nbsp;boards.... 
<BR>right away the bc mismatch errors showed up!&nbsp; and the probelm 
can only be fixed by HRR , 
<BR>coldstart (but not all the time) or power cycle. if you read board 
5 (n) with fdaq, then the error message will say " bunch counters slot 
4 and 5 (n) disagree...".&nbsp; clearly the reading 
<BR>of one board upsets the board. 
 
<P>the only way I&nbsp;can imagine that the&nbsp; fdaq tool can mess up 
the crate is somehow 
<BR>it writes to the board (such as changing the bc offset) or halt the 
board so that 
<BR>this board is out of sync with the rest of boards after the read.&nbsp; 
most likely the later to me. 
 
<P>but I don't know anything about fdaq. This morning I&nbsp;asked Jim 
Patrick about what exactly fdaq 
<BR>does when using it to read DIRAC.&nbsp; he took a hard look of the 
code and concludes that it is likely that the&nbsp; fdaq tool can upset 
DIRAC. but it is hard to find out exactly what fdaq tool does 
<BR>on DIRAC.&nbsp; I will ask the people who wrote that part of the code 
(Pedero?) about this later (he is on vacation). 
 
<P>I am not sure if this was the reason for the bc mismatch error occurred 
yesterday 
<BR>for b0l1cl02. if someone were using fdaq tool yesterday to read DIRAC&nbsp;(even 
once!) 
<BR>that would explain it. so if you ever used fdaq yesterday and read 
DIRAC boards, 
<BR>I&nbsp;would very much appreciate if you can let me know (thliu@fnal.gov). 
this would help 
<BR>me a lot to understand the problem.....&nbsp; meanwhile for those who 
uses fdaq, please 
<BR>don't use it on DIRAC crates, as it&nbsp; will for sure causing bc 
mismatch errors. and I&nbsp;have done this many times tonight. 
 
<P>I&nbsp;just learned to use this fdaq tool yesterday, hoping that I&nbsp;can 
use it to find out 
<BR>what is causing the bc mismatch problem.... without knowing that this 
<BR>very tool is the guilty one...... 
<BR>&nbsp; 
<BR>&nbsp; 
</BODY> 
</HTML>
<!-- Author: Ted Liu -->


<!-- Date: Sat Sep 30 10:26:48 2000 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Del Allspach has given permission for the following systems to turn on HV: 
IMU, 
CMU,  
NWX(EXCEPT for the disconnected and tagged out part of the miniskirt), 
SWX, 
SEX, 
SWA, 
TOP, 
BOT</pre></td></tr></td></table>
<!-- Author: Bill Carithers -->


<!-- Date: Sat Sep 30 10:27:40 2000 -->
<pre>Two crates had corrupt heaps while trying to start a run for testing the COT crates after triage.  b0tsi03 and b0ccal03 both 
had to be reboot due to corrupt heap.  Checked with Rick before 
rebooting b0ccal03, and he said that this crate regularly 
experiences memory leaks.  And Jim Patrick said that b0tsi03 
occasionally has this problem as well.</pre>
<!-- Author: Kirby -->
<!-- Comment: Sat Sep 30 10;54;20 comment by...Kirby -->
The next time I tried to start a run, b0pcal11 had something
possibly related where the FERMessenger was suspended and not
responding.  I rebooted the crate, and things worked again.


<!-- Date: Sat Sep 30 11:03:54 2000 -->
<pre>Started to bring HV on CMX and found that CAEN supplies had been locked and key gone. After few phone calls, found out that any CAEN supply key will activate any other CAEN and one did have a key. This allows us to continue but has a potential safety loophole if someone locked the supply but did not tag it and still thought that they were safe. In this case, the only cable terminations are either in the immediate rack or in the collision hall (plus north wall in assembly hall). We know that there is no one working in these areas because there are no keys out. We continue with bringing up CMX HV. 
</pre>
<!-- Author: Bill Carithers -->


<!-- Date: Sat Sep 30 11:16:00 2000 -->
<pre> 
Having gotten a debriefing from Greg Veramendi on the status of 
the COT TDC crates, I tried to diagnose and fix the BankSets to 
not readout TDCs in a group of crates.  The crates had failed the 
coldstart with board init failures.  I tried to issue a crate 
reset to recover the cards, but none of them came back to life. 
I will remove the cards from the readout list and put the crates into slow localDone readout mode. The crates that needed to 
be fixed are as follows: 
 
Crate	slot 
-----	---- 
03	9 
07	16,17 
10	17 
12	16 
17	16 
18	9,16 
19	16 
 
It seems interesting that the slot that failed was slot 16.  This 
was also the slot that appeared bad in the clean room crate.  As 
well, in the b0htdc00 crate, this slot had a blown buffer and so 
was removed.  <a href="/cgi/elog/det-system-elog.pl?nb=tdc"> Click here for TDC log</a>, look at Friday, Sept. 29, Ron's entry.  And note that Ron removed this card from the readout 
list. 
 
</pre>
<!-- Author: Kirby -->
<!-- Comment: Sat Sep 30 11;46;35 comment by...Ron Moore -->
<pre>Note that both CMU crates have TDCs in slot 9, but NOT 16.</pre>


<!-- Date: Sat Sep 30 12:05:16 2000 -->
<pre>SVX radiation trip. Called MCR and they did have an abort at A0. 
The radiation levels on SVX were very small. Post-trip levels are: 
West inner		.4814 rads 
West outer		.5362 
East inner		17.34 
East outer		23.21 
So, less than a rad in the abort. 
</pre>
<!-- Author: Bill Carithers -->


<!-- Date: Sat Sep 30 12:43:09 2000 -->
As per Monica's instructions, power cycled b0l2cl00 and did a hot 
insertion in b0l2cl02 for card 19.
<!-- Author: Kirby -->
<!-- Comment: Sat Sep 30 13;27;13 comment by...Kirby -->
After the power cycle, b0l2cl00 did not recover and so had to do
a hot removal of cards 4,5,6 and then the other cards recovered.
Then did a hot insertion of 4,5,6 and those boards responded as
well.  The early fix on b0l2cl00 worked for slot 19.  Note that
the only time that this problem is seen, is after a power cycle.
Yesterday, Steve Kuhlmann had been working on these crates and he
may have forgotten to check and make sure that they were working
after he finished.  With DCAS crates, please remember to make
sure that they are in working order after any power cycle.
<!-- Date: Sat Sep 30 13:23:23 2000 -->
<pre>Starting run with almost all of CCAL, PCAL, WCAL, MUON, and COT. 
no silicon.  while running keep getting erros from the 
reformatter.  The error say that 
 
RAWREF ERROR - LINK - CWORD - with code 11(n=2) -- START 
Line - 686 in File: src/library/rr_rawevent.c 
Buffer position: 7467 
 
Specific Message: 
control words are not what they should be: 
Correct control words: 
0x00000000 0xaaaaaaaa 0x55555555 TracerWord: 0x800000000 
What was found: 
0xaaaaaaaa 0x55555555 0x570027a4 0x00000000 0x00000000 
 
Error during scanning MINI structure! 
Last Unit prob. worked on (allcounts from 0); 
SCPU:1 
VRB:0 
LINK:5 
MINI:2 
 
Event Buffer Dump   etc...... 
 
NOTE: Event is rejected! 
 
 
 
This message just contintues to come.  I'm going to try and track 
down which crate it is that gives this problem on the VRB map and 
notify the expert.</pre>
<!-- Author: Kirby :: (run 100027) -->
<!-- Comment: Sat Sep 30 14;24;54 comment by...Kirby -->
after wading my way through the instruction from the L3 group
about how to track down reformatter errors, I found that at least two crates are giving problems:  XFT_FINDER_02 and COT_EAST_14
Interestingly enough, COT_EAST_14 isn't included in the run.  The
reason that it was excluded was that Jim Patrick had found errors
in the TRACER/VRB link and so told us not to run with the hEvb 
and this crate.  But I'm guessing that the connection may be bad
enough to cause noise even though the crate isn't included in the
run.


<!-- Comment: Sat Sep 30 14;42;13 comment by...Kirby -->
Sorry, got confused while looking up the VRB/Crate from L3.
COT_EAST_14 wasn't giving problems, but COT_EAST_13 and
COT_WEST_09 were.
<!-- Date: Sat Sep 30 14:32:07 2000 -->
<table><tr><td><a href=/elog/notebooks/2000/Sat_Sep_30_2000_Day.143135.ps align=top><img src=/elog/gifs/ghost64.gif  width='56' align='top'></td><td>This YMon canvas shows that for run 100027, crate b0ccal14 still appears to be missing. It was included in  the run.</a></td></tr></table>
<!-- Author: Heather Gerberich -->


<!-- Date: Sat Sep 30 14:36:02 2000 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2000&action=view&page=-244&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2000&action=view&page=-244  width='250' align='top'></td><td>For run 100027, b0ccal14 still seems to be missing.  It was included in the run.	</a></td></tr></table>
<!-- Author: Heather Gerberich -->


<!-- Date: Sat Sep 30 14:37:32 2000 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2000&action=view&page=-245&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2000&action=view&page=-245  width='250' align='top'></td><td>Run 100027, b0ccal14 still missing. Was included in run.</a></td></tr></table>
<!-- Author: Heather Gerberich -->


<!-- Date: Sat Sep 30 14:40:06 2000 -->
<pre>runYMon output repeatedly claims: 
Invalid Superlayer or wire for this hit!! SL:6 Wire:12 
Ace does not know why it is doing this.</pre>
<!-- Author: Heather Gerberich :: (run 100027, 100028) -->
<!-- Comment: Sun Oct  1 00;20;44 comment by...Greg V. -->
This output occurs whenever there is a hit that corresponds
to a SL or Wire that doesn't exist. In this case, there
is no wire 12. This is a known problem that one of the
TDC's puts out hit location information.

<!-- Date: Sat Sep 30 14:40:49 2000 -->
Restarted the previous run using a similar config, but also 
included the b0l2cl00 and b0l2cl02 after Monica had fixed them. 
The run shows the same problem with XFT_FINDER_02, but as noted 
earlier two additional crates gave reformatter errors, 
COT_EAST_13 and COT_WEST_09.
<!-- Author: Kirby :: (run 100028) -->
<!-- Comment: Sat Sep 30 14;43;39 comment by...Kirby -->
Also, had ID prom check errors in the XFT crates 01,03, and 05
during the coldStart.


<!-- Date: Sat Sep 30 15:40:13 2000 -->
<pre>Level3 Error Handling Test: 
 
When the ACE attempted to use the Run Set Menu to load  
ERROR_HANDLING_TEST, 1 an error pooped complaining: 
 
PhysicsTable ERROR_HANDLING_TEST, 1 not found in TriggerDB 
 
I have looked in the text Trigger DB and found no entries for this particular tag-set, I didn't make these entries myself. 
In fact after building this exe + tag-set I merely notified cdfl3debug about the new tar-balls.. 
 
The RUN SET menu lists ERROR_HANDLING_TEST 1 as a valid choice 
This is why the test has been delayed slightly. 
 
I will let Kirsten sort this out when she gets in. 
 
</pre>
<!-- Author: Farrukh -->
<!-- Comment: Sat Sep 30 16;33;56 comment by...W.Badgett -->
<pre>Run Control does a secondary check at download time to verify 
that the Level 1 and 2 download tables exist for a 
particular table.  If they do not, then the setup is aborted.

The code that does the verification is the TriggerDB product 
of Alexei Varganov.  He may be able to help you create (or 
otherwise) the L1/L2 download tables for this physics table.
</pre>
<!-- Date: Sat Sep 30 16:18:56 2000 -->
<font color=red>Shift Summary: </font><pre>Muon system (CMU,CMP,CMX) now has HV on. Setting up to run trigger 
 table to include muons in the trigger. 
 
One SVX trip due to Tevatron abort. Negligible rad exposure.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   thru 10028
Delivered Luminosity   0  pb-1
Acquired Luminosity    0  pb-1
Efficiency             
   
</td></tr></table></pre>
<!-- Author: Bill Carithers -->