<!-- Title: Eve shift. Mon Oct 16, 2000 -->
<!-- Leader: Henry Frisch -->
<!-- Ace: Nancy Lai -->
<!-- Co1: Konstantin Anikeev -->
<!-- Co2: Elena Vataga -->
<!-- Co3: Stan Thomson (SOS1) -->
<!-- Co4: Slawek Tkaczyk (SOS2) -->
<!-- Co5: &nbsp; -->
<!-- Notes: -->
<pre>Frank and Ilya are investigating garbled COT data problem.</pre>


<!-- Date: Mon Oct 16 16:44:10 2000 -->
<pre>Had problems starting the run after downloading the VRB fram. 
 
Had to change the dip switch to allow downloading. 
 
Now the west side has version  0A16.HEX</pre>
<!-- Author: Frank Chlebana -->


<!-- Date: Mon Oct 16 16:47:02 2000 -->
16:45 George Ott pag
<!-- Author: {Quick_Entry} -->


<!-- Date: Mon Oct 16 16:53:23 2000 -->
<pre>16:45 George Ott pages Steve Hahn about IMU 400 Hz for the IMU  
ASD crates; George sees no pilot lights on the crates with the camera. He sees no levels on the ECL differential outputs to the TDC inputs. They may have had a shunt trip during the flammable gas or power glitches earlier; Steve will come back and check.</pre>
<!-- Author: Henry  :: (run 00000000) -->


<!-- Date: Mon Oct 16 17:15:18 2000 -->
<pre>Running with only the east side COT and seeing problems again... 
 
MLE) b0l3gate1.fnal.gov:main:5:03:40 PM->Host b0eb14.fnal.gov, task tRec_0 
SCPU-P0-E-BadVrbByteCount: Event 373 total byte count is lt 32, gt 65528, or not divisible by 8 for VRB in slot 10. 
Count = 262580. 
(MLE) b0l3gate1.fnal.gov:main:5:03:40 PM->Host b0eb14.fnal.gov, task tRec_0 
SCPU-P0-E-VrbHeader: Dump of header words for event 373 from VRB in slot 10: 
0x00340074 0x00d401b4 0x00200060 0x00d401b4 0x0188015c 0x050c0000 0x00000198 0x03080000 
 
 
Earlier when we ran I noticed that the HV status was less than  
20 percent. Now it is at 100 percent (shown as a red band...). 
 
Giving up for now... 
 
Both VRBs have the same version of the firmware (0A16.hex). 
 
Should check the firmware version of the other VRB crates  
and try running with consistant versions...  
</pre>
<!-- Author: Frank Chlebana :: (run 101577) -->


<!-- Date: Mon Oct 16 17:22:20 2000 -->
<pre>Hyunsoo reports that CMU West HV is off. We don't see any instructions to the contrary, so he will turn it on. Steve Hahn is here investigating why the IMU ASD crate 400 Hz seems off.</pre>
<!-- Author: henry :: (run 000000) -->


<!-- Date: Mon Oct 16 17:28:03 2000 -->
<pre>Hyunsoo has run up the CMU West HV successfully. </pre>
<!-- Author: Henry :: (run 101577) -->


<!-- Date: Mon Oct 16 18:36:58 2000 -->
<pre>Running with all cal crates, l1 and l2 cal crates, and some cot 
crates: cot10 - cot14, cot17, and cot18.  2_Jet_2_test.  This  
run had a busy condition and a done timeout from b0pcal04.  This 
can be recovered with a HRR, and it runs for several hundred 
events, but then stops with the same busy and done timeout  
conditions from b0pcal04.  This also happens after repeated HRR 
cycles.  b0pcal04 is rebooted.</pre>
<!-- Author: Nancy :: (run 101579) -->
<!-- Comment: Mon Oct 16 20;08;12 comment by...Frank Chlebana -->
Error messages are seen in the error log file coming from 
b0eb14. This is the East side of the TDCs.

Data is not read out of the VRBs after this error and the data coming from PCAL04 fills the TRACER fifo.

This leads to the busy timeout... 

The problem is not with PCAL04 but the TRACER/VRB or ?? 
connection associated with the COT crates.

 


<!-- Date: Mon Oct 16 18:47:12 2000 -->
<pre>Put in the svtmon at the request of Subir and Marco.  They would 
like to test the svt monitor.  This run partitions and configures 
properly, but just get really low rate and a lot of messages  
like: 
 
----- 16 Oct 2000 17:30:54 CDT ----- b0eb10.fnal.gov ----- tsmDisp 
SM-P0-W-Loitering: Event 134 has had a load pending for a long time. Pending SCPU mask = 0x 
00000fff. 
 
Ending run and trying again.....</pre>
<!-- Author: Nancy :: (run 101580) -->


<!-- Date: Mon Oct 16 18:50:53 2000 -->
<pre>Jean Slaughter and Steve Nahn say that 'SVX is out of  
integration'; they have left it running in its own  
partition and will come back at 20:00. 
</pre>
<!-- Author: Henry :: (run 101577) -->


<!-- Date: Mon Oct 16 18:52:28 2000 -->
<pre>MCR calls- they have lots of SVX alarms on their ACNET 
page; we check, and it's true- lots of alarms are set. 
No alarms have been heard, however. We page Steve Blusk.</pre>
<!-- Author: Henry :: (run 101577) -->


<!-- Date: Mon Oct 16 18:54:31 2000 -->
<pre>This run ends with a done timeout and busy condition from 
b0pcal04 again.  There was also no new events for a space of a  
couple of minutes. The message on the green screen of death  
is written in the note below:</pre>
<!-- Author: Nancy :: (run 101581) -->
<!-- Comment: Mon Oct 16 18;55;18 comment by...Nancy -->
----- 16 Oct 2000 18:53:33 CDT ----- b0eb14.fnal.gov ----- tRec_0
SCPU-P0-E-BufferOverflow: Data for event 13 from VRB in slot 10 would not fit in remaining buffer space.
 Total buffer size = 51200 bytes, remaining buffer space = 51176 bytes.

----- 16 Oct 2000 18:53:33 CDT ----- b0eb14.fnal.gov ----- tRec_0
SCPU-P0-E-VrbHeader: Dump of header words for event 13 from VRB in slot 10:
0x000c000c 0x00d401f8 0x01dc0160 0x04400000 0x00000190 0x01cc0000 0x00000035 0x000d003b

----- 16 Oct 2000 18:55:32 CDT ----- b0eb10.fnal.gov ----- tsmDisp
SM-P0-W-Loitering: Event 13 has had a load pending for a long time. Pending SCPU mask = 0x00000010.


<!-- Comment: Mon Oct 16 18;56;31 comment by...Nancy -->
Ended with 34 events.


<!-- Comment: Mon Oct 16 20;09;47 comment by...Frank Chlebana -->
<pre>
Same problem as before...

COT East side.
</pre>
<!-- Comment: Mon Oct 16 21;41;11 comment by...patrick -->
The first two 32 bit words of the event,
which contain the total byte count, somehow
disappeared when they were read. The next
two words (0x000c000c/0x00d401f8) were then
interpreted as the total word count.
<!-- Date: Mon Oct 16 18:59:53 2000 -->
<pre>Stan Thompson (SOS) notices that the rack monitoring has turned red for 2RR29C, the crate fanout rack next to the L1 trigger racks. It's tagged by Peter W. to be off, so it looks ok; but 
in investigating Bill Ashmanskas notices that the fan failure  
light in 2R30C bottom is on- fans are turning- needs to be  
looked at though. </pre>
<!-- Author: Henry :: (run 101582) -->
<!-- Comment: Mon Oct 16 21;49;12 comment by...Ashmanskas -->
I think the lockout actually applied to a power supply
unrelated to the clock fanout crate, and that the clock
fanout crate is not intended to be locked out.  Perhaps
lockout tags should be dated?  This crate was definitely
powered on until some time early Monday (else SVT would
have had no clocks).
<!-- Date: Mon Oct 16 19:01:34 2000 -->
<pre>evb failed.  Will try again.</pre>
<!-- Author: Nancy :: (run 101582) -->


<!-- Date: Mon Oct 16 19:14:16 2000 -->
<pre>evb again failed on activate.  Checking EVB proxy and L3 proxy 
for good measure.  Both OK.  When a check of the Scanner Manager 
is done, it says "Server not found!"  evb paged..... Ivan tells 
me that the first time that the Scanner Manager check is  
performed after a hardware evb clean-up, it comes up as  
"Server not found!".... subsequent checks are OK.  Ivan is  
investigating why evb is failing.</pre>
<!-- Author: Nancy :: (run 101583) -->
<!-- Comment: Mon Oct 16 19;27;21 comment by...Nancy -->
b0eb13 has been reset.
<!-- Date: Mon Oct 16 19:21:27 2000 -->
<pre>On SVX  ACNET alarms: Paul Derwent responded to a page; 
taught us how to go to page E48 and turn alarms off 
(have to go to utilities page and give oneself permission 
to change things). No beam in machine- alarms probably due 
to power glitch etc. Channel 13 doesn't work- they're testing 
a `new' Channel 13 in MCR. MCR has linac going- getting ready 
to start ramp.</pre>
<!-- Author: Henry :: (run 101583) -->



<!-- Date: Mon Oct 16 19:43:38 2000 -->
<pre>After cleaning up the evb, we try again.  We get the following 
error: 
 
(MLE) b0l3gate1.fnal.gov:main:7:37:43 PM->Host b0eb14.fnal.gov, task tRec_0 
SCPU-P0-E-BadVrbByteCount: Event 8 total byte count is lt 32, gt 65528, or not divisible by 8 for VRB in slot 10. 
Count = 262580. 
 
It is the error that was seen by Frank and Ilya this morning. 
Apparently it has come back again.</pre>
<!-- Author: Nancy :: (run 101584) -->
<!-- Comment: Mon Oct 16 20;10;58 comment by...Frank Chlebana -->
and again and again...
<!-- Date: Mon Oct 16 20:02:28 2000 -->
<pre>Take out cot crates because they are not playing nice 
with the evb.  The goals right now are to take this run  
with the cal crates for Rick and to test the svtmon for  
Subir and Marcos.  </pre>
<!-- Author: Nancy :: (run 101586) -->


<!-- Date: Mon Oct 16 20:03:29 2000 -->
<pre>Nancy ends run 101586, a run for Rick Tesarek to look 
at single pm spikes, with about 57K events.  
There is no COTD bank due to dropping COT readout.</pre>
<!-- Author: Henry :: (run 101586 ) -->





<!-- Date: Mon Oct 16 20:11:32 2000 -->
<pre>Added svtmon.  It runs to over 15K events and going with other 
crates, no problem.  Next: try adding an svt crate.</pre>
<!-- Author: Nancy & Marcos :: (run 101587) -->


<!-- Date: Mon Oct 16 20:22:44 2000 -->
<pre>Run 101588 was using svt00, but the crate wasn't too happy, so 
in Run 101589, we use svt01 instead.  The run proceeds until 
there is a done timeout error from svt01.  Marcos says he thinks 
he knows what the problem is and will look at it further.</pre>
<!-- Author: Nancy & Marcos :: (run 101589) -->
<!-- Comment: Mon Oct 16 21;45;38 comment by...Ashmanskas -->
All of the SVT crates were supposed to be reserved for exclusive
use by Luciano Ristori and Alexei Semenov, who are running an
overnight test.  I guess this is a miscommunication.

<!-- Date: Mon Oct 16 20:48:05 2000 -->
<pre>tsi00 failed on activate (state 2).  Try again.</pre>
<!-- Author: Nancy :: (run 101590) -->


<!-- Date: Mon Oct 16 21:11:46 2000 -->
<pre>Run 101591 has cal, l1cal, l2cal, all muon, all xft, and xtrp crates in it .... is running happily now.  (Jean S. and company 
are working with the svx crates, and the svt crates depend on 
stuff from the svx, so that's why I haven't put the svx 
or svt in the run.)  Mako wants to reboot 
b0dau32 at some point, so we may be stopping the run in a few  
minutes in order to do this.</pre>
<!-- Author: Nancy :: (run 101591) -->


<!-- Date: Mon Oct 16 21:16:18 2000 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2000&action=view&page=-379&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2000&action=view&page=-379  width='250' align='top'></td><td>Just think this is not a very usual picture. People from respective project, I guess, might find it interesting/amusing</a></td></tr></table>
<!-- Author: K. Anikeev -->


<!-- Date: Mon Oct 16 21:34:58 2000 -->
<pre>Run 101591 ends after 135K events with a busy condition reported 
by xft00, 02, and 04.  This was taken with 2_jet_2_test,  
normal mode.  Will move on to cmu_low_pt_dphi_120 next.</pre>
<!-- Author: Nancy :: (run 101591) -->
<!-- Comment: Mon Oct 16 22;09;14 comment by...patrick -->
<pre>The root cause here was problems with XFT data
as seen by the event builder:

SCPU-P0-E-TracerEventId: Event 134930, crate 0, channel 1 has Tracer event ID of 0 instead of 2.
Tracer word = 0x04000000 (I think).
Channel data = 0x00000003 0x00000002 0x00000004 0x0000001f 0x0000003a 0x00000796 0x900007c0 0xffff07c0 ...
... 0x06000000 0x04000000 0x04000000 0x04000000 0x06000000 0x04000000 0x04000000 0x04000000
(MLE) b0l3gate1.fnal.gov:main:9:25:52 PM->Host b0eb12.fnal.gov, task tRec_0
SCPU-P0-E-TracerEventId: Event 134930, crate 0, channel 3 has Tracer event ID of 0 instead of 2.
Tracer word = 0x00000000 (I think).
Channel data = 0x00000003 0x00000008 0x0000000a 0x00000046 0x00000082 0x000000be 0x000000fa 0x00000136 ...
... 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
(MLE) b0l3gate1.fnal.gov:main:9:25:52 PM->Host b0eb12.fnal.gov, task tRec_0
SCPU-P0-E-TracerEventId: Event 134930, crate 0, channel 5 has Tracer event ID of 0 instead of 2.
Tracer word = 0x40004000 (I think).
Channel data = 0x00000003 0x00000008 0x0000000a 0x00000046 0x00000082 0x000000be 0x000000fa 0x00000136 ...
... 0x40006000 0x40004000 0x40006000 0x40004000 0x40006000 0x40004000 0x40006000 0x40004000
(MLE) b0l3gate1.fnal.gov:main:9:25:52 PM->Host b0eb12.fnal.gov, task tRec_0
SCPU-P0-E-VrbHeader: Dump of header words for event 134930 from VRB in slot 10:
0x00000000 0x00004970 0x00000011 0x00110d18 0x015c0d18 0x081c0d18 0x081c0000 0x081c0000
CT: 10/16/2000, 21:24'9"

In the 8 header words at the bottom, the upper 16 bits of the
third word are spurious. The rest of the header is shifted
right by 2 bytes, and thus gets misinterpreted
and leads to the preceding errors.</pre>


<!-- Date: Mon Oct 16 22:41:35 2000 -->
<pre>Try running CMU_LOW_PT_DPHI_120 (6,46) on the advice of Kirsten. 
However, we get the error when trying to partition:  
 
Client "LEVEL1_CAL_00" reports this error: 
"Failed to construct configuration message [Setup]" Has  
requested the following transition be performed: 
"Check configuration and database contents" 
 
This has already happened twice.....Runs 101593 and 101594.</pre>
<!-- Author: Nancy :: (run 101594) -->
<!-- Comment: Mon Oct 16 22;43;55 comment by...Nancy -->
b0l1cl00 looks fine on daqmon....


<!-- Comment: Tue Oct 17 08;29;03 comment by...W.Badgett -->
<pre>You are using the Dirac crates, but the Trigger Database 
is not providing the Dirac configuration information.  Perhaps 
triggerDb should provide some default values in this 
case.</pre>
<!-- Date: Mon Oct 16 22:52:00 2000 -->
<pre>Mike Church called from MCR- beam in machine in 5-10 min (est).</pre>
<!-- Author: Henry :: (run 101595) -->


<!-- Date: Mon Oct 16 22:52:07 2000 -->
<pre>Try CMU_LOW_PT_DPHI_120 (prescaled by 10) (6,56) now:  
 
Get "ERROR: bad L3TAGSET specification! (appears 9 times) in 
the terminal window displaying the number of acknowledgments 
received. 
 
Also get the same error: 
Client "LEVEL1_CAL_00" reports this error:...... 
message as before. 
 
We are about to get beam, so we'll move on for now..... Kirsten 
wanted us to test (6,56) and (6,46), so hopefully she has the 
answer now...</pre>
<!-- Author: Nancy :: (run 101595) -->


<!-- Date: Mon Oct 16 23:06:48 2000 -->
<pre>  
 Around 22:35 I turned on HV in wedge number 21 West  
 side of CMX detector. It tripped and crew shift could  
 not bring it to nominal voltage, it was in standby mode.</pre>
<!-- Author: Abraham Gallas -->


<!-- Date: Mon Oct 16 23:23:53 2000 -->
<pre>Trying to run with 2_JET_2_TEST (1, 47) with the cot crates 
1-3, 6-14, and 16-19 (these were the crates that Eiko used in 
the owl shift run earlier this morning).  Got 2 events, and then 
there was a done timeout from cot19.  A little time later, it 
decided that there was also a busy condition from cot19 also. 
The messages from the SCPU look like the ones listed in the note 
below: 
</pre>
<!-- Author: Nancy :: (run 101596) -->
<!-- Comment: Mon Oct 16 23;26;52 comment by...Nancy -->
----- 16 Oct 2000 23:16:39 CDT ----- b0eb13.fnal.gov ----- tRec_0
SCPU-P0-E-TracerEventId: Event 1, crate 241, channel 6 has Tracer event ID of 8 instead of 1.
Tracer word = 0xc0a82ef1 (I think).
Channel data = 0xc0a82ef1 0x404870e0 0x006ee2e2 0x00000000 0x0000007f 0x00010038 0x434f5444 0x00140008 ...
... 0xaaaaaaaa 0x55555555 0x00010034 0x80000000 0x00000000 0xc0a82ef1 0x404870e0 0x006ee2e2

----- 16 Oct 2000 23:16:39 CDT ----- b0eb13.fnal.gov ----- tRec_0
SCPU-P0-E-VrbHeader: Dump of header words for event 1 from VRB in slot 10:
0x00000000 0x00000c20 0x00000000 0x0000057c 0x00f400f4 0x00000000 0x000c01fc 0x01bc00bc

----- 16 Oct 2000 23:16:39 CDT ----- b0eb10.fnal.gov ----- tsmDisp
SM-P0-I-Suspending: Due to errors, data-taking in this partition is suspended until the next Recover transition. 



<!-- Comment: Mon Oct 16 23;28;23 comment by...Nancy -->
These are basically the same "event byte size dropped" errors
documented before and pointed out.

<!-- Comment: Mon Oct 16 23;39;52 comment by...Nancy -->
HRR doesn't work, of course, with problems like these.... get the
same symptoms.
<!-- Date: Mon Oct 16 23:27:44 2000 -->
<pre>MCR had 2 inhibits from us - SVX4MR and SVX6MR were inhibited, and we couldn't reset them. 
Called Paul Derwent, who led us through resetting them; 
had to up the threshold to 100,000 Hz on each before they  
would reset. Paul says these are no longer in the abort 
system, but still can give an inhibit. (We also had to 
add SVX4RM to page E48). Paul thinks the default settings 
were not downloaded correctly following the power 
glitch. Called MCR and told them to go ahead.</pre>
<!-- Author: Henry :: (run 101596) -->




<!-- Date: Tue Oct 17 00:01:12 2000 -->
<pre>Removed cot19 and try running with the remaining cot crates.... 
get no new events after 17 events. 
 
Mako wants to reboot b0dau32 (thanks for being patient - I had 
gotten caught up with other things....) so now would be a good 
time as we pass the baton along to the next shift.</pre><font color=red><cite>(Entry started on Tue Oct 17 00:01:10 2000)</cite></font>
<!-- Author: Nancy -->
<!-- Comment: Tue Oct 17 00;34;25 comment by...mako -->
<pre>Actually I did not need to reboot b0dau32; I stopped the
CS/L, dismounted the logger disks, and power cycled the
Ciprico FibreSTORE RAID system to get the Controller B
(the one on the FCC side) back to live. I then remounted
disk1 and disk3 and restarted the CS/L. I will try to
bring up disk2 and disk4 during the owl shift.</pre>
<!-- Date: Tue Oct 17 00:15:02 2000 -->
<font color=red>Shift Summary: </font><pre>Swing Shift, Monday Oct 16, 2000 
 
 
16:45 George Ott reports there are no ECL levels coming up from downstairs 
from the IMU; Looking with the cameras he also doesn't see any status lights  
on the crates in the collision hall. Steve Hahn comes in and investigates. 
 
16:47 Add SVTMON to run at Subir's and Marco's request. Testing to understand 
possible problems when run with Hardware EVB. 
 
17:28 Hyunsoo asks if we know why the HV is off on CMU West; we don't, so he 
turns it on. 
 
17:21 Reset SVX ACNET rad alarms- not due to beam. We reset it following  
instructions from Paul Derwent. 
 
21:00 Three requests from the SciCo: 
        1. can we get a telephone for the SciCo desk? 
        2. can I help get the TV channels for the stacking etc? 
        3. Can somebody please get rid of this silly Netscape bookmark message? 
 
21:00 Ran 2-JET_2_TEST and CMU_LOW_PT_DPHI_120 tables. Used the 
TAG=6 and Kirsten's new 56 L3 table- had errors- see Elag. 
 
23:00 MCR calls to tell us they can't put beam in due to 2 inhibits from us- 
SVX4MR and SVX6MR. Paul Derwent leads us through resetting them (see Elog). 
 
23:55 Situation is that COT19 is still giving troubles- Eric James looked at 
it, but it's still a problem. Other COT crates seem to be giving problems 
too- last run hung after 17 events. Protons in Tevatron on and off. 
We hand it to Owl shift. 
  
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   101577-101597
Delivered Luminosity   0  pb-1
Acquired Luminosity    0  pb-1
Efficiency             100%
   
</td></tr></table></pre>
<!-- Author: Henry -->