<!-- Title: 08:06:13  Wed Dec 26 2001 -->
<!-- Crewchief: L2  -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
l2


<!-- Date: Wed Dec 26 08:11:10 2001 -->
This morning shift crew started having troubles with L2.  Of course, the CO didn't notice the blatantly wrong MB word plot (<a href= 
'/cgi/elog/elog.pl?nb=2001&action=view&page=939&frame=2&anchor=07:59:59'> 
See shift log for rude comments and picture </a>.  The ACE didn't call the pager.  It appears that run 135684 (shown above in pic) went horribly bad.  The ace ended run and reset_crate.  Runs 135685, 686, 687 had many Decision TimeOuts again, but the MB plots look okay.  The only slightly funny thing I see is that the SVTlist, CLIST, ISOList all show the highest number of words sent to be 17-18 on the y axis.  I'm minicomed into the alpha now, and I don't see any error messages being printed when the ace is getting the TO and HRR-ing.  
<!-- Author: hray -->
<!-- Comment: Wed Dec 26 08;44;59 comment by...hray -->
From this latest run, 135688, there were again many dto from L2.  the vxlogin shows an error I haven't seen before: 
	b0l2de00-> ===> transfer size too big (TL2D) = 1109/1050
TRACER_enableDone: Front End Fatal Error (-1): Tracer done not set
Internal buffer too small.  The magicbus TrigMon plot again shows 
too many words for L1 interface and Muon board.  When I try to grep on "transfer" in L2run, alpha, or l2code directories I cannot find this message.

Tried powercycling the crate and Tracer DONE light stays lit as soon as I come up.  


<!-- Comment: Sun Dec 30 17;00;08 comment by...Arnd -->
<pre>For future use: you'd have to grep in $FER_DIR/src for 'transfer'
to find the error message.
</pre>
<!-- Date: Thu Dec 27 04:49:14 2001 -->
<ul> 
Many many things were tried yesterday.  I am going to try to list them in as orderly a fashion as possible.  Some things may not be chronological. 
<li>Run 135668 Cosmics_NoTracks[3,233] run on evening shift on Tuesday 12/25 with L2DE00 in.  No problems with the crate.  This was right before the last shot was dropped 
<li>Calibrations were done on Evening and Owl shift into Wednesday.  Run 135683 taken with Physics_0_03[1,238] during scraping of new shot, Owl shift. 
<li>Run 135684 is where things started to go bad with this crate.  Run was taken with above Physics table.  Here the Ace saw two different L2 errors: DTO and Internal Buffer too small (from Arnd's code, not from PWitt's code).  This run is the one mentioned above, where the MB plot showed words for the muon card and >1 for the l1 interface.  So at this point something was overwritting several memory sections in the alpha - possible collisions occuring. 
<li>A few weeks ago Arnd assured me that we will never see another DTO from the L2 crate, only a Decision TO.  He said if the Alpha is in a bad state the Decision TO which has a shorter timeout period will be pulled before we'll ever be able to get to Done TO.  Now, however we're seeing Done TO in the error handler. 
<li>The aces ended this run and did a reset_crate.  The next few runs, 135685, 135686, 135687 had DTO's for L2DE00, but the TrigMon MB plot appears normal. 
<li>At this point, ~7:30 am, I had come in and found these errors in the elog and TrigMon (I was never paged or informed by the shift crew there was a potential problem) 
<li>Run 135688 I had the minicom open to the Alpha.  I could tell the Ace was doing HRR, but NO ERROR MESSAGES appeared in the minicom window from Peter's error handling code.  The RC error handler said there were Internal Buffer too small errors, and Done TO from RXPT.  The minicom into the crate CPU showed TRACER_enableDone: Front End Fatal Error (-1).  This was the only error to show up in either minicom window 
<li>End run to powercycle the crate.  This is the point where everything went horribly wrong.  First the Tracer DONE light stayed lit after powercycling.  Tried reseting the CPU a few times, this did not help.  Around this time the crate started to fail CS.  The CPU would say it was config-ing, but the Alpha would never receive any messages.   Other things that had started to happen, too.  The Alpha would now not appear to be booting on CS.  The Alpha wasn't loading l2autoconfig file.  Sometimes after a manual reset of the crate the alpha minicom showed the bootp failed for the l2autoconfig, sometimes the printout was in a string of nonsense characters. 
<li>At this point I also began to fail Partition sometimes, even after a reset of the crate.  The CPU appeared to hang in the partition state. I started to remove board by board to see if that would help.  I removed the SVTlist first and I was able to pass partition but not CS - hung again in config.   
<li> Steve Nahn was here and he told me Bill Badgett had installed a VME scan that is done for each crate before the config message at CS.  Steve did a FER_printMode and FER_printStatus command and decided that this was the point where the system was hanging. 
<li>We had the Tracer Done error and now VME problems so I tried swapping the Tracer. I put the Tracer in that had been in b0l2de01.  I did change cardEditor (ie DB) to the new serial number of this tracer.  The crate would still not run - it was still hanging in CS.  After the hang in CS the crate CPU/Alpha would not respond to Abort/Reset/Partition commands from RC.  Only a physical reset of the crate would get it out of this condition. This config and the one above with the original Tracer was tried with all 3 D version Alphas before I decided to go for broke and remove all boards from the crate but the Tracer and the CPU.  I made sure all boards and alphas were marked offline in the cardEditor.
<li>Around this time I had been talking to both Bill and Peter Wilson on the phone.  I checked the HV supplies on the back of the l2de00 crate, and all appeared healthy (I don't remember if I checked this before or after I removed all the boards from the crate.) = 3, 5, 12 V supplies.
<li>The crate CPU would still hang on Partition.  Sometimes all 4 lights would remain on on the CPU.  While talking to Peter he had me remove the Tracer.  The lights went  off in the CPU (good), but the CPU wouldn't pass partition after a reset, next time around.  You don't need the Tracer for this, only for CS.  It was decided I should change the CPU with the one in the l2de00 crate with help from Bill/Steve Nahn for editing the DB/etc, and Peter would come in.  This was around 11/noon.
<li>In the meantime, the scope was connected to the backplane to StartLoad, Boss, ModDone08, DStrobe.  Nothing exceptional was occuring when we were hanging in CS, but you shouldn't expect to see anything then.
<li>The CPU was changed and this did not help - the CPU was hanging in Partition still.  Powercycling and reseting of the crate a few times did not help.  No other boards were in the crate but the transition board in slot 21 on the back and the CPU in slot 1.  I had checked the front and backplane for b0l2de00 and found no bent pins or anything visually out of place.
<li>Bill suggested I try to assemble the bottom crate and try to run with that.  I put the CPU and Tracer that were originally in the lower crate back, and had 1 Alpha, XTRPlist, SVTlist, CLIST and L1Int in the crate.  I have a few complaints about how things are cabled in the crate, but I'll send an email out later.  This configuration, with the new MB, passed partition and CS fine and appeared to run but I was using L2DE01 standalone.  I tried to include the crate in with the rest of the system and I got DTO immediately on activate.  Frank C. had been paged by Bill and he called in.  He said the only things he could think of were powercycling the TXPT crate, or physically changing the spigot out of TXPT.  Peter arrived at this time (~12/1?)
<li>We tried different combinations of swapping the TXPT cables that go into the tracer, and changing the flat grey cable between the alpha and the GL2 board.  Neither of these helped.  While I was fiddling with this Peter was reassembling the top crate.  When this failed we decided to go back to the top crate to understand what was going wrong.
<li>This is where things get rather murky and I can only relate what we did, not the correct order of things:
<li>With the original CPU and a new Tracer from the spare cabinet we were able to partition and CS in the top crate.  When we tried to install the (any of the 3) alpha in slot 3 the CPU did not acknowledge it's existence.  We definately changed cardEditor.  <b>Every time we made a change yesterday with cardEditor the cardEditor screen said the change was put into the DB, and a refresh from HWDB in RC showed the change picked up, but during run time the CPU for L2DE00 didn't appear to receive this information.  The same was also true when we tried much later to put the Alpha in tagging/cutting mode</b>
<li>So we tried to put any Alpha in and it would not be seen at CS/with VISIONdemo.  Peter put a TDC in the crate in a slot higher than the alpha and that was seen in VISIONdemo.  We removed the Alpha from the crate and put the TDC in slot 3, to see if there was a problem with the VME in slot 3.  The TDC was seen.  We put the Alpha in slot 6 and they both were seen.  We CS and nothing happens to the Alpha - it should boot and load a program but it didn't do anything.  VISIONdemo now would not see slot 3 (TDC). We removed the Tracer and TDC/Alpha was seen again. We put the Tracer in slot 7 and the Alpha in 3 and Alpha was seen.
<li>Through all of this we tried all 3 of the different blue ethernet cables for the alpha, too.  The one labeled 6 was working and downloading the executable so we know for sure this one is okay as of late last night.  We also found that changing the grey cable between Alpha and GL2 from the one labeled 00 to 02 (ie changing spigot) helped things along.
<li>We were able to run with L1Int and CLIST in the crate, so we tried to put the crate back into the run.  More Problems!  The Alpha would hang - it would get 4 events then do nothing.  We connected the scope to the tracer on l2de00 and l2cl00 and saw l2de00 had a healthy L1A signal coming in.  Backplane showed StartLoad had been asserted but no boards took BOSS.  The L1Int board wasn't flashing the red lights.  Reseating the CLIST/L1 board didn't help.  We tried changing to several different tables, with and without ReCES readout, and with and without PWitt's error handling code.  We tried unplugging a cable on the l2cl crates to force clusters but CLIST still didn't appear to be sending anything (L1 was still not flashing).  All cables between CLIST and the L2CL crates were attached. 
<li>We did get to the point where the Alpha was rejecting everything and the minicom printout said it was in tagging mode enen though we had never changed this parameter and the RC L2AA variable was set to 0 - to cut. 
<li>So where we left off was the crate partitioning and CS but something odd was happening in the processor - it was either rejecting all events or not doing anything it appeared.  We tried switching Alphas at this time, too but that didn't help.  
<li>We also tried reseting the TSI00 and TXPT crates.
<br><br>
<li>?'s: If the problem was Tracer and now that's fixed, is it possible that something is wrong with the GL2 board?  (and not just the grey cables)?  Would there be any way to narrow this down?
<li>If the problem is flakey VME in the l2de00 crate, why wouldn't the bottom crate run during activate?
<li>If the problem is flakey VME in the top crate what else could we(I) do to verify this 100%??
</ul>
<!-- Author: hray -->


<!-- Date: Thu Dec 27 12:05:12 2001 -->
<ul> 
Progress so for the past 6 hours so far: 
<li>power up l2de00, run with cpu, tracer in slot 7, alpha in slot 3, term/breakout and reces in back. alpha has tsi02 cable (which is spigot 3 in hwdb) and ethernet cable 06. Running with NO L1Int for now, just in calib continuous. ALL boards not mentioned above are marked OFFLINE in the DB using cardEditor. 
<li>I did double check this morning - there are no bent pins on the l2de00 crate and the jumpers are all in place.  
<li>Run with L2_TORTURE[3,232].  This exe is actually copied last night from one a few months old that doesn't readout reces.  Alpha in cutting rejects all events.  in tagging works to the extent that events are all accepted/passed through L2. 
<li>Disturbing thing present the entire time: <b>The Rates and Deadtimes Gui doesn't display any trigger table name, just [0,0].  The L1 pages show rates for triggers but no trigger names.  The L2 pages show nothing - no trigger names, no raw rate, no prescaled rate</b> 
<li>Run with TEST_ELECTRON_D_v-6.  I know there is no reces readout in this table.  Now we see the error - Number of L1 words sent = 0. 
<li>Run with COMSICS_NOTRACKS[3,232] in tagging - passes events through L2 but see no backplane activity other than sload being asserted. 
<li> run in cutting, get ClistMon fatal error: No Clist data. 
<li>put L1 Interface board into l2de00 crate now 
<li>Run with L1Int board in.  Get all CCAL/L1CAL/L2CAL/L1GL/L2DE00/TS crates.  Run with SEVB, L2_TORTURE[3,232] in tagging.  Pass all events through L2.  See SLOAD asserted, no DSTROBE or  BOSS or MODDONE.  L1 int board not flashing it's red light to indicate it sent anything on the MB.   
<li>Run with Physics_0_02[6,232].  Fails on activate with the VME errors we were seeing consistantly last night.  VISIONdemo maps tracer and alpha fine, but ~jsm/alpha/alpha_check_linux shows VME errors.  press reset on crate cpu to clear - only thing that works 
<li>L2_TORTURE[3,232].  okay, passes events.  Flip DIP switch on L1Int board to not accept Prefred info, just in case something is wrong with prefred.  Doesn't help.  Still no L1 info being sent on MB, no lights flashing. Check of alpha buffer bits show it is only requesting buffers 0 or 2 even though all 4 have 25% usage.  L1A rate ~10 Hz. 
<li>For these runs I had managed to track down Greg in Atlanta, GA.  He spend much time on the phone with me, checking everything he could think about the l1int/fred/prefred boards and cables. 
<br><br> 
<li>Move to bottom crate to see if problems from above persist with a different MB, CPU, Tracer, and Alpha.   
<li>Only use tracer, cpu, alpha, l1int.  connect tsi02 cable from top crate, change DB to spigot 3.  ethernet of 05.  mask everything offline in db but l1int.  Run with L2_TORTURE[3,232].  See exactly the same thing as above in top crate: passes events in tagging mode, all 4 buffers 25% full, ~10Hz L1A rate, no blinking on l1int board, BOSS never held.  again only see buffer 0 and 2 being requested by alpha. 
<li>try changing l1int board from sn1 to sn2, their spare.  this one holds boss high all the time.  change back 
<li>try Physics_0_02[4,224] (compile exe date 12/14, before reces readout made default).  alpha in tagging gives errors: L1nbus = 0.  Data is 0.  No events passed. 
<li>try another older table, Test4718_photon[1,226], compiled 11/21.  This table on activate sends the alpha into a stopped state.  The PS bits don't even get loaded.  This table has 6/7 triggers, all auto accept at L2.  It used to work.  Try [1,232] version (the l2_torture tagset for l3 that "worked" this am).  see the same thing - alpha immediately put into stopped state with activate 
<li>Try TEST_ELECTRON_D[6,225].  says error - Number of L1 words sent is 0.   
<li> I am in the process of trying to look at pins on l1int board to verify that buffers and strobes for valid data are coming in from the fred cables. 
<br> 
<li><b>SUMMARY SO FAR</b> It appears that regardless of the L2 exe date (code) or the L3 tagset, no trigger tables have transactions occuring on the MB.  This is present in both MB which leads me to believe it's either something wrong with the L1Int board (the only thing that is the same between the crates in terms of hardware aside from breakout board for alpha), the breakout board for the alpha on the backplane (highly unlikely), or something changed in the past two days from underneath us with the messages downloaded to the crate via RC - something with FER/DB/etc.  To have all trigger tables with all sorts of layers of l2 code not work in any of the three working version D alpha we have here in either crate is highly suspect and I think most definately does NOT point to a problem with either the MB or the Alpha or the CPU/Tracer.   
 
</ul>
<!-- Author: hray -->
<!-- Comment: Thu Dec 27 13;24;52 comment by...hray -->
I tried attaching a probe to pins and ended up removing a pin and pad from a chip on the SN1 L1Int board.  Steve Nahn took me to prep and we had it replaced.  The good news is now that I've put this board back in the 01 decision crate, the scope shows the Alpha is requesting ALL 4 BUFFERS.  The bad news is that means for some reason the L1 board never thinks it has good info for those buffers.  I am waiting for  Greg to be here in a few hours.

<!-- Comment: Thu Dec 27 15;24;57 comment by...Peter Wilson -->
<p>
Heather has made a rather complete summary of what happened yesterday.  One note about when the TDC was put into the crate.
At some point the could not be seen in the VISIONdemo crate map,  that appeared to have been caused by a VME reset or crate power cycle.  After doing a TDC FP reset the TDC was vivible in the map again.  
</p>

<p>
Note that we were definitely seeing the L1A signal coming from TS into the crate and the processor was correctly responding with StartLoad.  What is surprising is that neither the L1 interface nor CLIST was then asserting BOSS.  I would have though that if only the L1 interface was messed up then the CLIST would be still requesting the bus.  Is it possible that they have data bu for some reason think the bus is allready in use?  I am very puzzled.
</p>  



<!-- Date: Thu Dec 27 20:53:06 2001 -->
<ul> 
The final summary from ~4 pm - now. 
<li>I came in and continued to work with the bottom crate while BBadgett worked in the top crate.  I was seeing, with only L1Int in BOSSREQ held high, no BOSS, no flashing light.  when remove this board the BOSSREQ line goes low again, as it should. 
<li>Bill and I held a pow wow and decided to try the top crate again.  Bill moved the Tracer back to slot 2.  We put the L1Int board in the top crate and observed the same thing - BOSSREQ held high.  We were running with alpha ethernet 06 and TSI cable 02.  We found BOSSGRIN/OUT  bad between slots 8 and 9.  Changing the jumper on 8 fixed this.   
<li>Now, before anyone jumps to any conclusions, I would bet a kidney that this jumper was not in place when things started to go bad yesterday.  Jumpers were scrounged from all reaches of the earth when Peter and I were ripping apart the two decision crates yesterday.  The jumper that was bad was a black one.  There is a similar one in the bottom crate as well.  I've removed the offending jumpers and absconded with them. 
<li>So, changing the jumper caused everything to work - the l1 board now sending (red light flashing) on backplane, see healthy signals.   
<li>Ran with several different tables in tagging and cutting:L2_TORTURE[3,232], Physics_0_02[6,230], Physics_0_03[1].  During these runs we put CLIST in and changed the TSI cable to TSI00, and the spigot in HWDB to 2 to match. 
<li>The end result was when we added SVTList SN7 to slot 12 we started to get many DTO and Tracer Done Not set and Bunch crossing mismatch errors from the CPU minicom.  This is exactly what happened to the shift crew when things first started to go bad yesterday am.  Note: we DID NOT see any error messages from teh alpha checking code in either RC or the alpha minicom, and we were running with the PHYSICS_0_03 table (most recent exe that reads recess and has error handling).   
<li>Remove SVTlist and errors go away and shift crew is able to run. 
<br> 
<li>Dave calls and tells me to try one of the two spares.  Spare SN are 6 and 8 and Dave said 8 was the good one.  Problem:  neither of the spares have the xtrp jumper on (meaning they're both svtlist boards), but they have jumpers different from what is currently in the crate.  Specifically, SN8 was missing the SysCLK jumper that was in the crate and had DS jumper on 308 instead of 309 as in the crate.  I tried putting in SN8 with jumpers changed to match what was already in the crate and 0 events were getting through the L2 system.  This was with a COSMICS_NOTRACKS{3] table that had just worked for shift crew.  This table has 4/5 L2 triggers, all Auto Accept.  Removing the SVTlist board made the crate work again.  I put all jumpers back to original settings on spare board. 
<br><br> 
<b>LIST OF ALL CHANGES TO FINAL CONFIG IN L2DE00 CRATE</b> 
Tracer was changed.  A receiver chip on the L1Int board which handles one of the FRED Buffer No and STROBE (data is good) signals was replaced.  The pin that I broke off was the STROBE signal.  Even if this was "flakey" before I attacked it, I don't see how this receiver bit being bad could have had any affect on the l1 int board not sending at all, or screwing up the vme in the crate.  Changed jumper. 
<br> 
In short, there is NO REASON for this crate to have started working again.  Problems are still not understood.  It is possible that the SVTlist was sending a BC mismatch which started a cascade of events.  Why did they go away?  This is very very disturbing.   
</ul>
<!-- Author: hray -->


<!-- Date: Sun Dec 30 08:04:04 2001 -->
Yesterday am Matt and I put SVTlist into slot 12.  We saw several DTO's on the evening shift.  Two changes took place last night: arnd changed the number of TL2D words RC expected, and I moved SVTlist to slot 7.  There have been no DTO's since this happened.  I'm postulating it wasn't the change to the TL2D bank but the SVTlist board moving that made a difference and there was something buggy/flakey with slot 12. 
<!-- Author: hray -->



<!-- Date: Mon Dec 31 11:31:40 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-539&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-539  width='300' align='top'></td><td>Energy word bit 26 is not being set in L2.  This has been fairly  consistent through the last 2 stores (867 and 869).</a></td></tr></table>
<!-- Author: Matt -->
<!-- Comment: Mon Dec 31 14;05;46 comment by...Matt -->
This is with L1int SN01.


<!-- Date: Wed Jan 2 12:00:42 2002 -->
Moved SVTlist 08 back to slot 12.  Took physics data for approx. 1 hr, run 136043, with no errors.  Leaving the board there to make room for multiple alpha tests.  Shift crew is notified to keep an eye on L2 timeouts in the future.
<!-- Author: Matt -->


<!-- Date: Thu Jan 3 13:28:43 2002 -->
I just wanted to remind people that Jane is planning on 
removing the second SVTD board (with list of XTRP tracks) from the SVT crate next week, as discussed in the Dec. 7th review.  This board was there temporarily as an alternate copy of the XTRP readout.  Since the list in the TL2D bank has been stable for so long we will return this board its status as a hot spare.   The only impact I can imagine is if someone has code that reads the track list from that extra part of the SVTD bank.  If so, the code should be modified to take it from TL2D instead.   Ask Matt if you have any questions about the XTRPlist block in the TL2D bank.
<!-- Author: David -->


<!-- Date: Thu Jan 3 17:40:14 2002 -->
<pre>Several L2 tests were done today. 
 
1. Tested the L2Rate trigger. This works and is part of the  
DB and java code. 
 
2. Ran with 2 alphas for 10's of millions of L1A. The 2nd alpha 
had error checking and never reported any errors. There were 
2 cases where there was a L2 Decision timeout because the 
2 alphas were out of sync in such a way that the 2nd alpha  
didn't send a decision to the TS. Other than that things ran  
fine, although with a fixed rate using the BC trigger it  
is hard to say exactly how robust the code is for 2 alphas. 
I will leave 1 alpha as the default since old exes would  
need to be updated. 
 
3. I turned on the SVT bc error checking code and didn't get 
any errors. This is using fake track patterns from both xtrp 
and svt. 
</pre>
<!-- Author: Stephen -->


<!-- Date: Fri Jan 4 07:42:07 2002 -->
<ul>Stephen and I started to see some new features yesterday in the crate CPU minicom.  Prior to the L2DTO the following would show up:
<li>Sent ack to Run Control: 0 ACTIVE SUCCESS 
>>>>>Info: Alpha 0 is okay. Currently at spot 5: Read Fifo 
>>>>>interrupt: uninitialized vme interrupt 246 
>>>>>interrupt: uninitialized vme interrupt 134 
>>>>>interrupt: bad vme interrupt 0 
>>>>>interrupt: bad vme interrupt 0 
>>>>>interrupt: bad vme interrupt 0 
>>>>>interrupt: uninitialized vme interrupt 254 
>>>>>interrupt: uninitialized vme interrupt 20 
>>>>>interrupt: uninitialized vme interrupt 80 
>>>>>interrupt: uninitialized vme interrupt 140 
>>>>>interrupt: uninitialized vme interrupt 48 
>>>>>interrupt: bad vme interrupt 0 
>>>>>interrupt: bad vme interrupt 0 
>>>>>interrupt: bad vme interrupt 0 
<li>Arnd says this is probally from a board sending bad VME interrupts.  THis would mean CLIST, TrackList, or SVTList, (or the CPU or Tracer itself). 
<li>Last night I was paged because L2 was causing DTO from the start.  Problem was that the SVTlist and TrackList weren't seated properly.   
</ul> 
<li>
<!-- Author: hray -->


<!-- Date: Fri Jan 4 14:26:27 2002 -->
<pre>Notes from 1/2/02 L2 meeting 
 
1. It isn't exactly clear what caused all the problems with L2 
on Dec. 26th, 27th. Lessons learned include. 
 - don't do VME readout of SVTlist board in L2 crate (will happen 
       if the online flag is set in the HDWDB) 
 - don't use the little black jumpers for bossgrin/bossgrout 
 - if Reces is unplugged, then the alpha will hang in trying 
    to readout non-existent boards 
 
2. We need to make sure we have a full complement of spares 
on hand. 
  -L1: Greg will get Argonne to fix the current spare. Note that 
  the board currently in, has a bad sumet bit. 
  - Clist: Monica is working on getting a Clist spare functioning 
  - Track/Svt - Spares exist 
  - Iso - ??? 
 
3. In the L2 electron code, Masa found a bug which gives a 5%  
error. He fixed the error and will retest the code. 
   The photon triggers work. The difference between L2 clustering 
and L3 clusters (not extending across wedges) gives a higher 
rejection rate for diphoton and triphoton events at L2 than L3 
would normally give.   
 
4. Nate has a cron job which runs his validation code on the 
A stream data in the look area for every good run. 
 
5. We need to organize a pager schedule.  
</pre>
<!-- Author: Stephen -->


<!-- Date: Fri Jan 4 20:39:45 2002 -->
SVT is now configured to pull CDF_ERROR if any Merger board 
sees a BC mismatch.  This should be useful for understanding 
the occasional "empty SVTD" effect reported by Matt et al.  
<!-- Author: Bill A -->
<!-- Comment: Sat Jan  5 00;00;21 comment by...Bill -->
Hmmmm, no can do.  <b>If we pull CDF error, SVX trips off!!</b>
I did make two other small changes, though.  First, I eliminated
the "XTRP emulator," which was providing a BC check on the XTRP
input.  Second, I allow FIFO overflow errors to pull SVT error
in the Mergers (and hence turn on the yellow LEDs).  One time
when I happened to be looking, I saw a FIFO overflow in the
Merger that looks at XTRP and XTRPemu, but I lost the error
register contents that would have told me which input caused the
overflow.  Disabling the (unnecessary) XTRPemu input would test
the theory that it was the source.




<!-- Date: Sat Jan 5 12:38:10 2002 -->
I had to change a number of things last night just to get 
SVT running. 
<p> 
First, some background information. 
<p> 
Since late December, we have 
been using "wide" patterns, because we didn't know where the  
beam would be.  The wide patterns use both a larger beam spot 
and larger road sizes, so one gets more track combinations. 
<p> 
We also tried in December to loosen a few cuts that can reduce 
tracking efficiency: SVX clustering parameters, max clusters per 
SVX ladder, max roads per SVT wedge, max fit combinations per track. 
<p> 
We also have much more silicon now than before the shutdown. 
<p> 
Now some mysterious findings: 
<p> 
In late December, Matt W reported that occasionally the SVTlist 
board would receive no data from SVT.  Then Irene reported that 
~10% of events from a given run in late December had no EE word. 
<p> 
Since this smells like a lost-sync problem, I turned on CDF error 
for BC mismatch and FIFO overflow in the Merger boards.  After  
about 20 events, the Merger that merges the output from TF 0-3 
would detect FIFO overflow and BC mismatch.  Unfortunately, 
pulling CDF_error would then trip the silicon! 
<p> 
By elimination, I narrowed the source down to wedge 3.  I do not 
yet know, however, which board is the source of the problem.  I 
imagine svtspymon may shed some light on this.  I can also turn  
on SVT_error for FIFO overflow and BC mismatch in the other 
boards within the wedge and watch the LEDs. 
<p> 
When we simply used SVT_error, not CDF_error, to report this 
problem (i.e. kept running), TrigMon would report ~100% BC 
mismatch from SVT. 
<p> 
After disabling wedge 3, the FIFO overflow went away and the 
SVT internal BC mismatch errors went away, but TrigMon still 
reported a large rate of SVTD BC mismatches.  My only hunch 
here is that absurdly long tails in the Ntracks distribution may 
confuse the SVTlist boards. 
<p> 
I then took radical countermeasures: reduced max hits per  
superstrip in Hit Buffer from 8 to 4 for SVX layers; reduced 
max roads per wedge from 64 to 15; reduced max combinations 
per track from 4 to 2.  Reduced TF chi/2 cut from 3 to 2, which means cut starts having an effect around chisq=25 instead of 
chisq=49.  (Recall that we were losing beam data, 
so had to act quickly.)  With these changes, the overflows  
reported in the svtspymon Ntracks histogram became negligible, 
and the TrigMon BC mismatches vanished. 
<p> 
After these changes, I tried putting wedge 3 back in, but it 
still gave problems, so I took it back out.  I wonder if some 
board in wedge 3 is not properly respecting the hold signal? 
<p> 
I still see 100% overflow (above 300us) in the SVT processing 
time histogram.  Can this be real? 
<p> 
We need to be able to look at error registers from every board 
during a run.  We need a processing-time graph that is sensitive 
to long tails (maybe a log plot)?  We need some graphs that can 
track down long tails in data size, e.g. log2(words/event) for 
various spy buffers. 
<p> 
And we need to be able to stop the run when a fatal error, such 
as FIFO overflow or BC mismatch, is detected.  I guess our only 
option for now is to ask L2 to generate a decision timeout if 
either of those bits is set in the SVT EE word, or if the buffer 
number is not what is expected. 
<p> 
We urgently need to establish a stable operating point for SVT, 
to evaluate its timing, efficiency, etc., and then to make 
incremental changes. 
<p>  
<!-- Author: Bill -->


<!-- Date: Sat Jan 5 12:49:56 2002 -->
Meanwhile, here is some news about the effort to understand 
whether we understand what SVT is doing. 
<p> 
For some available files from runs 135965,135969, svtsim was able 
to match the complete SVT track list for ~90% of events.  The 
fraction of SVT tracks matched "perfectly" by svtsim was ~97%. 
Here, "perfectly" means that wedge, road number, hits, phi, and 
curvature match bit-for-bit; this definition will be refined over time.  I didn't notice many distinguishing features of the  
failures.  One frequent category is tracks found by SVT but not 
by svtsim, having poor chisq and zout=5; in fact this seems to  
be the dominant source of tracks found by SVT but not by svtsim. 
<p> 
  
<!-- Author: Bill -->


<!-- Date: Sat Jan 5 14:11:35 2002 -->
OK, after throttling the data flow to eliminate data-transmission problems, I see the following in runs 136280,136281: 
<p> 
98% of events match all tracks between svt and svtsim (where 
match means same road, same hits)<br> 
99.25% of svt tracks are matched by svtsim<br> 
99.25% of matches are "perfect" (crv,phi,csq exact)<br> 
of the 279/37753 svtsim tracks missed by svt,  
182 are in W0, 93 are in W1, and 4 are in W6 
<p> 
Note that crate b0svt00 (home of W0,W1) was the only place in October's 
data where I saw svt/svtsim disagreements that could not 
be attributed to known hardware or firmware problems. 
<p>  
<!-- Author: Bill -->


<!-- Date: Sun Jan 6 17:44:12 2002 -->
<u>SVTTrack tests from Run 136281</u><br> 
<br> 
I've looked at ~48K events from this run. (We had about  95K total before a beam hiccup seemed to cause a huge burst in  reformatter errors, at which point we ended the run.) &nbsp;I can't get at the other 50K events as I forgot to send all the files to the look area. Oops. &nbsp;We took the data in L2 tagging mode.<br> 
<br> 
The two track trigger has about 29K L1 prerequisites for these events,  and there were 1550 L2 accepts from this data. This agreed with my  offline checks, i.e., the L2 bit was never set when it shouldn't be and was always set when it should be.<br> 
<br> 
For the single track path, there were very few events with successful L1 bits set (176 of the same 48K events total). Of these, 12 passed the L2 requirements and again there was no disagreement with the offline checks.  
<br> 
<br> 
I also convinced myself that the values we had typed into the Java GUI  agreed with the values that the code was using to cut (i.e., the  translation of physical quantities such as GeV and micrometers to SVT  bins was good.) So I think all is ready to go. 
<!-- Author: Peter Wittich -->


<!-- Date: Tue Jan 8 21:25:53 2002 -->
With lots of help from Arnd, we now have a method for passing warnings from 
the alpha to run control. &nbsp;This means enabling other monitors in the 
alpha is now a lot less expensive - the runs won't stop, but the messages 
will go to the error handler. There is a prescaling mechanism in place to 
prevent the system from getting swamped. <br> 
<br> 
Please, now is the time to get your favorite monitors into the alpha code 
- so if you have any things you'd like to have added, please let me know.<br> 
<br> 
The monitors we currently have are all synchronization monitors:<br> 
<ul> 
  <li><font color="#ff0000">CList monitor</font>. &nbsp;Compares the buffer 
number in each cluster to the expected buffer number.</li> 
  <li>IsoList monitor. Similar to Clist Monitor.</li> 
  <li>SvtXtrpBc comparison. Compares the BC numbers in XTRP to that in SVT.</li> 
  <li><font color="#ff0000">L1Monitor</font>. Makes sure that at least one 
MBUS word is sent on every L1A.</li> 
</ul> 
Only the monitors in red are currently enabled, and they are all Fatal Errors 
- i.e., they halt the run. &nbsp;I would propose we consider turning on the 
other two at the warning level. Additionally, we should add monitors for 
other ``bad conditions'' such as<br> 
<ul> 
  <li>Internal Alpha buffers overrun. This happens, for instance, when the 
SVT is sending a lot of data to the alpha. Even with the recently expanded 
size for SVT tracks, we saw this &nbsp;happen on Friday. &nbsp;Currently, 
the alpha prints a message to minicom. We should add a monitor for this.</li> 
  <li>Checking various error bits that come from other systems.</li> 
  <li>MOD DONE not finishing when expected (i.e., STARTLOAD times out.)</li> 
</ul> 
Any other suggestions or comments?<br>
<!-- Author: Peter Wittich -->


<!-- Date: Tue Jan 8 23:30:05 2002 -->
We succeeded in loading the output of the 12 "wedge" Mergers 
(mrg_w00 to mrg_w11) from svtsim (636 events from Peter's run 136280 from  
Friday night) into the 36 Hit Finders' output spy buffers 
(because 36 HFs can hold more events than 12 HFs).  We then 
took a run replaying these hits through the 12 wedges in 
parallel, driven by the TS calib_continuous trigger. 
<p> 
As long as there are no pauses, no H/R/R's, and absolutely no 
deadtime, the sequence of BCs from the TS is predictable, and 
can be matched in the fake-event files, allowing a synchronicity 
check.  (With a modification, we can probably make the synch check 
verify internal consistenty without having to match the TS's BC, 
but for now this was a constraint.) 
<p> 
This should be useful for testing improvements to the monitoring 
software, cold-start code, etc., during Tevatron down days.  It 
could also be used to measure the post-clustering SVT timing if 
we vary the patterns, etc. to optimize trigger decision time. 
<p>  
<!-- Author: Bill & Marco -->


<!-- Date: Tue Jan 8 23:34:33 2002 -->
We implemented a work-around for SVT's once again being forbidden 
to pull CDF_ERROR:  We have the option to generate a Done Timeout 
on L2A if a BC mismatch has been detected by a Merger (currently 
mrg_tf_0 through mrg_tf_3) since the last H/R/R.  This is 
turned on/off at cold-start time via the hwset variable 
spy_tf_LostSyncDTO, so that we can use it differently when the 
silicon is in/out of the run.  
<!-- Author: Bill & Marco -->


<!-- Date: Tue Jan 8 23:46:29 2002 -->
Work plan from today's weekly SVT meeting: 
<p> 
<li>Alex will revisit the SVT efficiency study.<br> 
<li>Giovanni will make 1.2mm patterns centered at current beam spot.<br> 
<li>Marco & Bill will work to eliminate sources of data-flow corruption (e.g. lost sync, fifo overflow).<br> 
<li>Bill will work on diagnostic readout of hit data (to track down crate 0 discrepancies) and timing (to see how far over budget we are).<br> 
<li>Bill will teach svtsim to use the non-0123 constants that Giovanni has provided, so that more SVX wedges become usable.<br> 
<li>Bill will teach svtsim to initialize itself from database info.<br> 
<p>  
<!-- Author: Bill -->
<!-- Comment: Tue Jan  8 23;48;09 comment by...... -->
<pre>&lt;p&gt;
&lt;li&gt;Marco is tracking down SvtSpyMon-related crate CPU crashes.&lt;br&gt;
&lt;li&gt;Marco will try to implement freeze-on-error spy buffer
capture.&lt;br&gt;
</pre>


<!-- Date: Wed Jan 9 01:33:48 2002 -->
<table><tr><td><a href=/elog/notebooks/l2/Wed_Jan__9_2002_013329.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-541  align='top'></td><td>SVT timing (not including SVX readout or HF clustering) for Friday night data replayed through SVT, using wide patterns.  Ignore the overflows; they're from before I remembered to run gbinit.py to enable the timing data. Mean is 18usec, but we should probably add about 6-10usec to that for SVX + HF.</a></td></tr></table>
<!-- Author: Bill -->


<!-- Date: Thu Jan 10 15:01:54 2002 -->
Now that SVT is part of the trigger, 
Jimmy Proudfoot asked me what would happen if SVT 
made it impossible for CDF to run, e.g. by 
failing to send an event, by sending events that 
overflow buffers, or by finding so many displaced 
tracks that trigger rates got out of hand.  Ideally, 
one would be able to drop the SVT crates, and L2  
would know how to mask off inputs from sources that 
are not in its own partition.  In case this is not 
easy to do, one can start a run with SvtMode 
set to "hw=notracks", and SVT will send only end-event 
words, nothing else, to L2.  
<!-- Author: Bill -->


<!-- Date: Thu Jan 10 22:45:40 2002 -->
<ul> 
<li>Took several runs with the new CLIST.  Tried several different things - like putting the prom from SN1 (old, working) into new, etc.  The new CLIST (SN3) will not work here.  As soon as we activate there is the "Fatal Error no data from CLIST" error.  What the hell is going on between Michigan and here?? 
<li>On a side note, Steve K and I noticed for run 136717 the MBUS plot showed XTRPlist not sending data a good portion of the time.  This may be due to the cosmics situation/something known.  This run also had a lot of CLISTMon sync error in L2 pipeline in the RC error logger (which of course the ACE didn't notice -as usual) 
</ul>
<!-- Author: hray -->