<!-- Title: 23:25:25  Thu Dec 21 2006 -->
<!-- Crewchief: L2CAL -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
MEZZANINE CARD PROTOTYPE TESTING


<!-- Date: Thu Dec 21 23:28:59 2006 -->
Today we did the first mezzanine card tests. Started at UC with smoke tests and current measurements, the card draws 500mA. At FNAL then we were able to send data from the DIRAC board in the test stand and log it in the pulsar with one mezzanine card all the way to the control FPGA. We then moved to the trigger room and  set up the b0l2pu00 with one pulsar with one mezz card and one merger. Tevatron beam came in our way so we stopped testing. Tomorrow we should be able to determine the right delay to log the correct data and try some calibration runs.
<!-- Author: Laura, Lucas, Massimo, Gene, Vadim -->
<!-- Comment: Fri Dec 22 14;22;27 comment by...Ted  -->
<pre>sweet news!  as promised, a bottle of CHIMAY ALE GRANDE RESERVE is now in the teststand room...</pre>


<!-- Date: Fri Dec 22 14:34:00 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-370&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-370  width='300' align='top'></a></td><td>L2CAL input data timing</td></tr></table>
<!-- Author: Ted -->
<!-- Comment: Fri Dec 22 14;47;41 comment by...Laura and Ted -->
<pre>here is a trick to measure the L2CAL input data timing precisely(vs BC or L1A). This is a simple method Ted used 6 years ago to timing in the XFT for the first time. what's shown here are:

(top): Bunch Zero, from tracer front panel.

(middle): lower data bit of incoming data from the mezzanine FPGA. with accumulation, it is easy to see which one is the data from the very first bunch (one sees the beam structure clearly).
(bottom): L1A from tracer front panel. the very first one in the middle train is the L1A from the very first bunch.

so the latency from the input data arriving time (into mezzanine FPGA) to the final L1A is 3.61 us. this is the time we need to go back (after L1A) to capture the correct data slice in the pipeline.  the latency from BC to the input data arrival is 1.9 us. 

we took this scope capture and measurement around 2pm today during the official data taking. Laura is now updating the firmware with this timing information.

</pre>


<!-- Date: Fri Dec 22 14:51:51 2006 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-371&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-372' ></a><a href='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-373&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-374' ></a></center>
<!-- Author: Ted  -->
<!-- Comment: Fri Dec 22 14;54;42 comment by...Ted  -->
<pre>This bottle is for Vadim and his L2CAL team ...  </pre>


<!-- Date: Sat Dec 23 02:18:20 2006 -->
<pre>Random things (sh..) 
 
1) On DCAS connector 0 is at the top 
2) Same on DIRAC, connector 0 at the top 
3) </pre>
<!-- Author: v -->


<!-- Date: Wed Dec 27 12:07:00 2006 -->
<pre>HOWTO 
 
How to load a different delay between L1A and data in the dataio firmware to tune data 
 
1.All the scripts and the object file you need are in one of the two following directory: /cdf/onln/home/sartori/utils 
                        /cdf/onln/home/vrusu/pulsar/useful 
 
2. Load the object file on the crate: 
           ./vxtest nome_crate 
 
3. Write the register containing the delay 
          ./vxwritereg nome_crate slot fpga register_number delay 
   Register number is : 0x18 
   Fpga are: for dataio2 0x80000, for dataio1 0xC0000 
   The delay to be written is counted as 128 (number of locations of the memory - latency counted in 4xcdf_clk (33 ns)). So a delay of 3.61 us requires a writing of 128-110=18=0x12 
 
4. reset command to load the delay on the reading pointer register 
         ./vxwritereg nome_crate slot fpga Register Number value 
         the register number is 0x4 and the value is 0xfffff 
 
We are going to use this register today in trigger room running on cosmic. 
 
  
</pre>
<!-- Author: Laura & Vadim -->


<!-- Date: Wed Dec 27 14:00:53 2006 -->
<pre>Waiting for the end of the store......we are going to put online a Pulsar board with the following version of the firmware : 3.1.1 (08612271).  
Two mezzanines connected to dataio2 and 8 LVDS cables. More details if it works!</pre>
<!-- Author: Laura  -->


<!-- Date: Wed Dec 27 15:27:09 2006 -->
<pre>To close rogue partitions: 
 
Type this from the xterm:  
 
setup fer  
proxyViewer ResMgr  
 
Then, on the "ResMgr_Prd" line, press the "Stop" button.  
Wait 2 minutes and the partitions should clear up.</pre>
<!-- Author: Vadim (courtesy of BBadgett) -->


<!-- Date: Thu Dec 28 23:56:14 2006 -->
We finally were able to find the right delay today (0x17) and we took a long shot setup run so we had some activity through the channels that we have into the two mezzanines. Using some ghetto monitoring code that we put together in a pinch we found no errors in 579 events. 
 
So far so good. Will try an end of store tomorrow.
<!-- Author: Laura and Vadim -->
<!-- Comment: Tue Jan  2 11;35;58 comment by...Ted -->
nice progress. since the timing measurement was done at the mezzanine FPGA, the actual effective delay in Pulsar motherboard DataIO FPGA pipeline will have an offset. This offset could be determined by simple simulation at FPGA level, or be measured (as you have done effectively). cheers.
<!-- Date: Fri Dec 29 12:11:05 2006 -->
<pre>The test  Pulsar is in  
 
- crate b0l2pu00 
- slot 10 
- two mezzanines : mezzanine 0 and 1 (starting from the top of the pulsar) 
 
The cabes are labeled as follow, looking at the bottom side of the mezzanine (pulsar side) with the pulsar at the right side: 
 
 
 2  1    |  Pulsar side  
 4  3    | 
  
 
- mezzanine 0 , cable 1 : trigger tower phi=0, eta=23 and phi=0, eta=22 
 
- mezzanine 0, cable 2 : trigger tower phi=0, eta=21 and phi=0, eta=20 
 
-mezzanine 0, cable 3 : trigger tower phi=0, eta=19 and phi=0, eta=18 
 
-mezzanine 0, cable 4 : trigger tower phi=0, eta= 17 and phi=0, eta=16 
 
-mezzanine 1, cable 1 : trigger tower phi=11, eta= 1 and phi=11, eta=0 
 
-mezzanine 1, cable 2 : trigger tower phi = 9, eta=23 and phi=9, eta=22 
 
-mezzanine 1, cable 3 : trigger tower phi=1, eta=1 and phi=1, eta=0 
 
-mezzanine 1, cable 4 : trigger tower phi=3, eta=11 and phi=3, eta=10 
 
</pre>
<!-- Author: Laura  -->


<!-- Date: Fri Dec 29 15:41:43 2006 -->
<pre>RUN 231130, so far 340965 events with no errors!</pre>
<!-- Author: Laura & Massimo -->



<!-- Date: Wed Jan 3 14:15:06 2007 -->
<pre>Two additionals LVDS cable in the following position: 
-phi=4, eta=23,22 
-phi=0, eta=0,1 
</pre>
<!-- Author: Laura -->


<!-- Date: Wed Jan 3 14:21:41 2007 -->
<pre>The following firmware version has been load in the b0l2pu00 
 
dataio2: 08612281 (delay of 0x17 fixed in hardware, no DAQ2, just DAQ1) 
dataio1: 08612291 (a version of dataio with inputs fixed to 0, bacause the corresponding two mezzanines are missing) 
ctrl: 2411300 (the reces ctrl, without zero-suppression, and able to receive 32 data) 
 
In cardEditor we fixed the number of words for ctrl to 39. 
  
 </pre>
<!-- Author: Laura -->


<!-- Date: Wed Jan 3 17:24:07 2007 -->
<pre>run 231289 
223117 events with no errors 
 
readout code default 
firmware described before 
cables the same as the previous run 
 
</pre>
<!-- Author: Laura -->


<!-- Date: Fri Jan 5 11:02:47 2007 -->
<pre>We found errors in the previous runs, with the new version of the firmware. 
So we load again the previous version of the firmware. Basically only the delay register has been changed. 
 
Using the old configuration so far we have about 1200K events with no errors (run 231334)</pre>
<!-- Author: Laura -->


<!-- Date: Fri Jan 5 11:45:26 2007 -->
<pre>At the end of the store we want to test the first version of the readout code. (versione 1.0)  
Main features: 
- TP2D standard reading. we don't expect any problems and changing. 
- TP2D information ( only 2 mezzanines) is formatted as TC2D and added to the DCAS blocks as a new block (number 6) 
 
  </pre>
<!-- Author: Massimo & Laura -->
<!-- Comment: Mon Jan  8 12;15;46 comment by...Massimo & Laura -->
<pre>During this test we verified the new readout code doesn't affect the pulsar readout. (Previously we had problems trying to do that)

In the next step we want to verify the new TP2D information is formatted as TC2D bank without effecting then TC2D bank.</pre>
<!-- Date: Fri Jan 5 14:24:29 2007 -->
<pre>This is a crude way of running on recent data sent to tape but it does work at least for the moment. 
 
To run on recent data sent to tape: 
 
You can do the following 
 
in XXX.sh put  
--------------------------------------- 
 
source ~cdfsoft/cdf2.shrc 
setup cdfsoft2 development 
setup sam 
 
  
sam list files --dim="CDF.DATASET aphysr and RUN_NUMBER=231130" 
 
sam create dataset definition \ 
--defname="L2tryaphysr_12" \ 
--group=test --defdesc="L2 try aphysr" \ 
--dim="CDF.DATASET aphysr and RUN_NUMBER=231130"  
 
 sam take snapshot --defname="L2tryaphysr_10" --group=test 
 
----------------------------------------- 
 
then in your tcl file call the data in the following way: 
 
path enable AllPath 
module disable ConfigManager 
module input DHInput 
talk DHInput 
   cache set SAM 
  include dataset L2tryaphysr_12 
#  requireCatalog set false 
  show  
    maxFiles set 10 
#      sam 
#        station set cdf-caf 
#      exit 
exit 
 
 
mod enable TL2DAna 
mod talk TL2DAna 
exit 
 
 
 
begin -nev 100 
exit 
 
--------------------------------------------- 
 
if you get errors running this way that say there is no SAM_PROJECT defined you should be able to type the following  
 
 &gt;setenv SAM_PROJECT `whoami`.`date +%s` 
and try again. 
 
Note you may need to stop the project after you are done, to do this type 
 
&gt;sam stop project --project=$SAM_PROJECT --force  
 
There is no doubt better ways to do this but this should work in a pinch 
</pre>
<!-- Author: gene and laura -->



<!-- Date: Tue Jan 9 18:19:11 2007 -->
<pre>I leave our version of TrigMon running on the b0gateway, 
Release is set up in: 
/cdf/scratch/fedorko/tmon070105/ 
anyone should have permissions to run it from that directory. 
 
output is redirected to: 
/cdf/scratch/fedorko/tmon_log 
logs are in: 
/cdf/scratch/fedorko/results/ 
 
instructions for building and running online (and offline) are 
on our TWiki: 
http://www-cdf.fnal.gov/htbin/twiki/bin/view/Main/TrigMonInstructions 
 
By Wojtek 
</pre>
<!-- Author: Laura -->


<!-- Date: Tue Jan 9 18:21:00 2007 -->
<pre>Run 231334 
 
processed  174441 events, of which 7270 with errors 
 
The DAQ buffer of these events appears as follows: 
=============================================================================== 
Trigger 25642697 
Mon Jan  8 13:39:54 2007 
 
Event 25642697, L2CAL Monitor: Pulsar DAQ buffer  : 3 
               REAL TP2D:              SIM TP2D:      XOR Diff: 
******************************************************************************* 
Block 0 board 1 word 0            0                       0                       0 
Block 0 board 1 word 1       100000                  100400                     400 
Block 0 board 1 word 2            0                  200000                  200000 
Block 0 board 1 word 3       100400                  300000                  200400 
Block 0 board 1 word 4            0                  400000                  400000 
Block 0 board 1 word 5       100400                  500000                  400400 
Block 0 board 1 word 6       400000                  600000                  200000 
Block 0 board 1 word 7       500000                  700000                  200000 
Block 0 board 1 word 8          400                     400                       0 
Block 0 board 1 word 9       100000                  100000                       0 
Block 0 board 1 word 10       200001                  200001                       0 
Block 0 board 1 word 11       300001                  300001                       0 
Block 0 board 1 word 12       400401                  400401                       0 
Block 0 board 1 word 13       500000                  500000                       0 
Block 0 board 1 word 14       600000                  600000                       0 
Block 0 board 1 word 15       700000                  700000                       0 
 
Events affected by these behavior seem to be sequential from  
trigger number 25642697 to 28495088 and to be contained in the  
following files: 
 
root://fcdfnfs01.fnal.gov//export/look/aphysr/ar0387a6.05bbphys 
root://fcdfnfs01.fnal.gov//export/look/aphysr/ar0387a6.064aphys 
 
The complete error log file can be found in  
/cdf/scratch/cortiana/OutCAF/look231334/ErrorLog_LEVEL2_PULSAR_CAL_0_231334.log 
 
 
by Giorgio</pre>
<!-- Author: Laura -->
<!-- Comment: Tue Jan  9 18;22;24 comment by...Laura -->
<pre>This run was with the first version of the dataio firmware used in system. 
</pre>

<!-- Comment: Tue Jan  9 18;24;59 comment by...Laura -->
<pre>We are running with online monitor code (run 231693) to check if the problem can be connected to a reset (hrr, halt) of the system. </pre>


<!-- Date: Sun Jan 14 10:10:06 2007 -->
<pre>Last week a certain number of stream have errors. We discovered they are connected to the hrr operation. 
Starting with run 232062 a new version of the firmware (both for the mezzanine and the dataio) has been load to fix this problem.</pre>
<!-- Author: laura -->
<!-- Comment: Tue Jan 16 19;25;22 comment by...Laura -->
<pre>http://www.pd.infn.it/~cortiana/TrigMonOut/TrigMonOut.html

We analyzed 7,255,307 events and found 4 errors due to ADMEM-DCAS energy
mis-match!

By Giorgio</pre>
<!-- Date: Tue Jan 16 12:49:38 2007 -->
<pre>Please, note that the default FER code for the crate tstl2trg1 is, starting from now, the lucas test code</pre>
<!-- Author: Lucas & Laura -->


<!-- Date: Tue Jan 16 19:20:28 2007 -->
<pre>Data format for LVDS Receiver Pulsar Board : 
------------------------------------------------------------------- 
DataIO1 (an DataIO2 in similar way): 
daq1 (16 words) :  
                   Mezz0Word0.....Mezz0Word7, Mezz1Word0....Mezz1Word7 
daq2 (16 words) :  
                   Mezz0Word0,Mezz1Word0,Mezz0Word1,Mezz1Word1.....Mezz1Word7 
-------------------------------------------------------------------- 
Ctrl 
daq1 (39 data) :  
     2f000070 --&gt; 31..23=board type, 
                             22..19=0,  
                             18..13=Pulsar_ID,   
                             12..8=0 
                             7..0=DAQheaderWord(bunch count)   
     b0f00000 --&gt; Begin of fragment (fixed) 
     017001c0 --&gt; Header1: 
                             31..24=format ID, 
                             23..20=DataSource, 
                             19..18=RegionID, 
                             17..10=reserved,  
                             9..2=bunch count,  
                             bit 1..0=buffer number.  
                             Bit 31..10 fixed.  
     000d000c --&gt; Header2: Latency register (to be masked or maybe a maximum delay can be checked) 
 
-------&gt; 32 data = DataIO1 daq2 data + DataIO2 daq2 data 
 
0000ffff --&gt; fixed(???) 
0021eeee --&gt; data size + error flags (fixed) 
e0f00000 --&gt; end of fragment (fixed)      
 
 
Comments: we probably have to change few fields of the Slink data packet and maybe we can delete few SLINK words in order to reduce the latency of the transmission.</pre>
<!-- Author: Laura -->



<!-- Date: Wed Jan 17 15:10:38 2007 -->
<pre>Added 8 more cables. Right now we have four mezzanines in the Pulsar. 
By the way...The cabling start to be a mess!!!  
 
- Mezz2 cable1 : phi=4 eta=17, 16 
- Mezz2 cable2 : phi=4, eta=19,18 
- Mezz2 cable3 : phi=4, eta= 21,20 
- Mezz2 cable4 : phi=4, eta=23,22 
 
- Mezz3 cable1 : phi=7 eta = 1,0 
- Mezz3 cable2 : phi=8 eta = 23,22 
- Mezz3 cable3 : phi=6 eta = 17,16 
- Mezz3 cable4 : phi=6 eta = 9,8</pre>
<!-- Author: Laura -->


<!-- Date: Fri Jan 19 10:29:43 2007 -->
<pre>Thurs 18 ( day shift ) 
Attempted parasitic running between stores with L2cal ( using l2quad02 ) 
 
Setup: 
 
Full parasitic setup ( joint decision in TS ), we hard code the l2quad decision to be false all the time and send error packet. The packet contains: 
 
bunch counter ( this we modified to read the bc from the cluster data instead of muon ), reject bit, and we added latency. The latency looked to be zero all the time but we suspect we where victims of integer division... 
(when running psuedo parasitically ( no joint decision ) we see that the algorithms run via timestamp readout in Jet_New.cc ) 
 
Inputs new cluster input ( 32 trigger towers from l2pu00 ) on filar 1, muon on filar 2 , solar to TS. 
 
We modified the L2code to run the new jet alg on all events with no Level 1 prereq. 
 
//---------------------- 
The PC l2quad02 was using gentoo 2006.1 and gcc 4.1.1 ( old l2 pc uses 3.3.4 ). 2Gb  DMA was removed ( had 4 Gb ) to fix some problems related to  s32pci64 cards can only address 2GB for DMA ( at least that is the best guess ) 
</pre>
<!-- Author: l2cal -->
<!-- Comment: Fri Jan 19 10;30;46 comment by...gene -->
<pre>This is a belated entry...</pre>


<!-- Date: Tue Jan 23 10:18:06 2007 -->
<pre>Remember to write the register 0x18 of the SLINK  Merger to disable the mezzanines not used. Otherwise the SLINK merger doesn't send any data!! 
the command is: 
./vxwritereg b0l2pu00 6 0x0 0x18 0xc 
0xc disables mezzanines 3 and 4</pre>
<!-- Author: Lucas & Miguel -->


<!-- Date: Tue Jan 23 12:44:19 2007 -->
<pre>Tried to send data from the merger to l2quad02. At first this failed. Checked we could get data into the PC when sending directly from the pulsar- this worked. 
Laura replaced AUX card and the merger to pc test now works. 
 
Will need to modify the ptr code to match the increased number of towers compared to last weeks test. 
 
</pre>
<!-- Author: laura, giorgio and gene -->


<!-- Date: Thu Jan 25 15:53:00 2007 -->
<pre>205 LVDS cables tested. 
204 working, 1 broken.  
11 of the 204 working cables with little damages. 
 
Lucas: little damage in the fingers. 
Miguel: I dont't feel my fingers. 
 
</pre>
<!-- Author: Lucas and Miguel -->


<!-- Date: Fri Jan 26 01:23:45 2007 -->
<pre>During run 233018, in order to test that the new fer code is not corrupting the existing TC2D blocks, we unplug in the order:  
 - cable 2, mezz 0, board 0 (phi 8, eta 22-23) at 01:18; 
 - cable 2, mezz 1, board 0 (phi 4, eta 18-19) at 01:20; 
 - cable 4, mezz 1, board 1 (phi ?, eta  ?-? ) at 01:22. 
</pre>
<!-- Author: Laura, GIorgio and Massimo -->


<!-- Date: Fri Jan 26 03:25:11 2007 -->
<pre>&lt;pre&gt; 
Very brief overview of tests performed tonight. (more details later) 
------------------------------------------------- 
 
1) Readout code: tests using COSMIC RUNS (233012, 233018) with 
   max-minibank=12. 
   o) TC2D bank structure as expected in AC++ dump. 
   o) TrigMon crashes both official and custom version, still       
      investigating (Consumers?) 
   o) TP2P in place (in run 232452, at some point it was 
      disappearing, not understood the cause). 
 
1) Custum Monitoring  
   o) run online with default TC2D bank 
   o) crashes when running online, using new readout code 
 
 
 
&lt;/pre&gt;</pre>
<!-- Author: Laura, Massimo, Giorgio -->


<!-- Date: Mon Jan 29 10:35:23 2007 -->
<pre>Output of SLINK Merger for 1 Pulsar (4 mezzanines, 2 mezzanines, number 3 and 4,  have 0 as input): 
 
 2f00000b b0f00000 0180002c 00200020 0170002c 000c000c 00000000 00000000 
 00100000 00100000 00200000 00200000 00300000 00300000 00400000 00400000 
 00500000 00500000 00600000 00600000 00700000 00700000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 0000ffff 0021eeee 
 0170002c 000c000c 00000000 00000000 00100000 00100000 00200000 00200000 
 00300000 00300010 00400000 00400000 00500000 00500000 00600000 00600000 
 00700000 00700000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 0000ffff 0021eeee e0f00000 0049eeee e0f00000 
</pre>
<!-- Author: Gene & Laura -->


<!-- Date: Mon Jan 29 13:59:17 2007 -->
<pre>Follow Andrew's Instruction to switch between XFT and L2Cal parasitic test configuration 
 
http://www-cdf.fnal.gov/internal/people/links/AndrewIvanov/L2CalToL2XFT.html 
</pre>
<!-- Author: L2cal -->


<!-- Date: Mon Feb 26 17:07:48 2007 -->
<pre>Status of the pulsars in test room:  
 
- 5 pulsars from the trigger room (crate 2RR29C) when we cleaned in october, so, presumably, they 
are working (to be checked).   
 
They include:  
 
1 reces rx (4 mezzanines taxi)  
3 taxi rx (with 4 mezzanines)  
1 taxi   
 
- 6 pulsars (unknown status):  
 
1 hotlink rx (with 4 mezzanines)  
1 hotlink tx (with 4 mezzanines)  
1 slink  
1 taxi rx (with 4 mezzanines)  
2 taxi   
 
 
- 1 hotlink pulsar with ctrl jtag problems  
 
- 2 SVT pulsars   </pre>
<!-- Author: laura & virginia -->


<!-- Date: Wed Feb 28 18:14:10 2007 -->
 
   <h4>     The breakthrough in the L2 XFT commissioning </h4> 
 
    Yesterday we made one of the major breakthroughs in our quest of solving a long list of problems arisen during the L2 XFT commissioning.  <br><br> 
    We have been struggling for a few weeks now attempting to understand why the online stereo reconstruction disagreed with the offline simulation. We managed to build the offline executable based on the online library, and verified that the online software is doing stereo track reconstruction correctly so that results were in agreement with the offline simulation. We suspected that the data received  by the L2 PC was somehow not in-sync with the track information, and have performed series of L2 XFT parasitic runs with additional debugging information included into the TL2D bank. We appended a part of the XFT segments data received by the L2 PC into the output packet, and were surprised to see that the data seemed to be in-sync and was exactly as what it was supposed to be. Then we tried to verify that the data passed into the algorithms was consistent with the input XFT data as well as with what goes into the output bank. We proved that it was the case, which made the situation very confusing.   <br><br> 
    That was a conundrum: the software worked offline and returned the correct results, while online it lived its own life. Part of the problem involved in the debugging was due to the fact that unless our L2 PC was included into the data taking, the data was always going to be out-of-sync, and the stereo reconstruction should have obviously failed. That could be immediately seen since the slam confirmation bit almost always should match L2 stereo reconstruction bit. 
However, if it was included into the data taking, we couldn't do much debugging because too many printout statements into a log would cause L2 time-outs and would adversely affect the CDF data operation.<br><br> 
     While we were thinking on what minimum information we need to include into the next test, so that it would be maximally useful to locate the source of the problem, an ace issued the HRR. Our data was in-sync with the rest of the CDF data for the first few events, and luckily we had sufficient debugging information to see that the stereo reconstruction WORKED for the first few events, but started failing even before the data went out-of-sync. At that point we immediately knew where to look for a bug in the software. <br><br> 
      The L2 Stereo code requires a multi-layer unpacking. First we unpack the data received by each Pulsar, and find pointers to each Finder for each event. Next, during stereo track reconstruction we unpack the relevant Finder data (corresponding to 30deg slice in phi), and find pointers to more detailed phi-pixel and slope information. To avoid unpacking the same Finder data again, in case we need to reconstruct another track nearby, we stored that information. However, we forgot to reset it when we proceeded to the next event.<br><br> 
     That was obviously a silly bug, but especially difficult to find. On the other hand have anybody encountered *smart* bugs? <br><br> 
     We fixed the code and asked the ace to issue an HRR again, and we saw right away that the stereo reconstruction was working properly for all of the events being in-sync. We requested the L2 XFT parasitic run and later confirmed that the online stereo reconstruction is now working correctly. <br><br>
<!-- Author: Andrew Ivanov and Rob Forrest -->
<!-- Comment: Wed Feb 28 19;55;02 comment by...Ted -->
just came back after 1.5 months (out of town). need to hear some good news and this one seems good enough...:)


<!-- Date: Fri Mar 16 17:01:34 2007 -->
<pre>84 LVDS cables tested: 
 
83 working 
1 broken (connector broken) 
 
</pre>
<!-- Author: Virginia & Miguel -->


<!-- Date: Tue Mar 20 05:40:12 2007 -->
Tonight we had a problem related to l2ctrlb during data taking ("control node unable to contact decision node process"), and I happen to be on shift. I woke up Vadim to look into this. Thought that it's worth to make a note here, just in case it happens again so that others know what to do. see shift e-log <a href= 
'/cgi/elog/elog.pl?nb=2007&action=view&page=235&frame=2&anchor=02:15:11.453'> 
entry </a> here.  
<!-- Author: Ted -->


<!-- Date: Wed Mar 21 00:41:27 2007 -->
<pre>From eve shift e-log today: 
 
Tue Mar 20 21:36:29  Run 237002 ACTIVATE: RunSet: AAA_CURRENT Trigger: PHYSICS_4_01 [5,866,678]  - rob Tue Mar 20 21:38:54 Same problems, in starting this run. Already paged L2 a few minutes ago. I think this is a problem with the pulsar similar to earlier today. - rob :: (run 237002) 
-- Tue Mar 20 21:45:57 comment by...rob --  Got this error when restarting run: L2PULSAR_NODELOST! 
-- Tue Mar 20 21:47:16 comment by...rob --  Called Vadim's cell, he will take a look at this. 
-- Tue Mar 20 22:08:23 comment by...VR --  Multiple control processes left in by the previous XFT and L2CAL tests today confused the decision node. I restarted everything fresh. Will come up with procedures for tests regarding this thing so willnot happen again.</pre>
<!-- Author: Ted -->


<!-- Date: Wed Mar 28 18:51:20 2007 -->
<pre>Hardware: what we have, what we miss (the status of the test is not indicated just to avoid mess...) 
 
Right now we have : 20 new pulsars + 14 old 
 
- LVDS PULSARS:  
                         - 9 full + 9 news + 7 new used as spares   
                         - 13 Front panels (with strain relief)  ---&gt; 5 missing 
                         - 13 AUX cards ---&gt; 5 missing 
                         - 11 SLINK TX ---&gt; 7 missing 
 
- SLINK MERGER PULSARS : 
                         - 4 full +  2 new + 3 new used as spares 
                         - 7 Front Panels 
                         - 19 SLINK RX ----&gt; 5 missing 
                         - 6 full SLINK TX AUX  cards. 
 
- AUX card front panels are ok! 
 
---------------------------------------------------------------- 
Finally we are waiting for (at least): 
 
- 5 AUX cards 
- 7 SLINK TX  
- 5 SLINK RX 
- 5 LVDS Front Panels 
- 60 10A fuses (for new pulsars) 
- 100 jumpers (for new pulsars) 
- 80 Mhz Roboclocks 
- 100 screws for front panels 
- 50 stand-off for the SLINK RX 
- short stand-off for the SLINK TX </pre>
<!-- Author: Miguel & Laura -->
<!-- Comment: Sun Apr  1 11;36;59 comment by...Laura  -->
<pre>A new spreadsheet is around to take into account all the hardware materials  we have, we need and we are waiting for..</pre>


<!-- Date: Mon Apr 2 11:05:28 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-375&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-375  width='300' align='top'></a></td><td>During this weekend, I re-wrote the L2 PC NodeAlgo with optmized version of calls to Filar as discussed in my talk at TDWG ; this morning luckily I was granted time for a few quick cosmic runs with spare PC included.  
This plot shows the time when the L2D packet arrives for the spare PC and for the main PC. Before the optimization the spare PC was arriving at 24 usec. The optmization reduces the latency by 6 usec!.. </td></tr></table>
<!-- Author: Andrew -->


<!-- Date: Mon Apr 2 11:23:24 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-376&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-376  width='300' align='top'></a></td><td>The same plot with RECES+SVT data included. Note, no CLUSTER path was included since it was giving me errors, and I had to exclude it for these tests. Will have to invesigate it later.  
 
This plot shows that the optmized L2 with XFT included has about 1 usec then the current system with no optmization</td></tr></table>
<!-- Author: Andrew -->


<!-- Date: Tue Apr 3 18:59:44 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-377&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-377  width='300' align='top'></a></td><td>Another L2 XFT parasitic cosmic run test with the optimized version of the L2 PC code. The CLUSTER path has been fixed. SVT path is ignored, because SVT folks were doing some work during the test. The plots show when the L2D packet arrives to the TS board. The spare PC in addition to CLUSTER+RECES+MUON receives the data from the XFT crates. Plus it runs optmized FILAR commands. The spare PC seems to be about 1 usec slower than the main PC. The reason it did not show up in the previous tests is that apparently the CLUSTER path does matter. Additional room for improvement would be to set the FILARs into two different PCI buses, which was not yet done in these tests. </td></tr></table>
<!-- Author: Andrew -->


<!-- Date: Wed Apr 4 16:22:50 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-380&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-380  width='300' align='top'></a></td><td>Today I installed FILAR cards onto two different PCI buses. There are some plots in the main e-log. Here I attach the plot where we ran the system with no SVT. Since the SVT arrives late, the total L2 latency is driven by SVT. While the test with no SVT shows when all non-SVT data packets arrive and when the L2 PC starts its algorithms.  
This plots shows that tha algorithms start running on the spare PC about 0.75 usec later. However compared with the previous plot (one PCI bus configuration) the large tail in the spare L2 PC latency is significantly reduced, which means this is the way to go.</td></tr></table>
<!-- Author: Andrew -->


<!-- Date: Wed Apr 25 16:54:31 2007 -->
<pre>I Measaured the time it takes to access a filar register: 400 ns</pre>
<!-- Author: vr -->
<!-- Comment: Thu Apr 26 04;23;00 comment by...vr -->
<pre>Here are the totals for the whole filar_receive_channel_ptr method:

ret_buf - 0.3 usecs
memcpy - 0.5 usecs
every filar register read - 0.3 usecs</pre>


<!-- Date: Thu May 3 05:14:05 2007 -->
<pre>FILAR arrangements on the l2quad02: 
 
FILAR2 | FILAR1 | FILAR0 | SOLAR</pre>
<!-- Author: vr -->
<!-- Comment: Thu May  3 05;14;30 comment by...vr -->
<pre>this is looking from the back </pre>


<!-- Date: Thu May 3 07:01:09 2007 -->
<pre>To test channel link up on filar: 
 
Go to run/vadim_screwups and run bin/filarreset.exe --help to see options</pre>
<!-- Author: vr -->


<!-- Date: Thu May 3 17:28:10 2007 -->
<pre>tags for l2pulsar_algo, common, node, slink 
vl2cal_3</pre>
<!-- Author: vr -->
<!-- Comment: Thu May  3 18;33;49 comment by...vr -->
<pre>the current trigger table build command:
RemoteBuildL2Filar.sh --algotag vl2cal_3 --nodetag vl2cal_3 --commontag vl2cal_3 --tedtag vf_01_00 --iotag vl2cal_3 --triggertable L2_TORTURE --version 18</pre>

<!-- Comment: Fri May  4 11;18;57 comment by...vr -->
<pre>Dave compiles with these tags:

BuildL2Filar.pl: Using common tag &gt;vf_03_00&lt;
BuildL2Filar.pl: Using io tag &gt;vf_03_00&lt;
BuildL2Filar.pl: Using algo tag &gt;vf_05_07&lt;
BuildL2Filar.pl: Using node tag &gt;vf_03_00&lt;
BuildL2Filar.pl: Using ted tag &gt;vf_01_00&lt;


anything changes, code should be synced with the branch</pre>


<!-- Date: Mon May 7 10:33:51 2007 -->
<pre>latest build: 
 
RemoteBuildL2Filar.sh --algotag vl2cal_7 --nodetag vl2cal_3 --commontag vl2cal_3 --tedtag vf_01_00 --iotag vl2cal_3 --triggertable PHYSICS_4_02 --version 5 
</pre>
<!-- Author: vr -->


<!-- Date: Tue May 8 02:09:55 2007 -->
<pre>run 241007 is an exact copy of the old system. no l2cal fibers no algo l2cal</pre>
<!-- Author: vr -->


<!-- Date: Tue May 8 02:13:08 2007 -->
<pre>current build without l2cal: 
 
RemoteBuildL2Filar.sh --algotag vl2cal_12 --nodetag vl2cal_3 --commontag vl2cal_3 --tedtag vf_01_00 --iotag vl2cal_3 --triggertable PHYSICS_4_02 --version 5</pre>
<!-- Author: vr -->


<!-- Date: Tue May 8 02:48:01 2007 -->
<pre>run 241008 has all l2cal in running clustering on every accept</pre>
<!-- Author: vr -->