<!-- Title: 16:55:42  Mon May 21 2007 -->
<!-- Crewchief: L2  united team -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>start a new page... final commissioning stage...</pre>


<!-- Date: Mon May 21 16:58:25 2007 -->
<pre>======== Plan for the week ====== 
 
hi, Peter and Mary: 
 
This afternoon, we have successfully reconfigured the L2 system back to its normal state. The caused of the problem was indeed one L2-TS cable, so the fix was simple. The details are summarized in the e-log (attached below). 
 
The L2 united team had a long meeting today to discuss the plan for the rest of the week. We have learned A LOT from last beam runs last Friday morning. For L2CAL, there is no known problems so far. For L2XFT, the main problem is that we experienced some decision timeouts (shown up as XFT data corruption) when both XFT and CAL are included in the system. In the end, we all agreed to do the following for the next two stores: 
 
(1) the upcoming store (today):   
 
we request that this store be dedicated for L2XFT. We will still run the default system for decision making, and the new system will be in parasitic mode but with L2XFT (i.e. no L2CAL), using the integrated XFT+CAL code from last Friday beam run. The data will be still good for Physics, the only difference is that L2CAL will not be included.  
 
During this store, the L2XFT experts may need to start a new run from time to time (e.g. swap fibers to FILARs depending on what they learn), but should be quick. 
The idea is to study the decision timeouts (or XFT data corruption) seen last Friday, and see if this happens at all without L2CAL, esp. at high luminosity (and high L1A rate). 
 
(2) the store after this one: 
 
we request that the next store (after this upcoming one) be dedicated for L2CAL. We will run L2CAL in parasitic mode without L2XFT, and see if everything *really* works, with the same integrated XFT+CAL software code from last Friday beam run.  Again, the data should be still good for Physics. 
 
cheers, 
 
Ted for the L2 united team 
 
 
 
========== 
 
 
 
Summary of the L2 system re-configuration work ====  
 
 
This afternoon, we reconfigured the L2 system back to its normal state. Here is a summary:  
 
(1) we verified that the problem is really due to one bad L2-TS cable, and nothing else. This cable has caused problems in the past 7 years as far as I can remember (probably once or twice a year thing, usualy by reseating the cable connection solves the problem, but not this time).  
 
 
(2) We fully tested two new L2-TS cables, one is a spare, the other is the unused 03 L2-TS cable on the back of the TS crate.  This is to make sure that the spare cable is also fully tested (again) in the system.  
 
 
(3) We then configured the system back to its normal state. The default L2 system now uses the good cable (cable 00) which was used for the upgrade system in the past month or so.  The upgrade system now uses the new cable (labeled as 03). The spare cable is will be saved in the L2 cabinet (with a label attached "L2-TS spare cable fully tested on May 21, 2007... TL).  
 
 
(4) after the reconfiguration, we have tested the default system by itself, and also tested the upgrade system by itself, then the combination of both, made sure all works. In L2 torture mode, as well as in L2 cosmics.   
 
 
(5) There is one more important step needed: We need to save ALL the configurations back to normal again, so that ACE will not get confused. We haven't done that yet, since ACE is taking cosmic run for the L1 XFT folks. Andrew will  
save all the configurations as soon as the cosmic run ends, and he will make a comment after this entry. 
 - Ted with Andrew, Anadi, Alison and Laura ... 
-- Mon May 21 16:08:04 comment by...Ted with Andrew --   
Anrew just save all the configuration for L2_TORTURE, cosmics, AAA_shotsetup, AAA_silicon and non-silicon.  
 
 
For the upcoming store, the L2 folks will be here to do beam test with the upgrade system. we will send an official request soon. This is to inform the shift crew that the L2 experts will be around at the BOS (likely most of the rest of the store). 
 
 
On May 21, 2007, at 10:32 AM, Tiehui Ted Liu wrote: 
 
 
Mary and Peter: 
 
I am calling a L2 meeting,  at 11am today, with all the people working on the L2 upgrades. after the meeting, we will have a detailed plan for the rest of the week. the L2TS cable connection issue will be resolved right after the meeting, around noon time today. 
 
Ted 
 
On May 21, 2007, at 10:13 AM, Mary Convery wrote: 
 
Dear L2 experts, 
 
The TS cable/connector issue needs to be resolved before we can continue parasitic running.  The store will be dropped at 11:00 and there will be at least 4 hours before the next store goes in, so it would be great if the problem could be investigated now. 
 
Bill Badgett is at CERN, but he has cell phone and email contact and should be able to make necessary changes from afar. 
 
Greg is out of town this week, but we should try to continue our 9:30 planning meetings starting tomorrow. 
 
thanks, 
Peter and Mary</pre>
<!-- Author: Ted for L2 united team -->


<!-- Date: Mon May 21 17:18:46 2007 -->
<pre>folks; 
 
I started a new page in the L2 upgrade e-log. please use it to document all the details here... with enough information so that we know exactly what we did week/month later.  and label the fibers (e.g. to FILARs) with a name at least I can understand ... :) 
 
Ted</pre>
<!-- Author: Ted -->


<!-- Date: Mon May 21 17:54:37 2007 -->
<pre>folks: 
 
finally I have some quiet time to write down the to-do-list we discussed at the meeting (continued at lunch table): 
 
(1) Dan, Anadi and Giorgio: 
 
will compare the TL2D bank (default vs new) from the last Friday beam runs, event by event. for L2CAL+XFT run, and L2CAL only run. 
 
Dan is working on this with Anadi. 
 
(2) Andrew and Alison:  XFT data size vs luminosity, vs channels (XFT1,2,3). 
 
(3) Andrew and Alison: Use XFT fake patterns with L2 Torture run to reproduce the decision timeouts (or data corruption) problem seen from beam... by increasing 
the input data size, L1A, and L2A rates. 
 
** note: as I am writing this, Andrew and Alison are already able to reproduce the problem seen in beam with fake test patterns! this is great news. 
 
(4) Lucas and Anadi: label all FILAR inputs (XFT and CAL) with names that one can easily understand 
 
** work started 
 
(5) Dan and Giorgio: 
 
Merge the data checking monitoring and timing monitoring code into the same package. 
 
(6)Ted and Andrew: 
 
 reconfigure the L2 system back to normal, and take care of the bad cable problem. 
 
** DONE 
 
(7) Lucas with everyone's help: 
 
prepare a web page to document all the system configuration details (and changes), and keep track of all the changes (software and hardware) at each stage... for major beam test runs, and the summary of the test results. 
 
(8) Anadi and Alison: 
 
tag the current version of the code (used in the last beam test last Friday) and check into CVS.  This is the official version we will start from for the next two stores! any code changes in the future should be done like offline/online code releases... make every effort to document the changes as well (goes to Lucas's web page). 
 
(8) Ted:  send beam test request for the next two stores, the first one dedicated for L2XFT, and the 2nd one dedicated for L2CAL. 
 
** DONE 
 
(9) let me know if I missed anything... or there is anything should be added. 
 
 
</pre>
<!-- Author: Ted for L2 united team -->
<!-- Comment: Tue May 22 00;21;57 comment by...Ted -->
<pre>one missed item:

(10) will need to add/test the L2errorfilter at L3 soon, to check errors on-the-fly online. let's first make sure that the TL2D old vs new offline checking is more or less working first. then we can add/test the L2errorfilter at L3... say sometime around EOS during the 2nd L2CAL only store.</pre>
<!-- Date: Mon May 21 18:57:33 2007 -->
<pre>my cell phone is 630-988-9986.</pre>
<!-- Author: Ted -->
<!-- Comment: Mon May 21 20;04;51 comment by...Ted  -->
<pre>I am open for bussiness 24 hours x 7... for both L2CAL and L2XFT teams,
though starting next week (May 27th to June 8th), the phone call is no longer local, it will be international (to Paris).</pre>
<!-- Date: Mon May 21 19:07:20 2007 -->
<pre>Tagged version of NodeAlgo we used last Friday during the evening shift (the PLATINUM) : 
cvs tag vl2cal_4 l2pulsar_node/</pre>
<!-- Author: anadi -->


<!-- Date: Mon May 21 19:26:56 2007 -->
<pre>Summary of today's L2TORTURE tests: 
 
We made new patterns for the XTCs with hits for every cell (Friday we had hits every 3rd cell).  
This means about 1 in 5 pixels is set.  
The data size seems to be ~280 words for each XFT pulsar board (empty patterns are ~120 words I think) 
We carried out the following tests: 
1) L2 torture with XFT only in same fiber configuration as Friday 
-&gt; DTO for b0l2pu02 -&gt; crate complaining about TP2D bank being too big 
-&gt; L2 decision TO that come from XFT 2 being corrupted 
  
* removed all "extra" cards in b0l2pu02. Kept only: 
-XFT00 
-XFT01 
-XFT02 
-Merger board 
-extra test board (to have 1 card on ROL 0) 
 
2) L2 torture test 
-&gt; No more DTO from the crate  
-&gt; Seem to have solved the problem of the max size in TP2D bank! 
-&gt; Still data corruption on XFT 2 
 
* Swapped XFT 3 with CAL 3. 
Now we have XFT 3 on 1 PCI bus (filar 00) and XFT 1 and XFT 2 on the other (both filar 02). 
 
3) L2 Torture test 
-&gt; No problems seen 
We pushed the system to all corners of phase space: L1 up to 65 kHz and L2 up to 1.2 kHz 
-&gt; No problems seen 
-&gt; We are confident we should use this configuration in the beam tonight 
 
* Included L2CAL back into the system 
4) L2 torture test 
-&gt; L2 Decision Timeouts, same as seen before, from XFT 2 
-&gt; XFT and L2CAL together still causing problems 
Reduced rate down to only using min-bias trigger 
-&gt; Problems went away 
Increased rate very slightly (adding MET trigger) 
-&gt; Problem came back 
-&gt; Even at low rate we are pushing the system when XFT is "heavily loaded" 
 
Alison's conclusion: 
We really have a problem related to the size of the packets going onto the PCI buses 
 
 
Tonight's plan: 
Go in with XFT only at the beginning of the store with the "new" fiber configuration. 
</pre>
<!-- Author: DALTA -->
<!-- Comment: Mon May 21 22;13;53 comment by...Andrew -->
<pre>In the next between store time we have to continue these tests. I believe we can put more load on the bus with the Solar. The Solar packet is large only on the Level2 Accept, which have much lower rate compared to the input packets coming on every Level 1 Accept. </pre>


<!-- Date: Mon May 21 23:37:59 2007 -->
<pre>COPIED FROM MAIN ELOG: 
1) run 242289 (begin of store). Run with XFT only (removed cards as in yellow box).  Run was smooth 
including when UPS kicked in.  
2) run 242290 XFT+L2CAL in. Runs OK until UPS kicked in, then frequent L2 TOs as expected (confirmed 
what was seen on Fri)  
3) run 242291 XFT only. Seems to be fine.   
</pre>
<!-- Author: DALTA -->
<!-- Comment: Mon May 21 23;59;55 comment by...Ted -->
<pre>AA == Alison + Anadi, have stayed in the control room for these runs... great work!</pre>

<!-- Comment: Tue May 22 00;06;30 comment by...Aron -->
<pre>For run 242290, was the FILAR channel configuration the one with all three XFT fibers on the same PCI bus (as Friday) or with one of the XFT fibers moved to the other PCI bus (as run 242289)?</pre>


<!-- Comment: Tue May 22 01;20;24 comment by...Ted -->
<pre>as far as I know (I am not in the control room, only watching things from home tonight), that one of the XFT fiber is indeed on the solar bus... as from the L2-Torture run. From the shift e-log comments: 

" 
======
Swapped XFT 3 with CAL 3. 
Now we have XFT 3 on 1 PCI bus (filar 00) and XFT 1 and XFT 2 on the other (both filar 02). 
======

I do learn something from Aron's question: it would help us more if there are more comments in this e-log about the detailed configuration, so that people (like Aron) do not have to look back to the main shift e-log to search for answers. let's do that tomorrow: take a detailed snapshot about the configuration used for this store, hardware+software. 

in general, it would be good that any comment in this e-log be wrriten in such a way so that others from the L2 upgrade teams could understand. I have to admit that I even have a hard time to understand some of the earlier comments... I know that it is not easy for EXPERTs to do so.</pre>


<!-- Date: Wed May 23 00:51:20 2007 -->
<pre>NEW L2CAL UPGRADE WEB SITE: 
 
http://www-cdf.fnal.gov/upgrades/daq_trig/l2cal/l2cal_home.htm 
 
You'll find everything you need about the L2CAL upgrade. 
 
</pre>
<!-- Author: Lucas -->
<!-- Comment: Wed May 23 11;03;26 comment by...Ted -->
Here is the link you can click on: <a href=
http://www-cdf.fnal.gov/upgrades/daq_trig/l2cal/l2cal_home.htm>
Lucas's new L2CAL upgrade web site </a>. Great stuff! Please take a look and send him suggestions.
<!-- Date: Wed May 23 02:33:15 2007 -->
<pre>Wed May 23 02:31:55 
 
===== Summary of what we have done =====  
 
Tonight, we have been running the new upgrade L2CAL system in parasitic mode with the default L2 system. It has been running smoothly so far (more than 4 hours). We will leave it running like this and will take as much data as possible, so that we can compare the L2 decision (TL2D bank) bit by bit with that of the default trigger system offline. The run number is 242349.  
 
 
ps. the old L2-TS cable connection problem happened again today at the beginning of the store. The problem was cured by reseating the cable connection on the TSI end. This is still puzzling. see  
more details in the previous shift e-log. This problem has nothing to do with the new L2CAL system. If the problem happens again tonight, please call 630-337-6060 (Anadi's cell phone). We will investigate this problem more tomorrow.  
 
 
We put a spare L2-TS cable in the system in order to run the new L2CAL system in parasitic mode. The cable is still on the floor, and SciCo took a picture of our "professor proof" cable protection system, and he will post it soon. This cable is only there for tonight. we will remove during the day (go under the floor). 
 
 - Anadi/Laura/Dan/Ted :: (run 242349)</pre>
<!-- Author: Anadi/Laura/Dan/Ted -->


<!-- Date: Wed May 23 06:31:45 2007 -->
<pre>Monitoring results for run 242349: 
 
Run 242349, L2CAL monitoring summary 
-------------------------------------------------------------------- 
LVDS Pulsar      Tot error    ADMEM-DCAS mismatch     evts    status 
-------------------------------------------------------------------- 
         0              4             0               17911      OK 
         1              0             0               17911      OK 
         2              0             0               17911      OK 
         3              0             0               17911      OK 
         4              0             0               17911      OK 
         5              0             0               17911      OK 
         6              0             0               17911      OK 
         7              1             0               17911      OK 
         8              0             0               17911      OK 
         9              0             0               17911      OK 
        10              0             0               17911      OK 
        11              0             0               17911      OK 
        12              0             0               17911      OK 
        13              0             0               17911      OK 
        14              0             0               17911      OK 
        15              0             0               17911      OK 
        16              0             0               17911      OK 
        17              0             0               17911      OK 
-------------------------------------------------------------------- 
</pre>
<!-- Author: Giorgio -->
<!-- Comment: Wed May 23 10;42;00 comment by...Ted -->
<pre>fantastic! Giorgio. thanks again for the quick feedback. do we know more about
the non-zeros ("4" and "1") under Tot error?  was it at the beginning/end of the run or in the middler of the run?  let's go after them!</pre>



<!-- Comment: Wed May 23 13;26;02 comment by...Giorgio -->
<pre>LVDS pulsar 00: errors occurred on events: 
                9036270: for cable 4 mezzanine 2.
                12044754: for cable 4 mezzanine 2.
                12078401: for cable 4 mezzanine 2.
                13972378: for cable 4 mezzanine 2.
LVDS pulsar 07: the error occurred on event 8063608.
                the problem is related to an sim/real energy  
                mismatch on  cable 4 of mezzanine 2.

All errors appear to be un-correlated to RC action such as HRR, and  occoured in the middle of the run (last run event is ~15M).

I will try to understand whether or not the interested cables are source of error even in previous runs. 
</pre>
<!-- Comment: Wed May 23 14;02;11 comment by...Ted -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Giorgio:
take a detailed look at the event itself, take it apart! see which bit was mismatched... and whether it is the same bit for the same mezzanine. let's hunt them down. have fun hunting! great work.</pre></td></tr></td></table>
<!-- Comment: Wed May 23 20;21;16 comment by...laura  -->
<pre>Giorgio, running online the monitoring (run 242364) I think I've seen the same problem (event 1681301). 
In particular for cable 4, mezzanine 2:
sim data-&gt; 600401, 701405
real data-&gt;600000, 700001

i.e. 5 bits on the same cable bits didn't change from 0 to 1. 
</pre>
<!-- Comment: Thu May 24 04;48;28 comment by...Giorgio -->
<pre>Here are the words which contain errors for run 242349

Pulsar 00:  DATAIO1_1:
    wordnum    real word               sim word          xor diff
    ------------------------------------------------------------    
     w15       700001                  700801                 800     
     w15       700401                  700c01                 800
     w15       700400                  700404                   4
     w14       600001                  600401                 400
     w15       700403                  701407                1004

Pulsar 07:  DATAIO_1_1:
    wordnum    real word               sim word          xor diff
    ------------------------------------------------------------   
     w14       600001                  600003                   2
</pre>
<!-- Date: Wed May 23 12:08:20 2007 -->
<pre>email to the folks directly working on the commissioning: 
 
====== brief status update and plan for next two stores ===== 
 
folks: 
 
I stayed up late again last night, so I will email you guys instead of calling another meeting (even though I prefer to discuss with you in person, to see your faces at least!). I will be at lab after lunch. 
 
In the past stores, we have done first L2XFT only store, then L2CAL only store (still going), and we learned a lot: 
 
(1) For L2XFT only store, the first run was quite smooth when we were there at the control room. later at night, there were strange "out-of-sync" (as Andrew describes it) errors causing many decision timeouts. This is something new, Andrew/Alison  
think that it is due to xft upstream problem. we will try to get help from the upstream folks. more could/will be learned with beam (hope it is reproduceable, otherwise, could be painful). 
 
(2) For L2CAL only store, the runs were smooth, still going. offline analysis will tell us more. Giorgio's initial monitoring results look promising. details see L2 upgrade e-log. (for this store, the L2errorFilter was only ran towards the end of store with the most recent trigger table PHYSICS_4_02 [6,885,686], because the first L2CAL run started with the previous trigger table while debugging the L2-TS problem). 
 
Note that at beginning of this store, we again had the old L2-TS connection problem, in the end we "fixed" it by reseating the cable. This still remains to be a puzzle at CDF, since 7 years ago, I have discussed a detailed plan with Anadi/Laura/Dan last night in the control room, and we will try to fully understand this problem in-between the stores (but we need it to come back first!). If the problem doesn't show up (likely when you are waiting), we will test a simple configuration as a backup for ACE if this happens again with beam. We will also need to clean up the cable on the floor after this store ends. 
 
For the next two stores, let's do the same:  the next store will be dedicated for XFT again, while the store after that will be used for L2CAL. This store "ping-pong" makes things more predictable thus easier for planning for everyone, also good for  
experts to get some rest once in a while, to pace ourselves ... esp. at time like this when you folks work so hard and make rapid progress everyday! I have been there for both stores, and I can tell you that it is not easy. 
 
I will be there in the afternoon, after lunch. come talk to me if you have any questions/concerns. 
 
cheers, 
 
Ted 
 
 
</pre>
<!-- Author: Ted for the L2 united  -->
<!-- Comment: Thu May 24 00;02;14 comment by...Ted  -->
After talking to Andrew and Alison, for this high luminosity store, L2CAL is running with the system. Andrew and Alison have done really carefully studies to show that the "out-of-sync" problem they had earlier is really due to the upstream, as happened last Dec. 


<!-- Date: Thu May 24 01:50:14 2007 -->
<pre> 
=== Simple ideas on how to possibly avoid the FILAR FIFO full problem for xft ==== 
 
folks: 
 
as promised, here I will briefly describe the simple ideas I have on how to avoid the FILAR input FIFO full problem due to large XFT data size over busy PCI bus. There are two of them, the 1st one is the simplest, while the 2nd one is not as simple but still simple enough to be considered. I will only show the simplest version of both ideas... so you get the simplest of the simplest. 
 
(1) The simplest idea (to improve the situation): still existing setup, only a few lines of code change for PCI bus handling.  
 
So far, the way we handle the FILAR cards PCI data tansfer is by requesting effectively for all 4 buffer events at the beginning of the run for all channels, that is to say, we already request to transfer all 4 possible buffer events over PCI bus as long as they arrive to the FILAR input FIFO. What this means is that not only the data packages from the *current* event will compete with the PCI bus with XFT packages, but the early arrival data packages from next buffer event coming from other data paths will ALSO compete the PCI bus with the poor XFT data! such that the poor XFT data has to wait inside the FILAR input FIFO, and 2 such packages would fill it up due to the waiting. 
 
so here is the simple idea to improve the situation:   
 
(a) for all non-xft data channels, only allow them to send the current event package over the PCI bus. the packages from the next event have to wait in the FILAR input FIFO (this is acheived by simply not sending the request). 
(b) for xft data channels, allow them to get on PCI bus as soon as the bus is free... this could be done by request for all 4 buffer events in advance (just like we are doing now). 
(c) as soon as one gets all XFT data packages for the current event, then enable all other channels to send the next event package (and only next event). 
 
The essence of this idea, by playing tricks with the way we send the PCI data transfer requests to FILAR channels, XFT data packages have the much higher (or even highest) priority over the PCI bus ...  
 
This idea is simple enough that the coding is really simple (a few lines according to Andrew). may need a few hours testing time, with L2_Torture first, and if promising, with beam. but it may not solve the problem in a fundamental way (i.e. at higher  
and higher luminosity, if the xft package size is too large). 
 
(2) The second simple idea: add one more decision node, to receive all XFT data and a copy of muon/xtrp/L1 data.  
 
adding one more decision node is simple (almost free), the control node is designed to be able to control multiple decision nodes, just by running one more process (call it TEDCpp_XFT, along with TEDCpp and TEDCppTEST). for example, the current  
control node is already controlling 2 decision PCs (and we have 2 spares sitting on the private network doing nothing). 
 
the simplest version of this idea is to simply use another decision PC (call it XFT PC) to receive all XFT data and a copy of the muon/xtrp/L1 data, and the only thing this PC does is effectively doing "fast abort": if this event doesn't need steore tracking,  
simply send an empty SLINK package out to the main PC. The main PC receives all the other data, including the L2CAL, just like what we have right now running in the beam, with one more input from the XFT PC. If the event does need steore tracking  
info, the XFT PC can then pack the raw data in SLINK format and send over a solar (just like the main PC packs TL2D) to the main PC. In other words, to do fast abort in software instead of firmware (the later is much more involved). of course, with  
software, one could improve it later by only pass ROI (region of interest) xft data to the main PC, depending on the L1/XTRP information XFT PC has (say, only consider segments associated with tracks above certain Pt). 
 
the essence of this idea:  fast-abort + Region of interest, to suppress the xft data using another PC as a xft data filter (for suppresion). much easier (and flexible) to do than firmware approach. Since the xft data is done on a dedicated PC, the FIFO  
full problem can be avoid (esp. by using single channel FILARs, or S32PCI64, which has 2048 input FIFO depth!). effectively, this is as if one had 4 PCI segments on one PC. the extra timing (compared to hardware/firmware fast-abort) is  
highly unlikely a problem as it should be mostly hiding behind other things. 
 
of course, if all we care about is to only reduce the muon rate at L2 using the steore tracking, then all muon steore tracking algorithm can be done inside the XFT PC including the decision bits (while the main PC is busy clustering), and then send the  
results to the main PC. note that the overhead of sending an empty slink package from one PC to the other is on the order of say ~ 5 us or so. 
 
I explained this idea to Andrew/Anadi/Laura/Alison, but soon we were distracted into more complicated cases ... the simplest version of this idea is to just suppress the data, the main PC is still doing bussiness as usual (including steore tracking from time to time when needed), and now the XFT data will come from one fiber.  Note that now the xft data can be received by single channel FILAR (i.e. S32PCI64), with 2048 SLINK words input FIFO deepth (instead of 512). since large fraction of events will not need to do steore tracking, this should be effective.  
 
I could imagine it may take one week to try out this idea, including fast-abort software work and testing time. and the existing NodeAlgo in the main PC will not change, which makes things much easier.  
 
of course, one could combine both idea (1) and (2)...   
 
I can discuss these ideas at this Friday meeting. anyway, I have tried my best to come up with ideas to solve your problem (given the limited time I have had doing real think,  only had time to do thinking during beam abort gaps in the past week).  
Please think about it, and if you really like them, feel free to try things out. If you hate it, just ignore it ...:)   
 
there are other more elegant ideas, but they all take more time! I will NOT talk about them. this email is longer than I anticipated. 
 
cheers, 
 
Ted 
</pre>
<!-- Author: Ted  -->


<!-- Date: Thu May 24 01:55:50 2007 -->
<pre>==== smooth sailing over high luminosity ==== 
 
Wed May 23 23:27:00 Run 242381 ACTIVATE: RunSet: AAA_CURRENT Trigger: PHYSICS_4_02 [6,885,686]  - guido 
-- Wed May 23 23:39:13 comment by...Ted --  Good to see such high luminosity again..... just the right time to test the new L2 upgrade system. This run has the new L2CAL system running in parasitic mode, with the L2ErrorFilter running at L3, to check "on-the-fly" the decisions made by the new system with that of the default system. shift crew has done a great job starting this run so smoothly, without any L2 experts in the control room (I am at home).  
-- Wed May 23 23:42:55 comment by...Ted --  B0Lumi at 242E30 when scraping completed. the physics run already started when B0lumi is at 240E30. Impressive! L1A/L2A/L3/Deadtime = 20KHz/560Hz/140Hz/6%.  
-- Wed May 23 23:49:58 comment by...guido --  Thanks for all the work that the L2 crew did ;)  
-- Wed May 23 23:57:59 comment by...Ted for L2 --  if you want to know some of the details on how we are doing, take a look at the L2 upgrade e-log .</pre>
<!-- Author: Ted  -->



<!-- Date: Thu May 24 07:05:12 2007 -->
<pre>Run 242381, L2CAL monitoring summary 
-------------------------------------------------------------------- 
LVDS Pulsar      Tot error    ADMEM-DCAS mismatch     evts    status 
-------------------------------------------------------------------- 
         0              2             0               24995      OK 
         1              1             0               24995      OK 
         2              0             0               24995      OK 
         3              0             0               24995      OK 
         4              0             0               24995      OK 
         5              0             0               24995      OK 
         6              0             0               24995      OK 
         7              1             0               24995      OK 
         8              0             0               24995      OK 
         9              0             0               24995      OK 
        10              0             0               24995      OK 
        11              0             0               24995      OK 
        12              1             0               24995      OK 
        13              0             0               24995      OK 
        14              0             0               24995      OK 
        15              0             0               24995      OK 
        16              0             0               24995      OK 
        17              0             0               24995      OK 
-------------------------------------------------------------------- 
 
 
evt:      pulsar     DATAIO wordnum    real word               sim word          xor diff 
 
9655332    00         1_1     14       601000                  601401                 401 
                              15       700001                  701c05                1c04 
16092506   00         1_1     15       700003                  701003                1000 
5247594    01         1_1     14       600401                  600403                   2 
8738698    07         1_1     14       600001                  600005                   4 
13075588   12         2_1      6       600800                  600802                   2 
                               7       70040a                  70140e                1004 
 
At the end of the run, found the following errors on Merger 00. 
16660947   M00       missing info from 4th input in CONTROL DAQ 
16660948   M00       missing info from 4th input in CONTROL DAQ 
16660949   M00       missing info from 4th input in CONTROL DAQ                  
16660950   M00       missing info from 4th input in CONTROL DAQ                  
16660951   M00       missing info from 4th input in CONTROL DAQ                  
16660952   M00       missing info from 4th input in CONTROL DAQ     </pre>
<!-- Author: Giorgio -->
<!-- Comment: Thu May 24 07;07;41 comment by...Giorgio -->
See also 
<a href="http://www.pd.infn.it/~cortiana/TrigMonOut/TrigMonOut.html">   http://www.pd.infn.it/~cortiana/TrigMonOut/TrigMonOut.html</a>.

<!-- Comment: Thu May 24 09;33;09 comment by...laura  -->
<pre>These errors confirm an effect we saw yestarday. It seems that merger00 (slot 21) has some problems. Sometime the conf register that enable the inputs is reset to its default value. (power supply problem) At the following end of store we are going to replace this merger. </pre>

<!-- Comment: Thu May 24 11;55;04 comment by...Ted -->
<pre>yes. I discussed this with Laura last night. it's likely that something is wrong on this board (main board or mezzanine)... for example, something causing large current, with large voltage drop, to a point the board reset itself (by default, the merger expects 3 inputs, and we need to configure via VME to enable the 4th input). I have seen this long ago with a bad mezzanine card loaded on Pulsar in the teststand room. The FPGA is such that when it senses big voltage change (such as power up), the FPGA will reconfigure itself with the on-board firmware (this is a simplified picture)... so the merger will change from 4 inputs to 3 inputs merger... let's replace it and I will take a look at it later (tag that merger and leave it alone, together with the mezzanines). </pre>

<!-- Comment: Thu May 24 13;09;12 comment by...laura  -->
<pre>List of the pins causing errors  so far:

- Pulsar 0, mezzanine 2, cable 4, (bit0, bit10, bit22, bit30, bit31, bit32)
- Pulsar 7, mezzanine 2, cable 4 (bit2, bit1)
- Pulsar 7, mezzanine 1, cable 4 (bit2)
- Pulsar 12, mezzanine 3, cable 4 (bit1, bit22, bit23)

- The rate of the error is very low... </pre>


<!-- Comment: Thu May 24 14;28;52 comment by...Ted -->
<pre>interesting. what's so special about cable 4?</pre>
<!-- Comment: Thu May 24 15;09;04 comment by...laura  -->
<pre>Let's see if it's just a coincidence...I suggest do not change anything right now, let's simply keep check these bits in the following runs...
BTW, the most frequent error is in pulsar0 and it happens 8-9 times over 120.000 monitored events, and it happens in the less significant bits (i.e. the bits more fired).

P.S. If I should comment what's special about cable 4, well from the firmware point of view is the last cable we read...  </pre>
<!-- Comment: Thu May 24 15;39;32 comment by...Ted -->
<pre>just talked to Laura. this makes a lot sense to me now. she is right, the issue here is most likely to the fact that we read it last (at the end of cdfclk cycle, rigth before the data changes). we should be able to take care of this in the firmware.... nice to see that we are now already dealing with this low level of problems. a clear sign that the new system is getting ready for final operation.</pre>
<!-- Comment: Fri May 25 14;21;53 comment by...laura  -->
<pre>Lucas replaced the merger00 in slot 21.</pre>
<!-- Comment: Fri May 25 15;40;54 comment by...laura  -->
<pre>Problem merger00 seems to be fixed using a different merger.</pre>
<!-- Date: Thu May 24 13:21:50 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-407&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-408  width='300' align='top'></a></td><td>Timing differnce for new & old CPU on high lumi part of run 242381.</td></tr></table>
<!-- Author: Dan -->


<!-- Date: Thu May 24 13:22:26 2007 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-405&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-406' ></a></center>
<!-- Author: Dan -->
<!-- Comment: Thu May 24 13;22;50 comment by...Dan -->
<pre>The timing plots using 1 file of stream D at the highest luminosities taken are shown below.  Since it is only file, it is a small lumi range (230-240).  We are faster for the (L2 unbiased) previous event by ~1.5 mus and even faster than that for the (L2 biased) current event.  The current event timing difference plot shows an interesting 2 peak behavior which I am investigating further...</pre>

<!-- Comment: Thu May 24 13;23;19 comment by...Dan -->
<pre>Obviously, it should say shown above instead of shown below.</pre>





<!-- Date: Thu May 24 18:14:20 2007 -->
<pre>The error we just saw when trying to switch between L2CAL only and XFT only running was due to the fact that the output of the Pulsar boards were still connected as slaves (in order to use the merger while not reading out the xft into the PC). 
This error will therefore appear all the time a fiber is not connected properly. Would be nicer to have a nicer error message but for furture references the following messages are symptoms of a bad connection: 
LOG_OFFICIAL/052407_XFTOnly.log 
 
</pre>
<!-- Author: DALTAA -->
<!-- Comment: Thu May 24 19;23;01 comment by...Ted -->
<pre>let's document the procedure on the way you guys recovered from this problem this afternoon... and put it on Lucas's commissioning web page... </pre>


<!-- Date: Fri May 25 09:20:42 2007 -->
<pre>==== this was the cause of the problem for Jet trigger rates last night. clearly it was due to a DIRAC-DCAS cable. interesting thing is that the problem occured right after a long good run was stopped and a new run started. This cable problem needs to be fixed soon..... ==== 
 
 
 
Fri May 25 03:59:51 Summary of the trigger tower problem. We first saw in run 242405 high L2 jet rates (3ub versus 10nb expected). Carla masked tower phi=0, eta=5 for run 242406. We still saw high rates. Carla checked the masking and we tried again with run 242407. Still seeing high rates, Carla verified that she was seeing 512 counts per event in run 242407, as opposed to 512+some energy which is what she saw earlier. This indicated to her that the stuck bit problem was downstream of DIRAC, and that the tower had been properly masked. So she surmised that the problem was in the cable. I tried reseating the cable without success, so I went into the back room and found a box nicely labelled "DIRAC-DCAS Cables" so I grabbed one of those cables and put it in and the hot tower immediately went away. We started new run 242408 with the hot tower problem gone. There is a temporary cable running from the DIRAC board in slot 7 of B0L1CL00 to slot 6 of B0L2CL00. - KPitts :: (run 242408) 
-- Fri May 25 04:00:34 comment by...KPitts --  I would not have had any hope of finding an old cable if the teststand room on the second floor wasn't so clean and well organized.  
-- Fri May 25 04:30:37 comment by...KPitts --  Thanks very much to Carla for staying up half the night to figure this out.</pre>
<!-- Author: Ted -->
<!-- Comment: Fri May 25 14;29;41 comment by...laura  -->
<pre>So, cable 9, pulsar00, mezzanine 4, cable 2 is not connected anymore to l2cal system. 
Keep in mind for future monitoring...</pre>
<!-- Comment: Fri May 25 15;38;59 comment by...Laura  -->
<pre>Lucas changed the cable #9. </pre>
<!-- Date: Fri May 25 11:34:17 2007 -->
Monitoring results for runs: 242382, 242386, 242401, 242403 and 242404 are available from the following 
<a href="http://www.pd.infn.it/~cortiana/TrigMonOut/TrigMonOut.html"> web page </a>
<!-- Author: Giorgio -->






<!-- Date: Sun May 27 03:57:00 2007 -->
<h4> Summary of L2 Upgrade CAL+XFT Merging Studies </h4> 
 
The configuration, which was used during this week parasitic runs, worked with L2 CAL or L2 XFT only. Running together it used to cause the XFT data corruption. This configuration was suspected to have a high load on one of the PCI buses. I swapped RECES and L2 CAL 3 data packets, such that one PCI bus now has two XFT data packets, one L2CAL and the MAIN system, while the other has one XFT, two small L2CAL and one big L2CAL + SOLAR. I loaded large data patterns to XTCs, which incerased the XFT data packets sizes ranging from 300 to 400 for different events. The L2 torture test ran with the 60kHz Fred L1 rate, and L2A rate = 1.2 kHz for about 20 min, and did not produce any problems. It seems this configuration is very promising, and needs to be tested in beam. 
<br><br> 
I also implemented the controlled data flow mechanism by setting the high priorities to the large data packets (XFT and L2CAL3), and low priorities to all others by re-arming the FIFO for small packets one at a time. When testing the setup with the old configuartion it still produced the XFT data corruption (apparently due to L2CAL 3 having the high priority on the same PCI bus). However, setting the L2CAL 3 to lower priority introduced immediate problems, and caused the L2CAL 3 data corruption ( the input FIFO is not large enough for 4 L2CAL3 data packets). Therefore, even with the controlled flow, the data packets need re-balancing between PCI buses. 
<br><br> 
Another thing I tried is to push the XFT data packet sizes even larger. Using masks corresponding to the occupancy of 2 pixels/per cell, XFT data packets sizes for each event grew to 450 - 750. That introduced problems upstream, by making the second DATA IO in the XFT Stereo Pulsars out-of-sync. However, that was too extreme case, we do not expect so large XFT data packets coming on each event. We know that at lumi~240x10^32 only 0.3% of XFT data packets exceed size of 400. It is worth to note that the XFT data packets were not corrupted in the FILAR, but were corrupted upstream(!), which implies that there are other limitations to the system and for instance a single FILAR card for each XFT channel with large input FIFO is not a solution.<br><br>
<!-- Author: Andrew -->
<!-- Comment: Mon May 28 05;26;11 comment by...Ted -->
<pre>after I saw the XFT data size vs luminosity plot at last Friday trigger meeting, I am more worried than before... if the data size can indeed go up that high (and even higher with higher luminosity), then it is unlikely we will have a quick solution to the problem... unless the data size is kept under control at the *FINDER* sending side... let's hope for the best, but prepare for the worst. </pre>


<!-- Date: Mon May 28 07:38:55 2007 -->
<pre>Follows the overall error rate for L2CAL upgrade.  
For runs: 242037, 242040, 242195, 242349, 
          242352, 242354, 242381, 242382 
          242386, 242401, 242403, 242404. 
 
 
L2CAL monitoring summary 
---------------------------------------------------------------------------- 
LVDS Pulsar      Tot error    ADMEM-DCAS mismatch     evts          err.rate 
---------------------------------------------------------------------------- 
         0             20             0              180526      1.11 x10^-4 
         1              1             0              180526      5.54 x10^-6 
         2              0             0              180526      0.00 x10^-0 
         3              0             0              180526      0.00 x10^-0 
         4              1             0              180526      5.54 x10^-6 
         5              0             0              180526      0.00 x10^-0 
         6              0             0              180526      0.00 x10^-0 
         7              6             1              180526      3.32 x10^-5 
         8              0             0              180526      0.00 x10^-0 
         9              0             0              180526      0.00 x10^-0 
        10              0             0              180526      0.00 x10^-0 
        11              0             0              180526      0.00 x10^-0 
        12              3             0              180526      1.66 x10^-5 
        13              1             0              180526      5.54 x10^-6 
        14              0             0              180526      0.00 x10^-0 
        15              0             0              180526      0.00 x10^-0 
        16              0             0              180526      0.00 x10^-0 
        17              0             0              180526      0.00 x10^-0 
---------------------------------------------------------------------------- 
</pre>
<!-- Author: Giorgio -->
<!-- Comment: Mon May 28 07;40;19 comment by...Giorgio -->
Details can be found <a href="http://www.pd.infn.it/~cortiana/TrigMonOut/TrigMonOut.html"> here </a>.


<!-- Date: Mon May 28 16:21:17 2007 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-409&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-410' ></a><a href='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-411&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-412' ></a><a href='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-413&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-414' ></a><a href='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-415&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-416' ></a></center>
<!-- Author: Dan -->
<!-- Comment: Mon May 28 16;24;27 comment by...Dan -->
<pre>Above are the timing plots for run 242583 (L2CAL + XFT w/ new FILAR input configuration).  Note that there is still a double peak in the current event timing difference distribution.  The last plot shows that the larger peak is coming mostly from L2 bits 78 & 79 - the L2 inclusive rate limited triggers.</pre>


<!-- Date: Mon May 28 16:37:24 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-417&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-417  width='300' align='top'></a></td><td>XFT data sizes and instantaneous luminosity for part of run 242583.  The lumi range in this sample is small: 122-125.  The data volume is approximately 160 words per XFT pulsar at this lumi.</td></tr></table>
<!-- Author: Dan -->


<!-- Date: Wed May 30 15:29:34 2007 -->
<pre>New tags for the code: 
1) Node Algo (l2pulsar_node) tagged version  vl2cal_5: added the define for Reject; save the newest filar configuration 
2) libs (l2pulsar_algo) tagged version vl2cal_33: added the stereo info in the TL2D  
3) common (l2pulsar_common) tagged version vl2cal_4: added ignoreL2Cal and ignoreL2Stereo to the ControlObject </pre>
<!-- Author: anadi  -->
<!-- Comment: Wed May 30 18;42;13 comment by...anadi -->
<pre>In NodeAlgo we also have the new bus priorization written by Andrew</pre>
<!-- Comment: Wed May 30 19;35;04 comment by...anadi -->
<pre>both libs and NodeALgo now have TrackList in </pre>
<!-- Date: Wed May 30 17:08:43 2007 -->
<pre>Build Trigger Table 
PHYSICS  
  
./RemoteBuildL2Filar.sh --algotag vl2cal_33 --nodetag vl2cal_5 --commontag vl2cal_4 --tedtag vf_01_00 --iotag vl2cal_3 --triggertable PHYSICS_4_02 --version 6 
 
and the lib name is  
libl2algo.so.1_nodevl2cal5commonvl2cal4iovl2cal3tedvf0100algovl2cal33_0 
 </pre>
<!-- Author: anadi  -->




<!-- Date: Fri Jun 1 13:16:11 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-418&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=-418  width='300' align='top'></a></td><td>There's one new hot towers in CEM forware (eta -9, phi 5 upper)</td></tr></table>
<!-- Author: Alison -->
<!-- Comment: Fri Jun  1 13;16;41 comment by...Alison -->
<pre>Oups this should be in the main shift e-log... my fault...</pre>


<!-- Date: Fri Jun 1 14:53:44 2007 -->
<pre>Summary of last few days: 
 
run 242705 we went in when lumi was ~50e30 with the following configuration: 
 
l2cal only  
running new algorithms on L2A 
 
-Ran smoothly until run settings were changed for silicon problems 
 
The plan was to then go into the begining of the next store, however, after some checks regarding how to drive the system with the new PC we realised we did not understand things as well as we would have liked and thus backed out of the BOS test. 
 
The current plan is to run l2quad02 for 3 stores ( l2cal only NO ALGORITHMS) to check stability started today in run 242730. This test will end monday. 
Following this we will resume playing ping pong for a few stores to test xft and l2cal independently at high lumi. Inbetween stores continue on makeing combined system work. 
 
The L2cal algorithm test from yesterday was a repeat of the tests we did 4 weeks ago when we ran algorithms on L2A in the begining of a store ( L~170 ). Just wanted to make sure all the code still worked since we had not been running algo's for a while</pre>
<!-- Author: g for l2cal -->


<!-- Date: Sun Jun 3 10:49:38 2007 -->
<pre>New XFT Pulsar firmware loaded onto XFT04. Fibers are copies of XFT02. 
Look area file: ~3500 events 
No differences between XFT02 and XFT04 (except for the board ID and the timing is sometimes different by 0.1 us). 
 
Between stores I will look into changing the firmware to disable a channel and see what happens then. 
</pre>
<!-- Author: Alison -->
<!-- Comment: Sun Jun  3 14;47;01 comment by...Alison -->
<pre>Between store tests in AAA_SHOTSETUP_L2UP and L2_Torture (with XTC patterns) was sucessful. 

I managed to disable 1 of the channels and disconnect the fiber (after figuring out that the 12 fiber is actually not the bottom one but 2nd from the bottom). 

Visually the data is the same for the first 11 channels between XFT02 and XFT04.

I kept these changes in for the beginning of store and there does not seem to be any operational issues related to these changes.
</pre>

<!-- Comment: Sun Jun  3 15;29;47 comment by...Alison -->
<pre>It's running OK but when comparing the banks it seems that XFT04 gets out of sync and corrupted after the first event... investigating further</pre>

<!-- Comment: Sun Jun  3 15;41;38 comment by...Alison -->
<pre>Only 2 events out of 725 (from the runarea file) where the data is in sync. Maybe an HRR gets it back in sync?</pre>

<!-- Comment: Sun Jun  3 15;45;13 comment by...Alison -->
<pre>Indeed it's an HRR that solves the problem:
HRR on event 72846
Next event 72848: data is OK
Next event 72916: data is corrupted</pre>


<!-- Date: Mon Jun 4 14:12:05 2007 -->
<pre>Physics runs that make up the pc validation test ( ran from friday-monday ) without any problems ( no reboots etc ) 
 
The configuration was l2cal only ( NOT running any new clustering ) 
 
 
 
Run 242730  
Run 242732  
Run 242733 
Run 242734  
Run 242762  
Run 242764 
Run 242765  
Run 242766  
Run 242768 
Run 242769  
Run 242791 
</pre>
<!-- Author: gene -->