<!-- Title: 11:13:26  Thu Oct 11 2007 -->
<!-- Crewchief: L2 Team -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Let's start again using the L2 e-log</pre>


<!-- Date: Thu Oct 25 02:01:26 2007 -->
<pre>Summary of tonights tests: 
* Last week Bill, Nils and I changed the fer code to read the XFT Stereo finder fiber masks from the database at config and send that information in the form of an array to the Control Node. 
- I had checked that the information was indeed being sent to the control node 
* Today Vadim and I (well Vadim with me watching) change the control node code to check the information was being received and sending it to the L2PC at the end of the string of things being sent (like prescales and ignore bits). Changes were make to l2pulsar_ted and l2pulsar_common. 
* Rob and I then implemented the part of the code in the L2 PC to receive the information from the control node and pass it into the reconstructTrack routine. New executable and new library 
* Anadi made 2 new tags: 992 (cosmics) and 993 (l2_torture) that can only be run with the latest exe (official_test_rob_102407) 
* Tonight I tested these 2 new tags and the new exe and after finding a few minor bugs in the exe everything is working fine both with and without L2Stereo. 
 
WARNING: because none of the non-beam trigger tables ever call reconstructTrack we won't know until we get beam if this code is really doing what it should be doing or not (same as for event truncation and L1Abort, althrough the latter should never trigger an auto-accept unless the L1 bit mask is wrong). 
We can test this by looking at the trigger rates when masking on a whole bunch of fiber, setting the event size for the truncation to something tiny (in that case the trigger rate should be the same as for the 3D triggers). 
 
If all works to plan this is the last online major change on our list of things to be done. I'm sure we will find more but this should be close to a working system (fingers crossed). 
 
Think that's all from me for now. 
</pre>
<!-- Author: Alison -->


<!-- Date: Wed Nov 7 16:03:02 2007 -->
See <a href= 
'/cgi/elog/elog.pl?nb=2007&action=view&page=866&frame=2&anchor=16:00:02.635'> 
this entry</a> about TDC and XTC swaps today.
<!-- Author: jdl -->
<!-- Comment: Fri Nov  9 11;28;14 comment by...David, JDL, Alison -->
<pre>This swap did not seem to fix the problem.
It re-appeared last night.
We masked on now all the channels corresponding to those 2 XTCs: 192-207 in SL3.
Waiting to see if the problem comes back or not.</pre>


<!-- Date: Mon Nov 12 10:27:19 2007 -->
<pre>At the end of last week we masked on 2 XTCs.  
Over the weekend we saw that the problem with the finders getting out of sync had gone.  
I just unmasked 1 of the XTCs and we will keep an eye out to see if the problem comes back or not (using TrigMon).  
Currently SL 3 cells 192-199 are masked on (200-207 are the ones I just unmasked).  
This should not cause operational problems as L2Stereo is still ignored from Run Control.</pre>
<!-- Author: Alison -->
<!-- Comment: Mon Nov 12 18;15;32 comment by...Alison -->
<pre>Wrong one... well there was 50-50 chance...
Umnasked cells 192-199 and masked on 200-207.
Asked the shift crew to restart the run.
The XTC problem in the finders has gone (for the moment) but we are getting very strange problems with the Pulsars. Everything seems OK (bunch counter, buffer number) but TrigMon is showing errors... we are investigating.</pre>


<!-- Date: Mon Nov 12 19:34:58 2007 -->
<pre>To get some data with truncated events (for offline executable): 
Run number 253718: 
I set all XFT pulsar boards to truncate all events. 
The binary value is 66060304  
which corresponds to 3f00010 in hex 
Which corresponds to 16 ticks + data size 63 after which it truncates. 
Dumping boards shows me that all events are indeed truncated. 
This does NOT affect the data taking as L2Stereo is ignored from RC.</pre>
<!-- Author: Alison -->
<!-- Comment: Mon Nov 12 19;36;03 comment by...Alison -->
<pre>Run number 253717:
This run is with a "default" hardware configuration (no L1 Abort and no event truncation).
The only "non-perfect" setting is that 1 XTC is masked on (but current exe treats that as normal data.</pre>

<!-- Comment: Mon Nov 12 20;48;24 comment by...Alison -->
<pre>Run Number 253720:
Enable both L1Abort + Event Truncation.
L1Abort = 0
ABORTENABLE = 1 (maybe should stay 0 but don't think it matters as it's not a relevant quantity for our boards)
L1Abort rate seems to be very high (99%) when we expect ~75%. But TrigMon is telling me it agrees with what the TrackList is sending out. Will check offline.
</pre>

<!-- Comment: Mon Nov 12 20;50;10 comment by...Alison -->
<pre>MISTAKE IN MESSAGE ABOVE:
Run Number 253721:
Is what is referred to in previous message :-(

Run Number 253720:
L1 abort but NO event truncation.
L1 Abort rate also ~99%.</pre>


<!-- Date: Tue Nov 20 08:21:21 2007 -->
<p>This night <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2007&action=view&page=904&frame=2&anchor=04:13:24.297'> 
the L2 PC hang up with a kernel panic</a> once again. Preceding that there was a number of timeouts on RECES. Not sure whether it is related.</p> 
<p> Below is the excerpt from the L2 log :</p> 
 
<pre> 
NodeAlgo:: channel_masAddresCouldn't bind soNodeAlAddress already in use00 
L2Error object created 
NodeControl: CPU affinity: cpu_mask = 0x00000002 
Attempted to receive from unconnected communicator. 
NodeControl:: { 22:42:47, 11/19/2007 } Lost connectsolar_linkreset(0) 
setting up filar0 ... 
IO_PCIMemMap: before mmap 
size: 4096	pci_addr: 0xfc8ff000 
IO_PCIMemMap: after mmap 
IO_PCIMemMap: virt_addr = 0x41087c00 
filar_cardreset(0) 
filar_linkreset(0) 
setting up filar1 ... 
IO_PCIMemMap: before msetting up filar0 ... 
IO_PCIMemMap: before mmap 
size: 4096	pci_addr: 0xfc8ff000 
IO_PCIMemMap: after mmap 
IO_PCIMemMap: virt_addr = 0x41087c00 
filar_cardreset(0) 
filar_linkreset(0) 
setting up filar1 ... 
IO_PCIMemMap: before mmap 
size: 4096	pci_addr: 0xfc7ff000 
IO_PCIMemMap: after mmap 
IO_PCIMemMap: virt_addr = 0x41108c00 
filar_cardreset(1) 
filar_linkreset(1) 
setting up filar2 ... 
--> NodeAlgo: CPU affinity: cpu_mask = 0x00000001 
CMEM:  S_BUFSIZE set to 1024 
Waiting for Solar link to come up... 
Solar link is up! 
CMEM:  F_BUFSIZE set to 2048 
filar[0]->osr = 0x00001b01 
Waiting for Filar link to come up... 
Filar link is up! 
filar[0]->osr = 0x00001b01 
CMEM:  F_BUFSIZE set to 2048 
filar[1]->osr = 0x00101b01 
Waiting for Filar link to come up... 
Filar link is up! 
filar[1]->osr = 0x00101b01 
CMEM:  F_BUFSIZE set to 2048 
Sorry. Failed to allocate buffer #2 for filar 1 channel 1 
error: 0x704      => major: Error   4 in package   7 => CMEM RCC library: Error from call to ioctl function 
</pre>
<!-- Author: Andrew -->
<!-- Comment: Tue Nov 20 10;35;04 comment by...gene -->
<pre>After talking to the ace I think that this was not a kernel panic, the comment about the kernel ooops by the ace referred to 

UG: soft lockup detected on CPU#0!
&gt;
&gt; Call Trace:
&gt;  [&lt;ffffffff8020ae4a&gt;] dump_stack+0x12/0x17
&gt;  [&lt;ffffffff80252e37&gt;] softlockup_tick+0xdb/0xed
&gt;  [&lt;ffffffff8023a4c3&gt;] update_process_times+0x42/0x68
&gt;  [&lt;ffffffff80217f37&gt;] smp_local_timer_interrupt+0x23/0x47
&gt;  [&lt;ffffffff8021862b&gt;] smp_apic_timer_interrupt+0x41/0x47
&gt;  [&lt;ffffffff8020a222&gt;] apic_timer_interrupt+0x66/0x6c
&gt; DWARF2 unwinder stuck at apic_timer_interrupt+0x66/0x6c
&gt; Leftover inexact backtrace:
&gt;  &lt;IRQ&gt; &lt;EOI&gt;
&gt; BUG: soft lockup detected on CPU#0!

which we see in normal running. The excerpt from the log above tends to happen if the pc is not taken down cleanly - ie just power cycled not halted.

The ace also said he did have an xterm open for the pc and could type stuff- this may have just been the communication problem we have been seeing intermittently </pre>


<!-- Date: Thu Jan 3 19:24:32 2008 -->
<pre>Debugging L2 decision. 
Looking at /var/log/messages: 
1. We are "respawning too fast:disabled for 5 minutes". That's why if I kill the executable running I nedd to do telinit Q in order to have the executable running again. 
2. The coredump is disabled by default, now I've changed the space available for coredump from 0 to 2 GB. Next time the executable will crashes, we'll know why. 
 
</pre>
<!-- Author: Laura -->
<!-- Comment: Thu Jan  3 20;12;30 comment by...Laura -->
<pre>So, I edited the tcsh file using emacs in the l2 spare PC. I've edited the line starting the executable, restarted init file, and now it's working perfectly, no more respawing too fast and when i killed the process it restarted immediately.
I've tried to do the same in the main, but (very strange!!) from the trigger room console I can't open neither emacs nor xemacs ( instead I can from my the xterm on my laptop). Because they are going to restart in short time, I won't touch the main PC for the moment.
My suspect is that there is some strange characters let by xemacs that effects the correct behaviour of the p3 Id (our executable) and maybe emacs remove (???). By the way..is not the first time we had problem  opening xemacs. When we had time I'm going to reboot the PC, open using emacs the script file, checking space, column and file format and then restarted again.

 </pre>

<!-- Comment: Fri Jan  4 09;49;26 comment by...Laura  -->
<pre>Opened e saved init file and tsch file on the main PC. Now the init file is respawining correctly also in the main. I suspect some strange character let inside the tsch file opening from remote console, some version of xemacs...
I don't know if this fix also the problem of the l2node.exe pending, in case of segmentation fault. If it does we probably do not need to start by hand the executable.  In case of segmentation fault (that it will be debugged offline using the core dump) we'll need to start a new run in any case to get back the library, new ACE instruction will be provided, but let's wait to see the problem fixed. </pre>

<!-- Comment: Fri Jan  4 10;13;05 comment by...Laura  -->
<pre>For future reference (when this saga will be gone, I'll put a new item in the L2 expert web page) the important file to check if we are spawning correctly is /var/log/messages.

The lines indicating the process (script file in this case) has something wrong is:
Jan  4 02:02:07 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
Jan  4 02:07:08 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
Jan  4 02:12:09 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
Jan  4 02:17:10 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
 (p2 is our process)





</pre>

<!-- Comment: Fri Jan  4 11;55;28 comment by...Laura  -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>End of the respawning saga: DO NOT PUT  "&" in tcsh file after the following line:
/root/run/$version/bin/$prog $argv &gt;& $logdir$logname
 
</pre></td></tr></td></table>

<!-- Comment: Fri Jan  4 22;02;23 comment by...Laura  -->
<pre>Removed another additional & at the end of the another tcsh line..
The executable crashed, but we were able to restart automatically. I haven't found any core file, just additional message in the log file:
l2node.Fri_Jan__4_09:30:44_CST_2008
</pre>


<!-- Comment: Sun Mar 30 13;52;23 comment by...Laura -->
<pre>At the same time in /var/log/messages:

Mar 30 02:08:26 l2quad01 Leftover inexact backtrace:
Mar 30 02:08:26 l2quad01 &lt;IRQ&gt; &lt;EOI&gt;
Mar 30 02:09:06 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
Mar 30 02:14:07 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
Mar 30 02:19:08 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
Mar 30 02:24:09 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
Mar 30 02:29:11 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
</pre>
<!-- Date: Sun Mar 30 13:51:00 2008 -->
<pre>In the L2 Pc log file 
--------------------------------- 
Caught seg fault! 
 
 
dying gracefully ... 
 
unmapping filar 
unmapping filar 
unmapping filar 
unmapping solar 
finished. 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
NodeControl: ACK lcount&gt;10, quitting 
 
 
dying gracefully ... 
 
unmapping filar 
error: 0x803      =&gt; major: Error   3 in package   8 =&gt; IO RCC library: Library has not yet been opened 
unmapping filar 
error: 0x803      =&gt; major: Error   3 in package   8 =&gt; IO RCC library: Library has not yet been opened 
unmapping filar 
error: 0x803      =&gt; major: Error   3 in package   8 =&gt; IO RCC library: Library has not yet been opened 
unmapping solar 
error: 0x803      =&gt; major: Error   3 in package   8 =&gt; IO RCC library: Library has not yet been opened 
Warning: Failed to close the CMEM_RCC library for solar 0 
finished. 
</pre>
<!-- Author: Laura -->
<!-- Comment: Sun Mar 30 13;53;10 comment by...laura -->
<pre>At the same time in /var/log/messages:
---------------------------------------
Mar 30 02:08:26 l2quad01 Leftover inexact backtrace:
Mar 30 02:08:26 l2quad01 &lt;IRQ&gt; &lt;EOI&gt;
Mar 30 02:09:06 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
Mar 30 02:14:07 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
Mar 30 02:19:08 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
Mar 30 02:24:09 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
Mar 30 02:29:11 l2quad01 init: Id "p2" respawning too fast: disabled for 5 minutes
</pre>

<!-- Comment: Sun Mar 30 13;54;30 comment by...Laura -->
<pre>NodeAlgo and Control I think are doing the correct things: dying gracefully....and finished.</pre>

<!-- Comment: Sun Mar 30 14;01;06 comment by...Laura -->
<pre>Seems to me that: the executable died, but still not in a clean way. If I kill manually the executable (kill) the respawing is fine.</pre>


<!-- Date: Sun Mar 30 14:29:10 2008 -->
<pre>Here what happens If I disconnect SVT input (from pulsar board side), trying to reproduce this night problem: 
------------- 
ilar_linkreset(2) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
 
NodeControl: waiting for algo unlock (ACK) 
NodeControl: ACK lcount&gt;10, quitting 
 
 
dying gracefully ... 
 
unmapping filar 
unmapping filar 
unmapping filar 
Caught seg fault! 
 
 
dying gracefully ... 
 
unmapping filar 
unmapping filar 
error: 0x803      =&gt; major: Error   3 in package   8 =&gt; IO RCC library: Library has not yet been opened 
unmapping filar 
error: 0x803      =&gt; major: Error   3 in package   8 =&gt; IO RCC library: Library has not yet been opened 
unmapping solar 
error: 0x803      =&gt; major: Error   3 in package   8 =&gt; IO RCC library: Library has not yet been opened 
Warning: Failed to close the CMEM_RCC library for solar 0 
finished. 
unmapping solar 
error: 0x803      =&gt; major: Error   3 in package   8 =&gt; IO RCC library: Library has not yet been opened 
error: 0x803      =&gt; major: Error   3 in package   8 =&gt; IO RCC library: Library has not yet been opened 
... 
and so on, but the executable is still alive and restarting a new run is enough to fix the problem.</pre>
<!-- Author: Laura -->


<!-- Date: Mon Mar 31 17:41:39 2008 -->
<pre>L2quad02 passed memory test and has been upgraded to l2quad01: it's a real spare.</pre>
<!-- Author: laura and pedro -->


<!-- Date: Sat Apr 5 17:56:42 2008 -->
<pre>Spare nodes passed tests. 
 
Tested L2 decision node b0quad01 in combination with spare nodes b0l2cntl00 (192.168.0.20) and b0l2cntl01 (192.168.21), resp., while the corresponding other node and b0felix were deactivated. Tried various RC commands (CONFIG, ACTIVATE, HALT, RECOVER, RUN, etc) and monitored control and decision log files. Tried both L2_TORTURE and COSMICS runs. 
 
Smartsockets messages and acknowledgments indicate that TED communication of spares with RT server is fine (RC commands immediately reflected by TED messages as well as by actions on decision node).  
 
Also no obvious failures of tcp communication between control and decision observed (despite "connection refused error" just temporarily at the very beginning after starting l2node.exe, - happened also in case b0felix is involved). Also L2 decision TO observed once, which might be unrelated. 
 
Introduced environment variable TEDCONTROL_IPLOCAL in control node script log-run-l2node.tcsh to set the ip address of the TED control node you want to select on the decision node, which is then passed to the Communicator object (instantiated in NodeControl) via an alternate constructor. 
 
b0quad01 is running with old code, b0quad02 has above changes implemented. For future tests I suggest to switch forth and back between those nodes. 
</pre>
<!-- Author: Pedro (x2164) -->


<!-- Date: Wed Apr 9 00:30:13 2008 -->
<pre>L2 tests: 
 
* Control spare nodes b0l2cntl00 and 01 seem to work smoothly during latest end of store run.  I ran TED on both machines for 80-90min each and issued various RC commands to test response. No problems in transmission of  RC directives and prescale changes to decision node  b0quad01 observed. (Once the transition from ACTIVE to HALT stuck, but the reason might be unrelated to the new nodes since it was previously observed with b0felix as control node due to a failure of RTserver.) 
 
* No end-of-run crashes (due to SIGSEGV) observed during test, differently from the days before. Changed l2node.exe to invoke default gcc signal handler (instead of user defined version) in order to produce core dump useful for debugging, in case of future crashes.  The modified l2node.exe version is currently running as default.  Will switch back to previous version if major problems occur. 
 
</pre>
<!-- Author: Pedro -->


<!-- Date: Fri May 9 17:51:57 2008 -->
<pre>For the normal data taking I'd like to switch the L2 control process TEDCpp  from b0felix to the new spare b0l2cntl00, since both new spares (b0l2cntl00 and b0l2cntl01) have proven reliability in many tests (in cosmics and end-of-store runs). Please let me know if you agree. 
 
1. As on b0felix, the run script is 
       /cdf/code-L2/tld/current/l2pulsar_ted/src/runTEDCppTest.sh, 
 
and the log file is in 
 
       /cdf/code-L2/tld/current/log/*. 
 
To start the control process on the new spares: 
 
     ssh l2@b0l2cntl00 
     sudo xemacs /etc/inittab   and uncomment the last line 
     sudo /sbin/telinit Q 
 
Make sure that TEDCpp is not running on the other nodes. The process receives the IP address(es) of the active L2 decision node(s) via smartsockets messages (initiated by rc). In the update it writes the IP address(es) into  /tmp/.cntl_lock for ProcMon checks. 
 
2. Since the control process l2node.exe on l2quad01/02 is not connected to the RTserver, one has to manually tell the IP address of the active control node by  editing the variable "ted_ip" on top of the run script   /root/run/official/log-run-l2node.tcsh: 
 
   set ted_ip= 192.168.0.1  for b0felix 
   set ted_ip= 192.168.0.20 for b0l2cntl00 
   set ted_ip= 192.168.0.21 for b0l2cntl01 
 
and execute  "killall -INT l2node.exe" or alike to restart the decision process.  The process writes the IP address to /tmp/.dcsn_lock  for ProcMon checks. 
 
3. ProcMon is updated to complain if the communication between L2 control and decision is not setup correctly. 
 
4. The above changes also require to move the location of the trigger libs to b0home:~cprod area so they can be accessed via rsync server over the new control nodes. On the new control node, the location of the libs is (see L2-CAL path in /etc/rsyncd.conf) 
 
    ~/cdfprod/TriggerDB/triggerfiles/l2share 
 
differently from b0felix where L2-CAL points to a local directory /cdf/code-L2/build-l2cal. 
 
5.  I manually copied the trigger libs from b0felix to this new location and also updated and tested the L2 trigger build scripts to copy new builds to this new location, see 
 
   ~cdfprod/TriggerDB/current_UPDATE. 
 
When we switch to the the new control nodes we also need to switch to the updated build version to be consistent. 
 
If you don't mind I'd like to make the above changes this Monday between stores. In this case also  the ace L2 help page  has to be updated. 
 
Please let me know if you have comments or suggestions. Thanks.</pre>
<!-- Author: Pedro -->
<!-- Comment: Mon Jun  9 23;25;14 comment by...Pedro -->
Regarding points 1+2, please use new start stop utilities "l2dec" and "l2con" instead described below.
<!-- Date: Fri May 9 18:15:42 2008 -->
<pre>(Apr 28, 2008) 
 
I updated and tested ProcMon ($FER_DIR/src/{procAction,procActionHandler}) to check the configuration of  L2 control and decision nodes so we can use the new spare nodes during normal data taking w/o triggering a false ProcMon alarm. 
 
The new script scans the five available L2 control/decision nodes as user "ace" via the local network (handled by ssh-agent) and checks the overall configuration. There are more checks necessary than in the previous version (for example whether the right control and decision nodes are listening to each other) since we want to have hot spares connected to the network and the flexibility to switch between them w/o changing hardwired ProcMon settings. 
 
These are the L2 control/decision node checks performed by ProcMon: 
 
 1. Exactly one L2 control process is overall running 
 2. At least one L2 decision process (max. one per node) is overall running  
 3. At least one L2 decision process is setup to connect to one responding L2 control node, 
 4. Connected nodes must provide valid responses to this script (includes validation of ip addresses from the control and decision process lock files) 
 
 As long as the above requirements are met, 
  - more than one decision node processes listening to the same L2 control are allowed, 
  - one or more control and decision nodes may be disconnected, 
  - L2 decision processes ignored by L2 control are allowed, passive L2 control processes are prohibited. 
</pre>
<!-- Author: Pedro -->


<!-- Date: Tue May 20 11:00:01 2008 -->
<pre>Here the results for 5 wrong, but typical events, at high luminosity: 
Run:262490 
LumBegin:252 
 
For the following events every time we had a l2 decision TO or a Done TO for crate b0l2pu01, I put in HALT the system and dump what inside the Pulsars connected to the L2 PC (using script debug.py in B0L2PU01 directory under ACE home). The script dump what inside the Timing SLINK Merger to be faster. Unfortunately this information is missing for Tracklist. I analyzed 5 events. 
------ 
1) PC indicated mismatched between BC of Tracklist with Muon and l2CAL with MUON. The BC is fine for all L2CAL inputs and MUON. Also all the other paths are fine. 
------ 
2):PC indicated mismatched between BC of Tracklist with Muon and l2CAL with MUON.  The BC is fine for all L2CAL and MUON. XFT3 has different Bunch Counter for all the 4 Buffers. 
------ 
3): Done TO for b0l2pu01. XFT2 has 505 words and XFT3 has Bunch Counter mismatch for Buffer 2. 
----------- 
4): L2 Decision TO, no message from L2 log file. XFT3 has different Bunch Counter for all the 4 Buffers. 
------- 
5)    L2 Decision TO, no message from L2 log file. XFT3 has different Bunch Counter for all the 4 Buffers. 
------- 
 
IN CONCLUSION: 
- We probably need more statistic BUT: 
1. The Bunch Counter mismatches indicated for L2CAL/MUON in the PC log file doesn't seem to be correlated to the Pulsar side, I suppose data got corrupted once they are in the FILAR/PC. This is one typical error causing L2 Decision TO and then HRR. 
2. We need in any case info for Tracklist Board 
3. Most of the time XFT channel 3 has Bunch Counter mismatch in the pulsar side. This effect maybe is correlated to the high latency indicated by the L2latency plot. 
--------------------------------  
 
</pre>
<!-- Author: Laura -->
<!-- Comment: Tue May 20 11;07;46 comment by...Laura -->
<pre>When we had Done TO for b0l2pu01 the number of words for XFT2 was 505, very close to the limit set by the event truncation. But we need more statistic and the trigger table including the printout to be sure.</pre>


<!-- Date: Wed May 21 00:35:58 2008 -->
<pre>To make the handling of the decision and control executables (including /etc/inittab business) easier I have provided the safe start/stop utilities  "l2dec" and "l2con". 
 
L2 decision (on l2quad01/02):  /root/l2dec 
-------------------------------------- 
 
usage:  
l2dec [start|restart|stop|status] [b0l2cntl00|b0l2cntl01|b0felix] [option] 
l2dec [start|restart|stop|status] (default=b0l2cntl00)            (default=CURRENT) 
 
action examples: 
* l2dec  - usage info  
* l2dec status  - shows run status  
* l2dec start  - starts l2node.exe default version=CURRENT, listening to default TED node b0l2cntl00  
* l2dec restart b0felix  - restarts l2node.exe, listening to b0felix  
* l2dec restart b0l2cntl01 DEBUG - restarts l2node.exe with version=DEBUG, listening to b0l2cntl01  
* l2dec stop - stops l2node.exe 
 
 
L2 control (on b0l2cntl00/01):  /cdf/code-L2/tld/current/l2pulsar_ted/src/l2con 
--------------------------------------------------------------------- 
 
usage: 
l2con [start|stop|restart|status] 
 
action examples: 
* l2con  -  usage info 
* l2con status - shows run status 
* l2con start  - starts TEDCpp 
* l2con restart - restarts TEDCpp 
* l2con stop - stops TEDCpp 
 
(not supported on b0felix) 
</pre>
<!-- Author: Pedro -->


<!-- Date: Wed May 21 13:14:25 2008 -->
<pre>Problems with filar 3 on l2quad02 resolved (for now): 
----------------------------------------------------- 
On Friday, May-16, we observed the first time on l2quad02 that the executable wasn't able to setup filar 3: 
 
  error: 0x809 =&gt; major: Error9 in package8 =&gt;  
  IO RCC library: PCI Device not found 
 
The problem persisted until Tuesday, May-19. The same error can be provoked with other functioning filars by disconnecting them, thus confirming that the problem is related not the filar itself but the connection with the latch. (Also inserting a new card didn't help.)  It turned out that heftily joggling the filar in the slot helped.    
 
The advice for the future is to repeat this procedure with the affected filar if the function "filar_setup(&lt;filar number&gt;)" contained in NodeAlgo.cc exactly produces the above error message (with emphasis to "exactly" - don't try this if another kind of module error happens). </pre>
<!-- Author: Pedro -->


<!-- Date: Thu May 22 07:30:36 2008 -->
<pre>Run 262565. Trigger table tag 1089:printout of the object in TL2D every time the TL2D exceed the maximum number of word available in Pulsar board DAQ RAM. 
Here the printout for few events: 
 
 WARNING FROM THE DECISION MAKER! TL2D LARGER THAN 1020 words  
 N Jets07 = 12 
 N Jets04 = 21 
 N High E Photon = 0 
 N High E Photon pass 1 = 0 
 N Low E Photon = 12 
 N taus = 4 
 N tracks = 212 
 N SVT = 157 
 N Muon = 178 
-------------- 
 WARNING FROM THE DECISION MAKER! TL2D LARGER THAN 1020 words  
 N Jets07 = 10 
 N Jets04 = 15 
 N High E Photon = 1 
 N High E Photon pass 1 = 2 
 N Low E Photon = 23 
 N taus = 4 
 N tracks = 212 
 N SVT = 161 
 N Muon = 162 
----------------- 
 N Jets07 = 10 
 N Jets04 = 14 
 N High E Photon = 0 
 N High E Photon pass 1 = 0 
 N Low E Photon = 24 
 N taus = 1 
 N tracks = 231 
 N SVT = 151 
 N Muon = 142 
---------------- 
 WARNING FROM THE DECISION MAKER! TL2D LARGER THAN 1020 words  
 N Jets07 = 12 
 N Jets04 = 16 
 N High E Photon = 0 
 N High E Photon pass 1 = 0 
 N Low E Photon = 15 
 N taus = 3 
 N tracks = 231 
 N SVT = 171 
 N Muon = 146 
-------------- 
 WARNING FROM THE DECISION MAKER! TL2D LARGER THAN 1020 words  
 N Jets07 = 12 
 N Jets04 = 18 
 N High E Photon = 0 
 N High E Photon pass 1 = 0 
 N Low E Photon = 17 
 N taus = 3 
 N tracks = 220 
 N SVT = 169 
 N Muon = 164 
------------- 
and so on for 6-7 times. 
 
The log file is /log/l2node/l2node.Wed_May_21_13:48:09_CDT_2008_DEBUG. 
Look for "TL2D" 
We already known this message of big TL2D is associated to the b0l2pu01 Done TO.</pre>
<!-- Author: Laura -->
<!-- Comment: Thu May 22 08;14;36 comment by...Laura -->
<pre>This is the event with the maximum number of word in the TL2D found in this run (1090):
--------------
NodeAlgo:: { 4:11:38, 5/22/2008 } Waiting for MUON RECES TRACKLIST L2CAL 1 L2CAL 2 L2CAL 3 L2CAL 4 XFT 1 XFT 2 XFT 3  packet(s). Waited 5005 ms 
DataImp:: Packet warning | XFT02 event truncated after 555 - 5 SLINK  = 550 words. trailer_status = 2 word = 22b0002
DataImp:: Packet warning | XFT01 event truncated after 555 - 5 SLINK  = 550 words.  trailer_status = 2 word = 22b0002
DataImp:: Packet warning | XFT02 event truncated after 555 - 5 SLINK  = 550 words.  trailer_status = 2 word = 22b0002
 WARNING FROM THE DECISION MAKER! TL2D LARGER THAN 1020 words 
 N Jets07 = 12
 N Jets04 = 18
 N High E Photon = 0
 N High E Photon pass 1 = 0
 N Low E Photon = 17
 N taus = 4
 N tracks = 245
 N SVT = 191
 N Muon = 130
 In this event the packet lenght is = 1090
DataImp:: Packet error | XFT02 Trailerstatus  is not good (0xeeee) 2191 word = 68d088f
NodeAlgo:: { 4:12:9, 5/22/2008 } Error in setXftPointers [Nerr=1, Nev =36782297]  XFT 3 data is corrupt 
NodeAlgo:: { 4:12:9, 5/22/2008 } Error[Nerr=1, Nev =36782298]  XFT 3 filar error: 1
----------------------------------------
</pre>


<!-- Date: Thu May 22 13:09:26 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-963&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-964  width='300' align='top'></a></td><td>Number of SVT tracks</td></tr></table>
<!-- Author: Laura -->


<!-- Date: Thu May 22 13:17:00 2008 -->
I am testing updated L2 trigger build scripts. During this time it won't be possible for other to create new builds. Please let me know if you need to work on trigger tables, I will the restore the old configuration.
<!-- Author: Pedro -->
<!-- Comment: Thu May 22 19;22;42 comment by...Pedro -->
<pre>Tests are complete, reset L2 build to old configuration.</pre>


<!-- Date: Sat May 24 12:50:53 2008 -->
<pre>Printout and Dump combined to debug TL2D corruption. 
Please note that "N SVT" has to be intended as Number of  words 
---------- FIRST EVENT 
Printout: 
 N SVT = 171 
 N Muon = 182 
 In this event the packet lenght is = 1036 
------------------------ 
Dump 
MUON DAQ:192 Words- 10 words (SLINK trailer/header+L1 trigger bits)=182 
SVT DAQ:178 words-7 words(SLINK trailer/header)= 
 
-------- SECOND EVENT 
Printout: 
 N SVT = 175 
 N Muon = 126 
----- 
Dump 
MUON DAQ:136-10=126 
SVT DAQ:182-7=175 
 
-------THIRD EVENT 
Printout: 
 N SVT = 169 
 N Muon = 130 
----- 
Dump 
MUON DAQ:140-10=130 
SVT DAQ:176-7=169 
 
 
 
 
 
</pre>
<!-- Author: laura -->


<!-- Date: Mon Jun 9 23:21:01 2008 -->
(May 18th, 2008)  Time sync problem on decision nodes solved by re-installing ntpd.
<!-- Author: Pedro -->



<!-- Date: Mon Jun 9 23:37:46 2008 -->
(May 25th, 2008) <a href=http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=l2&action=copyentry&page=58&time=13:14:25> Concerning l2quad02 filar problem described here</a>: 
<br><br> 
Replacing filar 3 on l2quad02 (leftmost filar when looking at the backside of the PC) solved the problem.  Still need to check whether it is a filar or a connection problem. The socket is staggered w.r.t. the slot, which requires the filar board to be critically bent to allow connection with the fibers. As a consequence the configuration is under permanent mechanical tension and thus probably unstable. 
<br><br> 
I tested l2quad02 various times since then (L2TORTURE, COSMICS runs) - problem has disappeared. Also no PC freezing observed anymore. Spare decision seems to be remarkably stable now. 
<!-- Author: Pedro -->


<!-- Date: Mon Jun 9 23:50:23 2008 -->
(May 27th, 2008)  Updated L2 trigger build scripts to place target libraries on public network area instead of local b0felix disc. Builds are now directed to b0home:~cprod/TriggerDB/triggerfiles/l2share. Old libraries were manually copied to new area. For more information, see ~cprod/TriggerDB/current/{BuildL2.pl,BuildL2Filar.pl}.  
<!-- Author: Pedro -->


<!-- Date: Tue Jun 10 00:01:39 2008 -->
Configured L2 private network to allow old control node b0felix to be disconnected such that communication between the remainig nodes is still intact (a previous naive attempt caused  connection loss between b0l2cntl00 and l2quad01). This requires  disabling IP forwarding and  removing 192.168.0.1 (aka b0felix) as default gateway on the decision nodes, see /etc/conf.d/net.  
<!-- Author: Pedro -->


<!-- Date: Tue Jun 10 00:47:57 2008 -->
We started with dry installation exercises on backwinter (192.168.0.4) to gain experience for the new decision PCs that Laura ordered.  Ben upgraded the Gentoo version to be consistent with the version on l2quad01/02. We managed  to install the filar drivers (contained in /root/run/official/l2pulsar_slink) into the kernel and test the functionality using one of the spare filar boards.  The module was indeed successfully loaded and the device  detected by our test program, so it seems that we understand the installation procedure at least up to this point.
<!-- Author: Ben & Pedro -->


<!-- Date: Wed Jun 11 11:25:35 2008 -->
<pre>In an effort to track down the source of the high L2 latency in XFT 02 [3] we swapped the XFT 02 [3] (slot 8) and XFT 01 [2] (slot 6) pulsars in b0puls02 yesterday.  
TrigMon sees nothing unusual about the data during these periods of high latency. The recent appearances of the "Tracklist aborted sending XFT data when it should not have!!" message in the L2 log do appear to be correlated with the periods of high latency, this suggests that there may be a problem with something between the pulsars and the L2 PC.  
Swapping the two pulsars will allow us to determine if the high L2 latency is related to the pulsar board that was in slot 8 or if it is related to something else. The problem should be visible in two ways, either in the L2 log or the L2 latency plots available via the run summary. In the L2 log the "Tracklist aborted sending XFT ..." messages are preceded by a truncation and a data corruption in the problematic pulsar. In the L2 latency plots the problem has manifested in the past as XFT 02 [3] experiencing frequent high (&gt;40 mircoseconds) latency during running with high luminosity (&gt; 200x10^30 cm-2sec-1) 
</pre>
<!-- Author: David Cox & Alison -->


<!-- Date: Wed Jun 11 15:00:45 2008 -->
COT08 Slot13 has been masked on since November due to problems with the L2 path.  I unmasked the XTC and still saw <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/elog.pl?nb=2008&action=copyentry&page=488&time=14:21:41.393'> problems </a>.  See other entries from day shift today for more info.  After the test, I masked the XTC back on.  
<!-- Author: Heather -->


<!-- Date: Thu Jun 26 11:55:50 2008 -->
<pre>Swapped XFT 2 & XFT 3 pulsar transition cards. If the transition cards were causing the high XFT 3 latency we should see high XFT 2 latency with the next high luminosity store. </pre>
<!-- Author: David & Alison -->


<!-- Date: Thu Jul 3 12:04:21 2008 -->
<pre>Put spare XFT pulsar in slot 10 of bol2pu02. It isn't setup to properly pass timing info on yet.</pre>
<!-- Author: David & Olga & Laura -->


<!-- Date: Thu Jul 10 00:33:03 2008 -->
<pre>I changed the unused XFT 03 [4] Pulsar in the card database to correspond to the pulsar put in slot 9-10 of b0l2pu02. 
Changes: 
  Slot           11 -&gt; 10 
  NCHANNELS      12 -&gt; 4 
  ENABLEDCHANNEL 63 -&gt; 34* 
 
This one I'm fairly uncertain if it is correct.</pre>
<!-- Author: David Cox -->


<!-- Date: Tue Jul 15 12:26:54 2008 -->
<pre>Settings for XFT03 were incorrect. 
 
NCHANNELS = 4 -&gt; NCHANNELS = 12. 
 
I'll check the banks as soon as a new run's available.</pre>
<!-- Author: David -->


<!-- Date: Mon Jul 21 11:37:53 2008 -->
<pre>Swapped in different spare fibers for XFT 04. Now taking inputs from Finders from 1,4,7, and 10.</pre>
<!-- Author: David -->


<!-- Date: Thu Jul 24 10:22:04 2008 -->
<pre>Swapped in last set of spare fibers into XFT 4. Now taking data from 3,6,9,12. The spare fiber from 9 was marked with a black x.</pre>
<!-- Author: David -->
<!-- Comment: Thu Jul 24 11;21;28 comment by...David -->
<pre>TP2D bank for the x marked fiber doesn't look right so I've masked that fiber input on in the database, I'll keep an eye on it.</pre>

<!-- Comment: Sun Jul 27 08;05;34 comment by...David -->
<pre>I´ve now gathered enough data to conclude that XFT 4 does not show the high latency with just the three fibers enabled. I have reenabled the input. Will pickup the new settings during this morningś planned shot setup.</pre>


<!-- Date: Wed Aug 13 17:09:00 2008 -->
<pre>I've switched XFT 4 to enable the abort. XFT 4's configuration now matches that of XFT 3 in everything except it's read out list (which is 6 instead of 2.).</pre>
<!-- Author: David -->
<!-- Comment: Tue Aug 26 16;05;04 comment by...David -->
I've changed XFT 4 to read out list 2


<!-- Date: Fri Sep 5 03:46:39 2008 -->
<pre>I swapped the fibers leading from XFT 1 to the XFT SLINK Merger with the fibers leading from XFT 4 to the spare SLINK merger slot. </pre>
<!-- Author: David -->


<!-- Date: Mon Sep 8 14:59:27 2008 -->
<pre>We swapping b0xft06 slot 19 with slot 18. 
Slot 19 was the one causing the bunch counter mismatches at L2 (but the L1 data agrees with the L2 data). 
The problem in the L2 log-file was in [1][10] and if it moves to slot 18 it should appear as [1][9] 
</pre>
<!-- Author: David Cox + Alison  -->


<!-- Date: Tue Sep 23 10:55:38 2008 -->
<pre>b0xft06 slot 18 swapped with spare. 
Firmware downloads without problems. 
Shepherd the crate once in run control and take ~2000 events of shot setup without problems.</pre>
<!-- Author: David / Alison -->


<!-- Date: Tue Sep 30 10:25:58 2008 -->
<pre>I moved the PC and spare PC fiber connections to XFT 4 from XFT 3. I also changed swapped the sequence in which both pulsars are read out.</pre>
<!-- Author: David -->


<!-- Date: Fri Oct 3 13:47:40 2008 -->
<pre>The L2 PC is again receiving data from XFT 4. 
A test run was taken with the abort ignored from the pulsars. The data came into TrigMon was seen by XFTL2Monitor and no problems were observed.</pre>
<!-- Author: David -->


<!-- Date: Mon Nov 3 14:02:20 2008 -->
<pre>Swapped XFT pulsar -&gt; merger/PC inputs. 
 
XFT 1 now sends its merger inputs to where XFT 3 was. (XFT_2_2). 
XFT 3 now sends its merger inputs to where XFT 1 was. (SLINK_1_1). 
XFT 4 now sends its merger information via the slave output. 
XFT 4 now sends its information to the PC via the 2nd master output (was the 1st). 
XFT 4 now sends information to the Spare PC via the 1st master output (was the slave output).</pre>
<!-- Author: David Cox -->
<!-- Comment: Mon Nov  3 14;54;50 comment by...David -->
<pre>XFT 4 now sends its information to the Main PC via the 1st master ouput. (was the 2nd master)
XFT 4 now sends its information to the Spare PC via the 2nd master output. (was the 1st master)</pre>


<!-- Date: Tue Nov 18 17:33:41 2008 -->
<pre>Thu, 11/13/08: 
-------------- 
 
1. I tried to do a l2quad01/l2quad03 parasitic L2TORTURE run between stores. At the first attempt,  l2quad03 behaved asynchronously and reported many Bunch/Buffer Mismatches (also extra for XFT). Later the PC froze (didn't find useful info in /var/log/messages), so I rebooted it. After that I ran a very short L2TORTURE between stores for 2 minutes until protons were being injected. No complaints. I first assumed that the changed smp affinities (which are reset during boot) were responsible for the mess before, later I figured out that was probably not the reason. 
 
Fri, 11/14/08: 
-------------- 
 
2. Tried a few parasitic runs with l2quad01/l2quad03 using L2TORTURE/COSMICS/PHYSICS (AAA_SHOTSETUP) to establish that the configuration runs stably for a longer period (had also an eye to DataSync and TrigMon Errors). Very smooth runs. Did some HRR to provoke and test the system's response. Generally no problems here, except for very, very few cases during L2TORTURE where at HALT and RECOVER both nodes waited just for L2CAL03 or SVT (instead for all Pulsar inputs) and then further waited for those inputs during ACTIVE, which lead to L2TO. HRR helped to solve this. 
 
3.  I tested the behavior of the parasitic setup with l2quad01/03 when assigning the SMP affinities for the IRQs of l2quad03 to the first four cores, i.e. setting IRQs 1 4 8 9 12 14 21 22 23 1287 to 0x0F, which was the setting before the crash reported in 1. I did so in a PHYSICS and a L2TORTURE run. However,  no problems seen here. PC crash yesterday must have had another reason. 
 
4. I studied the total dead times during L2TORTURE in both parasitic and non-parasitic runs. In runs where l2quad03 is involved, the dead time is around 9.4%. In runs where l2quad03 is not included, the typical dead time is around 5%. Note that the typical Acc rates for L2TORTURE are L1Acc~39kHz, L2Acc~244Hz in BOTH cases. Not sure about performance conclusions for Physics runs when the L2acc are the same but the dead times are different. 
 
5. I studied whether the PCI bus &lt;-&gt; FILAR channel assignments can be modified/optimized and whether they have any impact on the L2TORTURE dead time.  Changed the l2quad03 FILAR order (as seen at  the backside of the PC)  from  (F1 F0 F2 S) to (F1 F2 F0 S) (and generated an executable version able to handle this order, l2node_pmf17.exe), where 
 
 S  = SOLAR 
 F2 = SVT MUON RECES TRACKLIST 
 F1 = XFTS0 XFTS1 * CALM3 
 F0 = CALM0 CALM1 CALM4 XFTS2 
 
On the old l2quad01/02, the order is (F2 F1 S F0) and F2/F1 and S/F0 share the same bus. Assuming that adjacent boards 1/2 and 3/4 share also the same bus on l2quad03, the new configuration on l2quad03 would then have the same traffic balance like on l2quad01. However, it turned out that the new configuration doesn't have any impact on the total dead time at all. 
 
6. a) 'taskset' tells me that the l2node.exe SMP affinity on l2quad03 is 0xFF, but according to the code it actually should be 0x03 - strange. That means that the control and the algo threads are NOT bound to specific cores (not sure). Or it might be that 'taskset' doesn't see how individual threads of a process are assigned to cores. Don't know yet how to check the affinities on l2quad01/02 where taskset isn't implemented. I am thinking that a failed smp affinity assignment might in principle contribute to differences in the dead time. 
 
b) The actual l2quad01/03 algo time differences cannot explain the dead time differences observed above, since for a L2 decision based on the L1 trigger bits that actually fired, the algo times are of O(1 micro second) on BOTH l2quad01/03, and the double peak affects only O(few %) of the events (these numbers are from offline test). This is different from the plot I showed in the meeting where the effect is dramatically and artificially enhanced because I set all L1 trigger bits to 1. 
 
7. I cabled the fibers back such that l2quad02 is ready for projected parasitic runs, and did a test run. Strangely, the XFT3 slot LED is not green - it apparently isn't receiving any data and caused permanent  L2TO. But XFT3 works fine with l2quad03.  So is there something wrong with the filar board? 
</pre>
<!-- Author: Pedro -->


<!-- Date: Tue Nov 18 17:43:52 2008 -->
<pre>Dan and I started debugging the problem with the slink connection of XFT3 into the spare PC reported in item 7 above (in parasitic run with l2quad02 the channel would not come up).  We put the filar in a loopback test today and it passed.  Then we put it back in l2quad02 and intermittently could get the link to come up via a filar reset.  Moving the inputs from one channel to another did not produce conclusive results.  Suspect that the problem is either bad cabling on the XFT aux card, a dirty fiber, or a loose PCI connection on the decision node, but it might be something else.</pre>
<!-- Author: Pedro and Dan -->


<!-- Date: Tue Nov 18 18:46:17 2008 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>1) For future reference, this is a short summary of the new directory structure on the decision nodes l2quad01-04: 
 
 /root/build/my_version     ... L2 decision executable sources 
 /root/run/official/bin     ... L2 binaries 
 /root/run/official/scripts ... L2 scripts invoked by executable and aces/experts 
 /root/test/offline         ... L2 offline test utilities 
 /root/test/filar           ... L2 filar test utilities 
 /cdf/trigger               ... L2 algo libraries fetched by executable 
 /log/l2node                ... L2 log files 
 /build-l2cal               ... L2 compile location used by (future) triggerGui/build script 
 
2) How to compile/install a new executable version: 
 
  cd /root/build 
  cp -rp latestdir newdir  
  cd newdir 
  [edit your changes] 
  make all 
  make install 
 
The last step copies the binaries l2node.exe and libl2algo.so.1.0.0 to  
 
  /root/run/official/bin/l2node_newdir.exe 
  /cdf/trigger/libl2algo_newdir.so.1.0.0, 
 
respectively. The directory name serves as tag for the binaries. (Note that the local algo library libl2algo_newdir.so.1.0.0 is only meaningful if the executable is compiled with the "local=yes" option - only for tests. Usually the the algo library  is fetched based on run control directives to L2 control.) Please read the master Makefile for more options. 
 
3) How to run a new executable version: 
 
 l2dec start control_node_name newdir 
 
where "newdir" is the tag generated in step 2).</pre></td></tr></td></table>
<!-- Author: Pedro -->


<!-- Date: Wed Nov 19 14:48:32 2008 -->
<pre>We tested different configurations of the slink and filar between XFT3 (slot 10 on b0l2pu02) and the spare PC (l2quad02).  The error is difficult to reproduce, but we find that we never saw the link go down in a configuration other than the left slink mezzanine on the slot 10 aux card with the XFT3 filar on the spare.  We verified that the problem is definitely NOT the fiber.  We have exchanged the aux cards between slot 10 and slot 8 on b0l2pu02 and taken a short run.  No problems were seen in this run.  </pre>
<!-- Author: Dan, Laura, and Pedro -->


<!-- Date: Sat Dec 6 18:55:36 2008 -->
<pre>The L2PC has unable to get past "Waiting for XFT 1". 
 
I've tracked this to XTC2/TDC COSWT3 and masked it on. (SL3, cells 128-135). 
 
Details on the process to identify the problem XTC: 
 
Step 1) The PC being unable to get past waiting for one of the XFT pulsars is our primary symptom of this type of problem (XTCs that need to be masked on). This tells me I need to check if we have a problem XTC. 
 
Step 2) Per instructions documented at  
 
http://www-cdf.fnal.gov/internal/people/links/AlisonLister/xft/L2_XTC_problems.html#mask 
 
 We're not getting any data so I can't use existing TrigMon to identify the problem finder, and I can't see any bunch / buffer mismatches in the L2 PC despite repeated HRRs so we switch to ignoreL2Stereo. 
 
Step 3) In TrigMon plot XFTL2Monitor_T2FD_Header_Format_Errors 
I see that there are problems in crate 7 slot 7. 
 
Step 4) I use the XFT Map java applet at: 
 
http://www-cdf.fnal.gov/~nils/xft/AppletFinal.html 
 
  to identify what two XTCs correspond to b0xft07 slot 7. 
 
Step 5) I mask on one of the XTCs (COSWB1) to test if it is the source of the problem, using the deadwires given me by the java applet and the instructions on masking dead wires available at  
 
http://www-cdf.fnal.gov/~nils/xft/deadwire.html 
 
Step 6) I turn off ingoreL2Stereo. The L2 PC is still hanging waiting for XFT 1.  
 
Step 7) Back to the java applet get the deadwires for the other XTC inputing to b0xft07 (COSWT3). Mask it on. 
 
Step 8) Try running with ignoreL2Stereo. The L2 PC is no longer hanging. 
 
</pre>
<!-- Author: David -->
<!-- Comment: Mon Jan 12 17;47;25 comment by...David -->
<pre>Problem is still present.</pre>


<!-- Date: Wed Feb 4 19:02:48 2009 -->
<pre>Details of tests regarding whether the fiber / finder was causing the inability to run without cells 128-131 masked. 
 
The first thing we did was swap the fibers for b0xft07-06F and b0xft07-07A on the side of the board where they are coming from the step before the finder. (ie we did not move the fiber that makes the last stage move to the actual finder board but rather the fiber before that). With this change we observed no difference in the behavoir of the problem. It still would not run withou cells 128-131 masked and it still identified b0xft07-07 as the board causing the problem. 
 
Then we restored those fibers to their original position and swapped b0xft07-07 with b0xft07-08. We were unable to run and saw errors with both b0xft07-07 and b0xft07-08. 
 
For b0xft07-08 we saw problems that manifested in the error log as (Note: SL3 cell 148-151 correspond to finder input A of b0xft07-08.): 
(SL;cell;subcell) = (3;149;4) T2FD data and simulation slope bits don't agree 
(SL;cell;subcell) = (3;149;5) T2FD mask expectation and XSFD data slope don't agree: 
(SL;cell;subcell) = (3;149;5) T2FD data and simulation slope bits don't agree 
For b0xft07-07 we saw problems that manifested in the error log as (Note: (1; 4) corresponds to the finder board in b0xft07-07): 
(1;4) Bunch Counter problem: Chip A (0), chip B (0), first card (79), first card previous event in buffer 0 (70) (event 9490 ) 
(1;4) The header bits 13-15 or 30-31 are not set correctly: 
 Bits 13-15 = 0 Bits 30-31 = 0 
(1;4) T2FD Couldn't find trailer 
(1;4) SL 3 & number of words expected from header not equal to 6 
(1;4) word = 5 T2FD: Mask words don't agree between data and simulation 
data = 00000000000000000000000000000000 
sim  = 00001111110000000000110000000000 
(1;4) T2FD data vs sim: Error in mask words 
 
   We masked on cells 148-151. We remained unable to run and we continued to see problems. 
 
   We restored the finders to their original position. 
 
   We were unable to run w/ ingoreL2Stereo continued to see errors in b0xft07-07 of the form: 
(1;4) Bunch Counter problem: Chip A (0), chip B (0), first card (2), first card previous event in buffer 0 (19) (event 12940 ) 
(1;4) The header bits 13-15 or 30-31 are not set correctly: 
 Bits 13-15 = 0 Bits 30-31 = 0 
(1;4) T2FD Couldn't find trailer 
(1;4) SL 3 & number of words expected from header not equal to 6 
 
   We masked on cells 128-131 and were again able to run without errors. 
 
   We conclude that the source of the error is not the fiber we swapped nor the finder board itself. It may be the finder which makes the final stage connection or it may be something connected to slot 7 of the b0xft07 crate. 
 
 
 
</pre>
<!-- Author: David,. Charles and Will -->
<!-- Comment: Thu Feb  5 10;24;44 comment by...David -->
<pre>The plot determining which finder was in error after the fiber swap was misread. The problem moved with the swapped fiber.</pre>


<!-- Date: Tue Feb 10 15:59:53 2009 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-965&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-965' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-966&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=l2&action=view&page=-966' width='300'></a></center>
<!-- Author: Dan and Pedro -->
<!-- Comment: Tue Feb 10 16;02;54 comment by...Dan and Pedro -->
<pre>Plot on the left is for parasitic run with l2quad03 (8 cores) and on the right for parasitic run with regular spare l2quad02 (4 core).  The run configuration was shot setup.  We observe a shift of ~ 8mus for the 8-core system.  Note that this shift is roughly of the order of the difference in the clock speed between the 4-core and 8-core computers.  We will next take some EOS data to check the timing with more realistic data.</pre>