<!-- Title: 05:37:39  Wed Jan 26 2005 -->
<!-- Crewchief: Pulsar folks -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Getting ready for the first beam test (since last shutdown).</pre>


<!-- Date: Wed Jan 26 06:05:52 2005 -->
<pre>Tonight, we have done some extensive tests for the Pulsar system 
in parasitic mode, in beam. We have learned alot, and here is a short summary: 
 
 
* At "high" luminosity or L1A above 20KHz: 
 
Using current official version of decision CPU software(with full algorithm running with full system), we found 4 types of problems (from software error checking in decision CPU code):  
 
(1) slink internal error bit on (already known) every ~ 10M events or so; 
(2) Start Control Word missing around every ~1-10M evts; 
(3) Trailerstatus word wrong (not 0xeeee), but event dumpe shows 
that the event itself (for non_SVT data) looks just fine, indicating that it is the package size wrong. 
(4)partial event from event dump. 
 
We then copied over Daniel's code and started to play with the code, by disabling one piece of error checking (and command checking) code at a time. still found the same problems. 
At that point, we called Kristian and got him to compile the 
good old muon/svt code (beam tested) code from last summer (stand alone code). We got that code running, and for up to 60M events there was no error... the luminosity was lower then and the L1A 
rate was at about 20KHz. 
 
* At lower luminosity or L1A belkow 20KHz 
 
then we went back to the official code, and tried to reproduce the problems we found earlier. To our suprise, the code ran fine for up to ~ 40M events. The only difference we can tell is that the L1A was at about 18KHz. We repeated the test a few times,  
half of the time it can run for long time without problem (PulsarMON also shows that the decisions made all make sense as far as we can tell). but half of the time we got some of the errors we saw earlier within ~ 1M events or so. 
 
To make the long story short, it looks like at low luminosity or 
low L1A rate, it is possible that we may be able to run without problem for quite a while. This means that the first beam test at very low luminosity is still meaningful. However, it is quite clear that there are issues we need to better understand at high luminosity/L1A rate, and we should do more tests at the beginning of next store. 
 
 
The system is now back to the condition as 6pm last night when we started these extensive tests. We will go to get some breakfast now and will come back for the beam test around 8am.</pre>
<!-- Author: Vadim and Ted -->
<!-- Comment: Wed Jan 26 07;30;04 comment by...Vadim and Ted -->
<pre>By the way, during the tests last night, we have been using PulsarMon to monitor the Pulsar crate data. As far as we can tell, all data Path Pulsar data looks fine and all events are in sync (even at times when there is error reported on the CPU side). 

There is one error (occured only once) reported by PulsarMon,
showing that Reces Pulsar#2 channel 3 had error at its input and caused trouble to downstream. This weak channel was already identified by Vadim a while ago and it is splitted with active splitter. We plan to replace it with passive splitter soon (the error rate is very low, once in ~ a few days). 

</pre>


<!-- Date: Wed Jan 26 12:21:26 2005 -->
<pre>So we have done our first beam test this morning. The goals we were trying to achieve were: 
 
(1) partial trigger table (like last summer, xtrp/muon/svt) with 
    alpha or Pulsar driving; 
 
(2) full trigger table with Pulsae in parasitic mode with alpha  
    driving (we cannot do driving alone in this mode because we             
    still do not have working TL2D readout code). 
 
For the first goal, we took a few short runs... with alpha driving as well as Pulsar driving. In case of Pulsar driving,the algorithm CPU runs fine without problem. However, we noticed that the SVT processing time has a long tail. This also happens when Alpha was doing the driving with the same trigger table. We then tried to use the same code (beam tested last summer) to do the driving, the same thing happens (long SVT processing time). This is something we need to understand right away. One thing we can take a hard look at is the partial trigger table (as this happens to both alpha and Pulsar, the only thing in common is the trigger table). In addition, last summer standalone code (beam tested) was based on the L1 trigger bits at that time, this may have been changed since.  
 
 
Now for the full trigegr table running Pulsar in parasitic mode, the test was very successful. This was run 192912. In this run, we disabled DPS (again, becuase we do not have TL2D readout) for this run so that we can compare the decisions made by Pulsar with that of Alpha directly. We have logged 14K (L3A) events over 1.7M L1A events. The Pulsar algorithm CPU stayed in sync all through the run. PulsarMon was checking the decisions while running. From the PulsarMon log file, we already know that for most events, the decision bits match exactly between alpha and Pulsar. There were a few events with decision mismatch after prescale, but agree before prescale. We will run through all events offline soon.  
 
Our current plan is to take a hard look at the data today. we will take some longer parasitic runs at different luminosity range with full trigger table in the next few days. We will need to understand the problem with (1), and really push to get the TL2D readout code for Pulsar working in the next few days so that we can take another beam test with full trigger table with Pulsar dring and providing TL2D to L3. In addition, soon we need to fully understand the (few types) problem events flagged by algorithm CPU error checking code last night (at high luminosity parasitic running) before we can attemp any high luminosity run with Puslar driving. 
 
 
 
 
 
 
</pre>
<!-- Author: Ted for Vadim, Cheng-Ju, Daniel, Chris etc -->


<!-- Date: Thu Jan 27 09:54:24 2005 -->
<pre>Hi, 
 
Last night I took a long L2T run without CES data in the merger.  We ran for 117.5M events in just under 1 hour.  There was 1 single error during that time, a filar input error.   
 
The new code in the node follows the error spec -- it marks the error trigger on for this event but does NOT send L2DTO unless it gets 3 in a row.  Below is the capture of the nonsvt data from the event.  I don't see anything obviously wrong with it -- but this error could be due to flakiness in the event data itself. 
 
 
End Control Word Error: 0xe0f00001  (should be 0xe0f00000) 
NodeAlgo:: { 13:14:8, 1/1a/7d5 } Non-SVT filar error: 8 
Data dump----- 
Non svt data is:  
Data is 21 from ptr = 0x640650 
0       0x1800134 
1       0x4c004c 
2       0x1100134 
3       0xa 
4       0x2 
5       0x0 
6       0x20e1f8 
7       0x221228 
8       0x229248 
9       0x209290 
10      0x2542a0 
11      0x2206d8 
12      0x235758 
13      0x60014d 
14      0xaeeee 
15      0x1300134 
16      0x220022 
17      0x0 
18      0xb0f00000 
19      0x1400134 
20      0x1b001a 
21      0x3412 
22      0x1eeee 
23      0xe0f00000 
24      0xb0f00000 
25      0x1500134 
26      0x130013 
27      0xeeeeeeee 
28      0x1eeee 
29      0xe0f00000 
30      0xdeeee 
31      0xe0f00000 
32      0x1eeeee 
Svt data is:  
Data is 0 from ptr = 0x642660 
End data dump --- 
NodeAlgo:: { 13:14:9, 1/1a/7d5 } event: 51500103         decision-&gt;._L2DecisionBeforePrescale[0]: 0      decision-&gt;_L2DecisionAfterPrescale[0]: 0 
</pre>
<!-- Author: Daniel Whiteson -->


<!-- Date: Thu Jan 27 11:10:40 2005 -->
<pre>Got paged last night about Pulsar crate pulling Done-Time-Out. 
I removed the Pulsar crate from the run (and also removed 
the crate from RC config files) while investigating the problem. 
It appears that the problem is due to the fact that the shift 
crew is still using the test version of RC from Jane's directory. 
I forgot to restart a default version of RC for the shiftcrew. 
I have asked the shiftcrew to restart RC.  Since Vadim and 
Sakari want to update the cluster firmware, I'll keep the 
Pulsar crate out until they are done.</pre>
<!-- Author: Cheng-Ju -->


<!-- Date: Thu Jan 27 18:48:57 2005 -->
<pre>System config 
 
I made 2 modifications to the decision setup to run without cluster 
 
in /root/run/official/l2node/src/NodeAlgo.cc 
- Commented out merger buffer/bunch checking (this required Clist slink headers) 
- Commented out algorithms 
 
We're running now in parasitic mode, and it's been many M of events. I don't want to interrupt it so we can see what kinds of errors we get. 
 
Daniel</pre>
<!-- Author: Daniel Whiteson -->


<!-- Date: Thu Jan 27 18:49:52 2005 -->
<pre>I've left the system in the following configuration: 
 
- Control node is running current fer version (NOT devel, where l2enable/l2decision are set) 
- Decision node is running latest code; 
    * error handling is as described in error note 
	- Most errors need 3 in a row before L2TO is sent 
	- Some errors (timeouts) send L2TO on first occurance 
   *  auto-respawn is turned OFF, so we can run in parasitic mode 
	- /root/run/official/nolog-run-l2node.tcsh     runs dumping to screen, no log file 
         - /root/run/official/log-run-l2node.tcsh          runs with log file generated 
- Trigger tables have been compiled for 
	PHYSICS_3_00, v2 [ l2tagsets = 560, 566, 571 ] 
          PULSAR_TEST, v4 [ l2tagsets = 571, 573 ] 
          L2_TORTURE  , v16 [ l2tagsets = 566, 561 ] 
 
- Known issues: 
	- Between runs, control & decision node are having problems clearing their ethernet sockets. It may require waiting for linux to clear the sockets before reconfiguring the next run. I'm trying to understand this error 
</pre>
<!-- Author: Daniel Whiteson -->


<!-- Date: Thu Jan 27 18:50:35 2005 -->
<pre>Please note that the previous 2 entries are reversed. The "modification" refers to the entry that follows it.</pre>
<!-- Author: Daniel Whiteson -->