<!-- Title: 05:29:30  Sat Aug 30 2003 -->
<!-- Crewchief: Pulsar folks -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
Getting ready to drop the MOAB (Mother Of All Boards) into beam...


<!-- Date: Sat Aug 30 06:58:48 2003 -->
 
** Tests to check Pulsar Muon path readiness for beam ** 
<p> 
 
Since we are burning lots of tax payer's money every second with beam on, 
we have tried our best to make sure that we have tested Pulsar 
hardware/firmware/software as much as possible in teststand mode as well as in non-beam mode, to make sure we are ready for the beam.  
<p> 
In the past 
few weeks, we have tested the hardware/firmware to death in the 
teststand room to make sure there is no single known problem left. After 
spending the whole (last) week failed to find and reproduce one single reformatter error and survived the so called  
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=l2upgrade&action=view&page=4&frame=2&anchor=12:30:17'> 
"reformatter error huricane",</a>  we have decided to move on to finish the 
rest of the business.  
<p> 
 
Tonight we have done extensive L2 torture runs to 
 
fully checkout the Pulsar Muon path readiness before beam. 
<p> 
The full path is: 
<p> 
L1 Muon/XTRP (TCMD,XFT/XTRP banks) --> Pulsar (T2PD bank) --> 
XTRPList-->MagicBus --> Alpha --> TL2D bank.  
<p> 
The test methodology 
we follow is to take L2 torture runs and compare all five banks data and make sure everything matches. The conclusion of these test runs: 
<p> 
  ** we are now ready for beam tests ** 
<p> 
Some of the details are described below. 
<p> 
  (1) Run 168483 (L2 torture, L1A~40Khz, L2A~300hz): 
      Pulsar muon board receives all 16 L1 muon fibers and fake track 
data 
      from Pulsar XTRP transmitter.  The merged&reformatted  track + zero-suppressed muon 
      data is then sent to  alpha via trackList board. The fake xtrp 
data is used 
      to simulate high  track volume events.  Comparison between 
     TCMD (L1 muon bank),  T2PD (Pulsar bank), and TL2D (alpha bank) 
      shows perfect agreement. 
<p> 
  (2) in order to make sure the path is roubst again HRRs, we have 
manually 
        induced ~50 HRRs during the run (168484, 168485), and again the 
data 
        analysis shows perfect agreement among the banks. 
<p> 
  (3) with L2 torture, the actual evt data size is rather small. To 
mimic high 
       occupacy & high rate  running conditions, we have taken a L2 
Torture 
      run with Pulsar Tx sending VERY large muon hit patterns as well as 
 
     XTRP tracks (run at high rate, ~30Khz). In the past, we have also 
     seen frequent L2 DTO when running Pulsar in this condition. 
     With the new firmware,  no L2 DTO occurred during this run 
     (and all the test runs).  Again, data show perfect agreement. 
<p> 
   (4) we have checked total of about 50K evts so far for all the runs 
taken. 
<p> 
  (5) we have also tried to unplug all the fibers to Pulsar to creat 
extreme 
        error conditions to make sure that this will not cause trouble 
to the 
        system running. 
<p> 
  (6) for all the runs we took, there was no single reformatter error. 
<p> 
  (7) All Pulsar hardware/firmware/software are now fully configured 
         ready to go for beam. 
 
<p> 
We conclude that we have done as much as we could in teststand mode and 
non-beam mode (as they say, "good enough for government work"), given 
the fact that there are only a few stores left before the long beam 
abort gap come (otherwise known as "shut-down"), we would like to 
request to have Pulsar muon path tested with beam to collect as much 
beam data as possible so that we will have something to chew on in order 
to survive the long beam abort gap while licking our wounds... 
<p> 
There are two kinds of beam runs for Pulsar muon path: 
<p> 
  (I) pure parasitic run with Pulsar receiving real L1 muon fiber data. 
<p> 
In this case,  Pulsar will just readout its input raw muon data, as well 
as zero suppressed muon output data (to be sent to XTRPList->Alpha). 
This is very simple, all ACE has to do is to include the Pulsar crate in 
the run. 
The data taken in this configuration will be used to check the 
robustness of the L1 muon to L2 Pulsar fiber interface by comparing the 
Pulsar 
fiber input DAQ buffers with that of TCMD. The data will also be used to 
check 
the robustness of the Pulsar firmware algorithm (zero-suppress&reformat&merging muon 
data with XTRP data) by comparing Pulsar output DAQ buffer data with 
TL2D. 
<p> 
IF Pulsar crate doesn't cause any trouble to the system running in the 
beam, we would VERY much like to keep the crate in the system in this 
configuration to take as much data as possible. 
<p> 
 (II) FULL chain test (send muons all the way into the ALPHA land). 
 
This test involves two simple changes: 
<p> 
(1) hardware: unplug the XTRP cable from XTRPList and plug it into 
Pulsar, 
      and plug Pulsar XTRP output  into the XTRPList; 
<p> 
(2) select Pulsar based physics trigger table in RC 
PHYSICS_1_05[7,334,400]. 
      This version of physics trigger table allows alpha 
      to add the zerosuppressed muon data from Pulsar to TL2D bank. 
 
      If there is no problem, the data&trigger in this configuration should be 
       the same as normal physics runs and thus the data should be 
useable for 
      physics analysis. The only difference is that now the TL2D bank 
will have 
       muon data in it (we will not run L2 muon algorithm in Alpha). 
 
 
  
<!-- Author: Cheng-Ju/Burkard/Ted -->
<!-- Comment: Sat Aug 30 07;35;38 comment by...Ted -->
For those who would like to learn more about Pulsar project, 
please visit:
<a href='http://hep.uchicago.edu/~thliu/projects/Pulsar/'>
Pulsar web page </a>.


<!-- Comment: Sun Aug 31 12;18;19 comment by...Ted -->
<pre>*********** Here is the Pulsar request for beam time ***

Sent: Saturday, August 30, 2003 12:35 PM
Subject: Re: Prioities for upcomping stores



Dear Jimmy:

   The Pulsar group have done extensive non-beam tests in the past
two days to get ready for the beam tests. The details are described
in Pulsar e-log:

http://www-cdfonline.fnal.gov/cgi-bin/det-system-elog.pl?
nb=l2upgrade&action=view&page=5&frame=2&anchor=06:58:48

Please take a look and let us know if there is anything else you and 
others think we have to do in order to meet your (&other heads) 
requirememts to be qualified to go into the beam. 

Hope this will help you and other heads to evaluate the Pulsar case
given where we stand at this point, and move Pulsar up or down on
your priority list accordingly. Please note that I (personally) still
have no idea what really caused the reformatter errors in the past,
so you and other heads have to take that into account in the risk
analysis. All we know so far about this exotic reformatoium has 
been summarized in the Pulsar e-log (the previous page)...

I don't really know who should be on this list, but I added some
people I think who needs to be informed about where is Pulsar on your 
priority list on this email list. hope you and other heads don't 
mind ...

cheers,

Ted

</pre>
<!-- Comment: Sun Aug 31 12;33;26 comment by...Ted -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>
*** This is Jimmy's reply to our request, *even* after we 
encounnted the nasty reformatter errors again in the beam 
with Silicon. It really shows his brilliant managemet
judgement! ***


----- Original Message -----
From: James Proudfoot &lt;jxp@hep.anl.gov&gt;
Date: Sunday, August 31, 2003 7:12 am
Subject: Re: Prioities for upcomping stores



Hi everyone,

the elog made very interesting reading this morning. 
As best I understand it, the muon board situation is 
as follows:


PULSAR -
========

This board met all requirements for being re-included yet 
when it was we again saw reformatter errors, but only 
with silicon in the readout as far as I can tell. 

With this caveat it is ready for a beam test with the 
PULSAR board sending data to the Alpha. 

They worked on the reformatter errors all last week with 
no success, so short of some inspiration I doubt if they 
will go away this week. 

So although I understand Masa's concern, 
_______________________________________

I would like the final test to be scheduled, with the 
exception that silicon not be integrated for the run in 
which the PULSAR sends the XTRP and Muon data to the alpha, 
where it is then unpacked by the algorithm 
running in the alpha. I think that 1 hr of data in this configuration 
should be sufficient for us to tell that this part of the 
system is functioning correctly.

I realise that both of these test runs will have an adverse 
impact on data taking efficiency. But that's life if we want 
to have a muon board after the shutdown.

I doubt if either of these tests will be on list list for 
the present store, so therre is time to exchange email on 
this issue (and clarify any misconceptions that I have about 
the status of the two boards). Frank or Jane might also like 
to comment on the reformatter errors.


cheers


        Jimmy


 
 
</pre></td></tr></td></table>
<!-- Comment: Sun Aug 31 13;18;23 comment by...Ted -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>I should mention that it was Kaori who insisted at the 8 am
 meeting today that Pulsar should have another chance (after the
reformatter error incident) to do our test... she suggested
that we should do it right after the 8am since we were all very
tired waiting for the beam.... she is our hero...</pre></td></tr></td></table>
<!-- Comment: Sun Aug 31 14;16;15 comment by...Ted -->
<pre>
**** here is my email to Jimmy after Pulsar DAQ readout caused
aother reformatter hurricane in the control room. ******

----- Original Message ----- 
From: "Tiehui Ted Liu" &lt;thliu@fnal.gov&gt;
To: &lt;thliu@fnal.gov&gt;; "James Proudfoot" &lt;proudfoot@anl.gov&gt;
Cc: &lt;cdf-ops-managers@fnal.gov&gt;; &lt;nahn@fnal.gov&gt;; &lt;trwright@umich.ed&gt;; &lt;mlindgre@fnal.gov&gt;; &lt;roser@fnal.gov&gt;; &lt;feild@fnal.gov&gt;; &lt;wittich@fnal.gov&gt;; &lt;jdl@fnal.gov&gt;; &lt;wallny@fnal.gov&gt;; &lt;reisert@fnal.gov&gt;; &lt;cjl@fnal.gov&gt;; &lt;sakari@fnal.gov&gt;; &lt;shochet@hep.uchicago.edu&gt;; &lt;nachtman@fnal.gov&gt;; &lt;chlebana@fnal.gov&gt;
Sent: Sunday, August 31, 2003 7:24 AM
Subject: Re: Prioities for upcomping stores


&gt; Dear Jimmy:
&gt; 
&gt;    Here is an update on Pulsar status. 
This morning, at the beginning of store, Pulsar crate was included 
(although Masa's
&gt; plan was to take data with
&gt; silicon but without Pulsar for two hours, then add Pulsar. 
But the Pulsar crate was included by mistake). 
And this caused lots of reformatter errors again.
&gt; It is *absolutely* clear now that Pulsar DAQ readout and silicon readout
&gt; cannot
&gt; happen at the same time. All our non-beam tests were done without silicon,
&gt; that was why we couldn't reproduce this error. Steve Nahn took one silicon
&gt; test run with Pulsar and even with software eventbuilder a while ago, and
&gt; also had reformatter errors. We planed to have non-beam test with silicon
&gt; last week to
&gt; confirm that, but never had the chance to do so.
&gt; 
&gt;   We clearly need to understand and fix this. but it will take sometime. For
&gt; that,
&gt; we need DAQ experts to help us. we are running out of time here...
&gt; 
&gt; The only other way we can test the L2 Pulsar muon path  is to disable the
&gt; Pulsar readout completely. Since Pulsar readout is purely for diagnostics
&gt; purpose,
&gt; for the full chain beam test, there is no real need to readout Pulsar banks,
&gt; as
&gt; we can compare the XTRP/TCMD (muon)  bank with that of TL2D (alpha).
&gt; This would eliminate the possibility of Pulsar readout causing any
&gt; reformatter errors.
&gt; 
&gt; Please let me know if this is acceptable to you, and especially to the
&gt; Silicon experts(!).  If it is acceptable, then we would like to do the tests
&gt; near the end of store.
&gt; 
&gt; Just got Jimmy's email, I will send this one first.
&gt; 
&gt; Ted
</pre>
<!-- Date: Sun Aug 31 01:55:13 2003 -->
The Pulsar crate configuration for beam: 
<p> 
(1) slot 16: This is the Pulsar Muon/XTRP Rx board. 
What this board does is to receive the L1 muon fiber 
data (16 fibers), as well as the XTRP data. On board 
firmware will then zero suppress the large raw muon 
data and them format them into XTRP format, and then 
append the zero-suppressed muon data to the XTRP data. 
The final combined package can be then delivered to 
the XTRPList board inside the L2 decision crate, where 
the XTRPList will transfer the data into Alpha. There is 
one bit (EP bit) in the XTRP package indicates whether 
the data word is XTRP or muon data (this is stamped by 
the Pulsar board while merging the data). With this  
approach, the only thing need to be changed in the L2 
decision crate is the alpha code (to check this bit 
to separate the XTRP data and muon). This board is currently 
running with 80MHz on board clock. The DataIO FPGA firmware 
is the working version (which has been tested in the beam before), the Control FPGA has the latest firmware with 
larger output FIFOs. 
 
<p> 
In addition, this Pulsar will readout the raw muon and 
XTRP input data, as well as the output data. These data 
allow us to monitor what's going on in the full chain 
by comparing the data among five banks: TCMD (L1 muon) 
bank, XFT/XTRP banks, T2PD Pulsar bank, and TL2D banks. 
<p> 
(2) slot 7: This is the Pulsar muon Tx, which is capable 
to send fake muon and XTRP data upon L1A. The data patterns 
are user defined and download via VME. We use this board 
to mimic high occupancy running conditions. 
<p> 
(3) slot 11: This is another Pulsar muon Rx. It is used to receive the XTRP output of the Pulsar muon Rx in slot 16  
for testing purpose.
<!-- Author: Ted -->


<!-- Date: Sun Aug 31 10:43:34 2003 -->
<table border=0><td><tr><td bgcolor=#e0e030>Today we took a special beam run for the FULL Pulsar muon path: 
 
muon/XTRP -> Pulsar -> XTRPList -> Alpha -> TL2D.  
 
and it worked! This is  
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=view&page=728&frame=2&anchor=09:05:17.362'> 
Run 168561 </a>. A run we will never forgot. TrigMon plots 
for the tracks for XTRD vs TL2D look perfect. Offline analysis 
show that muon data in TCMD and TL2D match perfectly as well. 
 
We started this project exactly three months ago. And all that 
hard work paid off! </td></tr></td></table>
<!-- Author: Cheng-Ju/Burkard/Sakari/Frans/Ted -->
<!-- Comment: Sun Aug 31 11;44;40 comment by...Ted -->
opps. forgot to add Shawn Kwang (Mel's student) here. 
He did the initial muon zero-suppression algorithm study 
and initial firmware implementation. He also wrote the code 
for the Alpha to separate the XTRP data from muon data.
He is great!






<!-- Date: Sun Aug 31 14:40:37 2003 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>*** SUMMARY of the Pulsar muon effort *** 
 
 
In summary, around 9am on Aug 31st, 2003, the DOE funded  
Pulsar MOAB (Mother Of All Board) has been droped in the  
beam... it first caused a huge reformatter error hurricane, but  
there was no single casualty reported, although three almost  
dead (due to the lack of sleep) Pulsar soliders spotted inside  
the CDF control room at Fermi National Lab in Batavia, IL,  
USA.</pre></td></tr></td></table>
<!-- Author: Pulsar group -->
<!-- Comment: Sun Aug 31 15;07;00 comment by...Pulsar group -->
<pre>Although there was no single casualty/damaged reported, the
reformatter error hurricane was so spectacular and strong that
it almost scared the shift crew and even the three soliders who 
droped it to death. Therefore, this DOE version of MOAB test
is considered to be truely successful, as they say, "good enough
for government work"!</pre>


<!-- Date: Tue Sep 2 09:59:32 2003 -->
<table border=0><td><tr><td bgcolor=#e0e030>A summary of the Pulsar L2 muon beam runs report can be found at: 
 
<a href= 
'http://www-cdf.fnal.gov/internal/WebTalks/0309/030930_800.html'>Cheng-Ju's talk at 8am meeting</a></td></tr></td></table>
<!-- Author: Ted -->


<!-- Date: Thu Sep 18 16:11:18 2003 -->
In the end, we took 2.6/pb beam data with Pulsar sending muon/XTRP data to alpha via tracklist board. The test results can be found in the  
<a href= 
'http://www-cdf.fnal.gov/internal/WebTalks/0309/030912_2bl2.html'>Summary talk by Burkard</a>.
<!-- Author: Ted -->