<!-- Title: 12:04:20  Fri Mar  3 2000 -->
<!-- Crewchief: Steve Tether -->
<!-- Op1: Jeff Tseng -->
<!-- Op2: Christoph Paus -->
<!-- Op3: Frank Wuerthwein -->
<!-- Op4: Ilya Kravchenko -->
<!-- Op5: Sasha Rakitine -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
Run II hardware event builder.


<!-- Date: Fri Mar 3 18:23:49 2000 -->
<p>Today I found a replacement ATM switching fabric waiting for me in the receiving area. I packed the faulty fabric and Fermilab dispatch picked it up this afternoon for shipping. 
<p> 
-Steve  
<!-- Author: Steve Tether -->


<!-- Date: Mon Mar 6 11:39:04 2000 -->
<p>Over the weekend Ilya ran a test with one SCPU generating fake data for one VRB and a fake Trigger Manager running on the Scanner Manager machine. It ran 9.5 hours at about 300 Hz before the fake TM code crashed. To me this means that the problem we see running with real TM and real data is unlikely to be due to a bug in SCPU or SM bookkeeping or SM-SCPU communications. We'll concentrate our search on the interface between TM and SM. 
 
<p>Our fake TM code generates triggers in bursts but after each one waits for the SM to acknowledge every message in the burst. We'll need to find out exactly what flow control policy the real TM code implements. The picture is complicated by the existence of a purely hardware feedback from VRB to Trigger Supervisor: when a VRB BUSY line is asserted the TS inhibits triggers. Ilya says that when this was implemented the problem didn't show up as soon or as often. 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Mon Mar  6 15;39;55 comment by...Steve -->
<a href=/cgi/elog/det-system-elog.pl?nb=evb&action=view&page=7&anchor=20:37:23 target=_top><img src='/elog/gifs/gr_menu.gif' width=19 height=10 border=0></a>The problem.
<!-- Date: Mon Mar 6 15:35:51 2000 -->
<p>We've seen that an SCPU becomes unresponsive when polling for an event for an empty VRB. Generally we've had to force another event through to clear the block. I've made a slight change in cdfevb_vrb v3_1 on b0l3pcom1 in an attempt to alleviate this problem. The body of the loop polling the the Universe's DMA status register for DMA completion is no longer empty; it contains a call taskDelay(0). I don't know why the tight loop should resist being interrupted for rescheduling, but this should force it. If the change makes a difference I'll put it into the repository. 
<p>
<!-- Author: Steve Tether -->



<!-- Date: Fri Mar 10 16:09:32 2000 -->
The Scanner Manager is screwing up when two SCPUs have VRBs that are marked as on-line in the hardware database but the readout list uses only one of those VRBs. <p>At the start of a run the SM is supposed to send every on-line SCPU two SCRAMNet messages that tell which of its VRBs are in the partition and what their enabled-channel masks are. Since only one SCRAMNet area is used  by the SM to talk to SCPUs, each message contains the SCPU ID number that was assigned at startup. The messages intended for the second SCPU have all fields correct except the SCPU ID number, which is not even the same for both messages. It should have been 1,1 but instead it is 2,3. As far as the second SCPU is concerned it never gets any messages defining the VRB side of the partition.  
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Fri Mar 10 17;23;23 comment by...Steve -->
This turned out to be a trivial mistake in hotline.c. I've fixed it, committed it, and made it part of cdfevb_sm v3_8 (which is what the SM startup script loads now).


<!-- Date: Mon Mar 13 16:34:39 2000 -->
I've put on the event builder web page a PDF version of the latest revision of hoary old CDF note 1864. The note describes the interface used to drive the reception of events and contains a brief description of the internal workings of the receiver. This is a corrected version of the note I put on peoples' desks last Friday. The central changes are that buffers are packet buffers instead of event buffers, and that buffers are no longer permanently associated with a particular builder but are assigned dynamically. Doing things this way simplifies the receiver and makes it easier to allow fragments to grow up to the maximum event size (which will be around 5 MB). As a consequence, a fragment is no longer a contiguous region of memory but is a bunch of packet buffers in a singly-linked list. I'm implementing the changes now. When I'm finished this will become cdfevb_l3recv v3_6.  
<!-- Author: Steve Tether -->


<!-- Date: Tue Mar 14 11:59:55 2000 -->
<pre> 
 Today we ran with 12 FE crates connected to 2 VRBs of 
b0eb12 and sent data all the way to consumer server. 
The rate was about 100Hz, as appears, limited by the  
processor node.  
  The first test lasted about 40min, stopped on "Done Busy" 
from one of the L1 crates. It appears that the crate was rebooted 
by someone. 
  The second attempt lasted 6h30min, stopped as well on "Done Busy" 
(difficult to say from which crate). I can not say for sure 
if somebody did it or not, likely yes.  
  For really long tests we will need to make sure that nobody 
touches the crates. 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Wed Mar 15 15:45:58 2000 -->
<p>Steve Nahn just told me that there should be a delay between readings of the fast FIFO when polling for the presence of a new event. Otherwise the first eight bytes of the event, containing the byte count, may get put into the FIFO twice. Steve gets good results with delays of several hundred microseconds to one millisecond. Mark Bowden is looking at the problem. 
<p>
<!-- Author: Steve Tether -->
<!-- Comment: Tue Mar 28 09;16;45 comment by...SCN -->
In Fact, Mark found a problem in the firmware.  He produced
a new version, which was tested and works without the
need of any delay.  All SVX crates have been updated
with a (parsed) firmware code date of 3/17/00.  "DAQ"
Vrbs not updated. Jim P et al notified.
<!-- Date: Fri Mar 24 21:55:27 2000 -->
<pre>  A short summary of today's integration tests: 
 
1. We read out 17 front end crates going into 4 VRBs 
of 2 SCPUs. Front end crates included 6 level1 and 6 level2, 
and 11 calorimeter (out of total 16) crates. We used the real 
cosmic trigger. 
   The data were received at the converter node of level 3 and 
sent to trash. 
 
2. For a while I had problem connecting to CSL. Tony said 
after some investigation that that was because of a glitch in  
CSL and cured it by restarting the CSL. 
 
3. The quality of the data is not always good, as we realized 
trying to send data to the CSL. Reformatter complains.  
  Here are random tests that I did: 
 
       crate                       reformatter says:       
 
    CCAL west 00 crate by itself - data are bad              VRB_00 
    CCAL west 06       by itself -          bad              VRB_00 
    CCAL east 12                 -          good             VRB_01 
    CCAL east crates 12->15      -          bad              VRB_01 
    LEVEL1_05                    -          good             VRB_09 
    LEVEL2_05                    -          good             VRB_10 
 
  So, more work will be needed to understand the data. Since the 
same event builder runs on every SCPU, I would think that VRBs/FE 
themselves have problems, but we will see. For these studies we 
need only a few CAL crates, hopefully we will get something before 
next Friday. 
 
4. As a result of the above, we could not right the events with  
all 17 crates to CSL to disk. 
 
5. I did save some data with CSL, these can be found on b0dau31 
in /logger/disk1/csl/0 directory, files dated between 4pm and 8pm  
of Mar 24. These data should contain small number of crates 
including L1, L2 and CCAL east 12 that passed reformatter control. 
 The files a viewable with  bank dump utilities. 
</pre>
<!-- Author: Ilya Kravchenko -->



<!-- Date: Tue Mar 28 18:17:44 2000 -->
<p>I've applied a bug fix to the L3 receiver package. The current version 
you get (no options to the 'setup' command) is now v3_5_patched. When 
not all of the on-line SCPUs were being used in a partition wait_next_event() and test_next_event() erroneously 
reported that fragments from the unused SCPUs were missing. 
<p> 
<p>The development versions of cdfevb_sm and cdfevb_scpu are now also the 
current versions, i.e., you don't have to say 'setup -d' any more but 
that will still work for now. 
<p> 
The old versions of all three packages can be gotten using 'setup -o'. 
  
<!-- Author: Steve Tether -->


<!-- Date: Wed Mar 29 23:34:45 2000 -->
<pre>Today's events and fixes: 
 
1. A VRB in slot 10 of b0eb15 crate appears to be bad, has stuck 
bit when we use 64bit transfer mode (although looks normal 
in 32 bit mode). Swapped with the VRB in slot 12. Now 
supposedly we can read out CCAL west crates (->slot10) but 
can not read out east crates (->slot12). Will try to get 
a replacement for the bad VRB. 
 
2. UPS version of level3 has been recompiled with the latest 
(fixed bug) l3recv v3_5_patched. Also, the latest reformatter 
v1_6 went in. Note that this level3 has a kludge for pstate 
array to make it compatible with the old and currently used 
filter executable (from Sept., Kevin). 
 
3. The configuration message from run control to L3proxy 
has been modified to include more numbers and options and 
has been propagated to fer devel. 
 
4. The problem with multiple TM processes for the same  
 partition appearing after Halt->Recover->Run has been 
solved. Frank Chlebana has modified the front end code  
(and reflexmem?). Works so far well. 
 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Fri Mar 31 17:40:53 2000 -->
<pre>  A brief report on today's integration test with hardware 
event builder. 
 
  Our time was spent on the following problems today: 
 
1. Getting a good VRB to replace the bad one. I have rejected 
two VRBs that have been situated in that crate. Both of 
them failed data test of the vrbtestnew for some of their 
channels ("test stalls after XX passes"). 
   One of them was formerly in slot 14. The other was in the crate 
but not inserted all the way. Right now I left all 3 VRBs that are 
suspected bad in that crate (with b0eb15), not incerted all the way. 
Two are marked by tape, one is unmarked. 
   Then I borrowed one VRB from the bottom crate in the same rack 
and put it in the slot 12 of the upper crate. This VRB so far 
has been good. I have also marked it by red tape (the only color 
I had), the writing there says where it comes from. 
   At that point we had good VRBs in both slots 10 and 12 corresponding 
to west and east calorimeter crates. 
 
2. While trying to send the data through level3, we observed 
complaits of the reformatter about some calorimeter crates. 
The complaint said that the bank instance number is larger than 
the total number of banks. It turned out that on the front end side 
the bank name was not recognized (CEMD, CHAD) and "2" was put into 
the bank header instead of "f"=16. The problem disappeared after 
rebooting the bad front end crates. The cause is unknown. This 
problem reoccured later and we had to reboot some of the crates  
again. 
 
3. The third problem has been indicated by the reformatter as well. 
For a fraction of events (sometimes 1/30, sometimes 1/1000) we 
saw the first control word from an arbitrary channel of the  
calorimeter VRB wrong. It is supposed to be x00000000 and it was 
x00000008. All the rest of the VRB structure was correct. Dirac crates  
appeared to be unaffected (did not  do long tests though). 
  This problem has been reproducing by running stand-alone reformat_vrb 
utility on b0eb15 without using the whole event builder. 
  After some investigations Frank Chlebana recompiled front end code 
and the problem disappeared. At least did not show up after 50000 events. 
Hopefully, we will verify this on Monday with more tests. 
 
4. As a side note, I had again multiple processes of Trigger Manager once. 
It appears that it happened RC crashed and I had to resturt it. 
(got out of memory in java exceptions). I wonder if it is possible 
during partition transition to clean up any leftover processes 
for the partition if there are any. Possibly not, I do not really 
know how TM is managed. 
 
  At our best time today we ran with 10 CCAL, 3 L1 and 3 L2 crates 
while observing occasional complaints from the reformatter about 
item 3. 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Mon Apr 10 11:22:31 2000 -->
<pre>  What happened on the weekend: 
 
1. On Friday evening I ran for about an hour with the 
2 muon TDC crates + usual calorimeter, up to CSL. This worked 
Ok. 
 
2. The long run with the real trigger, b0l1cl04, b0l1cl05, b0l1gl00, 
b0ccal06 lasted for about 26.5 hours. It stopped on busy timeout 
from the calorimeter crate. I found corresponding SCPU locked, 
polling on VRB. 
</pre>
<!-- Author: Ilya Kravchenko -->


<!-- Date: Fri Apr 14 09:54:54 2000 -->
<p>I've taken converter nodes 9 an 10 out of the event-builder SCRAMNet ring and made them into a two-node ring of their own using the PCI SCRAMNet cards that came back from Systran. Nearby I've set up a dual VME crate using the old 1604 CPUs b0eb03 and 04. It has the VME SCRAMNet module we got back from Systran  and our remaining spare from the cabinet, also set up in a two-node ring. 
 
<p>Both systems ran overnight tests in which the nodes took turns writing test patterns to all 512K of SCRAMNet memory, some tens of billions of messages overall. There were no failures. Recall that one PCI card had a transceiver replaced but that Systran couldn't find anything wrong with the the other PCI card or the VME card. Now I can't either. All nodes were writing to SCRAMNet as quickly as possible, relying on the "bus hold-off" feature to slow down bus transactions when the message FIFO on the SCRAMNet card fills up.<p>
<!-- Author: Steve Tether -->
<!-- Comment: Fri Apr 14 10;05;24 comment by...Steve -->
<a href=/cgi/elog/det-system-elog.pl?nb=evb&action=view&page=6&anchor=16:19:29 target=_top><img src='/elog/gifs/gr_menu.gif' width=19 height=10 border=0></a> Original log entry about VME card problem.

<!-- Comment: Fri Apr 14 10;10;31 comment by...Steve -->
 <a href=/cgi/elog/det-system-elog.pl?nb=evb&action=view&page=3&anchor=12:11:56 target=_top><img src='/elog/gifs/gr_menu.gif' width=19 height=10 border=0></a> Original log entry about PCI card problems.


<!-- Date: Fri Apr 14 10:14:55 2000 -->
<p>I've installed the TCP/IP version of the in-development L3 receiver 
package on b0l3040 (which is running Fermi RedHat 6). Just setup cdfevb_l3recv. While I was at it I 
changed the ownership of the products area to products:products and made 
it group-writable on both b0l3040 and b0l3pcom1.<p> 
  
<!-- Author: Steve Tether -->


<!-- Date: Mon Apr 24 10:22:45 2000 -->
<p>This weekend I fixed a bug in the SCPU code. When we went from 2-word to 6-word SCRAMNet message slots I didn't properly update the ERR_CODE_NO_STATS macro in scpu.h. This macro is used to place an error code in the SCRAMNet message being built as a reply to the last Scanner Manager SCRAMNet command. The macro was setting the entire SCRAMNet message, including tag, to zero before inserting the error codes. SCRAMNet acknowledgements must use the same tag as the original command, so event if this message had been sent to the SM it would have been rejected. However, the SCPU code examines tags in outgoing messages and won't send messages with tags of zero, so the SM got no reply at all and the SCPU sent Zephyr messages about "writing zero tag to SCRAMNet".  
 
<p>I've changed the macro so that it leaves the tag word alone. I've also changed the error message about zero tags to make it clear that the outgoing message is not sent if this happens. Since the actual code change is very small I've just updated SCPU v3_7 instead of making a new version.<p>
<!-- Author: Steve Tether -->


<!-- Date: Tue Apr 25 15:56:52 2000 -->
<p>As has been known for a while now, the Scanner Manager doesn't handle a normal end of run properly. One would get a lot of error messages about use of an undefined partition from the SCPUs and eventually the SM would hang. The SM would also print a message saying that it couldn't "receive from the Trigger Manager". 
 
<p>When the End Run message comes from the EVB proxy the SM sends the Delete Partition message to the SCPUs right away, then sets a flag indicating that any further triggers in the partition are to be ignored. This code must be broken because the SM keeps on processing triggers that were in the SCRAMNet pipeline, sending Load Event messages to the SCPUs, which then rightly complain that the partiton is no more. It looks like a synchronization problem between the hotline task and the dispatcher task. 
 
<p>The problem with the Trigger Manager communications seems to be due to the TM software being restarted for every run, with the side effect that it re-initializes SCRAMNet memory used for messages from TM to SM. The SM sees the change, but the all-zero message it gets is illegal and so it complains. 
 
<p>I've made a quickie development version of the SM to try to get around these problems. "setup -d cdfevb_sm" or "setyp cdfevb_sm devel". The end-run processing is delayed until after five consecutive seconds of inactivity in the partition, "inactivity" being defined as no Send Event or Delete Event messages being sent. This allows the end run to complete normally. Then the SM also closes and re-opens its SCRAMNet I/O object linkng it to the TM. This operation zeros all tag words so the re-init of the TM should no longer be seen, and both TM and SM will look for new messages starting in the first slot. <p>
<!-- Author: Steve Tether -->
<!-- Comment: Wed Apr 26 12;37;25 comment by...Steve -->
These changes work, but I don't know if resetting the TM->SM SCRAMNet area is the right thing to do if we're running multiple partitions.
<!-- Date: Tue Apr 25 17:35:32 2000 -->
<p>The new version of the L3 receiver which allows fragments to be sent in pieces has passed stand-alone tests using both TCP/IP and AAL5 for event transmission. It is tagged as v3_6 in the repository and is available on b0l3pcom1 via "setup -d cdfevb_l3recv". 
<p>
<!-- Author: Steve Tether -->


<!-- Date: Wed Apr 26 12:35:43 2000 -->
<p>"setup -d cdfevb_sm" now gets you the "v3_9" version instead of the 
"devel" version. This new version has the changes made to solve the 
end-run problem as well as other things I've been putting in for large fragments.<p>
<!-- Author: Steve Tether -->
<!-- Comment: Thu Apr 27 17;43;23 comment by...Steve -->
This now includes a more reasonable solution to the TM SCRAMNet synchronization problem. Re-initialization takes place just after the number of partitions drops to zero. The TM does the same.


<!-- Date: Thu Apr 27 17:52:07 2000 -->
<p>SCPU code is now available that uses the new fragment header format and is able to send a fragment in pieces. <b>setup -d cdfevb_scpu</b> or <b>setup cdfevb_scpu v3_8</b>. This completes the set of development versions of SM, SCPU, and L3 receiver able to deal with fragments that won't fit in a single AAL5 packet.  
 
<p>The new fragment header is defined in the <b>fragment.h</b> include file in the SM package.<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Apr 28 16:03:25 2000 -->
<p>Ilya and I found and fixed a few bugs today: 
<ul> 
<li>It was a bad idea to use an infinite reading loop to get around the VRB problem that caused a read error on the second event. The loop crippled the SCPUs ability to recover from other errors. Going back to one read attempt per event allows the event builder to recover and continue. The so-called "VRB_WAIT" option has been removed from SCPU 3.8.</li> 
<li>The Scanner Manager's event-flushing code would sometimes fail to send a Partition Flushed message back to the Trigger Manager when it was needed. 
<li>For the time being the EVB proxy does nothing on a Halt transition. It used to try to flush, but it seems better to do nothing and let the pipeline drain naturally. In the next Run Control meeting we'll discuss what the event builder should and should not do for Halt-Recover-Run. The actions will probably be different for DAQ and SVX SCPU crates. 
</ul> 
<p>We also tried an experiment. We pushed the reset button on a VRB while a run was in progress in order to simulate missing data from a front end crate. This resulted in a flurry of Stop Flow, Flow Stopped, and Partition Flushed messages going between TM and SM, plus complaints from the SCPU. After this the SM and SCPU were ready to continue but we weren't able to get more triggers even after multiple Halt-Recover-Run transitions. The Trigger Manager control panel showed a number of triggers unhandled by the SM, which should have been cleared to zero after a Partition Flushed was received from the SM. Ilya thinks that when this count gets too high the TM refuses to give out any more triggers.<p>
<!-- Author: Stephen Tether -->


<!-- Date: Fri Apr 28 17:59:41 2000 -->
<p>Version 1.1 of the EVB proxy has been installed in the evbproxy account on b0l3pcom1. This provides heartbeat messages and has the bug fix mentioned in the previous entry.<p>
<!-- Author: Steve Tether -->


<!-- Date: Fri Apr 28 18:22:32 2000 -->
<pre>  Here is a summary of our today's efforts in the Control Room. 
 
1. We have been mostly focused on interactions between the Event Builder, 
Trigger Manager and Run Control, in various transitions. 
 
  After modifications of the Event Builder (Steve) and Trigger Manager 
(Zhaoou) code done yesterday and today, we are able now to repeatedly  
go through the basic transition sequences without rebooting the CPUs: 
 
   Partition->XxxxStart->Activate->End->Reset  cycle (destroys partition) 
   XxxxStart->Activate->End-> ...              cycle (partition is preserved) 
   Pause->Resume-> 
   Halt->Recover->Run                          works under "nice" conditions 
 
 
There are occasional glitches that kill the run, such as task exceptions 
on the Event Builder CPUs or spurious TM processes on b0tsi00.  
It is sometimes a little difficult to disentangle bug effects from 
the generally noisy environment. 
 
    The Halt->Recover->Run procedures will have to be developed further 
in communication with all interested parties. We are missing 
the recovery procedures for the cases when some of the event 
fragments are lost and some have arrived to the VRBs. An example 
of a complication here is that we are not supposed to reset 
SVX VRBs. Steve Tether is thinking about proper ways to do it 
from the Event Builder side. At this time the Event Builder 
does nothing during this sequence and simply drains the 
remaining events from the pipline, while the Front End crates 
get actually the "Abort" transition. 
 
  We have tried to simulate data loss by pushing the Reset on  
the VRB in the middle of the run, doing Halt->Recover, then 
writing the enabled channels mask and resetting the VRB by 
hand with cdfvme_vme, and then trying to Run again. Did not 
work. No events go through. The VRB is not busy, but TS 
issues no triggers, the scanFIFO is empty. We gave up on this. 
 
2. Another tests that have been done today is running with 
with 2 partitions, Trigger Manager and hardware Event Builder.  
First I have verified that it works with the old level3 framework,  
then tested 2 partitions with the level3 proxy. Each partition 
had one SCPU receiving from one data source from Front End, 
and one subfarm of level3 (1 converter, 2 processors and 1 output), 
not running reformatter,executable, not connected to CSL. 
The above system was exercised for different transition 
sequences, mostly successfully.  I did experience occasional 
exceptions from tsmDisp task of Scanner Manager, esp. at 
high rates. 
 
3. We have not seen any scramnet-VME problems in DAQ_VRB_04 
although we have started and stopped 30 runs or so (this has been 
rather annoying problem lately). This is likely to be the result of 
pulling out all boards  from this crate except for SCPU and  
scramnet. Now I guess we can plug in back  a few VRBs and see what  
happens.  
</pre>
<!-- Author: Ilya Kravchenko -->
<!-- Comment: Tue May  2 09;33;03 comment by...SCN -->
I'd try putting the MVME 2301 back in first, see if it
is messing at all w/SCRAMNET.  Also might want to see if
it is processor 0 -> leaving RAM open to VME.