<!-- Title: 17:32:15  Thu Feb 23 2006 -->
<!-- Crewchief: Stefano -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>SHUTDOWN E-LOG</pre>


<!-- Date: Thu Feb 23 18:44:20 2006 -->
<pre>Updated svtvme with new simulation. New option in the mapset to ignore the isobit from XTRP since it is going to be used for stereo confirmation instead of short tracks.</pre>
<!-- Author: Stefano & Alberto -->


<!-- Date: Thu Feb 23 19:26:11 2006 -->
<pre>We added the new option in the mapset to ignore the isobit from XTRP. It is now active as default: 
xtfTrackDropMode 1</pre>
<!-- Author: Stefano -->


<!-- Date: Thu Mar 2 12:11:45 2006 -->
<pre>Summary of my work this morning - I took 70 of the tracks that Stefano found to have strange impact parameter in w00 (slightly different wrt simulation), and ran the simulated HB++ output from a merger to the TF++ and back again. Did this 300 times, and found no errors, so it is a very sbtle problem (it appears in ~0.5% of tracks).  
 
We should run vslice on the board once we get data again. Hopefully this will help us nail the problem and swap a board and be done with it. 
 
I also upgraded SVTVME to check the TF++ firmware version, and loaded a new hwset for the new version in the db. Edited stephen5, nosvx and fakehits_nobypass hwset files and added the version control switches to then as well. I think this should be all the hwset files that needed changing. The "current" link of svtvme now points to the development version (not sure if we want to change that or not) 
 
Ran fakehits just now, and the system is once again running OK.</pre>
<!-- Author: jahred -->
<!-- Comment: Thu Mar  2 14;59;56 comment by...jahred -->
As Stefano pointed out, I missed a file - I also changed fakehits_nobypass_beamfit.


<!-- Date: Sun Mar 19 18:49:40 2006 -->
<pre>vslice merger tests. 
 
Laura </pre>
<!-- Author: sartori -->


<!-- Date: Mon May 1 13:14:27 2006 -->
<pre>We got following errors during config. BEWARE!! openBoard called for 35 (HF) but is 0 (Empty) FER_E_SVT, hf(slot5, hf_w02_b2b3): unexpected board type in ID PROM We just powercycle b0svt01 and it looks good now. Since Ace didin't use svt this morning we think that the HF board didn't turn on correctly after power outage.</pre>
<!-- Author: Taichi and Stefano -->
<!-- Comment: Tue May  2 09;49;39 comment by...Taichi & Jahred -->
<pre>We took HF #5(slot5, hf_w02_b2b3, svt01) and changed to #51. We moved #5 upstairs and marked it bad.</pre>


<!-- Date: Wed May 3 20:14:02 2006 -->
<pre>b0svt09 is giving generating BTO errors and need to be shefferd to resume running. It's happening since the 1st of May. The crate was  excluded from the RC and we have been running w/out SVT all the week. I knew that only today when it has been incldued again in the Run configuration. I've asked to remove it from the run configuration for this night and will investigate the problem tomorrow.</pre>
<!-- Author: Stefano -->


<!-- Date: Mon May 8 12:03:53 2006 -->
<pre>(RC)   11:45:59 Reset -&gt; IDLE 
(MLE) b0svt03:Messenger:11:46:14 AM-&gt;FER_E_SVT,  
While trying to run fake hits we git this error. After shpehrding the crate we ran. 
 
write4MMap(ams_w06/AMMAP++): checkMemoryFragment  
(MLE) b0svt03:Messenger:11:46:14 AM-&gt;FER_E_SVT, mismatch (nerr=131067) 
 
(MLE) b0svt03:Messenger:11:46:14 AM-&gt;FER_E_SVT, AMSRW (slot 8) AMmap write failed 
 
(MLE) b0svt03:Messenger:11:46:14 AM-&gt;Setup failed with error code 1 
(MLE) b0svt03:Messenger:11:46:14 AM-&gt;Statechange failed 
</pre>
<!-- Author: Stefano -->
<!-- Comment: Mon May  8 12;04;35 comment by...Alberto -->
<pre>This error happened once. Then it cleared with a shepherd.</pre>


<!-- Date: Mon May 8 12:13:44 2006 -->
<pre>While trying to run SVT_500Hz_BEAMFIT (Fake hits with beam fit enable) we got this error from b0svt01,02,04,05: 
 
0x7351880 (SVTColdStart): memPartFree: invalid block 0x70d8880 in partition 0x2d89dc. 
 
The error appears at different points during CONFIG transition and doesn't appear when the beam fit is disabled</pre>
<!-- Author: Stefano -->


<!-- Date: Wed May 17 11:56:05 2006 -->
<p>Last night, the ace saw error messages complaining that no SPYMON task was found on b0svt09.  See for example the ErrorLog for <a href="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=216153">run 216153</a>.  According to the ace, shepherding didn&apos;t help.  But when I logged into the crate svtmon seemed to be running normally.  A perhaps irrelevant message that showed up when I ran svtStop/svtStart:</p> 
 
<pre>WARNING: Internal validation failed, file: ipcmsg.c, line 1314WARNING:Internal validation failed, file: ipcmsg.c, line 1314</pre> 
 
<p>In any case, this morning when I asked the ace to try including SVT again, everything worked normally.  So two possible explanations that I&apos;ve extracted/interpreted from real expert emails:</p> 
<ol><li>Svtmon failed to start on svt09 due to an rtserver connection problem.  By the time I logged in, the problem had corrected itself and things were back to normal, but the SVT wasn&apos;t included into a run again until this morning.</li> 
<li>A communication glitch caused daqmon to <i>think</i> the process wasn&apos;t running even though it was, resulting in the error seen by the ace.  Again, between last night&apos;s failures and this morning&apos;s attempt, the problem resolved itself.</li></ol>  
<!-- Author: brubakee -->
<!-- Comment: Wed May 17 12;01;41 comment by...brubakee -->
<p>Update: Benedetto said he restarted the SPYMON task last night between the ace&apos;s report and when I logged into the crate.  Problem solved.</p>



<!-- Date: Fri May 19 12:45:15 2006 -->
<pre>Today preliminary test of spymon with spyset files configuration. 
1) created spyset files for all the crates. 
2) tested on b0svt07 and b0svt05 : work well for the enable/disable spydumps, histogramsto bee tested. 
Before to leave the crates fakehits done with default configuration.</pre>
<!-- Author: diuzza -->


<!-- Date: Mon May 22 21:07:16 2006 -->
<pre>Summary of work done today. 
Test on configurable version of spymon (svtmon v4_0 in code area) 
this version load the configuration from spysets file located in 
~cdf_svt/svtconf/spysets/ 
 
Test done with fakehits beamfit: 
Succesfull for crate b0svt00...b0svt08, error "Done timeout" in crate b0svt09. After the error the task SPYMON keep alive. 
 
Benedetto </pre>
<!-- Author: Di Ruzza -->


<!-- Date: Tue Jun 13 14:27:44 2006 -->
<h4>I got paged!</h4> 
<p>Crate b0svt05 was causing high readout deadtime--when the shift crew logged into the crate they saw lots of spew like this: 
<a href='http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=397&anchor=10:03:59.468'>here</a>. 
Since there was no silicon in this run, I said it was fine to remove SVT while we figure out what's going on.</p> 
<p>Alberto recognized the error messages, and it turns out this is a problem only when the readout list is turned off, as was done for an XFT test.  Presumably all the SVT crates had the problem, but svt05 was the worst offender.</p>
<!-- Author: Erik -->
<!-- Comment: Sat Jun 17 22;40;56 comment by...Erik -->
This was incorrect; in fact, there were serious problems with b0svt05.  See following entries for details.
<!-- Date: Tue Jun 13 15:19:14 2006 -->
<p>Installed new patterns and constants and checked with fake hits.  The beam position used is (-0.16, 0.03, 0.00045, 0.00015), the closest one we have generated to the BeamMon position from last store.  Some of the instructions were updated/improved so that guinea pigs and even postdocs can follow them.</p>  
<!-- Author: Erik, Stefano, Alberto, Kohei -->


<!-- Date: Wed Jun 14 14:11:19 2006 -->
<pre>Today we saw again the following message on b0svt05. 
b0svt05 was spewing lots of messages:  
PULSAR_readOutFormat_stdd(), WARNING: iBuff == 6+numberDataBuffers = 18 means no  data is read out  
BTW, these are warning messages telling us that STDD readout is not working properly. 
 
We did debug the cause of this most likely to the L2buffer number being always 00 on the crate backplane. 
Just before run 218245 we did power cycle the crate and re-seat the trace. After this the problem went away. 
 
In order to debug this problem we did look at the AMSRW_HIT_DAQ0/1/2/3_LEN registers. 
They tell us the number of words that are to be readout in the 4 HITS DAQ buffers of the AMSRW. 
Both AMSRW in b0svt05 were reporting this behaviour: 
0x3 for register  AMSRW_HIT_DAQ0_LEN 
0x0 for registers AMSRW_HIT_DAQ1/2/3_LEN 
While for AMSRW in other crates the values were always 0x3. 
This was idicating that AMSRW boards in crate b0svt05 were always working with L2 DAQ buffer 0. Which was kind of weird. 
</pre>
<!-- Author: Alberto, Erik & Stefano -->


<!-- Date: Wed Jun 14 15:49:44 2006 -->
<pre>Summary of today SVT work:  
I tested some new internal SVT options.  
The first which is to enable KillOnBit18 in the HB++ (instead of AMSRW) looks fine.  
The second one is enabling early sorting of XFT tracks in the HB++.  
This will require more tests since there are some details to sort out.</pre>
<!-- Author: Alberto -->


<!-- Date: Fri Jun 16 17:38:25 2006 -->
<pre>It seems that tracer is crate b0svt05 was not working properly. 
Because of these errors were appearing and disappearing: 
PULSAR_readOutFormat_stdd(), WARNING: iBuff == 6+numberDataBuffers = 18 means no  data is read out  
See also previous entries for more debugging info. 
Today we also got serious problems with it, meaning no roads being found in wedges 10 and 11. This was traced to LostSync on each HF input in crate b0svt05 which points again to the tracer not working properly. 
So we did just replace it. 
Trace #22 out and #25 in. 
 
 
</pre>
<!-- Author: Alberto & Stefano -->


<!-- Date: Sat Jun 17 05:41:11 2006 -->
<pre>Trying to debug the problem on b0svt05 we did swap boards: 
AMSRW #06 out of slot  8 and replaced by #11  
HB++  #06 out of slot 21 and replaced by #09 
AMSRW #14 out of slot 17 and replaced by #07 
We have not understood it yet.....</pre>
<!-- Author: Alberto & Stefano -->
<!-- Comment: Sat Jun 17 05;46;17 comment by...Alberto & Stefano -->
<pre>It seems that this last swap did the job!!!!</pre>


<!-- Comment: Tue Jun 20 17;37;16 comment by...Alberto & Stefano & Taichi -->
<pre>So AMSRW #14 is suspicius</pre>
<!-- Comment: Thu Jun 22 18;53;55 comment by...Ted, with Alberto and Stefano -->
<pre>today, I asked Stefano and Alberto to show me that AMSRW board... turns out the jumpers for the P2 buffer chips are missing... in such a way that it looks like they were "hit" while plug/unplug into the crate... as the pins are bent towards the backplane.

those jumpers are for setting the direction (driving or receiving) of the buffer chips... and missing them should be the cause of the problem.  </pre>
<!-- Date: Sat Jun 17 05:53:18 2006 -->
<pre>A while ago I gave two gold cable to Greg Veramendi for tests 
</pre>
<!-- Author: Stefano -->


<!-- Date: Sat Jun 17 22:39:08 2006 -->
<h3>I got paged!</h3> 
<p>The ace <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=410&frame=2&anchor=18:33:30.616'>said</a> that there were "RXPT errors from b0svt06".  Happened several times and L2 halted while waiting for SVT data.  AutoHRR fixed it each time.</p> 
<p>OK, translation is that b0svt06 was pulling CDF_ERROR and issuing autoHRR.  From the <a href="https://www-cdfonline.fnal.gov/internal/ops/svt/doc/svtspymon/b0svt06.html">crate status page</a> we learned that the culprit was lost sync errors in the merger.  By looking at the spydumps (see "How to read spydumps"  on the <a href="http://www-cdf.fnal.gov/internal/upgrades/daq_trig/trigger/svt/newPagerHelp.html">new pager help page</a>), we could see that wedge 10 was always the one with the wrong bunch counter (a.k.a. "event tag") in the <a href="http://www-cdf.fnal.gov/internal/upgrades/daq_trig/trigger/svt/BoardDocs/data_words/end_event.html">end of event word</a> of both the TF++ and HB++.  Wedge 10 input comes, suspiciously, from b0svt05, which was poked and prodded all over last night to debug another problem.</p> 
<p>We couldn't trace the problem back further because only b0svt06 was dumping the spy buffers when it had the error.  After a configuration change to issue a global freeze on such an error, the aces started a new run, and we could look at the corresponding b0svt05 spydumps.  The wedge 10 AMSRW hit (input) buffer had the wrong bunch counter, but the four merger inputs had the correct one.  Suspicion: bad or poorly seated cable between the wedge 10 hit merger and AMSRW.</p>  
<!-- Author: Erik -->
<!-- Comment: Sat Jun 17 23;01;09 comment by...Stefano -->
<p>About the wrong BC and how we found it:
<ul><li> you get lost sync error and most of the board that check it now pull the CDF_ERROR
<li> with this information you can check the spybuffers and you find the bad bunch counter at the end of each spydump
<li> you can check if the BC that is different is wrong checking the followinng conditions
<ol><li> if (BC < 34) then the condition BC%3==0 is true 
<li> if (BC > 52 && BC<87) then the condition BC%3==2 is true
<li> if (BC > 105 && BC < 140) then the condition BC%3==1 is true </ol>
<li> if the BC is wrong SVT is taking good data but it's likely that some connections are not good or there's a problem in one board. So you have to trace the board that first send or recive the wrong BC
<li> if the BC is a good one it's likely that the system is really out of sync and this should be observable by looking at TrigMon plots or at the coverage of SVT. Again the SPYBUFFER are the best tool we have to trace the source of the problem
</ul>


<!-- Date: Sat Jun 17 23:06:48 2006 -->
<pre>I've added the following options to the hwset 
# b0svt07 
spy_fanout_SC_GERROR_ON_ERROR		1 
# b0svt06 
spy_tf_SC_GERROR_ON_ERROR		1 
# b0svt09 
spy_final_SC_GERROR_ON_ERROR		1 
# b0svt08 
spy_svt08_SC_GERROR_ON_ERROR		1 
#b0svt05 
spy_w10w11_SC_GERROR_ON_ERROR		1 
In this way any time one of these crates raise a CDF_ERROR a GERROR is issed causing a GLOBAL freeze. In this way all SPYBUFFERS are read out. 
For crate b0svt05 this is a temporary setting, given the problems  we had recently.  
The other crates cotains boards whose inputs/outputs come/go to other SVT crates so it's important to keep this setting so that we are able to trace the problem in the SPYBUFFERS 
</pre>
<!-- Author: Stefano -->


<!-- Date: Tue Jun 20 11:44:45 2006 -->
<pre>Today work. 
1) updated SVTVME code to include several changes from the past weeks. 
2) the new code is not a frozen release but it is a development release since I expect to change it again very soon. 
   I made a copy of previous code named "development.AAbackup20060619", which is actually almost the same as v4_0. 
3) I installed a new firmware version for AMSRW in wedge09. It has the ability to readout 5/5 roads during hit loading. This feature is not going to enabled now. 
4) I tried to install also the new version of the AM++ board but I found some problem so I'll need to further debug this. 
 
 
 
 </pre>
<!-- Author: Alberto -->




<!-- Date: Fri Jun 23 18:12:42 2006 -->
<pre>SVT test with 1.5GeV tracks: 
- we took run 219035 with the SVT processing XFT tracks down to 1.5GeV 
- we measured the SVT processing time in this configuration at low luminosity 12E30. 
- we will need to measure the processing time at high luminosity as well</pre>
<!-- Author: Alberto & Stefano -->


<!-- Date: Tue Jun 27 19:08:17 2006 -->
<pre>SVT work today: 
- we installed new SVTVME v4_1. It is basically same as before but is frozen. Which required to recompile SVTMON. 
- updated the DB to readout STDD bank from all TF++ boards. 
- we installed the new TF++ firmware in all 12 wedges. 
  So we took out TF++ #92 (it was in w11) since we cannot program it from the front panel. We put in TF++ #3169. 
- we installed we patterns and constants for the new beam position 
- we took out AMSRW #3171 in w09 that had a firmware version under be test. It tested fine, but since we are not yet ready to use the new features we went back to the normal AMSRW firmware.</pre>
<!-- Author: Alberto, Jahred & Stefano -->


<!-- Date: Tue Jun 27 21:58:30 2006 -->
<pre>I got paged! 
After today work I left the nosvx.hwset with FW key for the previous version of the TF++ firmware. 
The error did show up during coldstart on the frontend console as "FER_E_SVT, tfpp(slot5,tf_w00): TFPP_CONTROL_FIRMWARE reads 8675309, expect 8675326". 
I changed the "tf_TFPP_CONTROL_FIRMWARE 0x08675326" line inthe nosvx.hwset to fix it. 
</pre>
<!-- Author: Alberto -->


<!-- Date: Thu Jul 6 15:38:03 2006 -->
<pre>New TF++ with bug fix for FIFO overflow in w04 
all hwset changed with 
tf_w04_TFPP_CONTROL_FIRMWARE 0x08675327 
</pre>
<!-- Author: Stefano -->


<!-- Date: Tue Jul 11 05:41:36 2006 -->
<h5>I got paged!</h5> 
<p>The SVT beam fit table <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=480&frame=2&anchor=04:54:13.817'>was red</a>. 
When I called in, the ace was apologetic because it had already gone back to white.  The bottom half of the canvas was still red but with failures around 5e-4, not enough to trigger real alarm.  I watched as more stats accumulated and didn't see anything out of the ordinary.</p>
<!-- Author: Erik -->


<!-- Date: Tue Jul 11 12:48:56 2006 -->
Changed TF++ firmware for Control chip in all 12 wedges to 0x08675327 to hopefully solve FIFO overflows. 
 
<p>Had a problem with TF++ in wedge 8 (board 3190) - it wasn't sending data, or was sending data but not EE, but this could have been related to other problems with had in b0svt09 that we noticed later (problems went away after b0svt09 was reset) where dataflow was messed up. Replaced board 3190 with 3191, 3190 moved to tstsvt2. It should have latest firmware as a spare. 
 
<p>Updated all hwsets for not TF++ firmware version control. Fakehits and vslice script look OK.
<!-- Author: jahred and stefano -->


<!-- Date: Tue Jul 11 18:00:43 2006 -->
I tested the TF++ in tstsvt2 with the latest version of the firmware, and it's not working. I'm not sure why. It claims to be waiting always for EE, so maybe something broke while we were trying to load in new firmware? Sounds unlikely, but maybe possible? Do NOT use this for a spare. The spare TF++ boards we have now are old firmware version pre-DAQ buffer. I will load the latest firmware and test it in tstsvt2 after it has been validated in SVT for a few days with low autoSVTMon error rates.
<!-- Author: jahred -->
<!-- Comment: Tue Jul 11 18;09;22 comment by...jahred -->
<pre>This board (3190) is labeled as bad</pre>


<!-- Date: Fri Jul 14 13:45:45 2006 -->
<h5>I got paged!</h5> 
<p>SVT crates were <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=486&frame=2&anchor=00:39:35.715'>taking a long time to partition</a>.  I&apos;ve never heard of SVT crates taking a long time to partition.  But then it turned out it was not just SVT, but lots of crates.  The DAQ pager carrier was looking into it, so they let me off the hook.  In the end, it had something to do with a new version of EVB code...</p>  
<!-- Author: Erik -->


<!-- Date: Fri Jul 14 14:58:59 2006 -->
<h3>Lost sync errors in wedge 0</h3> 
<p>I looked into the lost sync errors that seem always to come from wedge 0.  I pulled out one hour&apos;s worth of spydumps and started combing through them.  My detailed notes are available at /cdf/onln/home/brubakee/svt/done.txt.</p> 
<p>Basically, at least during that hour, lost sync errors show up only in wedge 0 and are due to a bad BCID from the HitFinder in slot 6 of b0svt00, which can be traced back to a bad BCID coming from input fiber 2 (ISPY_2) to that board.</p> 
<p>This input corresponds to SVXII ladder e402 (SB4W0L2), which indeed has BCID mismatch errors at the <1% level according to <a href="http://b0sili01.fnal.gov/cgi-bin/bitreporter/makeladder.pl?beginrun=218245&endrun=219878&ladder=S402">this silicon BitReporter page</a>.  More BitReporter gymnastics show that this is in fact the only SVXII ladder with such a problem, at least in recent runs.  So it seems that we have a consistent picture of where these errors come from.</p> 
<p>One secondary question is: How does the HitFinder choose which BCID to pass on to the rest of SVT when its inputs disagree?</p>  
<!-- Author: Erik -->


<!-- Date: Tue Jul 18 11:58:49 2006 -->
<pre>Tested new version of spymon from benedetto 
(will allow configuration through spyset files), with  
fakehits and beam fitting (Run 220003). 
A bug has been found and corrected, minor tweaks still to  
do. 
</pre>
<!-- Author: Marco & Benedetto -->


<!-- Date: Tue Jul 18 19:28:22 2006 -->
<pre>Uploaded latest (0x8675327 Control) firmware in 2 spare TF++ boards. The board in tstsvt1 and the board upstairs in the locker are good spares (both should be clearly labeled). 
 
Board 3190 that is labeled as bad tests GOOD with the old firmware, but does NOT work with the new firmware (I tried one more time). 
 
This board is bad for TF++. Not sure what we should do with it.</pre>
<!-- Author: jahred -->


<!-- Date: Mon Jul 24 06:08:04 2006 -->
<pre>I was paged&lt;br&gt; 
&lt;br&gt; 
I was paged because b0svt09 became red during config state. ACE try to shephered it but it didn't help. I powercycled b0svt09 and shephereded it and it took long time to load up memory. After that it fixed b0svt09. </pre>
<!-- Author: Taichi -->
<!-- Comment: Mon Jul 24 14;25;42 comment by...Taichi -->
<pre>Sorry. There was typo. It was b0svt08 not b0svt09. I don't know how to correct this.</pre>
<!-- Date: Mon Jul 24 10:45:55 2006 -->
<pre>Hi, 
 
A list of what happened last night (not the b0svt08 problems that Taichi had, but the earlier problem in the evening). 
 
The shift crew started getting lots of L2DTOs. L2 complained that it was waiting for SVT (every minute or so). A look at the ip vs phi in TrigMon showed a strange problem at phi == 1 (wedge == 2) for all barrels, and the error rate in TrigMon went high. The HRRs from the L2DTOs clearly didn't help solve the problem (because L2 was still complaining). 
 
However, starting a new run somehow solved the problem. And this morning, I found that phi_XFT was == -1 for the period when SVT was having problems. 
</pre>
<!-- Author: jahred -->


<!-- Date: Mon Jul 24 10:46:50 2006 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-373&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-373' width='300'></a><a href='/elog/notebooks/svt/Mon_Jul_24_2006_104632.ps'><img src='/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-375' ></a></center>
<!-- Author: jahred -->
<!-- Comment: Mon Jul 24 10;47;12 comment by...jahred -->
<pre>Bad d vs phi in trigmon, and the phi_XFT == -1 during that period.</pre>



<!-- Date: Tue Jul 25 21:59:42 2006 -->
<h2> I was Paged! </h2> 
<p> Today the problem with the crate SVT09 appeared again and it was sending corrupted data to the EVB.  
<p> <h4> Temporary solution from Bill Badget: RE-SEAT THE RACER </h4> 
<p> NOTE: we took a cosmic run between two L2 Torture runs where SVT09 was causing problems. The problem appears only during L2 Torture 
 
  
<!-- Author: Taichi & Stefano -->


<!-- Date: Sun Jul 30 15:13:51 2006 -->
<pre>&lt;h1&gt; I was paged&lt;/h1&gt; 
I was paged bacause TrigMon crushed. TrigMon complained svtsim. 
 
here is the error message. 
 
&lt;b&gt; 
svtsimmodule::beginRun:  
  run=220727 
run=220727 m= mc=0 h= hc=0 
%ERLOG-e Error in trigger table query! ABORTING EXECUTION!!! (exit(1) follows):  
         /cdf/onln/code/b0cdfsoft/cdfsoftb0/6.1.4o/svtsim/src/svtsim_dbaccess.cc  
        : 189   GlobalLibraryLogger  
         svtsim::svtsim_build_xtf_bitmask_from_l1_table()  
         30-Jul-2006 11:55:14 CDT run = 220727 event = 0 
Termination Summary 
&lt;/b&gt; 
 
But we don't see anything wrong with svtsim. 
Consumer expert disabled svtsim in TrigMon tcl while we are investigating.</pre>
<!-- Author: stefano & taichi -->
<!-- Comment: Tue Aug  1 00;19;40 comment by...Taichi -->
<pre>svtsim was not bad. svtsim is good. It was DB problems. TrigMon is running with svtsim now.</pre>
<!-- Date: Mon Jul 31 14:19:21 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>TF++ cut on the tracks chi square turned on.  
The cut is at 16 (10)</pre></td></tr></td></table>
<!-- Author: Stefano -->



<!-- Date: Tue Aug 1 18:03:36 2006 -->
<pre>Found very strange situation for svtvme/svtmon setups and code.  
 
We were running with private Geum Bong version of spymon and current svtvme. THIS IS BAD!  
 
We should be more careful.  
 
I have changed svtb.table to load devel version of svtmon 
and current svtvme. Eliminated out-dated comments from that file. 
Now the portion of svtb.table read like this: 
 
# using current version of svtvme and svtmon instead of devel since 04/15/05 AA 
# changed back svtmon version to devel while current for svtvme MR 8/1/06 
# svtvme 
#      setupRequired(-j svtvme -d -q ${_dest_arch}) 
      setupRequired(-j svtvme -q ${_dest_arch}) 
# svtmon 
      setupOptional(-j svtmon -d -q ${_dest_arch}) 
#      setupOptional(-j svtmon -q ${_dest_arch}) 
 
Also devel version of svtmon now points to v3_6a which is  
the code we should have run up to know with the minor changes in beam width fitting from Chris and Geum Bong. To change back to v3_6 (previous version ) just change the devel link to v3_6 
in /cdf/code 
</pre>
<!-- Author: Marco  -->
<!-- Comment: Tue Aug  1 18;09;11 comment by...Marco  -->
<pre>Sorry the svtmon product area was misprinted, here is the correct one:

/cdf/onln/code/cdfprod/svtmon 
</pre>


<!-- Date: Thu Aug 3 02:29:26 2006 -->
<pre>&lt;h2&gt;SVT work&lt;/h2&gt; 
We tested GB in b0stv09 with MRGrun.py in 'CONFIG' state. 
We got same simptom as test of tstsvt1 for spare board. 
We swapped ghost buster board to spare ghost buster board in b0svt09.  
We confirmed that spare board works well with fakehits. If there is something wrong with svt at the beginning of next store, we immediately swap spare board to real board.</pre>
<!-- Author: Taichi & Taka -->
<!-- Comment: Thu Aug  3 08;45;32 comment by...Taichi -->
<pre>b0svt09 gives errors during L2 torture. Re-seat the tracer usualy fix this kind of problems, but in this case we were using spare board of b0svt09 and re-seat the tracer did't help. It seems that spare b0svt09 could cause this kind of error. So we put real board back to b0svt09. Now it seems good.</pre>

<!-- Comment: Thu Aug  3 11;45;52 comment by...Taka -->
<pre> It seems that you are not sure why this error is happened
from TRACER. Combination of spare GB and current TRACER 
cause trouble? </pre>

<!-- Comment: Thu Aug  3 16;45;53 comment by...maruyama -->
<pre>Date: Thu, 03 Aug 2006 15:57:57 -0500
From: William Badgett &lt;badgett@fnal.gov&gt;
Subject: Re: TRACER re-seat in b0svt09
To: Taka Maruyama &lt;maruyama@fnal.gov&gt;
Cc: taichi@fnal.gov


We have tried swapping tracers in the past with no
success.   This usually indicates a backplane
connectivity problem, or perhaps an impendance
discontinuity.   Things to do are to check that
all cards are well seating on the back-plane;
or change the position of cards on the backplane
to alleviate potential impedance problems.   Or
swap in a new backplane.

</pre>


<!-- Date: Thu Aug 3 17:44:56 2006 -->
<pre> Try another swapping of GB according to Bill's comment above. 
Currently, spare GB is in slot19, b0svt09, but if if I found 
something, I will swap the card again.  
 Stefano, if you are paged and it is coming from b0svt09, 
please call Taka (630-701-5767)</pre>
<!-- Author: Taka -->


<!-- Date: Sat Aug 5 12:05:24 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-376&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-376  width='300' align='top'></a></td><td>ip mismatched tracks</td></tr></table>
<!-- Author: Taichi -->
<!-- Comment: Sat Aug  5 12;08;43 comment by...Taichi -->
<pre>Disappear large amount of mismatched tracks in wedge00 since we swapped GB to spare GB.</pre>

<!-- Comment: Sun Aug  6 02;47;32 comment by...Taka Maruyama -->
<pre>Taichi and Taka should do followings according to facts above;
(a) put this GB card as default for a while. As Bill B.
   indicated above, relationship TRACER and GB is not an
   issue. 
(b) Same firmware were installed into these two GB cards,
   so that this symptom might be due to the hardware.
(c) Create one more additional spare GB card with certain 
   tests.
(d) Understand why current tests at teststand are troublesome
   although real operation of two GB cards has no issues.
(e) Taichi will calculate the error rate for mistamath/total    
   for phi=0,1,2,3 bit with swapped GB card in SVTD. 
   Also test the swapped card with teststand.   </pre>


<!-- Date: Mon Aug 7 20:58:49 2006 -->
<pre>We swapped spare board to original one. 
And we can test GB on teststand now.</pre>
<!-- Author: Taichi and Taka -->
<!-- Comment: Fri Aug 11 02;20;58 comment by...Taichi and Taka -->
Recent works on GB by Taichi and Taka were summarized at <br>
<a
href="http://www-cdf.fnal.gov/internal/WebTalks/Archive/0608/060808_svt_weekly/02_060808_svt_weekly_TaichiTaka_1_taichi_StatusOfGB060807.ppt">
 ppt file for SVT meeting </a> <br>


<!-- Date: Mon Aug 21 20:01:00 2006 -->
<pre>I've changed TF in wedge 9 
TF in: 2138 
TF out: 3176</pre>
<!-- Author: Stefano -->


<!-- Date: Tue Aug 22 14:55:25 2006 -->
<h2> I was not paged but this is important </h2> 
It looks like the cable from XTRP was loose and then it was causing  the high failure rate in the whole system. Why it became loose is a mystery since I think nobody has touched the crate recently, however I've found that several cable, and mostly the ones connecting boards in different crates were loose. This can be caused by moving cable of other systems or just by the time, in any case I think we should remember to check all the connector any now and then. 
<p> The rate of failures in TrigMon seems now under control
<!-- Author: Stefano -->


<!-- Date: Wed Aug 23 13:28:46 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-377&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=-377  width='300' align='top'></a></td><td><h2>Layer missing</h2> 
We have found the possible source of all the problems of this week. One layer in B5 is not present in the output of the HF while is perfectly working on the silicon side. <p> To find the problem we first looked at the report from rootplot and looked at the failures. There we saw that all the failures were due to different hitmask in the track. So this brought our suspects to the layers in the silicon. After we found in the plot here reported that one layer in the SVX was missing we have x-checked with silicon people and they told us that the layer is perfectly working. So we are going to swap the HF.</td></tr></table>
<!-- Author: Stefano, Alberto and Taichi -->
<!-- Comment: Wed Aug 23 16;39;33 comment by...Stefano & Taichi -->
We have replaced the HF in b0svt04 sl 15 (W9 and B4 and B5) because of the problem shown in the plot. The bad board is the 35 and is now in test room. It need to be tested. We forgot to check the number of the board we put in. Sorry.
<p> Btw now we have hits in the layer 1.



<!-- Date: Thu Aug 24 11:40:58 2006 -->
<pre>The HF I installed yesterday kept giving problems and here there's what I've found in the OUT_SPY: 
 
SB b0svt04 15 HF HF_OUT_SPY 1000 1584+ 0 
DA 22e1978 22687b8 22aa508 22a0988 22288cb 2268598 22a8b28 22a14a8 22288a3 22a1578 
DA 2228778 22a1598 2228758 22eaeb1 226c220 22e20bf 22a17b2 2228368 22681b9 232aa84 
DA 2268188 2329c78 2321e72 222c2e0 2268111 2329c58 2321f18 232808d 2322668 22a2178 
 
so it looks like there's an extra bit in the output word. So HF 17 is in the trigger room and I've put in HF 9. 
 
</pre>
<!-- Author: Stefano -->


<!-- Date: Fri Aug 25 05:19:10 2006 -->
I checked HF in b0svt04 and here is the HF_OUT_SPY in spydums. 
<p> 
SB b0svt04 15 HF HF_OUT_SPY 943 943+ 0<br> 
DA 228f28 269677 260053 2a0a38 26007c 320ae8 228ed4 26944f 26039c 2a10ef<br> 
DA 2aa4d8 2eaae2 228e7c 320e48 260538 2693e8 228e54 2a1725 2605b8 2a1748<br> 
DA 2aa3c3 268ccd 2a8ee8 2ea3c8 260818 2688d8 2a89aa 320ec8 22c910 2688a2<br> 
DA 26084c 2a1eb8 2e4d40 320f28 2286fa 321061 261066 26c858 2e0ec8 3211e0<br> 
</p> 
 
There seem to be no extra bit and we have hits in all layers in B5 in W9.
<!-- Author: taichi -->


<!-- Date: Fri Aug 25 16:55:44 2006 -->
<pre>After changing the HF board yesterday we kept having high failure rates in B4 and 5, W9. This time error came with Lost Sync from HF, that would have meant that the different fibers from SVX were not synchronized. We checked the SVX word that are fed in the HF and we found no reason for problems or Lost Sync. This pointed again to a problem in the new board. 
 
To test it we crossed the board with another one in the same wedge and we observed that the Lost Sync error followed the board. 
 
We then changed the board we another one and the Lost Sync error finally disappeared. 
 
The new board in slot 13 is HF #10 
The bad one is the HF #9 
 
&lt;b&gt;There are now 3 HF that need to be tested all in the test stand area while on the third floor there are a lot of HF tested by Bill that look to be working: please use them!!! &lt;/b&gt;</pre>
<!-- Author: Taichi and Stefano -->
<!-- Comment: Fri Aug 25 16;58;06 comment by...Stefano -->
<pre>HF #10 had a tag from Taka dated July 29 2004 saying heap corrupted, then there was a tag from Bill, from end 2005, saying tested and good.</pre>

<!-- Comment: Fri Aug 25 20;45;23 comment by...Erik -->
Note that of our many spare HitFinders, 09 and 17 were the two with problems.  They should have been clearly marked with tags or labels--if not, I don't know what happened!  See <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=svt&action=view&page=22&frame=2&anchor=13:17:08'>here</a>.



<!-- Date: Tue Aug 29 22:03:34 2006 -->
<h4>I was paged!</h4> 
 
because W4B1 has 0 occupancy for run 222326.<br> 
I was investigating this with Ricardo (CO).<br> 
The SVT is receiving hits from 4 layers of W4B1 as expected.<br> 
TrigMon is not reporting differences w.r.t. simulation.<br> 
Then Ricardo looked for problems in the silicon pages and found<br> that the layer 1 ladder is reporting errors.<br>  
<!-- Author: Alberto -->


<!-- Date: Tue Sep 5 23:33:38 2006 -->
<pre>I just tested STDD readout for crate svt08. 
With a 2304 CPU (i.e. tstsvt1 CPU) it works just fine. 
The corresponding fakehits run number is 222639. 
</pre>
<!-- Author: Alberto -->


<!-- Date: Mon Sep 18 21:03:24 2006 -->
<pre>We did test GB w00 issue. Details are shown in  
&lt;a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=689&frame=2&anchor=21:02:15.016'&gt; 
main elog &lt;/a&gt; 
We will investigate more.</pre>
<!-- Author: Taichi and Taka -->


<!-- Date: Mon Sep 25 09:23:50 2006 -->
<h6>I was paged!</h6> 
<p>There was a warning of no SPYMON task on b0svt09, which was coincident with a series of Done TOsfrom CMU_01.  They paged me just in case it was related somehow.  This was during shot setup.</p> 
<p>I logged into the crate and the task was running.  I checked a few plots etc. and everything looked OK, so I told them it was probably a temporary communication glitch, and unrelated to the Done TOs.</p> 
<p>Checked back when the store came in and still everything looked OK.</p>  
<!-- Author: Erik -->
<!-- Comment: Mon Sep 25 09;24;51 comment by...Erik -->
<pre>Just to be clear, the page was on Saturday evening, even though I'm only making the log entry now...</pre>


<!-- Date: Tue Oct 3 23:38:09 2006 -->
Taka and Taichi tested two spare board of Ghost Buster in b0svt09 with fakehits to investigate minor error in wedge00. These two spare boards still have errors although these two spare boards have no error using the test stand. We will investigate the output data of fakehits.
<!-- Author: Taka and Taichi -->


<!-- Date: Wed Oct 4 02:29:02 2006 -->
<h2> I was paged </h2> 
<p> 
Timeout Error persists due to b0svt09 while they were running L2 TORTURE. I   turned off b0svt09 and reseated tracer and GB board and then turned on b0svt09 as always we do when this kind of problems happen. This solved the problem.   
</p>
<!-- Author: Taichi -->


<!-- Date: Mon Oct 16 21:31:25 2006 -->
<h4>I was paged!</h4> 
<p>There were some very strange configuration-related error messages in <a href="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/ShowFile?FILE=/cdf/onln/data/cdfdaq/rcLog/errmon/log/errorfile225190.log">run  225190</a>, then in <a href="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/ShowFile?FILE=/cdf/onln/data/cdfdaq/rcLog/errmon/log/errorfile225191.log">run 225191</a> done timeouts from svt02,3,5,6 due to "Internal buffer too small".</p> 
<p>After some poking around I have no idea what is going on.  But the current theory is that a <a href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=773&anchor=18:34:50.070&target=_top">recent fer update</a> that included a fix for the b0svt08 "end of run hang" problem introduced some new problem as well.  The shift crew has given up on SVT for the night and will check with Bill tomorrow.</p>  
<!-- Author: Erik -->


<!-- Date: Tue Oct 17 16:29:38 2006 -->
<pre>Ankush and I removed the L00 VTMs from b0svt07</pre>
<!-- Author: jahred -->


<!-- Date: Tue Oct 17 19:38:21 2006 -->
<b> I was paged </b> 
 
Well, Erik was paged. SVT09 was sending corrupt data. I had Greg powerdown svt09, reseat the GB and the tracer, and then power back up.
<!-- Author: jahred -->


<!-- Date: Thu Nov 9 13:11:24 2006 -->
<pre>I'm taking a few fakehits run to test STDD after Bill's fixes. 
Run 227538 has 1 TF++ in crate svt08 enabled for STDD and is working. 
Run 227544 has all TF++ in both svt08 and b0svt06 enabled for STDD and it is working. 
We will try all of them with beam as an end of store test. 
 
We also upped SVTVME to version v4_2. 
 
</pre>
<!-- Author: Alberto & Stefano -->


<!-- Date: Thu Nov 9 15:47:08 2006 -->
<h2>I was paged</h2> 
<p> 
I was paged because EVB complained that b0svt09 sent the bad data. I reseated all the board on b0svt09 twice. Then it works fine. 
</p>
<!-- Author: taichi -->


<!-- Date: Fri Nov 10 15:43:14 2006 -->
<h2> I was paged </h2> 
<p>This problem is likely due to communication problems between the Tracer and the GB. Today we tried to change the slot of the Tracer, hoping in this way to solve the problem. The effect of this has been that this time we were not able to resume data taking even after re-seating the boards (we tried it 3 times) so we moved the board back to the previous position (Slot 2). Jason then took L2_TORTURE, COSMIC and AAA_SHOTSETUP runs w/out any problem. 
<p>We have asked Bill and Jonathan to investigate this problem. It is quite difficult to debug since it's likely an analogic issue and is rarely happening. As of today the recipe remain to re-seat the tracer.
<!-- Author: Stefano & Taichi -->


<!-- Date: Fri Nov 10 16:50:00 2006 -->
<pre>Run 227704 is the first one with STDD readout enable for all AMSRW and all TF++.</pre>
<!-- Author: Alberto -->


<!-- Date: Sat Nov 11 12:15:57 2006 -->
<pre>We insert the GB with the new firmware. We had usual problems with crate b0svt09. The first run with the new Gb is 227721.</pre>
<!-- Author: Stefano and Taichi -->