<!-- Title: 12:41:57  Mon Jun 26 2006 -->
<!-- Crewchief: Ulrich (MPC) -->
<!-- Op1: Ankush (SPL) -->
<!-- Op2: Ignacio (Observer) -->
<!-- Op3: the silicon crew -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
Take data with silicon, keep and eye on the beam, and continue to investigate the resonance and L1 DONE TO problem.


<!-- Date: Mon Jun 26 12:45:35 2006 -->
<h3>Store 4799</h3> 
Store 4799 came in without problems, initial luminosity: 105e30 cm<sup>-2</sup>s<sup>-1</sup>. The proton losses were rather low, 2 kHz at the most.
<!-- Author: Ignacio & Ulrich -->


<!-- Date: Mon Jun 26 12:52:12 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8857&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8858  width='300' align='top'></a></td><td>SVTSpyMon plot for the beginning of store 4799. Everything looks fine.</td></tr></table>
<!-- Author: Ignacio & Ulrich -->


<!-- Date: Mon Jun 26 13:02:44 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8859&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8859  width='300' align='top'></a></td><td>The losses for the beginning of Store 4799 seem fine. LOSTP are even falling now. We checked the MCR e-log but could not correlate this to anything we found there.</td></tr></table>
<!-- Author: Ignacio & Ulrich -->


<!-- Date: Mon Jun 26 13:34:08 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8860&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8861  width='300' align='top'></a></td><td>Pinkie disappeared after HRR by ACE.</td></tr></table>
<!-- Author: Ulrich & Ignacio -->


<!-- Date: Mon Jun 26 14:13:03 2006 -->
<h3>MCR: Serious Breach of Communication Protocol</h3> 
This issue almost got lost during the busy Monday morning: there was a serious breach of the communication protocol with CDF by MCR. The last store was terminated this morning around 9:25 <em>without informing CDF</em>. So all high voltages were up during the abort. The aces did a coldstart, and, luckily, nothing happened to the silicon. The ops manager (Argo-Bob) called MCR immediately. Willis informed me that AD (Bob Mau, I believe) has already officialy apologized for the incident. Here is the  
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=436&frame=2&anchor=09:28:25.787'> 
corresponding entry in the main e-log</a>.
<!-- Author: Ulrich -->


<!-- Date: Mon Jun 26 18:39:02 2006 -->
<h3>Spontaneous Switch-Off: Crate 15</h3> 
CAEN crate 15 just switched itself off. We switched the corresponding wedges back on (f32X, f54X, fa0X, fa4X, fa5X). fa5X tripped once more during the recovery, so we had to switch off the portcard and try again. Everything is back up now. Here is the entry in the  
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=437&frame=2&anchor=18:30:19.828'> 
main e-log</a> 
 
  
<!-- Author: Ignacio & Ulrich -->
<!-- Comment: Mon Jun 26 23;25;57 comment by...Ulrich -->
Crate 15 switched itself off again around 23:07, which fired the alpha-numeric pager. This time the aces managed to switch it on all by themselves, see
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=437&frame=2&anchor=23:15:48.949'>
main e-log</a>.
<!-- Date: Mon Jun 26 19:21:54 2006 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8862&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8862' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8863&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8863' width='300'></a></center>
<!-- Author: Ankush -->
<!-- Comment: Mon Jun 26 20;04;37 comment by...Ankush -->
<h3>Behold: A real resonance!!</h3>

<p>Today there was a resonance during data taking.  <b>BUT</b> only a SRC FATAL window appeared. No L1DONE TIMEOUT, No RXPT ERROR.  This is the first real resonance that has been caught.  The silicon  DAQ hardware, the Run control code and run control error handler are working perfectly well for real resonances.  So this is really starting to point to a problem somewhere in the trigger system.</p>

<p>The logic analyser traces above are what was recorded when the SRC pulled error.  The L1A rates do not suddenly go to a high rate.  what drives the resonance is the L2R line which is expected.  The trace on the left is over the whole data and the trace on the is a zoomed in section about the SRC error.</p>

<p>This trace is VERY significant.</p> 



<!-- Date: Tue Jun 27 00:02:35 2006 -->
Got paged because of consecutive red cells in L00 from SVXMon. Since this happened shortly after the spontaneous switch-off, I suggested to wait for the next update and see if it goes away — which it did. See 
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=437&frame=2&anchor=23:41:23.868'> 
entry in main e-log</a>.
<!-- Author: Ulrich -->
<!-- Comment: Tue Jun 27 00;23;05 comment by...Mark -->
Looks like the classic problem we see where the hardware inhibit clears but the IFIX inhibit has not engaged yet, so we take some data with the power supply off.

<!-- Comment: Tue Jun 27 00;55;20 comment by...Marcel -->
Well, Well<br>
The inhibits are actually coming from the inhibit crate to the TS and are much faster than the IFIX alarms. And these hardware inhibits stop the TS already. 


<!-- Comment: Tue Jun 27 08;27;12 comment by...Mark -->
Right, but the problem we have seen in the past is when we have these spontaneous switch offs, the inhibit from the crate seems to clear for a while.  In this case it looks even worse as the first inhibit RC sees is an IFIX:

<pre>
2006.06.26 18:22:07 	2043311 	6,176.43 	L00:iFix ISL HV:iFix
2006.06.26 18:22:11 	2043358 	6,179.43 	L00:iFix ISL HV:iFix ISL06 Trip
2006.06.26 18:22:19 	2043358 	6,188.43 	L00:iFix ISL HV:iFix
2006.06.26 18:24:09 	2043358 	6,299.43 	L00:iFix ISL HV:iFix SVX05 Trip
2006.06.26 18:24:14 	2043358 	6,302.43 	L00:iFix ISL HV:iFix
2006.06.26 18:24:42 	2043358 	6,329.43 	L00:iFix ISL HV:iFix ISL06 Trip
2006.06.26 18:25:35 	2043358 	6,383.43 	L00:iFix ISL HV:iFix ISL06 Trip SVX03 Trip
2006.06.26 18:25:40 	2043358 	6,389.43 	L00:iFix ISL HV:iFix ISL04 Trip ISL06 Trip
2006.06.26 18:25:44 	2043358 	6,392.43 	L00:iFix ISL HV:iFix ISL06 Trip
2006.06.26 18:25:48 	2043358 	6,395.43 	L00:iFix ISL HV:iFix
</pre>
<!-- Comment: Tue Jun 27 08;35;11 comment by...Marcel -->
that is Runcontrol seeing things, not the actual timestamp of the inhibit. Just keep in mind, RC polls status every 3 seconds at best
<!-- Comment: Tue Jun 27 10;11;36 comment by...Ulrich -->
Thew important thing here is that each CAEN crate has a trigger inhibit line. As soon as the trigger inhibit is pulled, the trigger is stopped. So as soon as the crate detects that it just has switched off itself, the trigger inhibit is set. 
<!-- Comment: Tue Jun 27 10;33;00 comment by...Mark -->
Of course it is RC, but that is what is important as far as taking data with a power supply off.  Even with a 3 second delay, it is still strange to see the iFix inhibits first.
<!-- Comment: Tue Jun 27 11;01;45 comment by...Marcel & Mark -->
So Marcel and I agree that the TS crate controls the data taking, but it looks like the CAEN crate is not holding the inhibit line while it reboots during one of these "spontaneous switch-offs".  A look at the error log from SVXMon shows nothing coming from these wedges:

<pre>
errlog 7 {Drop Readout (c2c2)} SIXDMon Silicon/I/B3/W2/L0 2040780 1151364127 {}
errlog 7 {No First Chip} SIXDMon Silicon/I/B3/W2/L0 2040780 1151364127 {}
errlog 7 {Drop Readout (c2c2)} SIXDMon Silicon/I/B3/W2/L1 2040780 1151364127 {}
errlog 7 {No First Chip} SIXDMon Silicon/I/B3/W2/L1 2040780 1151364127 {}
errlog 7 {Drop Readout (c2c2)} SIXDMon Silicon/I/B3/W2/L2 2040780 1151364127 {}
errlog 7 {No First Chip} SIXDMon Silicon/I/B3/W2/L2 2040780 1151364127 {}
errlog 7 {Drop Readout (c2c2)} SIXDMon Silicon/I/B3/W2/L3 2040780 1151364127 {}
errlog 7 {No First Chip} SIXDMon Silicon/I/B3/W2/L3 2040780 1151364127 {}
errlog 7 {Drop Readout (c2c2)} SIXDMon Silicon/I/B5/W4/L0 2040780 1151364127 {}
errlog 7 {No First Chip} SIXDMon Silicon/I/B5/W4/L0 2040780 1151364127 {}
errlog 7 {Drop Readout (c2c2)} SIXDMon Silicon/I/B5/W4/L1 2040780 1151364127 {}
errlog 7 {No First Chip} SIXDMon Silicon/I/B5/W4/L1 2040780 1151364127 {}
errlog 7 {Drop Readout (c2c2)} SIXDMon Silicon/I/B5/W4/L2 2040780 1151364127 {}
errlog 7 {No First Chip} SIXDMon Silicon/I/B5/W4/L2 2040780 1151364127 {}
errlog 7 {Drop Readout (c2c2)} SIXDMon Silicon/I/B5/W4/L3 2040780 1151364127 {}
errlog 7 {No First Chip} SIXDMon Silicon/I/B5/W4/L3 2040780 1151364127 {}
errlog 7 {Drop Readout (c2c2)} SIXDMon Silicon/I/B5/W4/L4 2040780 1151364127 {}
errlog 7 {No First Chip} SIXDMon Silicon/I/B5/W4/L4 2040780 1151364127 {}
errlog 7 {Drop Readout (c2c2)} SIXDMon Silicon/L/B1/W0/L0 2040780 1151364127 {}
errlog 7 {No First Chip} SIXDMon Silicon/L/B1/W0/L0 2040780 1151364127 {}
errlog 7 {Drop Readout (c2c2)} SIXDMon Silicon/L/B1/W0/L1 2040780 1151364127 {}
</pre>

We should study the behavior of a CAEN during a reboot to verify that this is the case.  In summary, we are in violent agreement.
<!-- Date: Tue Jun 27 07:37:40 2006 -->
<h3>Clock issues</h3> 
Got paged around 7:05 because of <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=438&frame=2&anchor=06:47:01.390'> 
clock problems</a>. Apparently, Roberto Davila was moving a NIM crate which was switched off. The problems occured when he unplugged the clock cable from that crate, so it may be a termination problem. Steve Hahn was paged and detected a  
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=438&frame=2&anchor=07:24:16.809'> 
clock error</a> at 6:43. Steve walked the aces through reseting the error and advised the shift crew to page me. I have checked the CCC in b0svx02. It is locked to RF and does not report any error. Roberto's further work on the clock cable is postponed until after this store. What I do not understand: why would a termination error in the clock fanout cause a clock error? 
<!-- Author: Ulrich -->
<!-- Comment: Tue Jun 27 10;33;48 comment by...Sebastian -->
The new RC/interrupt code will clear all errors other than lost of RF lock. You can check the interrupt counter by typing ccc_interrupt_counter on the mvme. There were several clock errors but lock was not lost.
<!-- Date: Tue Jun 27 07:40:34 2006 -->
<h3>SVX/L00 conductivity below warning limit</h3> 
As of this morning, the conductivity in the cooling circuit of the SVX chiller went below its warning limit of 4.0 μSi/cm. So I have un-ignored the corresponding alarm. For the ISL conductivity, we have to wait for the probe to be replaced.
<!-- Author: Ulrich -->




<!-- Date: Tue Jun 27 08:56:53 2006 -->
<h3>Another 2-shift D0 access?</h3> 
Willis mentioned in the 8:00 meeting this morning that there may be another 2-shift access for D0 upon us. We should start making plans for that occasion. The cooling work (exchange of conductivity probe) is a good candidate. Anything else?
<!-- Author: Ulrich -->


<!-- Date: Tue Jun 27 09:38:37 2006 -->
<table><tr><td><a href=/elog/notebooks/silicon/Tue_Jun_27_2006_093803.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8865  align='top'></a></td><td>Juan Pablo pointed out that CDF note 4375 had the information about the z side connections of the Hamamatsu detector I was asking around. Layer 0,1,3 are multiplexed 4:1, 3:1 and 2:1, 
respectevely.</td></tr></table>
<!-- Author: ignacio -->






<!-- Date: Tue Jun 27 12:32:22 2006 -->
<h3>Further work to catch the bursty L1A</h3> 
 
<p>The first step to catching the L1A is check the rest of CDF sees the same bursts of L1A that the SRC sees.  We already have one LA looking at a DIRAC TRACER and another looking at the SRC  The trick now is to make sure both are tirggered at the same time  .</p> 
 
<p>Thanks to Dervin's crew, a cable from the first to second floor was pulled which now allows the two LA to be connected.  The SRC LA now sends a TTL signal whenever it is triggered.  The TRACER LA is triggered by this TTL signal.</p> 
 
<p>We have tested this by making the SRC LA trigger on L1A and both LA saw the same signals. This is good.  Now the SRC LA is set up to trigger on SRC ERROR.  Now we just have to wait for another bursty L1A to trigger the GhostBuster.</p>
<!-- Author: Ankush -->
<!-- Comment: Tue Jun 27 19;57;57 comment by...Ankush, Marcel and Peter Wilson -->
<h3>A third LA has been added!!</h3>

<p>With Peter Wilson's help, we now have a third logic analyser added to our logic analyser chain.  This third LA is looking at the L1A signals from the TS and PREFRED TRACER.  So on the same LA, we can see how the TS and the TRACER (ie:- the rest of CDF) see the L1A signals.  We can then check the SRC sees the same signals aswell.</p>
<!-- Date: Tue Jun 27 12:39:12 2006 -->
<h3>2.5 hour access available after the store termination.</h3> 
 
<p>Today's store will terminate at 16:00.  There is a 2.5hr access avaialble.  The Diamond may go in to have a look at the CMS Diamond.  We can use the quiet time to have a look at a couple of ladders and think about further things to do to help catch the burtsy L1A.</p>
<!-- Author: Ankush -->

<!-- Date: Tue Jun 27 19:31:59 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB5W0L2 
<li>problem n: 1686 
<li>Status: needs_observation
<li>Type: AVDD
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Increase IMON  limits for AVDD as it has been going up lately.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>834.316</td><td>870</td></tr>
<tr><td><b>Lower</b></td><td>800</td><td>845</td></tr>
<tr><td><b>Upper</b></td><td>850</td><td>895</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B5W0&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Tue Jun 27 19:51:54 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W5L3 
<li>problem n: 867 
<li>Status: needs_observation
<li>Type: AVDD2
<li>Action: Chip_bit_changes
<li>entered by: husemann 
<li>description is: This ladder has not returned any data from the z-side (chips a3-a0) after the shutdown. We have chopped the z-side, and set all corresponding ISEL bits and Rdriver values to 0. After that, the ladder runs error-free in SVXDAQ. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B0W5&layer=L3">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Tue Jun 27 19:55:51 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB1W2L4 
<li>problem n: 1243 
<li>Status: needs_work
<li>Type: AVDD
<li>Action: Diagnosis_update
<li>entered by: husemann 
<li>description is: We tried to power up and run this ladder again. It was taken out of the HWDB recently because it tripped on AVDD. We could bring up the ladder with some difficulties (DVDD trips). Shortly after we started to run the ladder in SVXDAQ, it tripped on AVDD again. For a short time, we could see data coming from the first four chips. So the ladder stays out for now. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B1W2&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Tue Jun 27 20:00:49 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB5W0L2 
<li>problem n: 1686 
<li>Status: needs_observation
<li>Type: AVDD
<li>Action: Diagnosis_update
<li>entered by: husemann 
<li>description is: Just as a cross check, we ran this ladder in SVXDAQ. It returned reasonable data and ran fine for 5000 events. We will keep an eye on AVDD, though. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B5W0&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Tue Jun 27 20:04:57 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: IB0W2L4 
<li>problem n: 110 
<li>Status: needs_access_8hr
<li>Type: Chip_bit_error
<li>Action: Diagnosis_update
<li>entered by: husemann 
<li>description is: We have revisited this ladder which has a history of severe bit errors. Running in SVXDAQ, we could only see signals from two channels: 0 and 32. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B0W2&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Tue Jun 27 20:07:01 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB5W2L2 
<li>problem n: 383 
<li>Status: needs_work
<li>Type: DVDD
<li>Action: Diagnosis_update
<li>entered by: husemann 
<li>description is: This ladder still trips on DVDD whenever we tried to switch it on. The ladder stays out. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B5W2&layer=L2">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Tue Jun 27 20:17:00 2006 -->
We did some quiet time studies  during the access today: 
<ol> 
<li>e124 was taken out of the HWDB earlier in the week. We gave it a try again. It was difficult to put it in standby, as it was tripping on DVDD current overflow. We manage to run  it twice but it tripped on AVDD (as earlier during the week) after a few events.  
Only 4 (3) chips were giving data the first (second) time. 
<li>e053. After some tests SVXDAQ, the 4 z-chips were chopped from  the HWDB follwoing the instructions  documented <a href="http://www-cdf.fnal.gov/internal/silicon/expert.html">here</a>. 
  This ladder has a history of losing the z-side and getting it  back again. Observation. 
 <li>As it seemed stable we increased the e502 IMON limits by 35 mA. 
 <li> Light level measurement (units are μW) 
<pre> 
Bit   previous current 
OBVD  256      we forgot to measure this... 
bit0  304      we forgot to measure this... 
bit1  432      344 
bit2  376      304 
bit3  448      240 
bit4  352      216 
bit5  304      344 
bit6  noise    272 
bit7  352      256        
</pre> 
   The mapping can be found  
<a href="http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=304&time=14:03:31e262:"> 
here</a>. This result looks very weird. Maybe we did something wrong? 
  <li> Could not power up e522, trips on DVDD immediately.  
  <li>ADCMon plots for f024 looks very bad.  It seems it present multiple bit errors. SVXDAQ data showed only "1" and "32" values. Maybe it still works as a digital detector?   
  </ol> 
The bottom line: nothing fixed, one z-side chopped.
<!-- Author: Ulrich & Ignacio -->


<!-- Date: Wed Jun 28 00:20:33 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8866&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8867  width='300' align='top'></a></td><td><h3>Store 4802</h3> 
Stor 4802 came in with an initial luminosity of 108e30 cm<sup>-2</sup>s<sup>-1</sup>. All losses were <em>very</em> small from the beginning, everything smaller than 400 Hz. Obviously, silicon went in straight away. Also SVTSpyMon and SVXMON look fine.</td></tr></table>
<!-- Author: Ignacio & Ulrich -->

<!-- Date: Wed Jun 28 00:41:06 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W1L2 
<li>problem n: 1687 
<li>Status: needs_observation
<li>Type: AVDD
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Increase IMON  limits for AVDD as it has been going up lately.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>834.316</td><td>870</td></tr>
<tr><td><b>Lower</b></td><td>800</td><td>845</td></tr>
<tr><td><b>Upper</b></td><td>850</td><td>895</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B0W1&layer=L2">report</a></ul>
<!-- Author: Problem DB -->
<!-- Comment: Wed Jun 28 00;49;36 comment by...Ignacio & Ulrich -->
This entry is weird. We did not change any limits for e012 tonight. The entry looks like a duplicate of <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=307&frame=2&anchor=19:31:59'>
a previous entry for e502</a>


<!-- Date: Wed Jun 28 00:52:24 2006 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8868&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8869' ></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8870&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8871' ></a></center>
<!-- Author: Ignacio & Ulrich -->
<!-- Comment: Wed Jun 28 00;54;02 comment by...Ignacio & Ulrich -->
Our current candidates for pinkies: e012 has been jumping up and down for a while now. Currently it is down, and could not be brought up again by an HRR. We have changed the limits on e502 earlier today, but the current behaves quite erratic. 


<!-- Comment: Wed Jun 28 01;29;14 comment by...Ulrich -->
After the single event upset, e012 came back to a reasonable value.
<!-- Date: Wed Jun 28 01:25:23 2006 -->
<h3>Trip: e32X and e33X - single event upset in FIB?</h3> 
We just had a weird trip of two neighboring wedges. We tried to recover, but it would trip again. First, we tried to hockerize the corresponding crate (10), which did not help either. Then we had a second thought about the granularity of the problem: two neighboring wedges make one FIB board. So we rebooted the FIB crate (b0fib04), and everything is back again. Upon reboot, we saw the following error message: 
<pre> 
Fib Slot 9 MSRAM Corrupt (Calc - Expect) = ffffff80 
</pre> 
This is actually the FIB board which corresponds to the two wedges. So we assume this was a single event upset in the FIB board.Since it happened during the reboot: is the sequencer RAM still corrupted? 
<p> 
At the same time, we had L3 reformatter errors. We are not sure if they are related to the FIB problem. The ACEs started a new run, and now also this problem disappeared.
<!-- Author: Ignacio & Ulrich -->
<!-- Comment: Wed Jun 28 08;20;08 comment by...Ulrich -->
Just learned that we need a "master reset of the FIB" to clear this error. So it is not the reboot which recovers the FIB board but rather the Config.


<!-- Comment: Wed Jun 28 08;52;46 comment by...Mark -->
Here is my best explanation, somebody with more knowledge correct me if I am wrong.  The contents that are loaded in the Micro-sequencer RAM are stored on the FIB in an EEPROM.  They are loaded into the RAM upon a power cycle, and presumably upon a reboot (with some kind of check done first).  Since the two wedges seem to be happy now, I would guess that the EEPROM was not corrupted, and the correct contents were written to the MSRAM.
<!-- Comment: Wed Jun 28 08;55;27 comment by...Mark -->
<pre>Ok, looking at the previous message from Ulrich, I guess the MSRAM is written at Config.</pre>
<!-- Comment: Wed Jun 28 11;59;33 comment by...Mark -->
Final comment.  The FIBs are configured on a reboot, so both statements are correct.
<!-- Comment: Wed Jun 28 16;03;15 comment by...Intelligent Designer -->
Post-Final comment: Simple elog search on "MSRAM Corrupt" <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=106&frame=2&anchor=07:57:03'>
explains</a> it all.  Upshot is that during Config the MSRAM is checked-if faulty, a WARNING is issued (ask Badgett why Merlin Warnings are not logged and generally invisible) and the contents of the EEPROM are downloaded to the MSRAM, the check is performed again.  If it fails, you fail the Config, so the EEPROM was definitely not corrupt (I don't recall ever seeing the EEPROM corrupt.
<!-- Date: Wed Jun 28 08:40:38 2006 -->
<h3>Turbulent morning</h3> 
While I was checking the e-log from home, I saw a couple of things happening: 
<h4>b0fib00 went bad</h4> 
The L3 Reformatter complained: 
<pre> 
RAWREF Error - UTILSILDEC - with code 39 -- START 
 Line - 1152 in: src/library/rr_rawevt.c 
 Buffer position: 38929 
 Error count: 1 of 1 
 Reject rate: 0.015 percent 
 
 Detector ID not known (yet?). 
 FIB ids found: 
  0xc4c4c4c4  0xc4c4c4c4  0xc4c4c4c4  0xc1c1c4c4  0xc1c1c4c4  0xc1c1c4c4  0xc1c1c4c4  0xc4c4c4c4  0xc4c4c4c4  0xc1c1c4c4 
 
 Error location code: 0x01001290 
 
 
 Crate Id?: 0x00000010 
 
 Error during scanning MINI structure! 
 Last Unit prob. worked on (all counts from 0): 
  SCPU: 1 
  VRB : 2 
  LINK: 9 
  MINI: 0 
 
 Buffer (38909 until posi+2=38931) 
 0xfc5023f1  0x0000c1c1  0xfc5030f1  0x0000c1c1  0xfc5031f1  0x0000c1c1  
 0xfc5032f1  0x0000c1c1  0xfc5033f1  0x0000c1c1  0xfc5034f1  0x0000c1c1  
 0x0a007000  0x10044110  0x10044110  0x04000400  0x04000400  0x04000400  
 0x04000400  0x04000400  0xc4c4c4c4  0x00000000  0xc4c4c4c4  
 
 
 NOTE: Event rejected! 
 
RAWREF Error - UTILSILDEC - with code 39 -- FINISHED: L<  944 
</pre> 
The Aces realized that b0fib00 went bad, rebooted the crate according to our instructions, and it came back fine. 
 
<h4>Resonance</h4> 
A resonance happened at 7:15: 
<pre> 
15'41" b0svx02:Messenger:Jun 28, 2006 07:15:41 AM->SRC Fatal Error:Sl 6 Resonance Detected  
 -->   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
[MLE] 
(MLE) b0svx00:Messenger:Jun 28, 2006 07:15:43 AM->Silicon Resonator: Sl 16 Ch 8 -> e073: Bytes: 00092 
unction (S_errno_ETIMEDOUT) 
</pre> 
 
<h4>Clock glitch</h4> 
At around 7:20, the CCC in b0svx02 detected a clock glitch. At the same time, many more problems in the other subdetectors showed up. This seems to coincide with a thunderstorm in the area. The Aces overlooked the clock error in the first place, so I called in, reset the CCC, and a new run was started without problems. So the CCC interrupt mechanism works. Here is the error logger: 
<pre> 
22'3" b0svx02:Messenger:Jun 28, 2006 07:22:04 AM->Running on interal CLK (0x006E,365) -->   
 Additional Information:  
 
 Attention!!!. FERML_SILICON_CLOCK_ERROR !!! 
   Silicon DAQ has detected a clock error.   
   Run is HALTED.                             
   Please leave RunControl in the HALTED state  
   and page Silicon right away for further instructions.  
[MLE] 
</pre> 
and a little later this: 
<pre> 
22'9" b0svx06:Messenger:Jun 28, 2006 07:22:10 AM->SRC Fatal Error:Sl 6 L2A w/o L1A 2 L1A to Buff  
 -->   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx06: Sl 6 L2A w/o L1A 2 L1A to Buff  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
</pre>
<!-- Author: Ulrich -->
<!-- Comment: Wed Jun 28 09;08;09 comment by...sebastian -->
No trips due to the clk glitch?

<!-- Comment: Wed Jun 28 09;11;25 comment by...Ulrich -->
No trips. But the reason could be that the Ace was doing a couple of HRRs due to other problems.

<!-- Comment: Wed Jun 28 09;30;11 comment by...Sebastian -->
Yes, the "7:22:13 Halt -> HALTED" transition comming from the CCC interrupt is followed by "7:22:20 Recover -> RECOVERED", <b>which is good, since this is what we want RC to do anyway</b>. I will write to Donatella to ask her to change the pop-up window, to ask the ace to recover (I hope that eventually the transition to RECOVER will be automatic upon a CCC RF interrupt).

<!-- Comment: Wed Jun 28 09;34;31 comment by...Sebastian -->
Also note that the CDFCLK crate did not report any errors (see
<a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/ClockErrors'>here</a>).


<!-- Comment: Wed Jun 28 12;33;49 comment by...s -->
The message window will now read "Please take RunControl to the RECOVERED state" (thanks to Donatella). The automatic transition is not there yet, but should be ready soon. 
<!-- Date: Wed Jun 28 09:36:44 2006 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8872&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8872' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8873&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8873' width='300'></a><a href='/elog/notebooks/silicon/Wed_Jun_28_2006_092028.ps'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8875' ></a><a href='/elog/notebooks/silicon/Wed_Jun_28_2006_092037.ps'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8877' ></a></center>
<!-- Author: Ankush -->
<!-- Comment: Wed Jun 28 10;30;13 comment by...Ankush & Mark -->
Here is a snippet from the RunControl log from this time:

<pre>
(MLW) b0control21.fnal.gov:Thread-33:Jun 28, 2006 01:58:32 AM->Inhibit flag SET 
(MLW) b0control21.fnal.gov:Thread-33:Jun 28, 2006 02:00:12 AM->Inhibit flag CLEAR 
(MLW) b0control21.fnal.gov:Thread-33:Jun 28, 2006 02:00:15 AM->Inhibit flag SET 
(MLW) b0control21.fnal.gov:Thread-33:Jun 28, 2006 02:02:23 AM->Inhibit flag CLEAR 
CT: 2006.06.28 02:02:29
2'29" b0control20.fnal.gov:resmgr.ReplyMonitor 2:Jun 28, 2006 02:02:29 AM->Dead Time is High at 94.88036% is greater than 20.0% -->  
 Additional Information: 

 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 94.88036 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
[MLE]
CT: 2006.06.28 02:03:30
3'30" b0control20.fnal.gov:resmgr.ReplyMonitor 2:Jun 28, 2006 02:03:30 AM->Dead Time is High at 94.88036% is greater than 20.0% -->  
 Additional Information: 

 Attention!!!. FERML_HIGH_DEADTIME !!! 
 - Trigger/DAQ Deadtime is too high (at 94.88036 %). 
 - Look for source of hot triggers. 
 - Contact a trigger expert for further assistance. 
[MLE]
(MLW) b0control21.fnal.gov:Thread-33:Jun 28, 2006 02:04:08 AM->StateManager: User has requested an over-ride!
(MLW) b0control21.fnal.gov:Thread-33:Jun 28, 2006 02:04:48 AM->StateManager: User has requested an over-ride!
(MLE) b0svx04:Messenger:Jun 28, 2006 02:04:49 AM->Silicon Resonator: Sl 20 Ch 3 -> e4a3: Bytes: 00098
unction (S_errno_ETIMEDOUT)
(MLE) b0svx05:Messenger:Jun 28, 2006 02:04:49 AM->Silicon Resonator: Sl 08 Ch 3 -> e503: Bytes: 00052
unction (S_errno_ETIMEDOUT)
(MLE) b0svx01:Messenger:Jun 28, 2006 02:04:50 AM->Silicon Resonator: Sl 10 Ch 9 -> e134: Bytes: 00066
unction (S_errno_ETIMEDOUT)
(MLE) b0svx03:Messenger:Jun 28, 2006 02:04:50 AM->Silicon Resonator: Sl 10 Ch 9 -> e334: Bytes: 00052
unction (S_errno_ETIMEDOUT)
(MLE) b0svx08:Messenger:Jun 28, 2006 02:04:49 AM->Silicon Resonator: Sl 16 Ch 3 -> f843: Bytes: 00780

(MLE) b0svx07:Messenger:Jun 28, 2006 02:04:50 AM->Silicon Resonator: Sl 13 Ch 3 -> fa23: Bytes: 00780

(MLE) b0felix.fnal.gov:TEDCpp:Jun 28, 2006 02:04:51 AM->{ 2:4:51, 6/28/2006 }: L2-Pulsar-Linux: Halted while  waiting for SVT data
(RC)   2:04:51 Halt -> HALTED
CT: 2006.06.28 02:05:26
5'26" 23 crate/s: b0svt09(116), b0svt08(117), b0clc00(118), b0svt00(120), b0svt01(121), b0svt07(122), b0svt02(123), b0svt06(124), b0svt03(125), b0svt05(126), b0svt04(127), b0tsi03(138), b0puls00(139), b0l2cl00(144), b0l2cl01(145), b0l2cl02(146), b0l2cl03(147), b0l2cl04(148), b0l2cl05(149), b0l2pu01(150), b0l2pu00(152), b0htdc00(153), b0etdc00(154),  in error. [RXPT]
(RUN Comment) rundb.RunComment@1fab791
  RunNumber   219221
  CreateTime  2006-06-28 02:05:27.0
  TextComment b0tsi02 error.  start a new run.
  Name        gianluca
  State       TERMINATE
  ELog        true
</pre>

So it looks like there was a trigger inhibit where RunControl was not halted leading to high deadtime.  After the inhibit was cleared, an HRR was issued but b0tsi02 (the return crosspoint) was stuck in the recover transition.  Instead of contacting an expert for recovery procedures, the ACE manually overrode the Recover and Run transitions.  We were then in Activate with b0tsi02 in some weird state, which explains the LA trace where we see no L1As since the return crosspoint montitors the DONE signals. Finally the ACE realized that b0tsi02 was still in a bad state and ended the run.  This seems to be confirmed by the ACE's comment:

<pre>
we had a TOF trip (the fake trip with a quarter of the detector off , the trigger inhibit set,
but no red alarm on iFix).  I halted the run but the system was waiting for b0tsi02 going through
the partition.  It took a too long time so I decided to override the state manually.  It worked.  


I ended the run to start a new one.  During the end transition we lost the EventBuilder Proxy.  I
resetted and from START I cleaned the EVB.  EvbProxy came back for partition 0: now we're running
again.
</pre>

So one remaining question: Why are the SRCs sending so many L1Ds when we are not seeing any L1As from the TS?
<!-- Comment: Wed Jun 28 10;39;48 comment by...Sebastian -->
are those l1ds real? (what's the width of the signal?)
<!-- Comment: Wed Jun 28 12;54;30 comment by...Ankush -->
Sebastian is correct.  The width of the L1D lines is 5ns (see
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=307&frame=2&anchor=12:46:37'>
here</a>) which is too small.  A real L1A is 132ns wide.</p>
<!-- Date: Wed Jun 28 09:46:41 2006 -->
Since the logic analyzer setup had triggered on an SRC error at around 2:00, the following burst of errors was missed: 
<pre> 
(MLE) b0control21.fnal.gov:Thread-33:Jun 28, 2006 08:04:19 AM->FrontEnd Error: VRB_SVX_02  
CT: 2006.06.28 08:04:23 
4'23" 1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLE) b0control21.fnal.gov:Thread-33:Jun 28, 2006 08:04:29 AM->  Silicon Level 1 Done Timeout Error from SVX, ISL or L00 
 
CT: 2006.06.28 08:04:33 
4'33" L1 Done Timeout.[RXPT] 
CT: 2006.06.28 08:04:36 
4'36" 1 crate/s: b0svx02(112),  in error. [RXPT] 
CT: 2006.06.28 08:04:38 
4'38" L1 Done Timeout.[RXPT] 
CT: 2006.06.28 08:04:40 
4'40" 1 crate/s: b0svx02(112),  in error. [RXPT] 
CT: 2006.06.28 08:04:42 
4'42" L1 Done Timeout.[RXPT] 
CT: 2006.06.28 08:04:44 
4'44" 1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLE) b0svx03:Messenger:Jun 28, 2006 08:04:50 AM->Silicon Resonator: Sl 20 Ch 7 -> e3b2: Bytes: 00170 
unction (S_errno_ETIMEDOUT) 
(MLE) b0svx00:Messenger:Jun 28, 2006 08:04:50 AM->Silicon Resonator: Sl 08 Ch 7 -> e012: Bytes: 00324 
unction (S_errno_ETIMEDOUT) 
(MLE) b0svx08:Messenger:Jun 28, 2006 08:04:50 AM->Silicon Resonator: Sl 16 Ch 3 -> f843: Bytes: 00780 
 
(MLE) b0svx07:Messenger:Jun 28, 2006 08:04:50 AM->Silicon Resonator: Sl 13 Ch 3 -> fa23: Bytes: 00780 
 
CT: 2006.06.28 08:04:50 
4'50" b0svx02:Messenger:Jun 28, 2006 08:04:50 AM->SRC Fatal Error:Sl 6 Resonance Detected  
 -->   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
</pre>
<!-- Author: Ulrich -->





<!-- Date: Wed Jun 28 12:00:22 2006 -->
<h3>Another Clock Glitch</h3> 
 
<p>In my inbox, I just noticed an email from the logic analyser. I looked at the trace and I could not see a resonance. Then the ACE came in and said there was a Silicon problem.....we just had another Clock Glitch. Again, when this error occured, there was an "error box explosion" where the Clock Error was buried.  So the ACE HRR out of it and then we got other SRC errors as it was being driven by the CCC internal clock and not the CDF clock.</p> 
 
<p>I logged into the SVX02 and told the CCC to run off the CDF clock and then things started running as normal.  Below is the sequence you need to follow:</p> 
 
<pre> 
<svxii@b0dasvx01>setup fer 
setup-ppc: assuming PowerPC 603 processor 
<svxii@b0dasvx01>vxcom b0svx02 
 
Escape sequence is ^A. (ctrl-A period) 
One moment please... 
 
 
 
29916: Warning: No xauth data; using fake authentication data for X11 forwarding. 
 
 
b0svx02-> 
b0svx02-> ccc 
 
 Welcome to the CCCCC (CCC control code) 
 
CCC Base Address? 0x3000 
 
Your options: 
[0] enable IRQ and set vector to 0x80 
[1] change base addr 
[2] show status register 
[3] clear errors 
[4] toggle clk mode 
[5] reset board 
[6] read fwr version number 
[7] write to a register 
[8] read a register 
[9] show interrupt counter 
[f] Old board init (sets up a ISR...) 
[a] bye 
 
Your pick? 0x2 
Status: 0, Read: 7e 
BZ error 
BX error 
SY error 
GP error 
Master clk error 
Running on interal CLK 
 
Your options: 
[0] enable IRQ and set vector to 0x80 
[1] change base addr 
[2] show status register 
[3] clear errors 
[4] toggle clk mode 
[5] reset board 
[6] read fwr version number 
[7] write to a register 
[8] read a register 
[9] show interrupt counter 
[f] Old board init (sets up a ISR...) 
[a] bye 
 
Your pick? 0x4 
Status: 0, Wrote: 0x4 
Your options: 
[0] enable IRQ and set vector to 0x80 
[1] change base addr 
[2] show status register 
[3] clear errors 
[4] toggle clk mode 
[5] reset board 
[6] read fwr version number 
[7] write to a register 
[8] read a register 
[9] show interrupt counter 
[f] Old board init (sets up a ISR...) 
[a] bye 
 
Your pick? 0x3 
Status: 0, Wrote: 0x8 
Your options: 
[0] enable IRQ and set vector to 0x80 
[1] change base addr 
[2] show status register 
[3] clear errors 
[4] toggle clk mode 
[5] reset board 
[6] read fwr version number 
[7] write to a register 
[8] read a register 
[9] show interrupt counter 
[f] Old board init (sets up a ISR...) 
[a] bye 
 
Your pick? 0x2 
Status: 0, Read: 0 
CCC is locked to RF 
 
Your options: 
[0] enable IRQ and set vector to 0x80 
[1] change base addr 
[2] show status register 
[3] clear errors 
[4] toggle clk mode 
[5] reset board 
[6] read fwr version number 
[7] write to a register 
[8] read a register 
[9] show interrupt counter 
[f] Old board init (sets up a ISR...) 
[a] bye 
 
 
<em>this is the message from run control when I told the ACE to go to RECOVER and then ACTIVE</em> 
Your pick? 0xSent ack to Run Control: 0 RECOVERED SUCCESS 
Sent ack to Run Control: 0 ACTIVE SUCCESS 
a 
bye... 
value = 0 = 0x0 
b0svx02-> Connection to b0serial04 closed. 
<svxii@b0dasvx01> 
</pre>
<!-- Author: Ankush -->
<!-- Comment: Wed Jun 28 12;01;38 comment by...Ankush -->
<p>....also the CDF Clock crate did not report any error.</p>



<!-- Date: Wed Jun 28 12:46:37 2006 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8878&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8878' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8879&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8879' width='300'></a></center>
<!-- Author: Ankush -->
<!-- Comment: Wed Jun 28 12;52;18 comment by...Ankush -->
<h3>False L1D Recorded</h3>

<p>An earlier entry I posted showed a LA trace with lots of L1D without any L1A.  However Sebastian pointed that they may be fake and indeed they are.  The left trace is from the morning.  The width of the L1A is 5ns which is too small.  The trace on the right has real L1A width of ~132ns. This width also holds true for the L2A and L2R.  The captured left hand trace has a L2A and L2R (see 
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=307&frame=2&anchor=09:36:44'>
here</a>) and their widths are 132ns; so they are real.</p>



<!-- Date: Wed Jun 28 14:04:32 2006 -->
<h3>Beam Problems</h3> 
The aces put the silicon to STANDBY because of high losses. The losses seem to stabilize after a while. However, there are still spikes, and we have not yet heard back from MCR as to what went wrong. Here is my conjecture, from looking at a couple of ACNET plots:  
<ul> 
<li>The luminosities both at CDF and D0 are now half of what they were before. The same is true for the BLM dose rate (less radiation from collisions?) 
<li>The beam current only dropped by a little bit. 
</ul> 
Ergo: they did not lose half of the beam, but somehow the beam got defocused. 
Luckily, the whole incident was clean for the silicon: no trips, no pinkies.
<!-- Author: Ulrich -->


<!-- Date: Wed Jun 28 14:10:51 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8881&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8881  width='300' align='top'></a></td><td>While I was typing the above comment, Ron Moore put this plot into the Tevatron e-log. So my defocusing theory seems to be correct. Here is Ron's comment: 
<pre> 
-- Wed Jun 28 13:44:47 comment by...rm --  According to sync-light, both beams may have blown-up transversely. (This only shows P1 and A1).   
</pre></td></tr></table>
<!-- Author: Ulrich -->
<!-- Comment: Wed Jun 28 15;00;35 comment by...Ulrich -->
Diagnostics update from the Tevatron e-log:
<pre>
- Wed Jun 28 14:50:55 comment by...annala --  It looks like the biggest effects are on Proton bunches 28-35, and Pbar buches 2-11. The place where these bunches collide is about at the C4 separators. Our guess about what happened would be a C4 separator spark that did not report. Bruce and Dan are going to check out the response of the spark counters with a pulse generator. They already checked out the A17H spark counter and found it working, but the threshold was set too high (making it not sensitive enough). 
</pre>
<!-- Comment: Wed Jun 28 16;26;56 comment by...Ulrich -->
From AD Run Coordinator e-log:
<pre>
15:49:17-  The cause of the emittance jump/luminosity drop/loss spike is thought to be a C4 separator spark. Although the symptoms point to the C4 separator, the spark counter did not indicate that a spark occured. Experts will test the spark counting system to make sure it's working properly. - J. Morgan
-- Wed Jun 28 15:51:14 comment by...JPM --  D0 was hit pretty hard by the losses. They are discussing whether a rescrape would be enough to allow them to turn their silicon back on.
</pre>
Note that D0 had also called the CDF control room to ask if we are running with silicon...
<!-- Date: Wed Jun 28 14:13:19 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8882&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8882  width='300' align='top'></a></td><td>And here is the plot of the BLM dose rates, actually a screen shot of the new Java application to replace E2. At the point in question, the rate drops, compatible with a drop in the collision-induced radiation.</td></tr></table>
<!-- Author: Ulrich -->



<!-- Date: Wed Jun 28 16:18:01 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W8L2 
<li>problem n: 1394 
<li>Status: unfixable
<li>Type: Chip_other
<li>Action: Diagnosis_update
<li>entered by: boveia 
<li>description is: Chip 85 is sparsed out already.  The *observed* CAP ID error rate will continue to disappear during low rate data taking.  There is nothing to be done.   
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B2W8&layer=L2">report</a></ul>
<!-- Author: Problem DB -->




<!-- Date: Wed Jun 28 17:17:09 2006 -->
<h4>Re-scraping</h4> 
Per D0's request, MCR is currently rescraping the beam. The recult is that all losses come back to rather resonable values, the proton abort gap losses being the largest with 1.1 kHz. Essentially, the CDF losses were the same before and after the scraping. Silicon was included again.
<!-- Author: Ulrich -->


<!-- Date: Wed Jun 28 19:22:45 2006 -->
<h3>Lost the clock again</h3> 
 
<p>Again the clock was lost and the CCC went to its internal mode. The ACEs paged Ulrich first not realising Ignacio and I were still in the silicon office.</p> 
 
<p>This time two wedges ended up tripping.  They were recovered and the CCC was reset.  Again the CDF clock did not report any errors. Unfortunatly we don't have an SRC LA trace as I was playing with the LA at the time.</p> 
  
<!-- Author: Ankush -->
<!-- Comment: Wed Jun 28 20;04;41 comment by...Sebastian -->
Took 4min to go from halt to recover, so no surprise something trip: 
<pre>
18:14:26 Halt -> HALTED
18:18:14 Recover -> RECOVERED
</pre>
We will go to recover automatically, but for now the aces should be remind to follow the pop up window.


<!-- Comment: Thu Jun 29 15;25;31 comment by...Ankush -->
<p>Peter Wilson heard a rumour that someone in AD did something to the clock yesterday.  But this is just rumour.</p>
<!-- Date: Wed Jun 28 22:59:29 2006 -->
<h3>Permanent Light Level Measurement: Take 2</h3> 
 
<p>Yesterday's permanent light level measurement came back with some odd results.  So we used the quiet time to repeat the measurements.</p> 
 
<table border=1> 
<tr> <td>Bit</td> <td>Last Measurement</td>  <td>Today's measurement</td>  </tr> 
<tr> <td> OBDV </td> <td> 276 uW </td> <td> 280 uW </td> </tr> 
<tr> <td> 0    </td> <td> 352 uW </td> <td> 304 uW </td> </tr> 
<tr> <td> 1    </td> <td> 468 uW </td> <td> 450 uW </td> </tr> 
<tr> <td> 2    </td> <td> 404 uW </td> <td> 416 uW </td> </tr> 
<tr> <td> 3    </td> <td> 456 uW </td> <td> 430 uW </td> </tr> 
<tr> <td> 4    </td> <td> 420 uW </td> <td> 392 UW </td> </tr> 
<tr> <td> 5    </td> <td> 392 uW </td> <td> 400 uW </td> </tr> 
<tr> <td> 6    </td> <td>  48 uW </td> <td>  58 uW </td> </tr> 
<tr> <td> 7    </td> <td> 440 UW </td> <td> 392 uW </td> </tr>        
</table> 
 
<p>This time round, the mesurements agree much better.</p>
<!-- Author: Ankush & Ignacio -->


<!-- Date: Wed Jun 28 23:03:56 2006 -->
<h3>Playing with the BUSY TIMEOUT value for COLDSTART</h3> 
 
<p>As we did the LL rather quickly, we used the remaining time to play with the BUST TIMEOUT value.  We tried to search for an optimal value from 40,000,000 to 80,000,000.  However it appeared COLDSTART would only run error free got 80,000,000.  However in playing with the BUST TIMEOUT, 80,000,000 did not work error free.  At that point we gave up.</p>  
<!-- Author: Ankush & Ignacio -->


<!-- Date: Thu Jun 29 00:15:29 2006 -->
<table><tr><td><a href=/elog/notebooks/silicon/Thu_Jun_29_2006_001526.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8891  align='top'></a></td><td>In b0sili01:/data01/pll there is a small macro (pll.C) that compiles the permanent light level measurements done so far. It is a bit primitive: One have to type in the  the measured values and DS Day in pll.dat and change the number of measurements plotted (NM) in pll.C; the macro makes the mapping and plots the data. </td></tr></table>
<!-- Author: Ignacio -->


<!-- Date: Thu Jun 29 01:20:33 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8892&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8892  width='300' align='top'></a></td><td>Paul Derwent has updated the new Java GUI for the BLM system. Now it shows if the FIFOs are recording, and he re-arranged the radio buttons a bit. I have tried to plot the logged data, which kind of works. It takes a couple of seconds to bring up the menu, and then an error shows up "Error making datalogger plot", and the plot is empty. II will notify Paul about this.</td></tr></table>
<!-- Author: Ulrich -->


<!-- Date: Thu Jun 29 01:23:04 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8893&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8893  width='300' align='top'></a></td><td>This is how a FIFO plot looks like in the new GUI. This one has been taken from the running system, the one from the data logger is empty.</td></tr></table>
<!-- Author: Ulrich -->


<!-- Date: Thu Jun 29 01:34:21 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8894&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8895  width='300' align='top'></a></td><td>SVTSpyMon plot for the beginning of store 4804. 
Initial lumi was larger than previous stores 128 E30.  
Low and flat backgrounds: LOSTP 400 Hz, LOSTPB 120 Hz, AGIGI2 0.55E9. So silicon was in from the start.</td></tr></table>
<!-- Author: Ignacio & Ulirich -->


<!-- Date: Thu Jun 29 01:38:51 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8896&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8896  width='300' align='top'></a></td><td>Another try for the FIFO data, this time saved directly from the Java GUI.</td></tr></table>
<!-- Author: Ulrich -->


<!-- Date: Thu Jun 29 02:11:10 2006 -->
<h3>Event builder CPU in b0svx05 stuck</h3> 
b0svx05 turned red in Run Control due to an event builder problem. The ace instructions say that one should push the reset button on the event builder CPU to bring it back, which worked fine. Both the L3/evb expert on the phone and myself pointed out that it is potentially dangerous to reboot the wrong CPU. Is it understood why a stuck event builder CPU would stop the crate CPU as well? VME-bus blocked?
<!-- Author: Ulrich -->


<!-- Date: Thu Jun 29 10:49:05 2006 -->
<h3>SVXMon back on the web</h3> 
 
<p>Despite being Lammelled, SVXMon is back on the web...albeit in a different way.  Bill Badgett has added a "Consumer Root File Browser" to the Run Summary page.  So now you can check SVXMon from home again :-)</p> 
<!-- Author: Ankush -->
<!-- Comment: Mon Jul  3 06;17;46 comment by...Ignacio & Ulrich -->
The direct link for run 219388 is <a href="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/ConsumerBrowser?RUN=219388">http://www-cdfonline.fnal.gov/java/cdfdb/servlet/ConsumerBrowser?RUN=219388</a>
<!-- Date: Thu Jun 29 11:07:05 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8897&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8897  width='300' align='top'></a></td><td>Popup window when CCC looses lock to RF.</td></tr></table>
<!-- Author: Sebastian -->
<!-- Comment: Thu Jun 29 11;22;43 comment by...Sebastian -->
When the CCC looses lock to RF, RC goes through HALT to RECOVER and pops up the window above. The silicon expert on call has to check if she/he can sync up the CCC again and continue data taking or call a DAQ expert if the problem persists. <br>
Upon the request of Ulrich and Ankush I will setup a simple comand to lock the CCC and report problems.

<!-- Comment: Thu Jun 29 11;23;29 comment by...s -->
Forgot to say that the new RC code was tested in tstsvx1 and will go in asap.


<!-- Comment: Thu Jun 29 16;33;37 comment by...Ulrich -->
Bill and Donatella have just released a new version of Run Control with our changes included, see <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=446&frame=2&anchor=16:27:22.425'>
main e-log</a>.
<!-- Date: Thu Jun 29 13:22:07 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8898&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8898  width='300' align='top'></a></td><td>diamonds going nutso when solenoid tripped</td></tr></table>
<!-- Author: rainer -->
<!-- Comment: Thu Jun 29 13;22;46 comment by...rainer -->
<pre>eas diamonds are offset by a 1000</pre>




<!-- Date: Thu Jun 29 20:15:55 2006 -->
<h3>Magnet back up</h3> 
The magnet is back up, so we can continue data taking. The losses are still fine, although AGIGI2 is kind of high (4.7 right now). The SciCo has called MCR who claim this variable has been stable for hours (well, at least for the last hour, this is a true statement), and they expect it to stay there.
<!-- Author: Ulrich -->


<!-- Date: Thu Jun 29 20:21:46 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8899&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8899  width='300' align='top'></a></td><td>T:AGIGI2 and C:B0PAGC for the last couple of hours. Does is settle below 5e9?</td></tr></table>
<!-- Author: Ulrich -->


<!-- Date: Thu Jun 29 22:58:15 2006 -->
<h3>AGIGI2 Stuff</h3> 
Ankush and I had another couple of looks at AGIGI2. We asked the shift crew to call MCR and make them aware of the situation. At that point, AGIGI was at 4.5e9 and slowly creeping up, eventually hitting our warning limit of 5e9. We gave out the 
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=446&frame=2&anchor=21:38:14.171'> 
policy</a> to watch things closely and keep the silicon in if all losses are stable and AGIGI2 is only increasing slowly. Then we headed home... 
<p> 
I got paged around 22:15. MCR wanted to adjust some tunes (of the electron lens?) which should not influence the beam too much. I agreed to keep silicon in during this time. First AGIGI2 came down nicely, but them it shot up again, eventually surpassing 6e9. At that point, the shift crew decided to put the silicon to standby and page me. Within a couple of minutes, AGIGI was decreasing. I adviced them to wait for a couple of minutes, call MCR if they are done with the tuning and re-include silicon if everything looks nice and stable.  
<p> 
Currently, AGIGI2 is around 3.5e9 and rather stable. So we are running again with silicon and see if things stay stable tonight.
<!-- Author: Ulrich -->


<!-- Date: Thu Jun 29 23:05:08 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8900&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8900  width='300' align='top'></a></td><td>Development of AGIGI2 for the last 6 hours. Currently, it is still decreasing.</td></tr></table>
<!-- Author: Ulrich -->


<!-- Date: Fri Jun 30 02:22:00 2006 -->
<h3>MCR reports proton abort kicker problems</h3> 
Got paged because our shift crew has received a call from MCR. They have experts work a problem with one of the proton abort kickers. They warn us that this may cause a sloppy abort ans suggest to bring silicon to standby, which I emphatically agreed on. 
The shift crew will page me again in case something will happen. From MCR: 
<pre> 
The proton abort kicker #5 has faulted off on a "personnel interlock OK" bit. We have contacted Chris Jensen and Dirong Chen and they are coming in. We contacted CDF and D0 in case they want to power down until the kicker is repaired. D0 will power down when experts start the repair and CDF is contacting their Ops manager for guidence. 
</pre>
<!-- Author: Ulrich -->


<!-- Date: Fri Jun 30 05:18:26 2006 -->
Since the kicker business is very risky for the Silicon, the shift crew took it to standby for the repair work. A little later, also the COT was ramped down. 
More on the kicker problem from MCR: 
<pre> 
-- Fri Jun 30 02:30:49 comment by...jdc --  CDF silicon is down for the kicker repairs.  
-- Fri Jun 30 03:00:48 comment by...jdc --  Chris and Dirong are here. We have contacted D0 and they are powering down.  
-- Fri Jun 30 03:30:18 comment by...jdc --  Repairs are finished. CDF and D0 are turning back on.  
-- Fri Jun 30 03:32:31 comment by...djinn --  Chris reports that the kicker was off due to a "remote off" command (no one sent this command, but the card saw it anyway).  
-- Fri Jun 30 03:43:24 comment by...C. Jensen, D. Chen --  The kicker turned back on with a remote on command. Would like to inspect the kicker status and control module to check if the remote on and off command inputs have a filter on them. 
 
03:30:29-  Chris Jensen requests that the 465 card for C:PAK5 be replaced between stores. - djinn 
</pre>
<!-- Author: Ulrich -->
<!-- Comment: Fri Jun 30 08;57;15 comment by...Ulrich -->
Update on PAK5:
<pre>
08:29:02-   Rupe changed out the 465 card for PAK5, restored any diffs for the crate. - spb
-- Fri Jun 30 08:55:44 comment by...C. Jensen, D. Chen --  We added a filter to the remote on and off commands for the status and control card on PAK5. Visual inspection ok.
08:51:08-   PAK5 is ready to go, did a C49 restore. - spb
</pre>
<!-- Date: Fri Jun 30 05:28:29 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8901&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8901  width='300' align='top'></a></td><td>Got paged around 5:00 because T:AGIGI2 was rising to 6 momentarily. The shift crew took silicon to standby immediately. I advised them to watch the beam for 5 minutes and reinclude silicon. Shortly after that, a second excursion happened. I called the control room, and the SciCo told me that they already had spotted it but MCR claims they did not do anything. They will keep an eye on things.</td></tr></table>
<!-- Author: Ulrich -->



<!-- Date: Fri Jun 30 09:26:22 2006 -->
Just performed the regular reboot of SVXIICON.
<!-- Author: Ignacio & Ulrich -->


<!-- Date: Fri Jun 30 10:45:05 2006 -->
<h3>Just moved the Logic Analyser</h3> 
 
<p>The multi-logic analyser setup was getting in the way of Eric James' TDC tutorial.  Dervin got a long cable for us and now the logic analyser are in less obtrusive.  Also Peter Wilson helped tidy up the cables to the DIRAC TRACER</p> 
 
<p>I did a quick test and the LA all still fire....we are good to go !!</p> 
 
<p>...and I probably owe Dervin a few 6 packs (!!)</p>
<!-- Author: Ankush -->


<!-- Date: Fri Jun 30 11:42:31 2006 -->
<pre> 
<svxii@b0dasvx01>source sourceme 
Setting up caencontrol 
You can start the PSGui with "PS" 
<svxii@b0dasvx01>PS 
User: svxii 
Display: 
Attempting initialization of CAEN at svxinter.fnal.gov... 
Downloading /cdf/onln/data/svxii/caencontrol/PSfiles/PS.conf 
Loading startup files to the server.....Please be patient 
GUI Startup Done! 
Full thread dump Java HotSpot(TM) Client VM (1.4.2_09-b05 mixed mode): 
 
"DestroyJavaVM" prio=1 tid=0x4e20a878 nid=0x2845 waiting on condition [0..bfffcb24] 
 
"Thread-1" prio=1 tid=0x4e20a570 nid=0x2859 sleeping[4e8ff000..4e8ff8c4] 
        at java.lang.Thread.sleep(Native Method) 
        at CAEN.PSControlForm.run(PSControlForm.java:3026) 
        at java.lang.Thread.run(Thread.java:534) 
 
"AWT-EventQueue-0" prio=1 tid=0x4e2023d0 nid=0x2858 runnable [4e6ff000..4e6ff8c4] 
        at java.net.PlainSocketImpl.available(PlainSocketImpl.java:418) 
        - waiting to lock <0x44ceab00> (a java.net.PlainSocketImpl) 
        at java.net.SocketInputStream.available(SocketInputStream.java:217) 
        at java.io.BufferedInputStream.available(BufferedInputStream.java:337) 
        - locked <0x44ce2d30> (a java.io.BufferedInputStream) 
        at gov.fnal.robin.rRequest.get_response(rRequest.java:380) 
        at gov.fnal.robin.rRequest.invoke(rRequest.java:177) 
        at CAEN.CaenVme.CaenClient.getSiliconInfo(CaenClient.java:582) 
        at CAEN.PWR.resetValuesFromVMEFunction(PWR.java:530) 
        at CAEN.PWR.access$000(PWR.java:43) 
        at CAEN.PWR$resetValuesFromVMEThread.run(PWR.java:429) 
        at CAEN.PWR.resetValuesFromVME(PWR.java:475) 
        at CAEN.PSControlForm.refresh(PSControlForm.java:1627) 
        at CAEN.PSControlForm.access$000(PSControlForm.java:41) 
        at CAEN.PSControlForm$1.actionPerformed(PSControlForm.java:752) 
        at javax.swing.Timer.fireActionPerformed(Timer.java:271) 
        at javax.swing.Timer$DoPostEvent.run(Timer.java:201) 
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:189) 
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:478) 
        at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:201) 
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:151) 
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:145) 
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:137) 
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:100) 
 
"AWT-Shutdown" prio=1 tid=0x4e2021d0 nid=0x2857 in Object.wait() [4df5b000..4df5b8c4] 
        at java.lang.Object.wait(Native Method) 
        - waiting on <0x44cdad18> (a java.lang.Object) 
        at java.lang.Object.wait(Object.java:429) 
        at sun.awt.AWTAutoShutdown.run(AWTAutoShutdown.java:259) 
        - locked <0x44cdad18> (a java.lang.Object) 
        at java.lang.Thread.run(Thread.java:534) 
 
"Java2D Disposer" daemon prio=1 tid=0x081ed490 nid=0x2856 in Object.wait() [4eaff000..4eaff8c4] 
        at java.lang.Object.wait(Native Method) 
        - waiting on <0x44d65ed0> (a java.lang.ref.ReferenceQueue$Lock) 
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111) 
        - locked <0x44d65ed0> (a java.lang.ref.ReferenceQueue$Lock) 
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127) 
        at sun.java2d.Disposer.run(Disposer.java:100) 
        at java.lang.Thread.run(Thread.java:534) 
 
"TimerQueue" daemon prio=1 tid=0x0816a718 nid=0x2850 in Object.wait() [4e4ff000..4e4ff8c4] 
        at java.lang.Object.wait(Native Method) 
        at javax.swing.TimerQueue.run(TimerQueue.java:231) 
        - locked <0x44d15750> (a javax.swing.TimerQueue) 
        at java.lang.Thread.run(Thread.java:534) 
 
"AWT-Motif" daemon prio=1 tid=0x08144978 nid=0x284f runnable [4e15b000..4e15b8c4] 
        at sun.awt.motif.MToolkit.run(Native Method) 
        at java.lang.Thread.run(Thread.java:534) 
 
"Signal Dispatcher" daemon prio=1 tid=0x08099430 nid=0x284b waiting on condition [0..0] 
 
"Finalizer" daemon prio=1 tid=0x08095030 nid=0x2849 in Object.wait() [4ccc0000..4ccc08c4] 
        at java.lang.Object.wait(Native Method) 
        - waiting on <0x44cdae18> (a java.lang.ref.ReferenceQueue$Lock) 
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111) 
        - locked <0x44cdae18> (a java.lang.ref.ReferenceQueue$Lock) 
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127) 
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) 
 
"Reference Handler" daemon prio=1 tid=0x08093550 nid=0x2848 in Object.wait() [4cac0000..4cac08c4] 
        at java.lang.Object.wait(Native Method) 
        at java.lang.Object.wait(Object.java:429) 
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115) 
        - locked <0x44cdae80> (a java.lang.ref.Reference$Lock) 
 
"VM Thread" prio=1 tid=0x080903f0 nid=0x2847 runnable 
 
"VM Periodic Task Thread" prio=1 tid=0x0809bb60 nid=0x284d waiting on condition 
"Suspend Checker Thread" prio=1 tid=0x080989f0 nid=0x284a runnable 
 
 
 
 
 
Suspended 
<svxii@b0dasvx01>bg 
[1]    java -Xbootclasspath/p:/cdf/onln/code/cdfprod/robin/v2_4_10/classes -ss100m  ... & 
<svxii@b0dasvx01>pwd 
/cdf/onln/data/svxii/caencontrol/PSfiles 
[1]  + Done                          java -Xbootclasspath/p:/cdf/onln/code/cdfprod/robin/v2_4_10/classes -ss100m  ... 
<svxii@b0dasvx01>ls 
CAENcurrent.dat@                     DecrementedCAENvolts-dV30-L-1.dat 
caencurrentfiles/                    DecrementedCAENvolts-dV35-L0.dat 
caenIVcheck.pl*                      DecrementedCAENvolts-dV35-L-1.dat 
CAENLVcurrent.dat@                   DecrementedCAENvolts-dV40-L0.dat 
CAENLVcurrent.dat~                   DecrementedCAENvolts-dV40-L-1.dat 
caenlvcurrentfiles/                  DecrementedCAENvolts-dV45-L0.dat 
CAEN.map                             DecrementedCAENvolts-dV45-L-1.dat 
CAENvolts.dat@                       DecrementedCAENvolts-dV500-L-1.dat 
CAENvolts.dat~                       DecrementedCAENvolts-dV50-L0.dat 
CAENvolts.errors                     DecrementedCAENvolts-dV50-L-1.dat 
caenvoltsfiles/                      DecrementedCAENvolts-dV55-L0.dat 
CAENvolts_IVscan.pl*                 DecrementedCAENvolts-dV55-L-1.dat 
CAENvolts_IVscan.pl~*                DecrementedCAENvolts-dV56-L0.dat 
CAENvolts.orig                       DecrementedCAENvolts-dV5-L-1.dat 
DecrementedCAENvolts-dV0-L0.dat      DecrementedCAENvolts-dV60-L0.dat 
DecrementedCAENvolts-dV0-L-1.dat     DecrementedCAENvolts-dV60-L-1.dat 
DecrementedCAENvolts-dV100-L0.dat    DecrementedCAENvolts-dV65-L0.dat 
DecrementedCAENvolts-dV100-L-1.dat   DecrementedCAENvolts-dV65-L-1.dat 
DecrementedCAENvolts-dV10-L0.dat     DecrementedCAENvolts-dV70-L0.dat 
DecrementedCAENvolts-dV10-L-1.dat    DecrementedCAENvolts-dV70-L-1.dat 
DecrementedCAENvolts-dV110-L0.dat    DecrementedCAENvolts-dV75-L-1.dat 
DecrementedCAENvolts-dV120-L0.dat    DecrementedCAENvolts-dV80-L0.dat 
DecrementedCAENvolts-dV120-L1.dat    DecrementedCAENvolts-dV80-L-1.dat 
DecrementedCAENvolts-dV130-L0.dat    DecrementedCAENvolts-dV85-L-1.dat 
DecrementedCAENvolts-dV-130-L-1.dat  DecrementedCAENvolts-dV90-L0.dat 
DecrementedCAENvolts-dV130-L-1.dat   DecrementedCAENvolts-dV90-L-1.dat 
DecrementedCAENvolts-dV15-L0.dat     DecrementedCAENvolts-dV95-L-1.dat 
DecrementedCAENvolts-dV15-L-1.dat    log/ 
DecrementedCAENvolts-dV20-L0.dat     powerset.dat@ 
DecrementedCAENvolts-dV20-L-1.dat    powersetfiles/ 
DecrementedCAENvolts-dV25-L0.dat     PS.conf* 
DecrementedCAENvolts-dV25-L-1.dat    sourceme* 
DecrementedCAENvolts-dV30-L0.dat 
<svxii@b0dasvx01>PS & 
[1] 19193 
<svxii@b0dasvx01>User: svxii 
Display: 
Attempting initialization of CAEN at svxinter.fnal.gov... 
Downloading /cdf/onln/data/svxii/caencontrol/PSfiles/PS.conf 
Loading startup files to the server.....Please be patient 
GUI Startup Done! 
 
<svxii@b0dasvx01> 
<svxii@b0dasvx01>If same do nothing 
Status change prohibited 
ch = 5 
pcStatus = 0 
status = 1 
channelStatus = 0 
Status change prohibited 
ch = 5 
pcStatus = 0 
status = 1 
channelStatus = 0 
If same do nothing 
java.lang.ExceptionInInitializerError 
        at gov.fnal.robin.rRequest.unmarshal_response(rRequest.java:568) 
        at gov.fnal.robin.rRequest.get_response(rRequest.java:387) 
        at gov.fnal.robin.rRequest.invoke(rRequest.java:177) 
        at CAEN.CaenVme.CaenClient.getSiliconInfo(CaenClient.java:582) 
        at CAEN.PWR.resetValuesFromVMEFunction(PWR.java:530) 
        at CAEN.PWR.access$000(PWR.java:43) 
        at CAEN.PWR$resetValuesFromVMEThread.run(PWR.java:429) 
        at CAEN.PWR.resetValuesFromVME(PWR.java:475) 
        at CAEN.PSControlForm.refresh(PSControlForm.java:1627) 
        at CAEN.PSControlForm.access$000(PSControlForm.java:41) 
        at CAEN.PSControlForm$1.actionPerformed(PSControlForm.java:752) 
        at javax.swing.Timer.fireActionPerformed(Timer.java:271) 
        at javax.swing.Timer$DoPostEvent.run(Timer.java:201) 
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:189) 
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:478) 
        at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:201) 
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:151) 
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:145) 
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:137) 
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:100) 
Caused by: java.lang.NullPointerException 
        at org.omg.CORBA.CompletionStatus.<init>(CompletionStatus.java:23) 
        at org.omg.CORBA.CompletionStatus.<clinit>(CompletionStatus.java:6) 
        ... 20 more 
 
[1]    Done                          java -Xbootclasspath/p:/cdf/onln/code/cdfprod/robin/v2_4_10/classes -ss100m  ... 
<svxii@b0dasvx01> 
<svxii@b0dasvx01> 
</pre>
<!-- Author: Ankush -->
<!-- Comment: Fri Jun 30 12;04;26 comment by...Ankush -->
<h3>PS-GUI Barfed!!</h3>

<p>We about to start taking taking and then mutliple problems.  One being the PS-GUI barfs.  This core dump of error messages is to help Dean debug it.</p>
<!-- Date: Fri Jun 30 11:47:03 2006 -->
<pre> 
svxinter-> 
svxinter-> FRI JUN 30 10:58:43 2006  setPSStatus, detector = 0, ps_n = 10, status = 2 
FRI JUN 30 10:58:44 2006 Sending Request for fa46 
FRI JUN 30 10:58:52 2006  setPSStatus, detector = 0, ps_n = 11, status = 2 
FRI JUN 30 10:58:52 2006 Sending Request for fa56 
FRI JUN 30 10:58:56 2006 PS is still not ON, after 10 checks 
FRI JUN 30 10:58:56 2006  setPSStatus, detector = 0, ps_n = 11, status = 1 
FRI JUN 30 10:59:05 2006  setPSStatus, detector = 0, ps_n = 6, status = 2 
FRI JUN 30 10:59:05 2006 Sending Request for fa06 
Module is busy, sending code = 16 
FRI JUN 30 10:59:09 2006 PS is still not ON, after 10 checks 
FRI JUN 30 10:59:09 2006  setPSStatus, detector = 0, ps_n = 6, status = 1 
FRI JUN 30 10:59:24 2006 Sending Request for fa50 
FRI JUN 30 10:59:29 2006 Sending Request for fa51 
FRI JUN 30 10:59:31 2006 Sending Request for fa52 
FRI JUN 30 10:59:33 2006 Sending Request for fa53 
FRI JUN 30 10:59:41 2006 Sending Request for fa00 
FRI JUN 30 10:59:43 2006 Sending Request for fa01 
FRI JUN 30 10:59:47 2006 Sending Request for fa02 
FRI JUN 30 10:59:49 2006 Sending Request for fa03 
Unable to allocate return data buffer 
0xa0ec28 (t7): memPartAlloc: block too big - 71160 in partition 0x2d8ccc. 
msg_recv: bad size 0 reading length from socket 19, data 710e70, type 0 
errno is 0 
Connection 127.0.0.1:1057 aborted, status is -1 
FRI JUN 30 11:03:40 2006 Conf file is /cdf/onln/data/svxii/caencontrol/PSfiles/PS.conf 
/cdf/onln/data/svxii/caencontrol/PSfiles 
FRI JUN 30 11:03:40 2006 Mapping file is /cdf/onln/data/svxii/caencontrol/PSfiles/CAEN.map 
FRI JUN 30 11:03:40 2006 Voltage file is /cdf/onln/data/svxii/caencontrol/PSfiles/CAENvolts.dat 
FRI JUN 30 11:04:14 2006 PowerSet file is /cdf/onln/data/svxii/caencontrol/PSfiles/powerset.dat 
FRI JUN 30 11:04:14 2006 LV Currents file is /cdf/onln/data/svxii/caencontrol/PSfiles/CAENLVcurrent.dat 
Using local rtlm 
Zap TipcSrvLmCheck 
dummy add 10e58c; tipcadd 1ed2f0, newins 7ff2129c 
Connecting to <tcp:localhost> RTlm. 
Using tcp protocol. 
received connection from 127.0.0.1:1083, socket is 20 
Message from RTlm: Connection established. 
WARNING: Internal validation failed, file: ipcmsg.c, line 1314WARNING: Internal validation failed, file: ipcmsg.c, line 1314Message from RTlm: License for <RTclient> granted (132 licenses left). 
Connecting to project <cdf_daq> on <tcp:b0server05> RTserver. 
Using tcp protocol. 
Message from RTserver: Connection established.Start subscribing to subject </_svxinter_8955832>. 
Crate #: 1 
Group Name: "GROUP00" 
Crate #: 2 
Group Name: "GROUP00" 
Crate #: 3 
Group Name: "GROUP00" 
Crate #: 4 
Group Name: "GROUP00" 
Crate #: 5 
Group Name: "GROUP00" 
Crate #: 6 
Group Name: "GROUP00" 
Crate #: 7 
Group Name: "GROUP00" 
Crate #: 8 
Group Name: "GROUP00" 
Crate #: 9 
Group Name: "GROUP00" 
Crate #: 10 
Group Name: "GROUP00" 
Crate #: 11 
Group Name: "GROUP00" 
Crate #: 12 
Group Name: "GROUP00" 
Crate #: 13 
Group Name: "GROUP00" 
Crate #: 14 
Group Name: "GROUP00" 
Crate #: 15 
Group Name: "GROUP00" 
Crate #: 16 
Group Name: "GROUP00" 
FRI JUN 30 11:05:34 2006 Sending Request for e404 
FRI JUN 30 11:06:21 2006  setPSStatus, detector = 1, ps_n = 48, status = 1 
FRI JUN 30 11:06:28 2006  setPSStatus, detector = 1, ps_n = 48, status = 0 
FRI JUN 30 11:06:28 2006 Sending Request for e406 
FRI JUN 30 11:06:32 2006 PS is still not ON, after 10 checks 
FRI JUN 30 11:06:32 2006  setPSStatus, detector = 1, ps_n = 48, status = 1 
FRI JUN 30 11:06:50 2006 Sending Request for e405 
FRI JUN 30 11:06:56 2006 Sending Request for e400 
FRI JUN 30 11:06:58 2006 Sending Request for e401 
FRI JUN 30 11:07:03 2006 Sending Request for e402 
FRI JUN 30 11:07:06 2006 Sending Request for e403 
FRI JUN 30 11:07:08 2006 Sending Request for e404 
Module is busy, sending code = 18 
Module is busy, sending code = 18 
Module is busy, sending code = 18 
Module is busy, sending code = 18 
Module is busy, sending code = 18 
FRI JUN 30 11:09:20 2006 Conf file is /cdf/onln/data/svxii/caencontrol/PSfiles/PS.conf 
/cdf/onln/data/svxii/caencontrol/PSfiles 
FRI JUN 30 11:09:20 2006 Mapping file is /cdf/onln/data/svxii/caencontrol/PSfiles/CAEN.map 
FRI JUN 30 11:09:20 2006 Voltage file is /cdf/onln/data/svxii/caencontrol/PSfiles/CAENvolts.dat 
FRI JUN 30 11:09:53 2006 PowerSet file is /cdf/onln/data/svxii/caencontrol/PSfiles/powerset.dat 
msg_recv: bad size 0 reading length from socket 20, data 586550, type 0 
errno is 0 
Connection 127.0.0.1:1083 aborted, status is -1 
FRI JUN 30 11:09:53 2006 LV Currents file is /cdf/onln/data/svxii/caencontrol/PSfiles/CAENLVcurrent.dat 
Using local rtlm 
Zap TipcSrvLmCheck 
dummy add 10e58c; tipcadd 1ed2f0, newins 7ff2129c 
Connecting to <tcp:localhost> RTlm. 
Using tcp protocol. 
received connection from 127.0.0.1:1085, socket is 19 
Message from RTlm: Connection established. 
WARNING: Internal validation failed, file: ipcmsg.c, line 1314WARNING: Internal validation failed, file: ipcmsg.c, line 1314Message from RTlm: License for <RTclient> granted (132 licenses left). 
Connecting to project <cdf_daq> on <tcp:b0server05> RTserver. 
Using tcp protocol. 
Message from RTserver: Connection established.Start subscribing to subject </_svxinter_7104680>. 
FRI JUN 30 11:09:53 2006 Conf file is /cdf/onln/data/svxii/caencontrol/PSfiles/PS.conf 
/cdf/onln/data/svxii/caencontrol/PSfiles 
FRI JUN 30 11:09:53 2006 Mapping file is /cdf/onln/data/svxii/caencontrol/PSfiles/CAEN.map 
FRI JUN 30 11:09:53 2006 Voltage file is /cdf/onln/data/svxii/caencontrol/PSfiles/CAENvolts.dat 
FRI JUN 30 11:10:27 2006 PowerSet file is /cdf/onln/data/svxii/caencontrol/PSfiles/powerset.dat 
FRI JUN 30 11:10:27 2006 LV Currents file is /cdf/onln/data/svxii/caencontrol/PSfiles/CAENLVcurrent.dat 
Using local rtlm 
Zap TipcSrvLmCheck 
dummy add 10e58c; tipcadd 1ed2f0, newins 7ff2129c 
Connecting to <tcp:localhost> RTlm. 
Using tcp protocol. 
received connection from 127.0.0.1:1087, socket is 28 
Message from RTlm: Connection established. 
WARNING: Internal validation failed, file: ipcmsg.c, line 1314WARNING: Internal validation failed, file: ipcmsg.c, line 1314Message from RTlm: License for <RTclient> granted (132 licenses left). 
Connecting to project <cdf_daq> on <tcp:b0server05> RTserver. 
Using tcp protocol. 
Message from RTserver: Connection established.Start subscribing to subject </_svxinter_10448736>. 
Crate #: 1 
Group Name: "GROUP00" 
Crate #: 2 
Group Name: "GROUP00" 
Crate #: 3 
Group Name: "GROUP00" 
Crate #: 4 
Group Name: "GROUP00" 
Crate #: 5 
Group Name: "GROUP00" 
Crate #: 6 
Group Name: "GROUP00" 
Crate #: 7 
Group Name: "GROUP00" 
Crate #: 8 
Group Name: "GROUP00" 
Crate #: 9 
Group Name: "GROUP00" 
Crate #: 10 
Group Name: "GROUP00" 
Crate #: 11 
Group Name: "GROUP00" 
Crate #: 12 
Group Name: "GROUP00" 
Crate #: 13 
Group Name: "GROUP00" 
Crate #: 14 
Group Name: "GROUP00" 
Crate #: 15 
Group Name: "GROUP00" 
Crate #: 16 
Group Name: "GROUP00" 
msg_recv: bad size 0 reading length from socket 28, data b51c38, type 0 
errno is 0 
Connection 127.0.0.1:1087 aborted, status is -1 
FRI JUN 30 11:10:57 2006 Sending Request for e404 
FRI JUN 30 11:11:32 2006 Sending Request for e404 
FRI JUN 30 11:12:12 2006 Sending Request for e404 
FRI JUN 30 11:12:44 2006  setPSStatus, detector = 1, ps_n = 48, status = 1 
FRI JUN 30 11:12:51 2006  setPSStatus, detector = 1, ps_n = 48, status = 0 
FRI JUN 30 11:12:51 2006 Sending Request for e406 
FRI JUN 30 11:12:55 2006 PS is still not ON, after 10 checks 
FRI JUN 30 11:12:55 2006  setPSStatus, detector = 1, ps_n = 48, status = 1 
FRI JUN 30 11:13:03 2006  setPSStatus, detector = 1, ps_n = 48, status = 0 
FRI JUN 30 11:13:04 2006 Sending Request for e406 
Timeout 
FRI JUN 30 11:13:08 2006 ERROR: not all Fibs are initialized 
FRI JUN 30 11:13:08 2006 ERROR: FIB does not respond to Request for handshake, ps_n = 48FRI JUN 30 11:13:08 2006 setPSStatus,err = -1, detector = 1, ps_n = 48 
Module is busy, sending code = 18 
Module is busy, sending code = 18 
 
 
svxinter-> 
svxinter-> 
</pre>
<!-- Author: Ankush -->
<!-- Comment: Fri Jun 30 12;05;23 comment by...Ankush -->
<h3>....more PS-GUI barf</h3>

<p>These are the errors from SVXINTER during the PS-GUI barf.</p>
<!-- Date: Fri Jun 30 12:02:30 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W0L4 
<li>problem n: 37 
<li>Status: needs_work
<li>Type: BIAS
<li>Action: Removed_hdwdb
<li>entered by: amitra 
<li>description is: <p>The biggest post-shutdown store came in and of course this ladder starts tripping on Bias.  After lots of swearing I took this ladder out.</p> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W0&layer=L4">report</a></ul>
<!-- Author: Problem DB -->





<!-- Date: Fri Jun 30 12:24:56 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8904&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8905  width='300' align='top'></a></td><td> 
 
</td></tr></table>
<!-- Author: Ankush -->
<!-- Comment: Fri Jun 30 12;49;02 comment by...Ankush -->
<h3>Store 4805</h3>

<p>The recycler stash was really large and so promisied a really high inital lumi.  The store came in fine and the losses after scraping were fine; all < 3kHz.  So Silicon came in straight away with an initial lumi of 146.54e30.</p>

<p>...then we had problems......</p>

<p>In going from STANDBY to ON, 3 L00 wegdes (LB1W0, LB1W4, LB1W5) were stuck at STANDBY.  In addition SB4W0L4 was tripping on BIAS.  So I thought just powering them up manually would be sufficient.</p>

<p>......then....</p>

<p>the PS-GUI then decided to stop updating and then crash.  All this in the middle of trying to power up the 3 L00 wedges and recover the trip on e404.  The PS-GUI then became more and more unresponsive.  I got very angry and started swearing a lot (so much so, I passed Mary Convery's Silicon SPL swear test). </p>

<p>Eventually the run started without silicon.  I rebooted SVXINTER, IMON & PS-GUI.  The L00 wedges came back.  However e404 continued tripping on bias.  Looking at the PDB, this ladder tripped on bias back in 2002.  We put it back in during the PDB review.  So after a month of data taking it has gone again.</p>

<p>After taking out e404, Silicon was put back in and silicon is OK again.  SVTSPYMon looks fine and so does SVXMon.</p>


<!-- Date: Fri Jun 30 13:14:55 2006 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8906&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8907' ></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8908&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8908' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8909&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8909' width='300'></a></center>
<!-- Author: Ankush -->
<!-- Comment: Fri Jun 30 13;20;53 comment by...Ankush -->
<h3>Downstream Z-side chips on e012 have gone again</h3>

<p>I spotted this PINKY on AVDD for e012.  I asked the DAQ ACE for an HRR but that didn't help.  Looking at the SVXMon plots, chips  a3, a2, a1, a0 have gone but a4 is still up.</p>

<p><em>pooey</em></p>


<!-- Date: Fri Jun 30 13:40:29 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8910&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8911  width='300' align='top'></a></td><td><p>AVDD on e012 came back</p></td></tr></table>
<!-- Author: Ankush -->


<!-- Date: Fri Jun 30 13:56:00 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8915&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8915  width='300' align='top'></a></td><td><h3>Wisdom from Ron Moore about AGIGI2</h3> 
 
<p>from the Tev e-log</p> 
 
<p> 
-- Fri Jun 30 13:09:35 comment by...rm --  Tune changes over the store and the effect on abort gap beam intensity. The proton tunes have been a little high at collisions lately, so that would decerase the TEL's abort gap cleaning efficiency. The steps in T:AGIGI2 generally followed the routine pbar tune changes...the tune mults haven't been calibrated in a while, so I'm not surprised that pbar tune changes also affect the protons. Reducing the proton horizontal tune helped reduce T:AGIGI2. We can probably push the proton tunes down a little bit more at collisions. 
<br> 
-- Fri Jun 30 13:10:35 comment by...rm --  ...also calibrate the tune mults, implement a longer TEL e-pulse (waiting for some controls work), verify TEL electron beam close enough to proton orbit... 
<br> 
-- Fri Jun 30 13:43:19 comment by...rm --  This plot is for store 4804.   
</p></td></tr></table>
<!-- Author: Ankush -->
<!-- Comment: Fri Jun 30 17;03;41 comment by...Ulrich -->
Just talked to rm and invited him over for coffee...
<!-- Comment: Fri Jun 30 17;13;59 comment by...rm -->
<pre>Sometime Monday works for me...page me if you want to chat sooner.</pre>
<!-- Date: Fri Jun 30 14:00:18 2006 -->
<b>Germany - Argentina: 1-1 (5-3 after penalties)</b>
<!-- Author: Berlin, Berlin, wir fahren nach Berlin! -->


<!-- Date: Fri Jun 30 16:22:12 2006 -->
<h3>Fun at FCC Teststand</h3> 
Ignacio and I headed over to the FCC testand this afternoon for more studies of FIBs 119 and 162.  Both of these FIBs exhibit a rare error were they seem to not recogonize the last chip/last channel and sit waiting for it (and sending d0d0 to the VRB) before timing out and sending c2c2.  We found that this error only shows up for specific combinations of last chip, chips read and channels sent by the DEM.  We wanted to test the FIBs in FCC to see if there are any problems with real ladders before handing the them off to prep.  Before we could get the testand working we had to reconnect the DOIM cables to the SVX setup and reconnect the half of the junction card that had been borrowed for the ISL setup.  After doing this and adjusting the svxdaq settings for reading out with 2 chips, we were able to run without errors with HDIs 0, 3 and 4.  Unfortunately we also ran error free with FIBs 119 and 162, so we did not learn anything today.
<!-- Author: Mark&Ignacio -->
<!-- Comment: Sat Jul  1 11;54;26 comment by...sasha -->
Weird...you were not suppose to see anything at all from HDI3. Only HDIs 0 and 4 are working. But it's only HDI-0 which is really OK, because HDI-4 starts giving errors after a few runs.
<!-- Date: Fri Jun 30 20:03:08 2006 -->
<h3>Another clock glitch</h3> 
Just came home and saw in the e-log that the shift crew just reported a clock error. I have reset the clock and asked the ace to go to RUN again. That caused an SRC error (SRC Fatal Error Sl 6 Too Many L1A), which cleared after another HRR. Why would the SRC pull another error? Anyway, they are up and running again. Here is the error message: 
<pre> 
45'18" b0svx02:Messenger:Jun 30, 2006 07:45:18 PM->Running on interal CLK (0x007E,736) -->   
 Additional Information:  
 
 Attention!!!. FERML_SILICON_CLOCK_ERROR !!! 
   Silicon DAQ has detected a clock error.   
   Run is in RECOVERED state.                             
   Please page Silicon right away for further instructions.  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx06: Sl 6 Too Many L1A 2 L1A to Buff  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
</pre> 
Note also that they had paged me, but I never received the page.
<!-- Author: Ulrich -->
<!-- Comment: Fri Jun 30 21;57;02 comment by...Ankush -->
<p>After the LA caught the clock glitch, I reset the LA to now trigger on the GhostBuster output rather than a general SRC error....we wanna catch those bursty L1A.</p>
<!-- Comment: Sat Jul  1 15;09;23 comment by...Sebastian -->
after relocking RF, you have to clear the error register, so have to HRR again.
<!-- Date: Fri Jun 30 21:12:01 2006 -->
<b>PS pager temporarily unmanned</b>: for any PS problems, call my cell 291-3454.  I'm in Chicago so give me at least 2 hrs notice if I need to come in.  Thanks!<br>
<!-- Author: Jen -->



<!-- Date: Sat Jul 1 07:57:51 2006 -->
<h3>B2 W4 L2 not returning data</h3>
<p> 
After a quiet night, I got paged around 7:20. A DQMon error pointed the shift crew to a ladder which did not return data, e242 (phi side, the z-side is chopped already) This is bad since e244 is already missing and it would create a new hole in SVT. The shift crew has tried HRR, without success. I had a look at SVXMon and ADCMon. The ADCMon plot was indeed empty. So just as I was calling the shift crew to try and start a new run, they found that the problem went away. I thought it would be prudent to start a new run anyway and mark the other one (just one hour worth of data) bad for Silicon. What is our policy on this? 
</p> 
While I was on the phone, the solenoid tripped on low He, the run was ended, and all HV where brought to STANDBY.
<!-- Author: Ulrich -->
<!-- Comment: Sat Jul  1 07;58;34 comment by...Ulrich -->
<p>Endtag</p>


<!-- Comment: Sat Jul  1 18;55;41 comment by...Ankush -->
<p>Fixed :-)</p>
<!-- Date: Sat Jul 1 08:00:37 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8916&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8917  width='300' align='top'></a></td><td>B2W4 really shows a drop in efficiency for the current run.</td></tr></table>
<!-- Author: Ulrich -->


<!-- Date: Sat Jul 1 08:10:04 2006 -->
The solenoid is back up (see entry in <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=cryo-gas&action=view&page=1972&frame=2&anchor=07:46:16'> 
cryo e-log</a>). The shift crew ask for permission to reinclude the silicon. Since the beam is currently very stable, I gave the permission. I also told them to keep an eye on SVXMon in case the problem with e242 returns. Forgot to mention before that e242 has gone temporarily in the past, see the 
<a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B2W4&layer=L2">Problem DB</a>.
<!-- Author: Ulrich -->


<!-- Date: Sat Jul 1 08:18:02 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8918&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8919  width='300' align='top'></a></td><td>The SVT inefficiency seems to be gone in the new run. So it seems e242 is back.</td></tr></table>
<!-- Author: Ulrich -->


<!-- Date: Sat Jul 1 08:49:18 2006 -->
<h3>Tevatron abort</h3> 
The current store has just been aborted. According to MCR: 
<pre> 
Store lost. Investigating. - Mike V. 
-- Sat Jul 1 08:41:47 comment by...Mike V. --  Clean abort, no quench. 
BQ9 pulled the abort. - Mike V. 
</pre> 
So they may have lost a power supply for the B0 low beta quads. The post-quench checkout went  
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=451&frame=2&anchor=08:42:56.449'>fine</a>.
<!-- Author: Ulrich -->
<!-- Comment: Sat Jul  1 10;57;01 comment by...Ulrich -->
After some difficulties with the new Java GUI, the aces pasted the
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=451&frame=2&anchor=10:44:13.696'>
BLM FIFO plots</a>.




<!-- Date: Sat Jul 1 16:01:46 2006 -->
<h3>It's hot outside, baby</h3> 
Just saw in the main e-log that there are chilled water problems  
(see <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=cryo-gas&action=view&page=1972&frame=2&anchor=13:47:17'> 
cryo e-log</a>). RAM has been called in and are about to fix things. Let's hope we still get enough flow to our chillers.   
<!-- Author: Ulrich -->


<!-- Date: Sat Jul 1 18:43:56 2006 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8920&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8920' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8921&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8921' width='300'></a></center>
<!-- Author: Ankush&Ignacio -->
<!-- Comment: Sat Jul  1 18;50;56 comment by...Ankush & Ignacio -->
<pre>Those are the diamonds' plots for  last week. They seem to have suffered a pedestal shift after the magnet quench.</pre>
<!-- Date: Sat Jul 1 18:49:19 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8922&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8923  width='300' align='top'></a></td><td> </td></tr></table>
<!-- Author:   -->
<!-- Comment: Sat Jul  1 18;54;03 comment by...Ignacio & Ankush -->
<h3>Store 4808</h3>

<p>The store came in after 6pm and the losses after scraping were good; all less than 3kHz.  The initial SVTSPYMon plot shown above look OK; it even looks like we are getting tracks in SB4W8 which supposed to be an SVT hole.  SVXMon looks fine.  Unfortunatly CDF didn't start data taking straight away due to EVB/L3 problems.  So we were not to blame this time; otherwise a smooth start to this store.</p>

<p>Initial Lumi: 123.11E30</p>



<!-- Date: Sun Jul 2 04:17:47 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8924&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8924  width='300' align='top'></a></td><td>pink SVX channel , HRR didn't work.</td></tr></table>
<!-- Author: Chunlei -->
<!-- Comment: Sun Jul  2 04;43;20 comment by...Ulrich -->
What a service, the plot is already in our e-log! I got paged for this pinkie in AVDD for e554 around 4:20. As it turns out, the shift crew overlooked it from around 3:00 to 4:00 and tried and HRR around then. The cell is now pink again. From the plot it looks like the current is just below the limit, so I told the shift crew to leave it as it is for now and document it again in a couple of hours.


<!-- Date: Sun Jul 2 04:46:55 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8925&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8925  width='300' align='top'></a></td><td>SVX PINK channel</td></tr></table>
<!-- Author: CHunlei -->
<!-- Comment: Sun Jul  2 04;51;07 comment by...Ulrich -->
And an AVDD pinkie in e012. This ladder has chips coming and going for a while now. We should find a way to stabilize it.


<!-- Date: Sun Jul 2 04:52:19 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8926&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8926  width='300' align='top'></a></td><td>History of AVDD current for e554. The current (as well as DVDD have become a little flaky lately.</td></tr></table>
<!-- Author: Ulrich -->


<!-- Date: Mon Jul 3 00:34:23 2006 -->
<h3>Tevatron quench during shot setup</h3> 
The Tevatron quenched at E48U after final protons have been loaded. MCR diagnoses a slow abort and finds problems with the RF signal: 
<pre> 
00:07:43-  
	Whatever happened, it was quite slow, starting about .9 sec before the quench, as shown by this snapshot logger picture of T:IBEAM. 
 - m.m 
The datalogger shows the RF drooping before beam fallout. The snapshot buffer shows a sine wave pattern on the RF signal. - DJC 
</pre> 
I have asked the shift crew for the post-quench checkout. The BLM FIFOs recorded a  
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=456&frame=2&anchor=00:31:07.073'> 
slow increase of the radiation level</a>, although on a very low level. Also according to T40, the abort was clean for CDF. Now shot-setup is delayed.
<!-- Author: Ulrich -->


<!-- Date: Mon Jul 3 04:44:44 2006 -->
<h3>Master Clock Error: Major Trip</h3> 
Got paged around 3:10 because of an error with the master clock which tripped half of SVX and ISL. Here is the current state of the CCC: 
<pre> 
Your pick? 0x2 
Status: 0, Read: 3e 
BZ error 
BX error 
SY error 
GP error 
Master clk error 
CCC is locked to RF 
</pre> 
It is locked to RF, but the master clock has a problem.
<!-- Author: Ulrich -->
<!-- Comment: Mon Jul  3 05;06;42 comment by...Ulrich & Ignacio -->
<pre>The error may be connected to some of the action at MCR:
--------------------------------------------------------------------------------
 03:58:40-  We rebooted both the Main Injector LLRF and MRF which fixed the communication problems and the frequency error at the start of ramp. I also rebooted the TeV LLRF which had no effect on the noise plots during cogging. Tan and I looked at the proton RF phase detector during cogging and the noise level is very low. It is possible that we had a power glitch at MI-60 as there were many network errors logged as well as the other problems. - Brian Chase 
-------

So, most of SVX and L00 tripped as a result of the clock glitch.
We started to recover a couple of ladders but they trip and the whole ISL went into pink (low current state) in IMON.
Tried a  cold start but got EEPROM errors from several fibs. 
This happened as S. Hahn was resetting the clock.  After 4:25
all errors were clear and  the clock status was green on: 
http://www-cdfonline.fnal.gov/java/cdfdb/servlet/ClockErrors.

The EEPROM errors  and the low current state in ISL disappeared by  repartition and config in RC, once the clock was in OK state

As ISL L00 and B0  from SVX were on standby a new clock error
at 4:55 sent L00 and ISL to low current state. Wait.
</pre>


<!-- Comment: Mon Jul  3 05;19;23 comment by...Ulrich -->
The CCC detected another clock error which Steve Chappa and Steve Hahn are trying to reset the clock:
<pre>
Status: 0, Read: 2c
BX error
SY error
Master clk error
CCC is locked to RF
</pre>
<!-- Date: Mon Jul 3 05:07:28 2006 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8927&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8928' ></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8929&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8930' ></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8931&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8932' ></a></center>
<!-- Author: Ignacio & Ulrich -->
<!-- Comment: Mon Jul  3 05;10;48 comment by...Ignacio & Ulrich -->
Three stages of the clock error: 1) trip after the master clock error. 2) pinkies during attempts to bring the clock back.. 3) pinkies after the second attempt to bring the clock back.




<!-- Date: Mon Jul 3 06:50:01 2006 -->
<h3>CCC Status during Clock Problems</h3> 
We are confused about the way the CCC acts during the clock problems.  It detected several problems with the master clock, but it never lost lock to the RF. Is this a bug or a feature?
<!-- Author: Ignatio & Ulrich -->
<!-- Comment: Mon Jul  3 07;06;23 comment by...Ignacio & Ulrich -->
Now Steve Hahn and Steva Chappa power-cycled the clock crate. The CCC switched to internal clock.
<!-- Date: Mon Jul 3 06:57:26 2006 -->
<h3>Warmed up SVX</h3> 
Since we are sitting now for more than 3 hours with 5 out of 6 SVX bulkheads off, we change the temperature setpoint from -10&deg;C to -8&deg;C, just to keep COT warm and fuzzy. ISL is still on, so it should not be too bad for them.
<!-- Author: Ignacio & Ulrich -->
<!-- Comment: Mon Jul  3 07;06;16 comment by...Ankush -->
<p>If the CDF clock is having problems...can&#39t we run just from the CCC internal clock ?</p>

<!-- Date: Mon Jul 3 07:05:23 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8933&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-8934  width='300' align='top'></a></td><td><h3>Pinkies during clock problems</h3> 
During the clock problems, some SVX/L00 AVDD lines and almost all ISL AVDD and DVDD lines get pink. Why did they go into a state with a lower current?</td></tr></table>
<!-- Author: Ignacio & Ulrich -->




<!-- Date: Mon Jul 3 07:40:31 2006 -->
<h3>Problems rebooting clock crate, Silicon completely off</h3> 
The Steves have rebooted the main clock crate. At this point, the CCC switched to its internal mode (finally!). 
We tripped all of ISL and most of L00 during the clock crate reboot. After that, we switch off the entire Silicon detector. 
Currently, they still have problems with the clock crate. It got stuck during the reboot, due to problems with the boot server(?). 
<!-- Author: Ignacio & Ulrich -->
<!-- Comment: Mon Jul  3 07;41;33 comment by...Ulrich -->
We also encountered RTserver problems with IMON. We restarted the program after the RTserver was responding again.

<!-- Comment: Mon Jul  3 08;00;00 comment by...Ingnacio & Ulrich -->
<pre>We also rebooted b0svx07, which got stuck.</pre>

<!-- Comment: Mon Jul  3 08;59;48 comment by...Ulrich -->
Probably, the reboot of b0svx07 screwed up the ISL VRB chain. So we reset all VRBs, starting from left to right on b0svx06, b0svx07, b0svx08.


<!-- Date: Mon Jul 3 09:09:51 2006 -->
    <h3>Summary of the Week</h3> 
 
    <h4>Monday, June 26</h4> 
    <ul> 
      <li>Previous store aborted without warning CDF: Silicon was on!</li> 
      <li>Store 4799: losses fine</li> 
      <li>CAEN crate 15: 2 spontaneous switch-offs, discussion on trigger inhibit after switch-off (trigger inhibit line is not held continuously)</li> 
      <li>A real resonance! Ankush gets excited.</li> 
    </ul> 
 
    <h4>Tuesday, June 27</h4> 
    <ul> 
      <li>Clock problems while moving a NIM crate, we are not affected.</li> 
      <li>SVX chiller: conductivity back to normal</li> 
      <li>Set up 3 logic analyzers to understand L1 DONE TO problems</li> 
      <li>Quiet time studies: e124 still bad, e522 still bad, e052 z-side chopped, confirmed bit errors in f024, screwed up permanent LL measurement</li> 
    </ul> 
 
    <h4>Wednesday, June 28</h4> 
    <ul> 
      <li>Store 4802: all losses very low</li> 
      <li>Single event upset in FIB board</li> 
      <li>b0fib00 went bad, causing reformatter errors</li> 
      <li>Real resonance</li> 
      <li>3 clock glitches: had to reset CCC</li> 
      <li>Burst of L1 DONE TO</li> 
      <li>Silicon in STANDBY during beam problems (half of luminosity lost, beam blew up transversely) and during rescraping later on.  
      <li>Successful permanent LL measurement 
      <li>Played with BUSY TIMEOUT for COLDSTART</li> 
    </ul> 
 
    <h4>Thursday, June 29</h4> 
    <ul> 
      <li>New Java application replaces E2 for BLM monitoring</li> 
      <li>Store 4804: all fine</li> 
      <li>CDF solenoid tripped at 6:00: no data taking until 20:00</li> 
      <li>SVXMon can be monitored from Consumer Root File Browser again</li> 
      <li>RC goes to RECOVER automatically after CCC detects clock error</li> 
      <li>AGIGI2 too high, peaked during Tevatron tune changes</li> 
    </ul> 
 
    <h4>Friday, June 30</h4> 
    <ul> 
      <li>PAK5 (proton abort kicker) problems: Silicon in STANDBY</li> 
      <li>Store 4805: highest luminosity so far, losses fine 
      <li>Startup problems:  
	<ul> 
	  <li>PS GUI: ladders stuck in STANDBY, crashed later</li> 
	  <li>e404 trips on BIAS: taken out</li> 
	</ul> 
      <li>e012: pinkie coming and going (for a while now)</li> 
      <li>Investigating strange FIB errors at FCC teststand</li> 
      <li>Clock glitch: CCC reset</li> 
    </ul> 
 
    <h4>Saturday, July 1</h4> 
    <ul> 
      <li>e242 does not return data for 1 hour, comes back automagically</li> 
      <li>Tevatron abort</li> 
      <li>Store 4808: everything fine</li> 
    </ul> 
 
    <h4>Sunday, July 2</h4> 
    <ul> 
      <li>New pinkie: e554 AVDD (just below limit) 
      <li>Tevatron abort after proton load: RF problems</li> 
    </ul> 
 
    <h4>Monday, July 3</h4> 
    <ul> 
      <li>Major clock glitch trips SVX and L00</li> 
      <li>Almost all of SVX is off: bring SVX temperature setpoint to -8&deg;C</li> 
      <li>After the clock came back: SVX back to -10&deg;C, all powered up again</li> 
    </ul>  
<!-- Author: Ignacio & Ulrich -->


<!-- Date: Mon Jul 3 09:18:11 2006 -->
<h3>Silicon back up</h3> 
 
<p>Silicon is now up again and taken through COLDSTART. However this time I had to increase the BUSY TIMEOUT by x100 to prevent BUSY TIMEOUT erorrs.  Also the replies from the crates in run-control was <em>really</em> slow.  I suspect something else is interfering with our COLDSTART.</p> 
 
<p>In the end, Silicon went through COLDSTART with no problems.</p>
<!-- Author: Ankush, Ulrich and Ignacio -->