<!-- Title: 22:22:47  Sun Jan  1 2006 -->
<!-- Crewchief: Dr. TOF -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>New File for a New Year</pre>


<!-- Date: Fri Jan 20 10:40:25 2006 -->
<pre>The west TOF LV power supply has had the current of the iFix 
monitored channel 7 drifting higher than the alarm limit.  We 
will try to figure out if this is associated with the power 
supply or the detector by unplugging individual cables and 
measuring the currents that result.</pre>
<!-- Author: M Jones/N Ranjan -->


<!-- Date: Fri Jan 20 11:48:31 2006 -->
<pre>Here's what we learned: 
 
If the cables on the front of the LV power supply are 
numbered from 0..5 going from left to right, then the 
iFix channels 4,5,6,7 correspond to the left-hand Vicor 
modules and iFix channels 0,1,2,3 correspond to the right- 
hand Vicor modules. 
 
With only one yellow LV cable plugged in at a time, the 
currents were as follows: 
 
   Nothing unplugged:  I1=7.46 A, I3=7.70 A, I5=7.18 A, I7=8.44 A 
Cables 3,4 unplugged:  I1=2.52 A, I3=2.63 A, I5=7.21 A, I7=8.44 A 
Cables 1,2 unplugged:  I1=7.42 A, I3=7.67 A, I5=2.45 A, I7=2.87 A 
Cables 0,2 unplugged:  I1=7.46 A, I3=7.74 A, I5=2.42 A, I7=2.87 A 
Cables 0,1 unplugged:  I1=7.46 A, I3=7.74 A, I5=2.42 A, I7=2.84 A 
 
One thing I noticed was that when all the cables were unplugged, 
the right-most yellow LED came on, but when at least one of 
cables 0,1,2 was plugged in, the LED went off again. 
 
Given that there is no difference in the current drawn by any 
of the sets of channels on cables 0,1,2 we can conclude that the 
high current reading is associated with the power supply and 
not with a problem inside the detector, unless some problem 
affected all channels on the west side in a similar way, which 
is rather unlikely. 
 
I think it is safe to leave the power supply as it is until we 
can replace it or fix it during the "March" shutdown.</pre>
<!-- Author: M Jones/N Ranjan -->


<!-- Date: Wed Feb 1 15:21:25 2006 -->
<pre>Got paged this morning because of 'Done timeouts' in the TAC calibration. Various tries just before and just after the 8 am shift crew change failed; errors reported for b0wcal00, b0wcal05 and (b0wcal05+06). Matthew suggested (see email below) to try to run the calibration in a different partition.  
 
It seems to have worked: The e-log shows a succesful calibration run taken at 14.08. (run 211096). 
 
 
------ 
Matthew Jones wrote: 
 
A problem in more than one crate is almost always due to a problem with 
the DAQ system or run control.  Done timeouts could be caused by clock or 
other types of glitches. 
 
On the other hand, there have also been cases where the flash RAM in 
the ADMEM's gets corrupted and needs to be downloaded again, but from 
what I can see, they are running fine with all the crates in a cosmic 
run, so I don't think that is the case. 
 
I suspect that this must have been a momentary glitch, so I would 
encourage them to try to take the TAC calibration again.  Another thing to 
be aware of is that if the TAC calibration doesn't complete, the consumer 
is left waiting for events.  This can happen, for example, when there is 
even one done timeout and up to 4 events get flushed before making it to 
the consumer.  When a new one is started the old consumer sometimes gets 
the new events and proceeds to do the fits when it thinks it has got 
enough events...  The way to avoid this is to take the new TAC calibration 
in a different partition. 
 
----- 
Gerry Bauer: 
  um.....i chased around with such mysterious problems 
for awhile once or twice. i dunno if there is a real problem 
causing this to happen (didn't used to in old times), or 
it is just unlucky glitches. however, one one occasion 
it seemed to be the consumer server/loger was goofing 
us up. so another desperate measure is to restart it? 
(changing partitions may be a milder way of trying 
to achieve the same thing, but it may have a milder 
effect too....a more definitive action is to restart or 
reset the consumer server/loger.....i'm not sure what 
the exact terminology is for ace instsructions though....) 
 
--- 
 
 
 
</pre>
<!-- Author: aart -->


<!-- Date: Tue Feb 28 19:33:52 2006 -->
<pre>test</pre>
<!-- Author: Stefano -->


<!-- Date: Wed Mar 15 09:53:15 2006 -->
Trying to fix wcal00 slot 5 and wcal04 slot 9.<BR>  The wcal04 
problem first showed up on <A HREF= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=184&frame=2&anchor=18:29:43.581'> 
March 3</A>...
<!-- Author: M Jones -->


<!-- Date: Tue Apr 4 09:10:58 2006 -->
<pre>Running a TAC calibration - right now the following ADMEM's are 
excluded:  wcal04 slot 9.  Actually that's the only one.</pre>
<!-- Author: M Jones -->


<!-- Date: Wed Apr 19 23:01:38 2006 -->
<pre> ****** TOF TUBE REPLACEMENT ******* 
 
Due to aging issues we started some experimental tube replacement: 
 
Removed PMT ID 229 --- Assembly 419 with tube DA885 
(West side tube, the center tube of three-pack starting  
at 20 degrees [usual coordinates: 0 degrees is horizontal 
pointing North) 
 
Replaced it with tube DA0050, with NEW preamp P0194 & NEW cookie 
 
replacement went pretty smoothly, except we (i.e. I) forgot  
a "center" tube has a different endcap holder. So we removed 
the one from Assembly 419 and put it on. 
 
After tube removal, we restored LV, and put 1kV on this  
tube an looked at signals at the 1st floor patch panel using  
the TOAD monitor feature. Looked fine. Also current draw looked 
fine too. 
 
 
Dervin, Gerry, & Tom Phillips</pre>
<!-- Author: bauerg -->


<!-- Date: Thu Apr 20 17:20:56 2006 -->
 <br>    
<br> 
   ********** MORE TOF TUBE REPLACEMENT ***************<br> 
<br> 
<br> 
...a continuation of our tube replacement adventures....<br> 
<br> 
1)<br> 
moved to the east side, looking for another "bad" tube 
to replace per Matthew's list. After some discouragement, 
we decided to try to replace PMT 176<br> 
<br> 
 
PMT 176 is in 290 degree three pack (#58) on the East 
side  containing 
"assebly" A136 and tube DA0414, preamp P0412<br> 
<br> 
 
we removed the tube needing to pop two covers and to 
lift out of the way an obstructing 1/4(?) in stainless 
tube.<br> 
<br> 
 
This tube was replaced by DA0530, where we recycled 
the preamp and the end cap. The cookie came out with  
the tube, and we used a NEW cookie.<br> 
<br> 
 
2) <br> 
the second tube replacement was to be the matching 
partner to the tube we replaced yesterday 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=18&frame=2&anchor=23:01:38'> 
the tube we replaced yesterday </a> (tube 229), that is, 
we wanted to replace PMT 13. However this looked too 
"dangerous". After consulting with Matthew, we got 
(one more time) a list of "good" tubes to remove. 
We settled on PMT25. <br> 
 <br> 
Removed PMT25 in the 40 degree 3-pack on the east side, 
which is Assembly  A243 containing tube DA0385 and 
preamp P0505 and replaced it with DA0923---again recycling 
preamp and endcap assembly, but putting on a new cookie. <br> 
 <br> 
 
After installation checked out all the tubes we put 
in AND their neighbors in the same three-pack, ie 
tubes 24, 25, 26, 174, 175, 176, 228, 229, 230. 
Applied 1kV and looked at signals from the TOAD 
monitor. All were ok EXCEPT PMT 26 seems to have 
NO VOLTAGE. SMACS reads back NO CURRENT. Used  
a DVM to look at load on HV cable (from upstairs) 
and it is open. Logical conclusion is that we 
damaged the HV connection for this tube. 
Discussing how to proceed with the the management 
and experts...... <br> 
 <br>       :-( 
 <br>    
 <br>   
<!-- Author: Gerry/Dervin/Tom -->


<!-- Date: Fri Apr 21 11:07:39 2006 -->
<br> 
<br> 
   ********** AFTERMATH TOF TUBE REPLACEMENT ***************<br> 
<br> 
<br> 
After 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=18&frame=2&anchor=17:20:56'> 
yesterday's tube replacement</a> 
we found that Tube 26 (~40 degrees east side) was not drawing any current. 
We went down this morning to look for what was wrong. 
We found that the black connector on the back of the "oval" 
pc-board was a little bent. Its hard to tell visually 
for sure, but it looked like the HV pin was not going 
into the black connector.  
We got some epoxy, and put a few dabs on to fix it 
in place, and used a dummy connector for alignment. 
Then we put the oval board back onto the the 3-pack, 
and saw we are now drawing reasonable currents. 
Subsequently we also see signals from this tube, 
as well as the others in the 3-pack. <br> 
<br> 
So finally, all looks ok with all the molested 
3-packs.---and we are done!  
<!-- Author: Gerry/Dervin -->


<!-- Date: Tue May 2 10:38:45 2006 -->
<pre>Replaced the west preamp LV power supply with the spare in 
the cabinet.  Channel 7 on the iFix page shows 7.49 A, which 
is now close to the nominal value of 7.5 A.  However, channel 
1 indicates a low current of 6.65 A.  An unplugged cable would 
give 5 A, so I expect that this is a monitoring problem. 
I'll take the LV power supply that was removed from the detector 
back to Purdue to have it "reworked".</pre>
<!-- Author: M Jones & N. Ranjan -->


<!-- Date: Thu May 4 10:19:33 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-261&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-261  width='300' align='top'></a></td><td>Checking recent TAC calibrations in case anything got broken 
during the shutdown...  run 0x33c82 is from Feb 22, just after 
the shutdown started.</td></tr></table>
<!-- Author: M Jones -->


<!-- Date: Thu May 4 10:21:40 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-262&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-262  width='300' align='top'></a></td><td>...and this one is run 0x34962, taken yesterday (May 3rd). 
They are largely indistinguishible and don't indicate any 
gross failures anywhere.  But there's still lots of time left 
in the shutdown for stuff to get broken...</td></tr></table>
<!-- Author: M Jones -->


<!-- Date: Thu May 18 12:22:07 2006 -->
After replacement of the optocoupler I have done several test to check the communication beetween PC and Caen system. 
Due to absence of magnetic field, I had loaded several time different setup file with very low voltage (50, 60 and 100 Volt). 
No error occurred at all. 
 
Unfortunally the channel 2-7-4 (Pm=138) has problem on setting the V0Set parameter, maybe the board has to be replaced. More check in next days.
<!-- Author: Daniele -->


<!-- Date: Mon May 22 20:33:48 2006 -->
<pre>On Matthew request I have changed the voltage values for  
 
  Channel 229: 1750 volts 
  Channel 176: 1400 volts 
  Channel  25: 1500 volts 
 
The new values has been saved on the same setup file currently in use taht is  
 
DEFAULT050512-today-b14kG_g30k.suf 
 
then restarting the system the new values will be loaded. 
</pre>
<!-- Author: Daniele -->


<!-- Date: Mon May 22 20:47:58 2006 -->
<pre>Checking the behaviour of channel 2-7-4 (PM=138) that some days ago shown problems (see my earlier entry) now it seems work very fine. Indeed that bad behaviour could be due to the very low voltage used in that tests. We need more check and careful watch in next days</pre>
<!-- Author: Daniele -->


<!-- Date: Wed May 24 08:21:39 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-263&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-263  width='300' align='top'></a></td><td>Plots for last night's cosmic run.  Channel 201 appears to be hot, but it's not clear this correlates with any other 
recent issues...</td></tr></table>
<!-- Author: M Jones -->


<!-- Date: Wed May 24 17:03:01 2006 -->
<pre>Smacs reports about 170 error reading the status word. 
All this errors appear on the first six boards of crate 1. 
 
Watching at logger file seems be not a serious problem, nevertheless we have to check periodically if the behaviour goes worste.</pre>
<!-- Author: daniele -->


<!-- Date: Sun May 28 18:44:40 2006 -->
<pre>Some other tests loading and reloading setup file in order to verify the correct trasmission on caenet. No error to report. 
 
I leave system running 
</pre>
<!-- Author: Daniele -->


<!-- Date: Fri Jun 9 23:14:30 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-264&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-264  width='300' align='top'></a></td><td>Data from first collisions... still a hot channel, but it 
doesn't show up in the occupancy plot.  Perhaps it is a 
problem on the TOAD board.</td></tr></table>
<!-- Author: M Jones -->


<!-- Date: Sat Jun 10 13:52:53 2006 -->
<pre>I took a test run with the discriminator threshold set to 2 mV. 
With the HV off, no discriminators fired in 50,000 events, so the 
intrinsic noise floor is somewhere less than 2 mV.  Perhaps things 
change when the HV is turned on. 
 
This study also showed that channel 201 reads out 4 values separated by approximately 2^13, suggestive of a problem on the 
CAFE or ADMEM.  We swapped the CAFE card a while ago, so it is 
probably the ADMEM.</pre>
<!-- Author: M Jones -->


<!-- Date: Sat Jun 10 14:03:52 2006 -->
<pre>With the discriminator threshold set to zero, we now see many channels firing randomly.  This mainly confirms that we are sensitive to noise at some level.  </pre>
<!-- Author: M Jones -->


<!-- Date: Sat Jun 10 14:13:44 2006 -->
<pre>Found evidence of noise with 1 mV discriminator thresholds, but 
it was limited to a handful of channels.  So, with the HV off, 
the noise is below 1 mV for most channels and around 1 mV for 
a limited number of channels.  I'll try to quantify this...</pre>
<!-- Author: M Jones -->


<!-- Date: Sat Jun 10 17:41:46 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-266&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-266  width='300' align='top'></a></td><td>This is the noise occupancy for 1 mV thersholds with the HV 
turned off.  The noisiest part is wcal02 slot 10 TOAD 2.</td></tr></table>
<!-- Author: M Jones -->


<!-- Date: Sun Jun 11 14:56:34 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-268&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-268  width='300' align='top'></a></td><td>This is the noise occupancy with 1 mV discriminator 
thresholds with the HV turned on.  The channel that was 
noisiest with the HV off is now quiet with the HV on which 
is presumably due to a small (<1 mV) baseline shift when 
the HV is on but I don't know what might cause this shift. 
 
So, the conclusions from this study are: 
1.  Noise levels are < 1 mV in almost all channels 
    regardless of the HV state. 
2.  In about 10 channels, the level of noise is around 1 mV 
    regardless of the HV state. 
3.  The size of baseline shifts might be of order 1 mV in 
    some channels when the HV is on.</td></tr></table>
<!-- Author: M Jones/N Ranjan -->


<!-- Date: Sun Jun 11 15:24:30 2006 -->
<pre>We might want to replace TOAD #2 in slot 9 of wcal02, which 
is the noisy one seen with 1 mV discriminator thresholds.</pre>
<!-- Author: M Jones/N Ranjan -->


<!-- Date: Wed Jun 14 14:57:16 2006 -->
<pre>We will replace the TOaD board on slot 9 of wcal02 and replace the ADMEM in 
slot 9 of wcal07.</pre>
<!-- Author: M Jones/N Ranjan -->


<!-- Date: Wed Jun 14 16:23:48 2006 -->
<pre>In the end we did not touch the transition board for slot 9 of 
wcal02 since the CMX arch was not pulled and there was not enough 
space to pull it without purtubing adjacent channels. 
The ADMEM we put in has the following PROM label: 
b0wcal07 slot 9: '0342 003 ADMEM_V4.0          460' 
 
The TAC and QIE calibrations appear to be fine, we will try to 
do a low discriminator threshold run since the problem showed 
up when we did this last time.</pre>
<!-- Author: M Jones -->
<!-- Comment: Wed Jun 14 21;21;35 comment by...M Jones -->
<pre>The ADMEM we removed had serial number 138.  I expect that
all the CAFE and deCAF cards on it are good and could be
used as spares.  However, we'll see how Peter wants to get
it checked out before removing them.</pre>
<!-- Date: Wed Jun 14 19:16:16 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-269&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-269  width='300' align='top'></a></td><td>Here's the latest TAC calibration after downloading the 
flash ram on the QIE cards.  The pedestals do not look 
unreasonable. 
</td></tr></table>
<!-- Author: M Jones -->


<!-- Date: Wed Jun 14 19:22:26 2006 -->
<pre>Based on the 1 mV threshold test (run 0x353c1), the problem 
with channel 201 appears to be gone.  But we need to wait 
for the next store to be sure.</pre>
<!-- Author: M Jones -->



<!-- Date: Wed Jun 14 23:15:49 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-270&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-270  width='300' align='top'></a></td><td>With limited statistics from cosmics it appears that channel 201 is fine.</td></tr></table>
<!-- Author: M Jones -->


<!-- Date: Thu Jun 15 12:05:28 2006 -->
<pre>I turned on pulse monitoring for both ends of bar 0: 
UPDATE CARDS SET TOADCTRL0=2 WHERE CR_CRATENAME='WCAL_00' AND SLOT=5; 
UPDATE CARDS SET TOADCTRL0=2 WHERE CR_CRATENAME='WCAL_04' AND SLOT=5; 
We should disable this at some point before we take serious 
physics quality data.</pre>
<!-- Author: M Jones -->


<!-- Date: Fri Jun 16 09:15:43 2006 -->
<pre>We are trying to re-start Smacs so that we can load new HV settings for a min-bias run.  However, Smacs is getting stuck 
at the point where it says it is  "Updating iFix data base". 
It stays in this state for a long time, like more than 10 minutes.  Right now we have lost at least 1/2 an hour of beam 
time just trying to load the HV settings and we haven't even 
begun to take data yet.  We will need to go through this 3 more 
times.</pre>
<!-- Author: M Jones -->
<!-- Comment: Fri Jun 16 10;22;45 comment by...M Jones/Bruno -->
<pre>We now believe the problem was related to the cr/lf issues
associated with files prepared on a unix machine and then transferred to the PC.  The solution seems to be to open the
file in excel and then save it again as a .suf file in text mode
with tabs as field delimiters.</pre>
<!-- Date: Fri Jun 16 09:49:38 2006 -->
Run <A HREF="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218420">218420</A> is being taken with TOF scan point B...
<!-- Author: M Jones -->


<!-- Date: Fri Jun 16 10:07:06 2006 -->
Run <A HREF="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218421">218421</A> is being taken with TOF HV scan point A.
<!-- Author: M Jones/Bruno -->



<!-- Date: Fri Jun 16 10:29:04 2006 -->
Run <A HREF="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=218422">218422</A> is being taken with TOF HV scan point C.
<!-- Author: M Jones/Bruno -->


<!-- Date: Mon Jun 19 11:03:51 2006 -->
<pre>Got paged at ~10.10 am, because of trip. 
 
When I arrived, bars from crate 1 were  
blue in ifix and smacs showed V=I=0 for them.  
 
Ace tried to bring HV down and UP, but bars from crate 1 remained off. Then it tripped, so everything was brought down again. 
 
Turned HV on from smacs -&gt; no problems. 
taking data now...  
</pre>
<!-- Author: aart -->


<!-- Date: Thu Jun 22 09:32:59 2006 -->
There were problems in WCAL01 associated with the QIE and TAC 
calibrations <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=422&frame=2&anchor=17:03:09.222'> 
last night</a>.  I have asked the shift crew to take another set 
of calibrations to help determine if this problem persists.
<!-- Author: M Jones -->


<!-- Date: Thu Jun 22 13:02:38 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-271&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=tof&action=view&page=-271  width='300' align='top'></a></td><td>I just ran the TAC calibration and everything appears to be fine now.</td></tr></table>
<!-- Author: M Jones -->


<!-- Date: Wed Jul 5 06:42:25 2006 -->
<pre>TOF inhibit: 
 
Trigger Inhibit is set from the components: 
 TOF00 Trip  		[107] Time of Flight Trip 00 
 TOF HV:iFix  		[132] Time of Flight High Voltage from iFix 
 
 
When I clicked on the D on the IFix, ~ 1/3 of the detector was off.. corresponding to one crate. Went down to the 1st floor and checked. Had to power cycle the crate (crate one).  
 
I also noticed the following message on the PC: 
 
7/3/06 12:02:25.6 !SYSTEM ALERT! The alarm queue 'summary list' has overflowed. Atleast one alarm may have been lost 
 
I didn't quite understand this.</pre>
<!-- Author: niharika -->


<!-- Date: Thu Aug 3 14:31:06 2006 -->
<pre>Got paged this morning regarding a discrepancy in the TOF TAC calibration results. Here is the link to the &lt;a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=549&frame=2&anchor=06:22:04.245'&gt; 
plot &lt;/a&gt; 
 
All the channels corresponding to crate b0wcal01 seemed dead (there is a white band corresponding to the crate in the plots above). However, the TOF QIE calibration results looked ok. We re did the TOF TAC calibration a couple of times, tried a different partition and also tried re-starting CSL. But, the calibrations failed.  
 
Finally talked to Matthew over the phone and he suggested that do the following: 
- Check the TOMAIN board voltages 
- Monitor the clock signals 
- Check YMON plots 
 
Matthew helped me in checking the TOMAIN board voltages over the phone. I checked the values for VCC5, VEE5, VCC15, VEE15, VTT for crate b0wcal01 and they looked fine.  
 
We looked at the YMON plots for the physics run 220847 (Owl shift , august 2nd, store 4868) and the plots looked fine. We also looked at YMON plots (TOF ave charge and time plots) for cosmics run 220971 (Day shift, august 3rd) and they looked fine (no gaps). So, all the channels seem to be working fine.  
 
In the meanwhile we were monitoring the clock signals present on the TOMAIN boards using the BNC patch pannel on the first floor. Here is what we did : 
 
- Looked at the signal corresponding to RESET, STOP, OUT_CLOCK, ADC_CLOCK, QIE_CLOCK, QIE_RESET for the crates listed below and the signals looked fine (atleast we did not notice any discrepancies between them) 
b0wcal00 slot 5 channel 48 (on the patch pannel) 
b0wcal01 slot 5 channel 60 
b0wcal03 slot 5 channel 84 
 
The signal for OUT_CLOCK, ADC_CLOCK, QIE_CLOCK, QIE_RESET was flat which was normal I guess since it comes from the ADMEMs and can probably seen only when the ADMEM is jumpered (what does this actually mean ?). 
 
Fianlly we looked at the CAL signal for the same crates listed above by re-doing the TOF TAC calibration multiple times (Is there a better way to look at this signal ie. instead of re-doing the entire calibration everytime ?). 
 
We saw the CAL signal for crates b0wcal00 and b0wcal03 on the scope but did not see it for crate b0wcal01 (slot 5 in each case). We tried a couple of calibrations but noticed the same (missing CAL signal for crate b0wcal01) and the TOF TAC calibration failed each time.  
 
Finally we tried a new partition the second time and the calibration went fine this time! The plots looked fine and we also saw the CAL pulse for crate b0wcal01 !!  
 
This was pretty strange since it the problem seemed to have cured itself by changing the partition! It's unlikely that we reset something by just monitoring the clock but I am not sure now. The last calibration run taken today (run 221012) looked fine. But we don't quite understand what caused or cured the problem. 
 
 
 
 
</pre>
<!-- Author: Niharika, Guillemo, Aart -->
<!-- Comment: Thu Aug  3 14;31;56 comment by...Niharika -->
<pre>Sorry forgot to paste it as html!</pre>


<!-- Date: Sat Aug 5 03:21:45 2006 -->
<pre>HV settinig for run 221168 was  
newHVSetting_700ADC_2200limit_08042006.suf. 
 
For next run the HV setting is changed to 
newHVSetting_fix_2200limit_08042006.suf</pre>
<!-- Author: Bruno -->


<!-- Date: Sat Aug 5 03:58:58 2006 -->
<pre>Until the end of store 4884 data will be taken with HV setting: 
newHVSetting_fix_2200limit_08042006.suf 
</pre>
<!-- Author: Bruno -->


<!-- Date: Sat Aug 5 20:35:22 2006 -->
<pre>Brief summary of ToF HV studies: 
1.- Tested both HV settings on quiet time, no problem was found. 
 
2.- During store 4884 data was taken with both HV configurations: 
  run 221168: newHVSetting_700ADC_2200limit_08042006.suf 
  runs 221169-221174: newHVSetting_fix_2200limit_08042006.suf 
 
3.- At the end of the store default HV setting was loaded. 
 
For store 4884 TOF was marked bad. </pre>
<!-- Author: Bruno -->


<!-- Date: Wed Aug 23 18:11:25 2006 -->
<pre>We started having problems with the TOF HV at the beginning of the evening shift on Monday.  
 
We had multiple TOF trips (crate 3), we checked that the communication with the crate was still established and the HV monitoring program was still updating. However, the PC was really slow. I and Guillelmo rebooted the PC and started the HV monitoring. Different channels on crate 3 still continued to trip. We tried to pin down the problem by checking whether it was the same channel/board that continued to trip.  
 
However, everytime there were different channels that tripped. We powercycled the crate twice (using the key) but this didn't fix the problem.  We even tried turning the channels ON one by one, but they tripped after a few mins.  
 
After talking to Sandro over phone, we gave it another try (ie powercycled the crate and tried to reset the 
trip) but couldn't reset the trip. Since, we were having problems with other detector components and were not taking useful physics data Sandro suggested that we look into the TOF HV problem in the morning as we might have to replace the crate.  
 
We tried to reset the trip in the morning but failed. At one point the PC froze and had to rebooted again. Sandro powercycled the crate once again and noticed that the LED next to 'FAN FAILURE' was lit. Finally, we decided that the problem was with the crate and not individual channels or boards.  
 
Roberto, Steve and another technician changed the crate. After which we were able to bring up the TOF HV. </pre>
<!-- Author: Niharika, Sandro, Guillelmo -->


<!-- Date: Wed Aug 30 09:13:40 2006 -->
Updated calibration instructions/reference histograms.<BR> 
Added some <A HREF="http://www-cdfonline.fnal.gov/ops/tof/info/expert_web.html">documentation</a> to help maintain TOF web pages.
<!-- Author: M Jones -->


<!-- Date: Thu Aug 31 20:48:14 2006 -->
<pre>We got TOF trip alarm. I went down to check on the PC and saw that one of the channels in crate 2 was drawing too much current. I have turned off this channel and masked the inhibit for now. And we are taking with TOF included (with only one channel off). 
 
Info about the channel:  
Crate 2 , board 2 , channel 0 - 96  
VMon 3126.8  
VSet 1906  
IMon 2140  
ISet 250 
 
Since it's just one channel, we have told the shift crew NOT to mark TOF as bad for the run(s), till the end of the store (4932). We will turn this channel ON at the end of store (tomorrow) and see how it behaves. </pre>
<!-- Author: Niharika, Bruno, Guillelmo -->
<!-- Comment: Wed Oct  4 09;55;42 comment by...Bruno -->
<pre>Channel 96 is off since run 222418.</pre>
<!-- Date: Thu Sep 14 13:41:11 2006 -->
<pre>Testing new default HV setting.  
There was a problem with channel 400 that now is solved.  
Also channel 96 trips as it did 2 weeks ago. HV for this channel is fixed to a low value (until we figure out what is wrong with this channel) so it shouldn't trip.</pre>
<!-- Author: Bruno -->


<!-- Date: Thu Sep 14 14:23:17 2006 -->
<pre>New default file for HV configuration for the next stores: 
 
DEFAULT060914-today.suf</pre>
<!-- Author: Bruno -->
<!-- Comment: Thu Sep 14 17;33;24 comment by...Bruno -->
<pre>This setting is being used since run 223029</pre>


<!-- Date: Fri Sep 15 15:12:55 2006 -->
<pre>I'm about to replace the CAEN module that has problems with channel 96.</pre>
<!-- Author: m jones -->


<!-- Date: Fri Sep 15 15:51:38 2006 -->
<pre>The problem with channel 96 is NOT with the CAEN module.  I replaced the one with FNAL property number 575511 with a new 
one from the spares cabinet that had FNAL property number 575523 
but the problem persists.  Channel 96 draws 123 uA at 10 volts 
but shows up as 10 Mohm when the resistance is measured on the 
cable.  I propose that we leave this channel unplugged until 
we can get a good long access to look at it downstairs.</pre>
<!-- Author: m jones -->


<!-- Date: Fri Sep 15 17:58:23 2006 -->
<pre>New HV configuration file (DEFAULT060914-today.suf) such as channel 96 is out of monitoring until we figure out what is wrong with it. 
This should be usefull for shift crew if they need to restart TOF HV PC.</pre>
<!-- Author: Bruno -->
<!-- Comment: Fri Sep 15 17;59;08 comment by...Bruno -->
<pre>Sorry the name of the file is: DEFAULT060914-today_NO_CHANNEL_96.suf</pre>


<!-- Date: Mon Oct 2 10:00:23 2006 -->
<pre>Last night TOF was paged but the batteries was dead, that's why I didn't call back. TOF was paged because HV tripped three times, but now is working smoothly. I apologize for the inconvenience.</pre>
<!-- Author: Bruno -->



<!-- Date: Thu Oct 12 09:25:06 2006 -->
<pre>Mark Mattson is willing to replace the CAFE card that generates 
the errors during the QIE calibration: 
 
Code: 0x6070804421446  Mnemonic: FERML_QIECALIB_ERROR 
Node Name: b0wcal01; Node Id: 131.225.237.109 
Process: Readout_Calib; PID: 12403272; Time: 9:23:46 AM 
Code Line: 2633; Routine: FER_mess.c 
Parameters: 8 16 Fail 14: Sigma(off) big      1.0975677 0 2  
 
Run 224714 is the QIE calibration before replacing it.</pre>
<!-- Author: M Jones -->
<!-- Comment: Thu Oct 12 09;37;39 comment by...M Jones -->
<pre>The replacement CAFE card will have serial number 02774.</pre>


<!-- Date: Thu Oct 12 11:01:58 2006 -->
<pre>Replacement seems to have been successful.  QIE calibration run 224722 had no errors or warnings.  Downloaded QIE flash ram.</pre>
<!-- Author: M Jones -->


<!-- Date: Wed Oct 18 10:14:07 2006 -->
<pre>HV Cable for channel 96 has been replaced, the serial numbers are: 
Old cable: 53898 TOF ST E, HV PMT 96 
New cable: 53910 TOF ST E, HV SPARE 2</pre>
<!-- Author: Bruno -->
<!-- Comment: Wed Oct 18 10;15;01 comment by...Bruno -->
<pre>Run Number: 225250</pre>


<!-- Date: Sat Oct 21 05:17:24 2006 -->
<pre>West LV preamp power supply was replaced on Friday morning. 
A problem showed up where the current was low and the voltage 
monitored was slightly higher.  A hole showed up in YMon so Dervin replaced the supply and things are fine now.</pre>
<!-- Author: M Jones -->


<!-- Date: Mon Nov 6 10:46:24 2006 -->
<pre>Got paged around 10am: tof crates were dropped because of a flamable gas alarm. Superficial inspection of the pc on 1st floor shows things seem to have recovered ok. Will keep an eye on things 
during next few hours (when turning hv on, calibrations...) 
</pre>
<!-- Author: aart -->


<!-- Date: Tue Nov 14 15:59:08 2006 -->
<pre>Starting TOF discriminator threshold scan during store 5066. Run 227834 was taken with standard threshold setting (15mV). Began a new run (227867) with first threshold point of 5mV.</pre>
<!-- Author: Bruno -->
<!-- Comment: Tue Nov 14 19;21;53 comment by...Bruno -->
<pre>run 227881: threshold set to 2mV.</pre>

<!-- Comment: Wed Nov 15 07;47;00 comment by...Bruno -->
<pre>threshold set to 10mV for runs 227882, 227883, and 227884 with total integrated luminosity of 1113.3 nb-1.</pre>

<!-- Comment: Wed Nov 15 09;57;08 comment by...Bruno -->
<pre>run 227886: nominal value of 15mV is set. </pre>


<!-- Date: Sat Nov 18 13:26:02 2006 -->
<pre>Following up on Matthew's email (which I'll paste below), I had a look at the timing signals on the 1st floor patch panel. I  
checked the following channels 
 
for reference: 
wcal03 5 (bnc connector 84)  see both STOP and RESET 
wcal01 5 (bnc connector 60)  see both STOP and RESET 
 
then in the problematic crate: 
wcal00 5 (bnc connector 48)  see only STOP, not RESET 
wcal00 10 (bnc connector 58) see only STOP, not RESET 
 
So it's clear: the RESET signal is absent for wcal00 
channels. 
 
aart. 
 
 
------------ matthew's email ----------------- 
 
It appears that part of the TOF system did not survive the power glitch yesterday.  Right now we aren't getting any valid data from wcal00. Here's what I can piece together from the e-log: 
 
On Thursday, Nov 16 at 23:45 the TAC calibration (228196) was taken and the CO claimed to have checked it but in fact the data from an entire crate (wcal00) was missing.  They declared that it was okay in the e-log. 
 
On Friday Nov 17th at 18:11 the TAC calibration was described as having failed so nobody bothered to check the results. 
 
On Saturday Nov 18th at 2:14 the TAC calibration was taken again and the CO claimed it was fine even though (again) there was no data from wcal00. 
 
So, we proceeded to take data and at 5:10 it was noticed that indeed there was no data coming from wcal00. 
 
Anyway, never mind how this got missed, we have to figure out how to diagnose the problem and fix it.  Here's what we know so far: 
 
1.  The QIE calibration is okay indicating that the ADMEM is fine. 
2.  I read out all the voltages in the crate and they all look fine. 
    In particular, I *think* this rules out a problem with the -2 volt 
    supply which is one potential point of failure. 
3.  It might be a problem with either the local clock fanout module 
    or one channel on the master clock fanout module.  To check these 
    we would need to do the following: 
 
    - Follow the instructions on 
       http://www-cdfonline.fnal.gov/ops/tof/tof_electronics.html 
      to look for clock signals from wcal00 (and compare them with 
      wcal01) using the oscilloscope near the patch panel in the 1st 
      floor counting room.  If we see a STOP signal but no RESET signal 
      or a RESET signal but no STOP signal, then the problem is with the 
      local clock fanout module.  If we don't see either RESET or STOP 
      then the problem is either the LCF module or one channel on the 
      master clock fanout module.  Either way, I think we need an access 
      to track it down. 
 
    - Assuming we need an access, we would need to have the south muon 
      wall opened and go up to the master clock fanout module in the ADMEM 
      side of wcal00. 
      o  If we swap the common stop cables for wcal01 and 
         wcal00 but the problem remains with wcal00 then we know it's the 
         LCF module in the back of wcal00.  If the problem moves to wcal01 
         then we know we have to replace the master clock fanout module. 
         I understand that we have a spare one from Italy in our spares 
         cabinet. 
      o  If we swap the common stop cables but the probem stays with 
         wcal00 then we know the problem is with the LCF module and we 
         will have to open up the west IMU and north west CMX arch to get 
         access to the back of wcal00 to replace the LCF module. 
 
    - Either way, after the change we need to take a TAC calibration to 
      verify that the problem has been fixed. 
 
    - It's not clear who is available to do this work... possibly a 
      combination of TOF pager carriers (such as Niharika) ADMEM experts 
      people from Italy who still have some connection with the clock 
      fanout system and/or CDF operations people like Dervin, etc. 
 
I should be available by cell phone (630-309-8187) if you want to discuss 
this further.  As it stands, TOF should be marked bad since 1/4th of the 
detector is giving no valid ata. 
 
Have a good weekend! 
 
----------------------------------------------- 
 
 
 
</pre>
<!-- Author: aart -->


<!-- Date: Wed Nov 22 08:12:27 2006 -->
<pre>Yesterday Dervin replaced the LCF module in wcal00 and the problem 
eventually went away after power cycling the crate.  It is possible 
that power cycling the crate in the first place would have fixed 
this.  I vaguely recall that there was another instance where we 
replaced a LCF module a long time ago but when we looked at it in 
the test stand it was fine.  So, there might be a "feature" of the 
LCF module that gets it into a wierd state that will be recovered 
by power cycling.  Next time, if we see the stop signal (which is 
just fanned out with no logic) but no reset signal (which is 
generated using more sophistocated logic and communication with 
a TOMAIN board), then we should try power cycling that crate. 
 
Also, I think it is probably NOT a good idea to remove transition 
boards without turning off the power in the crate, not because 
the boards will die from being hot swapped, but because there is 
a significant chance of making contact with the adjacent module, 
shorting out one of the voltages and blowing a fuse.</pre>
<!-- Author: mjones -->