<!-- Title: Owl shift. Tue Jun  5, 2001 -->
<!-- SciCo: Christoph Paus -->
<!-- DAQAce: Michael Riveline -->
<!-- MonAce: Koji Ikado -->
<!-- CO: Andrei Solodsky -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre>Shift Plan 
========== 
 
a) Reset all calorimeter and COT crates by hand. Good exercise 
   to see whether everything works properly. 
 
b) Do CLC QIE if there is quiet time. 
 
c) Keep the detector running as much as possible. 
 
d) Keep an eye on the radiation montoring. 
 
</pre>


<!-- Date: Tue Jun 5 00:19:15 2001 -->
<pre>BLM integrated dose at the starting time of the owl shift : 
SVRAD0 : 0 
SVRAD1 : 0 
SVRAD2 : 0 
SVRAD3 : 0</pre>
<!-- Author: Koji -->


<!-- Date: Tue Jun 5 00:19:56 2001 -->
<pre>End run 116799. 
Start minicom exercise as requested by Rik</pre>
<!-- Author: michael -->


<!-- Date: Tue Jun 5 01:04:29 2001 -->
<pre>I looked through the process system alarms and did not 
see any problem. But "GAS" menu is always yellow.</pre>
<!-- Author: Koji -->


<!-- Date: Tue Jun 5 01:44:43 2001 -->
<pre>PTM heartbeat box on the GLOBAL ALARMS is always violet  
(Local PC unaccessible).</pre>
<!-- Author: Koji -->


<!-- Date: Tue Jun 5 02:22:15 2001 -->
<pre>PHA trips on crate 8 at 2:19. 
Sound alarm system worked fine. 
It recovered automatically by itself.</pre>
<!-- Author: Koji -->


<!-- Date: Tue Jun 5 03:00:11 2001 -->
<pre>Goes fine for cal except: 
 
b0wcal00 -&gt; send doesnot work, even after resetting the crate. 
 
b0pcal01 -&gt; reset_crate indicate that the crate was reset but there is no boot message in the minicom window. 
 
b0cot01 -&gt; send doesnot work. 
 
 
 When resetting the b0cot06 crate, some unusual lines at the end of the boot message: 
 
b0cot06: 
------- 
 
    DAC(0): b0cot06 (0xE680) opened, CSR(0) 0x02 = 0x5800 (module 2) 
  
    ATN = +0000 mV   --&gt;   C01(0) C07(0) C13(0) C19(0) 
    DTH = +1000 mV   --&gt;   C02(0) C08(0) C14(0) C20(0) 
    TRE = +0000 mV   --&gt;   C04(0) C10(0) C16(0) C22(0) 
    TRO = +0000 mV   --&gt;   C03(0) C09(0) C15(0) C21(0) 
    QEN = +0000 mV   --&gt;   C05(0) C11(0) C17(0) C23(0) 
    QDR = +0000 mV   --&gt;   C06(0) C12(0) C18(0) C24(0) 
  
    BID(0): 0x00 = 0x1780 
    CSR(0): 0x02 = 0x5880 
    ADC(0): 0x04 = 0x00F8 
  
    C01(0): 0x40 = 0x0800 (+0000 mV)       C17(0): 0x60 = 0x0800 (+0000 mV) 
    C02(0): 0x42 = 0x099A (+1001 mV)       C18(0): 0x62 = 0x0800 (+0000 mV) 
    C03(0): 0x44 = 0x0800 (+0000 mV)       C19(0): 0x64 = 0x0800 (+0000 mV) 
    C04(0): 0x46 = 0x0800 (+0000 mV)       C20(0): 0x66 = 0x099A (+1001 mV) 
    C05(0): 0x48 = 0x0800 (+0000 mV)       C21(0): 0x68 = 0x0800 (+0000 mV) 
    C06(0): 0x4A = 0x0800 (+0000 mV)       C22(0): 0x6A = 0x0800 (+0000 mV) 
    C07(0): 0x4C = 0x0800 (+0000 mV)       C23(0): 0x6C = 0x0800 (+0000 mV) 
    C08(0): 0x4E = 0x099A (+1001 mV)       C24(0): 0x6E = 0x0800 (+0000 mV) 
    C09(0): 0x50 = 0x0800 (+0000 mV)       C25(0): 0x70 = 0x0800 (+0000 mV) 
    C10(0): 0x52 = 0x0800 (+0000 mV)       C26(0): 0x72 = 0x0800 (+0000 mV) 
    C11(0): 0x54 = 0x0800 (+0000 mV)       C27(0): 0x74 = 0x0800 (+0000 mV) 
    C12(0): 0x56 = 0x0800 (+0000 mV)       C28(0): 0x76 = 0x0800 (+0000 mV) 
    C13(0): 0x58 = 0x0800 (+0000 mV)       C29(0): 0x78 = 0x0800 (+0000 mV) 
    C14(0): 0x5A = 0x099A (+1001 mV)       C30(0): 0x7A = 0x0800 (+0000 mV) 
    C15(0): 0x5C = 0x0800 (+0000 mV)       C31(0): 0x7C = 0x0800 (+0000 mV) 
    C16(0): 0x5E = 0x0800 (+0000 mV)       C32(0): 0x7E = 0x0800 (+0000 mV) 
  
    DAC(0): b0cot06 (0xE680) closed   
 
 
 
Then for b0cot07 to 16: 
------------------ 
Connecting to project &lt;cdf_daq&gt; on &lt;tcp:b0dau30&gt; RTserver. 
Using tcp protocol. 
RTserver denied access. 
Message: Number of client licenses exceeded 
Message from RTlm: Checking in &lt;RTclient&gt; license. 
Message from RTlm: License for &lt;RTclient&gt; granted (72 licenses left). 
Connecting to project &lt;cdf_daq&gt; on &lt;tcp:b0dau30&gt; RTserver. 
Using tcp protocol. 
RTserver denied access. 
Message: Number of client licenses exceeded 
Message from RTlm: Checking in &lt;RTclient&gt; license. 
Message from RTlm: License for &lt;RTclient&gt; granted (72 licenses left). 
Connecting to project &lt;cdf_daq&gt; on &lt;tcp:b0dau30&gt; RTserver. 
Using tcp protocol. 
RTserver denied access. 
Message: Number of client licenses exceeded 
Message from RTlm: Checking in &lt;RTclient&gt; license. 
Message from RTlm: License for &lt;RTclient&gt; granted (72 licenses left). 
Connecting to project &lt;cdf_daq&gt; on &lt;tcp:b0dau30&gt; RTserver. 
Using tcp protocol. 
RTserver denied access. 
Message: Number of client licenses exceeded 
Message from RTlm: Checking in &lt;RTclient&gt; license. 
Message from RTlm: License for &lt;RTclient&gt; granted (72 licenses left). 
Connecting to project &lt;cdf_daq&gt; on &lt;tcp:b0dau30&gt; RTserver. 
Using tcp protocol. 
RTserver denied access. 
Message: Number of client licenses exceeded 
Message from RTlm: Checking in &lt;RTclient&gt; license. 
Message from RTlm: License for &lt;RTclient&gt; granted (72 licenses left). 
Connecting to project &lt;cdf_daq&gt; on &lt;tcp:b0dau30&gt; RTserver. 
Using tcp protocol. 
RTserver denied access. 
Message: Number of client licenses exceeded 
Message from RTlm: Checking in &lt;RTclient&gt; license. 
FER_smartInitMS: Could not connect to RTserver!           
 
 
b0cot08, 09, 10, 11, 12, 13, 14, 15: see b0cot07 
 
 
b0cot16: b0cot07+ 
  
rlogind: This system *IN USE* via rlogin by user: vejcik. 
shell restarted.                                                                 
 
Reset again and no message at the end. 
 
RT server licences have disappeared!!! 
 
 
Expert page -&gt; Problem with the RT server: number of tokens exceeded. 
 
 
Look at the documentation: SmartSockets 
 
1) Obtain the number of rtserver connections: 
on b0dau30, do /usr/etc/netstat | grep 5101 | sort -k5 -u | wc -l 
 
--&gt; shows 201 connections!!! 
 
 
1) exit from all the consumer windows. 
 
2) look again: number of connections goes up! 
b0dap16 and 19 seem to have the most connections 
 
3) find b0dap19 and reboot 
 
4) number of connections drops back to 201. 
 
5) Don't know what to do so restart rtserver as indicated: 
 
setup smartsockets 
rtserver 
 
6) look again-&gt; still 201 connections??? 
 
7) Try to reset b0cot17 -&gt; still no connection to RT server 
 
Call the expert 
                                         </pre>
<!-- Author: michael -->


<!-- Date: Tue Jun 5 03:04:38 2001 -->
<pre>Call the expert: says he will restart the RT server himself.</pre>
<!-- Author: michael -->


<!-- Date: Tue Jun 5 03:15:04 2001 -->
<pre>The rtserver had NOT been restarted when I logged in, 
but had been running since this afternoon: 
 
b0dau30:21:~&gt;ps -fu cdfdaq 
     UID        PID       PPID  C    STIME TTY     TIME CMD 
  cdfdaq    2076276    2076114  0 03:05:41 pts/36  0:00 -tcsh 
  cdfdaq       1328          1  0   May 31 ?      24:48 /cdf/code-common/products/smartsockets/v5_1/bin/iris_irix/rtlm.x 
  cdfdaq    1761091          1  0 16:44:10 ?      57:25 /cdf/code-common/products/smartsockets/v5_5/bin/iris_irix/rtserver.x 
   
I logged into b0dau30 as user cdfdaq and killed it. 
The automatic restart script restarted it. 
Now the number of connections is less: 
 
b0dau30.fnal.gov&gt; /usr/etc/netstat | grep 5101 | sort -u -k5 | wc -l 
           161 
 
In general if one reboots vxworks nodes by SYSRESET then 
the connections will not be cleanly closed. 
It is best to reboot via the vxboot command like script, 
or the "reboot" button on the vxmon graphics pages. 
In this case the connections will be cleanly closed. 
 
Also, simply rebooting PCs without closing the applications 
will also NOT cleanly close the connections and reclaim 
the licenses. 
This is the way TCP/IP is designed to work.</pre>
<!-- Author: patrick -->


<!-- Date: Tue Jun 5 03:33:58 2001 -->
<pre>Expert resets RT server. 
 
So we reset all the COT crates (07 to 19). 
 
This time, there is no problem and the crates reset fine.</pre>
<!-- Author: michael -->


<!-- Date: Tue Jun 5 03:41:42 2001 -->
<pre>After reseting all the crates, we start taking a new run. 
Set up smoothly. </pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?116800' target=_top>116800</a>) -->


<!-- Date: Tue Jun 5 05:00:03 2001 -->
<pre>End run after some time and start again.</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?116801' target=_top>116801</a>) -->


<!-- Date: Tue Jun 5 06:15:36 2001 -->
No beam, so stop the run to take a CLC QJE calibration: run 116802.
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?116802' target=_top>116802</a>) -->


<!-- Date: Tue Jun 5 06:35:45 2001 -->
<pre>Did 2 CLC QIE calibration (no beam in the machine). 
Only crate CLC00 is included but this crate is labelled dead.  
I thought it was a known problem. 
The data does not appear in the look area.  
 
 
</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?116803' target=_top>116803</a>) -->


<!-- Date: Tue Jun 5 06:50:56 2001 -->
<pre>Start a new run: 116804</pre>
<!-- Author: michael :: (run <a href='/cgi-bin/runSummaryRun.cgi?116804' target=_top>116804</a>) -->


<!-- Date: Tue Jun 5 08:01:42 2001 -->
<pre>Problems with the last run: times out in a lot of crates: 
 
(MLE) b0cot02:Messenger:7:56:53 AM->Event 623200: Bunchcounters in slot 2 and slot 3 disagree (at 
least) 
(MLE) b0ccal01:Messenger:7:56:53 AM->Runtime Error 1, Event 623200: See nonsequential event number (old 15, new 12) in L2 message 
(MLE) b0ccal00:Messenger:7:56:53 AM->Runtime Error 1, Event 623200: See nonsequential event number (old 15, new 5) in L2 message 
(MLE) b0ccal06:Messenger:7:56:52 AM->Runtime Error 1, Event 623200: See nonsequential event number (old 15, new 9) in L2 message 
(MLE) b0ccal07:Messenger:7:56:53 AM->Runtime Error 1, Event 623200: See nonsequential event number (old 15, new 5) in L2 message 
(MLE) b0cot01:Messenger:7:56:53 AM->Runtime Error 4, Event 623200: See nonsequential event number 
(old 15, new 5) in L2 message 
(MLE) b0cot09:Messenger:7:56:53 AM->Runtime Error 2, Event 623200: See nonsequential event number 
(old 15, new 12) in L2 message 
(MLE) b0pcal05:Messenger:7:56:53 AM->Runtime Error 1, Event 623200: See nonsequential event number (old 15, new 12) in L2 message 
(MLE) b0pcal01:Messenger:7:56:53 AM->Runtime Error 1, Event 623200: See nonsequential event number (old 15, new 12) in L2 message 
(MLE) b0wcal00:Messenger:7:56:53 AM->Runtime Error 1, Event 623200: See nonsequential event number (old 15, new 12) in L2 message 
(MLE) b0wcal03:Messenger:7:56:53 AM->Runtime Error 1, Event 623200: See nonsequential event number (old 15, new 12) in L2 message 
(MLE) b0cot02:Messenger:7:56:53 AM->Runtime Error 1, Event 623200: Bunch counter mismatch, mismatch count = 10 
(MLE) b0ccal01:Messenger:7:56:53 AM->Runtime Error 3, Event 623200: SMXR in slot 3 is not responding, header 0x0x3d002d 
(MLE) b0ccal00:Messenger:7:56:53 AM->Runtime Error 3, Event 623200: SMXR in slot 3 is not responding, header 0x0x2b002f 
(MLE) b0ccal07:Messenger:7:56:53 AM->Runtime Error 3, Event 623200: SMXR in slot 3 is not responding, header 0x0x2d0047 
(MLE) b0pcal01:Messenger:7:56:53 AM->Event 623200: Bunchcounters in slot 14 and slot 4 disagree (at least) 
(MLE) b0pcal05:Messenger:7:56:53 AM->Event 623200: Bunchcounters in slot 15 and slot 4 disagree (at least) 
(MLE) b0ccal01:Messenger:7:56:54 AM->Runtime Error 5, Event 623200: SMXR in slot 4 is not responding, header 0x0x220027 
(MLE) b0ccal00:Messenger:7:56:53 AM->Runtime Error 5, Event 623200: SMXR in slot 4 is not responding, header 0x0x3c0039 
(MLE) b0ccal07:Messenger:7:56:53 AM->Runtime Error 5, Event 623200: SMXR in slot 4 is not responding, header 0x0x3d003b 
(RC)   7:56:53 Run -> ACTIVE 
(MLE) b0pcal01:Messenger:7:56:53 AM->Runtime Error 5, Event 623200: Bunch counter mismatch, mismatch count = 3 
(MLE) b0pcal05:Messenger:7:56:53 AM->Runtime Error 5, Event 623200: Bunch counter mismatch, mismatch count = 3 
(MLE) b0ccal01:Messenger:7:56:54 AM->Runtime Error 7, Event 623200: SMXR in slot 5 is not responding, header 0x0x2f002e 
(MLE) b0ccal00:Messenger:7:56:53 AM->Runtime Error 7, Event 623200: SMXR in slot 5 is not responding, header 0x0x330036 
(MLE) b0cmx01:Messenger:7:56:53 AM->Runtime Error 1, Event 623199: TDC in slot 17 not ready for readout 
(MLE) b0ccal07:Messenger:7:56:53 AM->Runtime Error 7, Event 623200: SMXR in slot 5 is not responding, header 0x0x310025 
(MLE) b0ccal01:Messenger:7:56:54 AM->Event 623200: Bunchcounters in slot 16 and slot 3 disagree (at least) 
(MLE) b0ccal00:Messenger:7:56:53 AM->Event 623200: Bunchcounters in slot 16 and slot 3 disagree (at least) 
(MLE) b0ccal07:Messenger:7:56:53 AM->Event 623200: Bunchcounters in slot 17 and slot 3 disagree (at least)     
                             (MLE) b0ccal07:Messenger:7:56:53 AM->Runtime Error 5, Event 623200: SMXR in slot 4 is not responding, header 0x0x3d003b 
(RC)   7:56:53 Run -> ACTIVE 
(MLE) b0pcal01:Messenger:7:56:53 AM->Runtime Error 5, Event 623200: Bunch counter mismatch, mismatch count = 3 
(MLE) b0pcal05:Messenger:7:56:53 AM->Runtime Error 5, Event 623200: Bunch counter mismatch, mismatch count = 3 
(MLE) b0ccal01:Messenger:7:56:54 AM->Runtime Error 7, Event 623200: SMXR in slot 5 is not responding, header 0x0x2f002e 
(MLE) b0ccal00:Messenger:7:56:53 AM->Runtime Error 7, Event 623200: SMXR in slot 5 is not responding, header 0x0x330036 
(MLE) b0cmx01:Messenger:7:56:53 AM->Runtime Error 1, Event 623199: TDC in slot 17 not ready for readout 
(MLE) b0ccal07:Messenger:7:56:53 AM->Runtime Error 7, Event 623200: SMXR in slot 5 is not responding, header 0x0x310025 
(MLE) b0ccal01:Messenger:7:56:54 AM->Event 623200: Bunchcounters in slot 16 and slot 3 disagree (at least) 
(MLE) b0ccal00:Messenger:7:56:53 AM->Event 623200: Bunchcounters in slot 16 and slot 3 disagree (at least) 
(MLE) b0ccal07:Messenger:7:56:53 AM->Event 623200: Bunchcounters in slot 17 and slot 3 disagree (at least) 
(MLE) b0cmx00:Messenger:7:56:55 AM->Runtime Error 1, Event 623199: TDC in slot 15 not ready for readout 
(MLE) b0ccal01:Messenger:7:56:54 AM->Runtime Error 8, Event 623200: Bunch counter mismatch, mismatch count = 3 
(MLE) b0ccal00:Messenger:7:56:54 AM->Runtime Error 8, Event 623200: Bunch counter mismatch, mismatch count = 3 
(MLE) b0ccal07:Messenger:7:56:53 AM->Runtime Error 8, Event 623200: Bunch counter mismatch, mismatch count = 3             
                                                       56'54
<!-- Author: michael -->


<!-- Date: Tue Jun 5 08:03:05 2001 -->
<font color=red>Shift Summary: </font><pre>Shift Summary 
============= 
 
 a) Beam in the machine basically all the time with short interuptions. 
 
 b) Verified list of crates in the readout; including IMU_01 which 
    every now and then failed; HRR usually fixed the problem. It would 
    be good to fix this once and for all. 
 
 c) As an exercise we reset all calorimeter crates and COT crates over 
    minicom to see whether it is working. Worked well except for some 
    exceptions which are mentioned in the log. Duriong the testing we 
    ran into the problem of smartsocket license limitations. Some 
    processes would not properly handback their license. Jim helped to 
    rectify the situation. Documentation for smartsocket reset on the 
    WEB needs improvement. 
 
 d) **NO** problems with hadron TDC (HTDC00). 
 
 e) Taking lots of cosmics. 
 
 f) CLC calibration failed to appear in the look area?! Experts need 
    to check what is going on! 
 
 g) No radiation dose observed whatsoever. 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   116804 - 116804
Delivered Luminosity   0  pb-1
Acquired Luminosity    0  pb-1
Efficiency             100 %
   
</td></tr></table></pre>
<!-- Author: Christoph Paus -->


<!-- Date: Tue Jun 5 08:04:00 2001 -->
<pre>We got alarms on CMU, CMP and CMX. 
Beam loss caused this. But we had no baem, so turned them on.</pre><font color=red><cite>(entry outside this shift's time range )</cite></font>
<!-- Author: Koji -->