<!-- Title: 18:59:06  Mon May 26 2008 -->
<!-- Crewchief: Hyunwoo Kim -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Start of A New CDF SAM Shift</pre>


<!-- Date: Mon May 26 18:59:57 2008 -->
<pre>Tuesday 09:00 AM at Korean Local Time</pre>
<!-- Author: Hyunwoo -->


<!-- Date: Mon May 26 19:12:15 2008 -->
<pre>(Presumably) Known Problems from DHAAG page: 
1. No of cells Inactive: 4: r-fcdfdata320-{1,4} 
2. No of cells OFFLINE : 7: r-fcdfdata154-{1,3}, data157-{1,4} 
3. Bad Movers: 9940B22, 26</pre>
<!-- Author: Hyunwoo -->
<!-- Comment: Tue May 27 11;54;03 comment by...Angela Bellavance -->
<pre>The machine fcdfdata154 being down was not a known issue. Email has been sent to the dCache admins.

By the way, it is fcdfdat157 that has 3 pools and fcdfdata154 has 4 pools (unlike previous comment).

- Angela</pre>
<!-- Comment: Tue May 27 15;45;39 comment by...Angela Bellavance -->
<pre>My mistake - it was a known issue. The machine will not power up. A service call has been placed.

- Angela</pre>
<!-- Date: Tue May 27 02:37:49 2008 -->
<pre>Total number of files to restore in dCache 
have been increasing since around midnight. 
</pre>
<!-- Author: Hyunwoo -->



<!-- Date: Tue May 27 11:57:48 2008 -->
<pre>Queue has developed on dCache pool r-fcdfdata138-2. Pool has been moved to TestPools group to drain. 
 
- Angela</pre>
<!-- Author: Angela Bellavance -->



<!-- Date: Tue May 27 20:52:05 2008 -->
<pre>- As of 10:45(8:45PM Chicago), 
  CDF-LTO3 seems back online. 
  (CDF-9940B has been online since 6:15PM Chicago) 
- A Bad Tape IAE661 in CDF-LTO3 showed up. 
  an email sent to enstore-admin ccing cdfdh_oper 
- 2 Bad Movers in CDF-LTO3 as well. 
  will keep on an eye on it for a few more hours</pre>
<!-- Author: Hyunwoo -->


<!-- Date: Tue May 27 21:50:35 2008 -->
<pre>Total number of files to restore are high (3800) now 
but appear to disincrease gradually as enstore is back online.</pre>
<!-- Author: Hyunwoo -->
<!-- Comment: Tue May 27 21;51;34 comment by...Hyunwoo -->
<pre>disincrease-&gt;decrease(sorry)</pre>


<!-- Date: Wed May 28 02:42:33 2008 -->
<pre>A bad mover in CDF-LTO3 LTO3_23 for &gt;4 hours. 
An email sent to enstore-admin.</pre>
<!-- Author: Hyunwoo -->
<!-- Comment: Wed May 28 02;44;42 comment by...Hyunwoo -->
<pre>Also a bad tape IAB446 in CDF-LTO3, reported to enstore-admin</pre>


<!-- Date: Thu May 29 02:05:11 2008 -->
<pre>All of CDF-LTO3 movers are marked as "bad" 
and all transfers in LTO3 are "pending".. 
Sent an email to enstore-admin around midnight. 
CDF ACE emailed cdfdh_oper saying that  
"ENCP process is red in ProcMon since about five hours ago" 
aroudn 1AM. 
I just talked with CDF ACE and heard that this problem 
will not affect CDF data taking even for the next 6 hours 
when enstore people come to office around 8AM. 
I.e. I am not sure whether to call SAM oncall shifter now(2AM) 
and decided to wait until morning..</pre>
<!-- Author: Hyunwoo -->


<!-- Date: Thu May 29 03:04:34 2008 -->
<pre>Forgot to log the following that occurred on Wednesday 1PM(Chg) 
"The LTO3 libraries on CDFEN and STKEN are paused due to  
 an LSM failure on the GCC SL8500 robot.   
 A service call has been placed with Sun/STK." by Messer 
And around 4:20 PM Wednesday, 
"SL8500 robot arm has been replaced,and the LTO3 libraries   
 on both CDFEN and STKEN are back in service." 
</pre>
<!-- Author: Hyunwoo -->


<!-- Date: Fri May 30 08:44:59 2008 -->
<pre>A bad mover LTO3_05. 
email to enstore-admin</pre>
<!-- Author: Hyunwoo -->


<!-- Date: Sun Jun 1 01:47:44 2008 -->
<pre>a new bad mover, 9940B16 
and a bad tape, IA4030. 
email to enstore-admin</pre>
<!-- Author: Hyunwoo -->


<!-- Date: Mon Jun 2 15:47:52 2008 -->
<pre>I have removed some very old products from the online machines (b0gateway.fnal.gov, username "sam") to clear up some disk space. The products include ximagetools, imagelibs, imagemagick, xfig, xanim, and sam v7_6_3. If any problems arise that look like they could be from these products being removed, first try to upgrade the external product to sam v8_1_6. If that fails, we can consider re-installing the needed product. 
 
- Angela</pre>
<!-- Author: Angela Bellavance -->


<!-- Date: Mon Jun 2 18:55:16 2008 -->
<pre>A queue built up on r-fcdfdata155-4. I have moved it from ReadOnlyPools to TestPools to drain. 
 
- Angela</pre>
<!-- Author: Angela Bellavance -->