<!-- Title: 10:26:02  Tue Apr 17 2007 -->
<!-- Crewchief: Rick St. Denis -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Things running, small issue of raid on fcdfdata119</pre>


<!-- Date: Tue Apr 17 10:26:23 2007 -->
<pre>fcdfdata119 has been rebooted.  The RAID controller monitoring software is giving good 
results.  To be safe, I checked the filesystems and they show up clean.  I think we're good to 
go. 
                                                                                                   
Jason M. Harrington         </pre>
<!-- Author: r -->


<!-- Date: Tue Apr 17 10:29:55 2007 -->
<pre>From Seth Graham: 
 
 
I added accounts as requested last Friday, and haven't seen any mails since reporting problems 
with the configuration of the machine. 
                                                                                                   
So I assume the dCache group is working on it now. 
                                                                                                   
Doug Benjamin wrote: 
&gt; Dear dCache group and run2sys group, 
&gt; 
&gt;  What is the status of adding fcdfdata323-fcdf325 into CDF dCache system 
&gt; in the raw pool group?  I checked this morning and did not see the nodes in the 
&gt; list of pools.  This is help desk ticket 94913 
&gt; 
&gt; Thanks, 
&gt; 
&gt; Doug Benjamin 
</pre>
<!-- Author: r -->


<!-- Date: Tue Apr 17 16:02:23 2007 -->
<pre>This is the week where we had no shifter and e-mail with the shift reminder was sent to the dummy entry which was cdfsam-auto. I'll cover the shift as indicated in my earlier e-mail. Krzysztof</pre>
<!-- Author: K. Genser -->


<!-- Date: Tue Apr 17 16:09:03 2007 -->
<pre>problem with fcdfdata136. I had temporarily moved it from the dCache readPools to the TestPools group so that it does not get any more of the files staged to it beyond those requested already to be staged. Jason (FEF) is working on the file systems</pre>
<!-- Author: K. Genser -->


<!-- Date: Wed Apr 18 07:30:52 2007 -->
<pre>One mover in a bad state 9940B18 with no reported tape in a bad state.</pre>
<!-- Author: K .Genser -->


<!-- Date: Wed Apr 18 19:41:49 2007 -->
<pre>CDFEN servers, like volume clerk and 9940B library manager were shown as dead @ ~18:50. e-mail sent to enstore-admin. Seem ok now.</pre>
<!-- Author: K. Genser -->


<!-- Date: Thu Apr 19 10:24:42 2007 -->
<pre>CDF-LTO3 library paused due to a downtime ~07:00-12:00; dCache plots indicate no much demend for files from LTO3 as most requests are still for files on 9940B tapes; fcdfdata136 still being worked on, a hardware call open on it;</pre>
<!-- Author: K. Genser -->


<!-- Date: Fri Apr 20 06:56:45 2007 -->
<pre>CDF-LTO3 library was set online at about ~13:00 yesterday; fcdfdata136 still down; tapes IA6669 (xbhd0h) and IAA934 (cphysr) in NOACCESS state; movers: 9940B24 9940B26 down; IA6669 in 9940B24?</pre>
<!-- Author: K. Genser -->


<!-- Date: Sat Apr 21 08:19:10 2007 -->
<pre>fcdfdata136 still down for repairs; no movers in bad state; no tapes in NOACCESS state</pre>
<!-- Author: K. Genser -->


<!-- Date: Sun Apr 22 08:36:28 2007 -->
<pre>fcdfdata136 down; Alex K. sent summary on corrupted files in dCache , 15 files total. 8 files were read from the same tape IA2744. All file restores started within 50 min interval, Apr 17 15:20 to 16:07. Other info from SSA: We encountered problems this (that? klg) afternoon with both of the Sun machines running acsls. We are not positive that they are a direct cause and effect. All in Helpdesk ticket 96005. 
</pre>
<!-- Author: K. Genser -->


<!-- Date: Mon Apr 23 08:06:25 2007 -->
<pre>fcdfdata136 down; Alex K. removed corrupted files form dCache, more files were reported by the crc checking scripts this morning, to be looked at;</pre>
<!-- Author: K. Genser -->
<!-- Comment: Mon Apr 23 17;13;02 comment by...K. Genser -->
<pre>Alex has some evidence the corrupted files were a fallout from the fcdfdata136 failure and pool to pool copies</pre>
<!-- Date: Mon Apr 23 17:11:10 2007 -->
<pre>problems with monitoring of fcdfdata321/322 due to a recent change on those nodes</pre>
<!-- Author: K. Genser -->