<!-- Title: 02:29:18  Tue Dec  4 2007 -->
<!-- Crewchief: Valeria Bartsch  -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>SAM shifter</pre>


<!-- Date: Wed Dec 5 02:37:12 2007 -->
<pre>downtime from the 4.th of December is over. </pre>
<!-- Author: Valeria -->


<!-- Date: Wed Dec 5 02:38:32 2007 -->
<pre>Raw data logging:  
 
CSL To Enstore Status / Rate:	    Warning / 0.0 
 
No action required 
 
================================ 
Wed Dec  5 02:30:45 2007 1196843445 Warning: 
b0csl23 -- Csl_fetch encountered problems 
b0csl24 -- Files remain in .open for too long 
b0csl25 -- Files remain in .open for too long 
b0csl26 -- Files remain in .open for too long 
b0csl27 -- Files remain in .open for too long 
b0csl28 -- Files remain in .open for too long 
b0csl29 -- Files remain in .open for too long 
Mon Dec  3 19:40:52 2007 1196732452 Action: Experts have been notified, please wait 
Mon Dec  3 23:45:46 2007 1196747146 Action: Experts have been notified, please wait 
Tue Dec  4 03:50:45 2007 1196761845 Action: Experts have been notified, please wait 
Tue Dec  4 07:50:51 2007 1196776251 Action: Experts have been notified, please wait 
Tue Dec  4 11:55:46 2007 1196790946 Action: Experts have been notified, please wait 
Tue Dec  4 16:00:46 2007 1196805646 Action: Experts have been notified, please wait 
Tue Dec  4 20:00:51 2007 1196820051 Action: Experts have been notified, please wait 
Wed Dec  5 00:01:05 2007 1196834465 Action: Experts have been notified, please wait</pre>
<!-- Author: Valeria -->


<!-- Date: Wed Dec 5 02:48:04 2007 -->
<pre>DCACHE: 
 
No. of cells Inactive / OFFLINE:	    8 / 4 
 
============== 
r-fcdfdata139-1 
r-fcdfdata139-2 
r-fcdfdata139-3 
r-fcdfdata139-4 
r-fcdfdata320-1 
r-fcdfdata320-2 
r-fcdfdata320-3 
r-fcdfdata320-4 
============== 
r-fcdfdata124 also in trouble 
 
Problems with fcdfdata139 known and are going to be repaired.  
fcdfdata124 and fcdfdata320 are pingable. email to dcache-admin 
sent. </pre>
<!-- Author: Valeria -->


<!-- Date: Wed Dec 5 08:29:40 2007 -->
<pre>On the DCACHE problem Krzysztof wrote: ignore fcdfdata124,139,320 
124 is being decommissioned, 320 also has a known problem.</pre>
<!-- Author: Valeria -->


<!-- Date: Thu Dec 6 01:50:32 2007 -->
<pre>Large increase in total files to restore to nearly 500. No action required.</pre>
<!-- Author: Valeria -->


<!-- Date: Thu Dec 6 01:52:53 2007 -->
<pre>ENSTORE: 
 
CDF-9940B bad mover 9940B27 
CDF-LTO3   bad mover LTO3_29, LTO3_31 
</pre>
<!-- Author: Valeria -->


<!-- Date: Thu Dec 6 01:54:29 2007 -->
<pre>Krzysztof made a comment that the memory on the dCache doors increases</pre>
<!-- Author: Valeria -->


<!-- Date: Thu Dec 6 04:30:58 2007 -->
<pre>Number of files to restore goes down again.</pre>
<!-- Author: Valeria -->


<!-- Date: Thu Dec 6 11:27:30 2007 -->
<pre>Alex wrote: 
 
kerberized dcap door logs are full of messages when job with wrong  
credentials (=expired cred.) tries to access door. 
Please grep dcache logs to get host IPs, there are few of them. 
 
There records are like follows : 
 
[enstore@fcdfdcache5 dcache-log]$ fgrep "Tunnel verification failed"  
~enstore/dcache-log/dcapK02-fcdfdcache5Domain.log 
[enstore@fcdfdcache5 dcache-log]$ host 131.225.240.20 
20.240.225.131.in-addr.arpa domain name pointer fcdflnx8.fnal.gov. 
 
</pre>
<!-- Author: Valeria -->


<!-- Date: Fri Dec 7 03:44:28 2007 -->
<pre>In order to enable new sam project summary and recovery features 
related to the new CAF-SAM interface being deployed on CAFs we had made 
sam v8_1_6 current on the central clusters. 
This version, among others, provides more information in the output of 
the sam get project summary command. 
 
( $SAM_DIR/RELEASE_NOTES provide more details) 
 
Krzysztof (for DH) 
</pre>
<!-- Author: Valeria -->


<!-- Date: Fri Dec 7 11:08:48 2007 -->
<pre>Pool node fcdfdata138 has become unpingable.  
A Helpdesk ticket has been opened. 
- Angela</pre>
<!-- Author: Angela Bellavance -->
<!-- Comment: Fri Dec  7 13;57;28 comment by...Angela Bellavance -->
<pre>Node has been rebooted.
- Angela</pre>


<!-- Date: Fri Dec 7 21:05:28 2007 -->
<pre>Pool node fcdfdata138 crashed again at 20:42. A Helpdesk ticket has been made, but they are currently in off-hours mode. The pools have been moved into the TestPools group. 
- Angela</pre>
<!-- Author: Angela Bellavance -->
<!-- Comment: Mon Dec 10 10;17;08 comment by...Angela Bellavance -->
<pre>Node has been power-cycled.</pre>
<!-- Date: Fri Dec 7 21:21:52 2007 -->
<pre>Two of the RawPools are getting hammered while the other two are practically idle, so I moved them to TestPools to get the files distributed off to the idle pools. 
- Angela</pre>
<!-- Author: Angela Bellavance -->


<!-- Date: Fri Dec 7 22:02:30 2007 -->
<pre>The queue on r-fcdfdata325-1 has drained some, so I'm moving it back to the RawPools group.</pre>
<!-- Author: Angela Bellavance -->
<!-- Comment: Fri Dec  7 23;04;32 comment by...Angela Bellavance -->
<pre>I've moved the other pool (r-fcdfdata323-1) back to the RawPools group.</pre>


<!-- Date: Mon Dec 10 03:22:21 2007 -->
<pre>Enstore has a new bad mover LTO3-22 and a bad file  IAB352</pre>
<!-- Author: Valeria -->