<!-- Title: 09:01:44  Tue Jul 22 2008 -->
<!-- Crewchief: Mike Albrow -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Mike Albrow comes on CDF shift</pre>


<!-- Date: Tue Jul 22 09:53:11 2008 -->
<pre>(0) Continuing issues 
 
  (0a) Unresolved to-be-ignored "semi-permanent problems" 
 
  Dcache cells  fcdfdata320-1/2/3/4 inactive 
 
  (0b) long term residual problems left over from previous week 
 
  DCache Number of cells OFFLINE= 8--fcdfdata146/7  
   (since about 2 weeks--supposedly being tested by FEF) 
</pre>
<!-- Author: John Yoh -->


<!-- Date: Tue Jul 22 11:56:16 2008 -->
<pre>I am aware of the Warning from CSL to Enstore (online) about stager  being above quota. I will be working on it when there are lulls in data taking. 
 
- Angela</pre>
<!-- Author: Angela Bellavance -->


<!-- Date: Tue Jul 22 14:08:35 2008 -->
<pre>The "number of files to restore" just shot up by about 1000 
starting 13:40--most likely due to user wketchum (who has 
525 logins). There was a request this morning while discussing the 4200+ files to restore last Tuesday July 15--that we should find out who is doing this, and whether it is legitimate. I have just inform Angela about this and she will investigate. </pre>
<!-- Author: John Yoh -->


<!-- Date: Wed Jul 23 10:36:39 2008 -->
<pre>Email from Ken Schumacher of SSA group (see below)  
---Enstore Tape access will be down Thursday from 07:15 
to perhaps as late as 18:00 
---DCache PNFS mounts will also be down starting Thursday 
07:15 --but, hopefully, will be back up much sooner  
 
Please plan your jobs accordingly.  I will update this logbook when Enstore and DCache/PNFS caome back up 
 
 
---------------------- 
This is an updated reminder about tomorrow's Public Enstore and Public dCache downtime for systems and software upgrades. 
 
There is a downtime scheduled for Thursday, July 24, which will affect 
all of the public Enstore (STKen) systems.  CD/SCF/DMS will be upgrading the Enstore software to an SDE configuration.  A similar upgrade has already been done for Dzero and CDF Enstore systems.  This will be an all day outage. 
 
Enstore will begin "draining" at 06:30 CDT.  At this point further writes to tape will be held until after the upgrade.  Enstore and dCache services will be shut down at 07:15.  These services are scheduled to be back on-line by 18:00. 
 
This outage will also affect PNFS access for all public dCache areas. 
PNFS mounts will be unavailable while the PNFS server node is being 
upgraded. 
 
During this outage, some maintenance will be performed on the SL8500 
tape robot located at GCC. This will make LTO tape drives used by CDF and Dzero unavailable as well.  We will strive to have the SL8500 back on-line as early as we can. 
 
Ken Schumacher 
SSA Group </pre>
<!-- Author: John Yoh -->


<!-- Date: Wed Jul 23 11:55:51 2008 -->
<pre>I had moved the pools on fcdfdata098 to the TestPools because we suspected there was a problem with it, but it might be okay, and considering the queue of restores, I'll move it back into the StagePools group. 
 
- Angela</pre>
<!-- Author: Angela Bellavance\ -->


<!-- Date: Wed Jul 23 15:22:51 2008 -->
<pre>The dCache machine fcdfdata146 has been brought back up with a new motherboard and new memory. It has been put back in to service as a member of the ReadOnlyPools group. 
 
- Angela</pre>
<!-- Author: Angela Bellavance -->


<!-- Date: Thu Jul 24 13:29:59 2008 -->
<pre>Most LT03 tape drives are back online (just 20 and 21 are still offline--being investigated). Apparntly, DCache was not down.</pre>
<!-- Author: John Yoh -->


<!-- Date: Thu Jul 24 13:54:05 2008 -->
<pre>I am back in business after a very busy 2 days; thanks John for "minding the shop" up to now.</pre>
<!-- Author: Mike -->


<!-- Date: Thu Jul 24 14:28:06 2008 -->
<pre>Lots of errors (about 100 reds) reported on Enstore list. 
E.g: 
======================================= 
Noticed the local file changed: [Errno 2] No such file or directory: '/export/data1/read-pool-1/data/0018000000000000000BD530' : (SRC: /pnfs/fs/usr/.(access)(0018000000000000000BD530)) (DST: /export/data1/read-pool-1/data/0018000000000000000BD530) 
==================================== 
John reported to Angela, suggestion we wait as LT03 restarted an hour ago. 
 
Bad tape : IAE482 
Bad movers LT03_20 & _21 still reported.</pre>
<!-- Author: Mike -->


<!-- Date: Thu Jul 24 14:32:10 2008 -->
<pre>Angela says: 
Since those two drives are the ones we are borrowing from STKEN, they will come up when the STKEN work finishes at 18:00. Let's just give the rest more time. 
</pre>
<!-- Author: Mike -->


<!-- Date: Fri Jul 25 09:10:31 2008 -->
<pre>Enstore: Red Letter warnings: Cannot Update Status on CDF9940B 
(B11-B28 bad) and LT03-02,04,20,21,24,27,30,31 bad.</pre>
<!-- Author: Mike -->


<!-- Date: Fri Jul 25 15:18:32 2008 -->
<pre>Frequent alarms as: 
&gt; Fri Jul 25 14:05:22 CDT 2008 
&gt; 
&gt;  From: alarm_server 
&gt; 
&gt;  ['1217012722.35', 1217012722.353673, 'fcdfdata110.fnal.gov', 2812,  
&gt; 'root', 'C (1)', 'ENCP', 'DEVICE ERROR', None, None, None, {'text':  
&gt; {'status': '[ Error 16 ] Device or resource busy: write() system call stuck  
&gt; inside kernel too long', 'outfile':  
&gt; '/export/data3/read-pool-3/data/00020000000000000198D170', 'infile':  
&gt; '/pnfs/fs/usr/.(access)(00020000000000000198D170)'}}] 
========================== 
Angela emails: 
&gt; We've been seeing these alarms frequently since just before 07:00 this morning. Could someone please investigate? 
======================================= 
Enstore admins (?) response: 
This error is originating from encp(s) claiming that the  
/export/data3/read-pool-3/data/ filesystem is hung on fcdfdata110.  I'd recommend that CDF check the disks. 
================================ 
Who can do this? Angela? 
</pre>
<!-- Author: Mike -->


<!-- Date: Fri Jul 25 15:20:38 2008 -->
<pre>Oh, I just now see this from Krzysztof: 
Hi,  FYI: 
I just moved 'fcdfdata110.fnal.gov' to testPools to let it drain. 
I also moved several large nodes from ReadOnlyPools to readPools 
to balance the space/load before the weekend. 
(without changing it in PoolManager.conf) 
Krzysztof 
</pre>
<!-- Author: Mike -->


<!-- Date: Sun Jul 27 08:45:40 2008 -->
<pre>In enstore, CDF-LT03 IAE482 is marked as bad tape (since Friday I think). Details given as: 
===================================== 
Sun Jul 27 08:30:13 CDT 2008 
Brought to You by: noaccess-tapes  
 
IA1853              0.23GB   (NOTALLOWED 0201-0820 full     1227-2115)   CDF-9940B        cdf.xbmu0i.cpio_odc                  sent to STK in drive 9940B22 - Feb 4, 2008 
IA2312            200.00GB   (NOTALLOWED 0810-1727 none              )   TST-9940B        cdf.none.none                        
IA2934             63.76GB   (NOTALLOWED 0124-1051 readonly 1112-1445)   CDF-9940B        cdf.mjkim.cpio_odc                   File 9 is bad, need to copy 
IA5057            200.00GB   (NOTALLOWED 0110-1316 none     1227-1645)   CDF-9940B        cdf.none.none                        Broken Leader on a blank tape 
IAB837            400.00GB   (NOTALLOWED 0314-1110 none              )   CDF-LTO3         cdf.none.none                        
IAB973            400.00GB   (NOTALLOWED 0721-1151 none              )   CDF-LTO3         cdf.none.none                        9 consecutive mount failures 
IAC226            200.00GB   (NOTALLOWED 0622-1405 none              )   CDF-9940B        cdf.none.none                        To be replaced 06/22/07 
IAE482              1.02GB   (NOACCESS   0724-1211 full     0319-2322)   CDF-LTO3         cdf.cdfpstn.cpio_odc                 
 
Sun Jul 27 08:30:13 CDT 2008 
============================================== 
 
</pre>
<!-- Author: Mike -->


<!-- Date: Mon Jul 28 08:42:22 2008 -->
<pre>(Sorry Angela, connected late) 
1 CafStation is closed since ~ 4 a.m. 
</pre>
<!-- Author: Mike -->
<!-- Comment: Mon Jul 28 09;13;10 comment by...Angela Bellavance -->
<pre>The station log files indicate it is running fine. I'll check the web server connections.

- Angela</pre>

<!-- Comment: Mon Jul 28 09;32;58 comment by...Angela Bellavance -->
<pre>I didn't see anything in the log files, but I restarted the station just in case. A few minutes later, the plot started showing data again while I was looking at the cdfsamweb log files.

&lt;shrug&gt;

- Angela</pre>


<!-- Date: Mon Jul 28 19:59:38 2008 -->
<pre>CDF-LTO3 is now fine, no bad tapes reported. 
DCache: Number of files to restore has climbed to ~ 700, requested not queued.</pre>
<!-- Author: Mike -->