<!-- Title: 06:43:14  Tue Jul 17 2007 -->
<!-- Crewchief: Valeria Bartsch  -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>SAM shift</pre>


<!-- Date: Tue Jul 17 07:16:08 2007 -->
<pre>50 sam upload errors at fcdfdata321 and 8 errors at fcdfdata322 =&gt; email sent 
 
one bad mover at CDF-LTO3, called LTO3_02 =&gt; email sent 
 
6 dCache cells offline: 
r-fcdfdata124-2	r-fcdfdata124-2Domain	OFFLINE 
r-fcdfdata124-3	r-fcdfdata124-3Domain	OFFLINE 
r-fcdfdata124-4	r-fcdfdata124-4Domain	OFFLINE 
r-fcdfdata124-5	r-fcdfdata124-5Domain	OFFLINE 
r-fcdfdata124-6	r-fcdfdata124-6Domain	OFFLINE 
w-fcdfdata124-7	w-fcdfdata124-7Domain	OFFLINE =&gt; email sent 
 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Tue Jul 17 13:13:39 2007 -->
<pre>CSL To Enstore Status / Rate:	    DEAD / 0.0 
Dehong also saw the problem. The problem was fixed very quickly by the CSL team.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Tue Jul 17 14:40:13 2007 -->
<pre>dbs_caf_int_dbs died =&gt; Angela is working on this 
LTO3_01, LT03_5 and LTO3_28 were stuck in status "busy writing" =&gt; now back working</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Wed Jul 18 06:05:33 2007 -->
<pre>The CDF-LTO3 library manager for Enstore is down at 11pm. However it seems that the problem: 
http://www-cdfen.fnal.gov/enstore/CDF-LTO3.library_manager.html 
 
</pre>
<!-- Author: Valeria -->


<!-- Date: Wed Jul 18 06:08:58 2007 -->
<pre>opt_int2 optimizer died at cdfsam18 =&gt; Angela is working on this 
 
LTO3 library manager up one hour after it was done </pre>
<!-- Author: Valeria -->


<!-- Date: Wed Jul 18 06:21:51 2007 -->
<pre>Ray asked about the staging of gtj1bj and gjt2bj which are going very slowly =&gt; under investigation</pre>
<!-- Author: Valeria -->


<!-- Date: Wed Jul 18 06:43:23 2007 -->
<pre>expired GridKa-CA certificate in CDF dcache =&gt; has been solved, reported by Thomas</pre>
<!-- Author: Valeria -->


<!-- Date: Wed Jul 18 09:39:07 2007 -->
<pre>Ray's problem seems to be caused because of the enstore backlog. So problem seems to be solved unless we hear otherwise</pre>
<!-- Author: Valeria -->


<!-- Date: Wed Jul 18 09:44:04 2007 -->
<pre>The seems to be one door not responding for dCache: 
DOOR11   dcap://cdfdca2.fnal.gov:25146 =&gt; email sent to dcache-admin</pre>
<!-- Author: Valeria -->


<!-- Date: Wed Jul 18 13:49:31 2007 -->
<pre>door not responding for dCache: DOOR11 dcap://cdfdca2.fnal.gov:25146 
door was restarted, analysis of what exactly happened ongoing.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Wed Jul 18 13:50:21 2007 -->
<pre>2 bad movers at CDF-9940B =&gt; email sent to enstore-admin</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Thu Jul 19 05:25:44 2007 -->
<pre>dCache door responding again</pre>
<!-- Author: Valeria -->


<!-- Date: Thu Jul 19 08:38:57 2007 -->
<pre>CDF-9940B has one bad tape IAA133: 
 
Thu Jul 19 08:30:08 CDT 2007 
 
IA2312            200.00GB   (NOTALLOWED 0810-1727 none              )   TST-9940B        cdf.none.none                       
IA2934             63.76GB   (NOTALLOWED 0124-1051 readonly 1112-1445)   CDF-9940B        cdf.mjkim.cpio_odc                   File 9 is bad, need to copy 
IA5057            200.00GB   (NOTALLOWED 0110-1316 none     1227-1645)   CDF-9940B        cdf.jphysr.cpio_odc                  Broken Leader on a blank tape 
IAA133              1.46GB   (NOACCESS   0718-1032 full     0522-1438)   CDF-9940B        cdf.cdfhstn.cpio_odc                
IAB837            400.00GB   (NOTALLOWED 0314-1110 none              )   CDF-LTO3         cdf.none.none                       
IAC226            189.05GB   (NOTALLOWED 0622-1405 none              )   CDF-9940B        cdf.cphysr.cpio_odc                  To be replaced 06/22/07 
 
Thu Jul 19 08:30:08 CDT 2007 
 
=&gt; email sent</pre>
<!-- Author: Valeria -->


<!-- Date: Fri Jul 20 04:46:33 2007 -->
<pre>14 dCache cells offline: 
r-fcdfdata124-2	r-fcdfdata124-2Domain	OFFLINE 
r-fcdfdata124-3	r-fcdfdata124-3Domain	OFFLINE 
r-fcdfdata124-4	r-fcdfdata124-4Domain	OFFLINE 
r-fcdfdata124-5	r-fcdfdata124-5Domain	OFFLINE 
r-fcdfdata124-6	r-fcdfdata124-6Domain	OFFLINE 
r-fcdfdata149-1	r-fcdfdata149-1Domain	OFFLINE 
r-fcdfdata149-2	r-fcdfdata149-2Domain	OFFLINE 
r-fcdfdata149-3	r-fcdfdata149-3Domain	OFFLINE 
r-fcdfdata149-4	r-fcdfdata149-4Domain	OFFLINE 
r-fcdfdata149-5	r-fcdfdata149-5Domain	OFFLINE 
r-fcdfdata149-6	r-fcdfdata149-6Domain	OFFLINE 
r-fcdfdata149-7	r-fcdfdata149-7Domain	OFFLINE 
r-fcdfdata149-8	r-fcdfdata149-8Domain	OFFLINE 
w-fcdfdata124-7	w-fcdfdata124-7Domain	OFFLINE 
=&gt; on fcdfdata124 and fcdfdata149 (I thought 149 is back again?) 
=&gt; email sent to helpdesk and run2-sys</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Fri Jul 20 09:25:30 2007 -->
<pre>CDF-9940B has 2 bad movers: 9940B20, 9940B25</pre>
<!-- Author: Valeria -->


<!-- Date: Fri Jul 20 09:43:48 2007 -->
<pre>14 dCache cells offline 
fcdfdata124,fcdfdata135,fcdfdata148 are being reconfigured, fcdfdata149 is not pingable</pre>
<!-- Author: Valeria -->


<!-- Date: Fri Jul 20 09:56:44 2007 -->
<pre>The tape IAA133 at CDF-9940B has been cleared and it available for use</pre>
<!-- Author: Valeria -->


<!-- Date: Fri Jul 20 11:28:26 2007 -->
<pre>fcdfdata149 apparently online again, reduction in inactive dCache cells due to this.</pre>
<!-- Author: Valeria -->


<!-- Date: Fri Jul 20 11:29:04 2007 -->
<pre>CDF-9940B movers o.k. again.</pre>
<!-- Author: Valeria -->


<!-- Date: Sat Jul 21 06:31:49 2007 -->
<pre>Two LTO3 movers that were in the public enstore configuration but not 
online have been moved to the CDF enstore configuration instead. They 
are now online, so CDF currently has 11 LTO3 movers (one is still dead, 
waiting for replacement by manufacturer). 
 
The web pages may look a little confusing, because these two nodes are 
still called stkenmvr118a and stkenmvr119a, but they are in fact cdfen 
nodes, corresponding to movers LTO3_18 and LTO3_19. 
 
David also modified the configuration to make the priorities and discipline 
identical for 9940B and LTO3 libraries. This is something Dehong and David 
had discussed. 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Sat Jul 21 06:35:32 2007 -->
<pre>Ricardo has seen a problem with the diskpool, although I have not seen any problem occuring on the monitoring lately: 
 
Server error message for [1]: "internalError : User eusebi/cdf/glidecaf@FNAL.GOV 
 is not authorized" (errno 2). 
 Failed open file in the dCache. 
 SysError in &lt;TDCacheFile::TDCacheFile&gt;: file 
 dcap://fcdfrdc3.fnal.gov:22125/pnfs/diskpool/top/varganov/topLJAcc/bhel0d/TopTree/bhel0d_lepton_outpu 
 t_cafjob3315717_section001.root can not be opened for reading ("internalError : 
 User eusebi/cdf/glidecaf@FNAL.GOV is not authorized") 
 Ran over 0</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Sat Jul 21 06:38:15 2007 -->
<pre>It looks like the number of requests to CDF-LTO3.library_manager has 
 gone quite high over the last few (~2.5+) hours and only a few tape  
 drives are being used (mainly for writing). 
 http://cdfsam-prd.fnal.gov/~sam/DHAtAGlance/enstore_transfers.html 
 
Enstore-admin set SG limit to higher than number of  
drives. This should help to increase performance of pending requests  
processing. This also could be a reason of the problem reported by you.  
Enstore-admin saw this pattern for the first time.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Sat Jul 21 06:41:52 2007 -->
<pre>The diskpool problem was the issue of authorizing a user. Many of our collaborators are in the system but some want to run on various caf systems other than default CAF such as JPCAF FermiGRID etc. Alexei has added the requests yesterday so there should not be any more issues. 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Sat Jul 21 14:49:26 2007 -->
<pre>The database response time is quite slow. However this seems to be because it is very busy. It recovers itself every few minutes.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Sat Jul 21 14:50:18 2007 -->
<pre>Alex has increased max number of logins on all 30 kerberized dcap doors from 400 to 4000.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Sat Jul 21 16:08:43 2007 -->
<pre>Problem is that I do not see any of these problems in the Data Handling at a Glance.  I am seeing one mover off out of 10, but that was the status the whole week. Only that I see one more bad tape. Either I have not seen that by mistake or this was not caught by the monitoring 
 
Report from Ken: 
 
At about 11:15 today, the CDF-LTO3 library tried to mount tape IAB973.  
Each mover that tried to mount the tape and suffered problems, going  
offline.  The Enstore system generated a helpdesk ticket (AR #101317)  
paging me with a 'Ball is Red' alarm due to six LTO3 movers being  
unavailable.  All but one of the movers ended up going offline.  The common  
alarm said "Supposedly a serious problem with tape drive positioning the  
tape: ftt.FTTError FTT_SUCCESS."  I marked volume IAB973 as "NOT-ALLOWED"  
and warm-started the movers. 
 
As those movers were restarting, I was paged for AR #101319 saying that  
"cdfensrv6 has gone unpingable."  This node hosts the drivestat server and  
the Accounting server.  I came into the office and the node would not POST.  
  I am working to replace the failed node with help from Chih-Hao of the  
Enstore developer group. 
 
We hope to have the CDF Enstore systems back on-line soon. 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Sun Jul 22 06:17:05 2007 -->
<pre>Ken wrote: 
 
It appears that we have the replacement node for cdfensrv6 up and running.  
  Chih-Hao has started the accounting and drivestat servers which normally  
run on that node.  The Enstore SAAG web monitor is showing a Green ball  
indicating that things look good.  There are several CDF data transfers in  
progress now and all movers are on-line. 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Sun Jul 22 12:32:27 2007 -->
<pre>bad mover CDF-9940B called 9940B11</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Mon Jul 23 05:17:24 2007 -->
<pre> 
I am seeing a lot of farm processes pending: 378         (11:46 - 03:39). See: 
http://cdfsam-prd.fnal.gov/~sam/DHAtAGlance/enstore_transfers.html 
 
I am not allowed to login at cdf-fncdfsrv0. Therefore I can not give more 
details. Also the link from the main page to: 
http://fnpcc.fnal.gov/ 
is broken which is meant to lead to the farms webpage. 
 
=&gt; sent email to cdf-farms</pre>
<!-- Author: Valeria -->


<!-- Date: Mon Jul 23 09:41:05 2007 -->
<pre>Krzysztof found that the number of pending read requests is growing since yesterday. I see two tapes in this state: 
 
Pending read of IAC348 from fcdfdata109 by root [VOL_BUSY] 
Pending read of IA5815 from fcdfdata108 by root [VOL_BUSY] 
 
</pre>
<!-- Author: Valeria -->


<!-- Date: Mon Jul 23 09:50:49 2007 -->
<pre>DF-9940B called 9940B11 has been restarted</pre>
<!-- Author: Valeria -->


<!-- Date: Mon Jul 23 11:41:10 2007 -->
<pre>number of pending transfers of farm is decreasing.</pre>
<!-- Author: Valeria -->