<!-- Title: 08:13:33  Tue Oct 23 2007 -->
<!-- Crewchief: Barry Blumenfeld -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>New Shift</pre>


<!-- Date: Tue Oct 23 09:45:36 2007 -->
<pre>Bad Mover CDF-LTO3   
Bad Tape IAB070 
 
IAB070              0.83GB   (NOACCESS   1023-0914 full     0212-1600)   CDF-LTO3         cdf.cdfptnt.cpio_odc                 
 
 
</pre>
<!-- Author: Barry -->


<!-- Date: Tue Oct 23 12:14:44 2007 -->
<pre>fcdfdata318 is offline but pingable.</pre>
<!-- Author: Barry -->


<!-- Date: Tue Oct 23 12:20:19 2007 -->
<pre>Another bad mover/bad tape:  LTO3_01 
 
IAD373              0.79GB   (NOACCESS   1023-1044 full     1019-0757)   CDF-LTO3         cdf.cdfpewk.cpio_odc                 
 
</pre>
<!-- Author: Barry -->


<!-- Date: Tue Oct 23 13:57:13 2007 -->
<pre>Resonse from David Berg about the bad movers: 
 
These two movers were in error state after mounts failed during cap 
activity in the SL8500 at GCC. The two noaccess volumes were home. I 
cleared the volumes and restarted the movers. 
</pre>
<!-- Author: Barry -->


<!-- Date: Tue Oct 23 14:00:07 2007 -->
<pre>Response about fcdfdata318: 
 
Solution: ettab@fnal.gov sent this solution:  
fcdfdata318 has been rebooted.  Etta B 
 
</pre>
<!-- Author: Barry -->


<!-- Date: Tue Oct 23 14:17:39 2007 -->
<pre>  Despite the reboot, fcdfdata318.fnal.gov does not respond to a ping and remains offline. 
</pre>
<!-- Author: Barry -->


<!-- Date: Tue Oct 23 15:05:39 2007 -->
<pre>Second response on fcdfdata318: 
 
fcdfdata318 should be fine now. 
A machine having it's operating system re-installed was 
grabbing the ip address of fcdfdata318 - that has been corrected. 
Etta B 
 
However, DCache still reports it  as offline: 
 
r-fcdfdata318-1 r-fcdfdata318-1Domain OFFLINE  
r-fcdfdata318-2 r-fcdfdata318-2Domain OFFLINE  
r-fcdfdata318-3 r-fcdfdata318-3Domain OFFLINE  
r-fcdfdata318-4 r-fcdfdata318-4Domain OFFLINE  
</pre>
<!-- Author: Barry -->


<!-- Date: Tue Oct 23 15:45:21 2007 -->
<pre>Bad mover offline: 
 
9940B19.mover alive : OFFLINE cdfenmvr19a </pre>
<!-- Author: Barry -->


<!-- Date: Tue Oct 23 16:53:10 2007 -->
<pre>Problem using dccp: 
 
I have a question.  I am trying to pull some files over using "sam get dataset --fileList". 
 
 
When I do this, the dccp command sits there and hangs for ~hours, but no file gets copied over.  I can log in to fcdflnx5, and look at /pnfs/ where the file is being copied from, and I see that it is sitting there ready to be copied.  So I'd think that the file is already staged to disk.    I also tried the dccp command stand-alone as it is written below, and get the same result.  The debug option on dccp doesn't tell me anything either. 
 
 
 
 
Why is it that the dccp copy is hanging ? 
 
 
~Ben 
---------------------- 
 
 
More detail : 
 
 
# command I tried : 
sam get dataset --fileList="bc02efb8.0001hgs1" --group=test --downloadPlugin=dccp 
 
 
# resulting output : 
RetryHandler.establishConsumer(...)&gt; initial retriable exception ProjectNotFound(' Project 'bjk_20071023160255' on station 'cdf-caf' not responding.') 
RetryHandler.establishConsumer(...)&gt; will retry in 8.47 seconds 
RetryHandler.establishConsumer(...)&gt; retry number 1 succeeded, returning to caller . 
dccp dcap://cdfdca3.fnal.gov:25161/pnfs/fnal.gov/usr/cdfen/filesets/NS/NS00/NS0064 /NS0064.7/bc02efb8.0001hgs1 . 
 
 
(hangs like this without copying any output for 
 
 
# checking disk : 
fcdflnx5 Tue 16:03 &lt;NS0064.7&gt; pwd 
/pnfs/cdfen/filesets/NS/NS00/NS0064/NS0064.7 
fcdflnx5 Tue 16:04 &lt;NS0064.7&gt; ls -ltr bc02efb8.0001hgs1 
-rw-r--r--    1 sam      sam      1323224876 Apr  5  2007 bc02efb8.0001hgs1 
 
Response from Krzysztof: 
 
this file is not cached yet: 
dc_check (after setup dcap) in place of dccp will say that. 
 
project not responding if seen only once and then 
retry number 1 succeeded is ok. 
 
Someone initiated ~1.7 staging requests: 
http://cdfsam-prd.fnal.gov/~sam/DHAtAGlance/requests_2_dcache_pool_groups.html 
which may be the reason for the long delay. 
 
 
 
 
</pre>
<!-- Author: Barry -->


<!-- Date: Tue Oct 23 20:20:31 2007 -->
<pre>Follow up on fcdfdata318 at 14:55 CDT: 
 
dcache did not start, I'm restarting dcache on this node. Alex.</pre>
<!-- Author: Barry -->


<!-- Date: Wed Oct 24 08:43:36 2007 -->
<pre>File access problem from Kohei Yorita: 
 
I have been trying to access etlpmh/etlpmi dataset on SAM for a 
week or so, and having some problems - A bunch of files are not 
accessible. 
For instance, a file 
/pnfs/cdfen/filesets/NT/NT00/NT0023/NT0023.3/eh02fbf6.0258tlpm 
seems like exist and cached, but can not be readable/accessible. 
</pre>
<!-- Author: Barry -->


<!-- Date: Wed Oct 24 12:22:17 2007 -->
<pre>The above problem seems to have cleared up by itself.</pre>
<!-- Author: Barry -->


<!-- Date: Wed Oct 24 12:27:21 2007 -->
<pre>Bad movers: 
 
LTO3_26.mover alive : OFFLINE cdfenmvr126a 2007-Oct-24 12:24:43    
  
LTO3_27.mover alive : OFFLINE cdfenmvr127a 2007-Oct-24 12:24:43    
  
LTO3_28.mover dead  
</pre>
<!-- Author: Barry -->


<!-- Date: Wed Oct 24 14:00:22 2007 -->
<pre>Enstore Ball is Red</pre>
<!-- Author: Barry -->


<!-- Date: Wed Oct 24 14:32:25 2007 -->
<pre>Added  
 
 /DC=org/DC=doegrids/OU=Services/CN=sam/cdfsam1.cr.cnaf.infn.it  
 
to sam_gridftp grid-mapfile  
</pre>
<!-- Author: Barry -->


<!-- Date: Wed Oct 24 14:45:18 2007 -->
<pre>Enstore Ball is no longer red. 
However, LTO3_23,24,25,26,27 are offline and 28 is dead.</pre>
<!-- Author: Barry -->


<!-- Date: Wed Oct 24 16:26:25 2007 -->
<pre>LTO3_27 and 28 are back in service, 
but LTO3_23,24,25,26 are marked suspect.</pre>
<!-- Author: Barry -->


<!-- Date: Thu Oct 25 08:26:26 2007 -->
<pre>     One cell of cdfdata16 is offline.  The machine is pingable. 
 
r-fcdfdata316-2 r-fcdfdata316-2Domain OFFLINE  
 
Also, there is one bad 9940B tape: IA6978 
 
IA6978            181.47GB   (NOTALLOWED 1024-1728 none     0619-1350)   CDF-9940B        cdf.sgi2test.cpio_odc</pre>
<!-- Author: Barry -->


<!-- Date: Thu Oct 25 09:13:20 2007 -->
<pre>r-fcdfdata316-2 r-fcdfdata316-2Domain is now OK.</pre>
<!-- Author: Barry -->


<!-- Date: Thu Oct 25 09:40:42 2007 -->
<pre>The cell from fcdfdata316 is offline again. 
 
r-fcdfdata316-2 r-fcdfdata316-2Domain OFFLINE </pre>
<!-- Author: Barry -->


<!-- Date: Thu Oct 25 09:51:00 2007 -->
<pre>r-fcdfdata316-2 is back again.</pre>
<!-- Author: Barry -->


<!-- Date: Thu Oct 25 15:30:35 2007 -->
<pre>LTO3_19.mover shows a red dot 
 
alive : ERROR - ('1', "mount IAB633 failed: ('ERROR', 13, ['ACSSA&gt; mount IAB633 0,2,1,3', '2007-10-25 15:15:39 16615 Volume VOF309 reactivated.</pre>
<!-- Author: Barry -->


<!-- Date: Thu Oct 25 15:34:13 2007 -->
<pre>A user had the following problem: 
 
  Error on control line [19] 
  Failed to create a control line 
  Failed open file in the dCache. 
  Error on control line [19] 
  Failed to create a control line 
  Error in &lt;TDCacheFile::TDCacheFile&gt;: file dcap://cdfdca3.fnal.gov:25164/pnfs/fnal.gov/usr//cdfen/filesets/NT/NT00/NT0015/NT0015.1/zm0366a6.0003ewkm does not exist 
 
 
Resolution of the problem found it was due to incorrect forwarding of the Kerberos ticket.</pre>
<!-- Author: Barry -->


<!-- Date: Thu Oct 25 15:43:50 2007 -->
<pre>LTO3_19.mover is OK now, but IAB633 is bad 
 
IAB633              1.31GB   (NOACCESS   1025-1520 full     0610-1704)   CDF-LTO3         cdf.cdfpstn.cpio_odc </pre>
<!-- Author: Barry -->


<!-- Date: Fri Oct 26 07:42:40 2007 -->
<pre>fcdfdata099.fnal.gov is not pingable.</pre>
<!-- Author: Barry -->


<!-- Date: Fri Oct 26 10:26:38 2007 -->
<pre>Resolution of problem with fcdfdata099 from Ling: 
 
Machine brought back up. Two of the three power supplies have failed.</pre>
<!-- Author: Barry -->


<!-- Date: Sun Oct 28 08:35:53 2007 -->
<pre>Bad Mover/Bad Tape 
IAB434              0.86GB   (NOACCESS   1028-0809 full     0520-0815)   CDF-LTO3         cdf.gjt10j.cpio_odc       
 
LTO3_19.mover alive : ERROR - ('1', "mount IAB434 failed: ('ERROR', 13, ['ACSSA&gt; mount IAB434 0,2,1,3', 'Mount: Mount failed, Unreadable label.', 'ACSSA&gt; logoff', ''], '', 'MOUNT 13: mount IAB434 0,2,1,3 =&gt; 0,Mount: Mount failed, Unreadable label.')")         </pre>
<!-- Author: Barry -->


<!-- Date: Sun Oct 28 11:53:53 2007 -->
<pre>cdf-ekpka on ekpsam.physik.uni-karlsruhe.de:6789 is unreachable.</pre>
<!-- Author: Barry -->


<!-- Date: Sun Oct 28 12:50:34 2007 -->
<pre>cdf-ekpka is back.</pre>
<!-- Author: Barry -->


<!-- Date: Mon Oct 29 13:16:30 2007 -->
<pre>LTO3_19.mover is back.</pre>
<!-- Author: Barry -->


<!-- Date: Mon Oct 29 15:47:32 2007 -->
<pre>Enstore Ball is red</pre>
<!-- Author: Barry -->


<!-- Date: Mon Oct 29 15:49:27 2007 -->
<pre>LTO3_26.mover alive : ERROR - ('MOUNTFAILED', 'max_consecutive_failures (3) reached') cdfenmvr126a 2007-Oct-29 15:47:48 </pre>
<!-- Author: Barry -->


<!-- Date: Mon Oct 29 16:00:18 2007 -->
<pre>Enstore ball is no longer red.</pre>
<!-- Author: Barry -->