<!-- Title: 08:38:20  Wed Aug 15 2007 -->
<!-- Crewchief: Barry Blumenfeld -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>New shift</pre>


<!-- Date: Wed Aug 15 12:29:33 2007 -->
<pre>On http://cdfsam-prd.fnal.gov/~sam/DHAtAGlance/ there are some errors under upload server for fcdfdata321 and fcdfdata322. 
 
Response: Thanks for the warning.  The upload processes got stuck on these 2 machines in the same time!  It seems the 2 machines are out of memory. 
 
I will contact run2-sys. 
 
Best regards, 
Dehong 
</pre>
<!-- Author: Barry -->


<!-- Date: Wed Aug 15 15:33:43 2007 -->
<pre>From Dehong: It turned out that the DB browser was down.  We need it to get the description for the datasets.  It is ok now --maybe Randy did something</pre>
<!-- Author: Barry -->


<!-- Date: Thu Aug 16 10:03:27 2007 -->
<pre>Inquiry on e-mail message: 
New mail will not be send if the server crashes again  
within the same hour. 
[Accepting connections on port 9005] 
System exception `COMM_FAILURE' 
Reason: no such host is known 
Completed: no 
Minor code: 1330577422 (gethostbyname() failed) 
Killed process: 18543 
 
Response from Angela: 
A helpdesk ticket has already been opened regarding  
this message. You can ignore it for now. Also, note  
the tag "int" in the name of the service ("ns_int").  
This indicates that it is an 'integration' service  
and not a 'production' service and therefore has a  
lower priority than 'production' (which is the  
official users' version). 
</pre>
<!-- Author: Barry -->


<!-- Date: Thu Aug 16 14:47:19 2007 -->
<pre>Oscar Gonzalez Lopez inquiry: 
I am trying to run jobs on files that are on SAM but the release of the  
files is continuosly failing since yesterday (or before). The error is, e.g. 
Sam2DiskCacheInterface_samapi::getNextFile-&gt; The check of file  
dcap://cdfdca1.fnal.gov:25140/pnfs/fnal.gov/usr/cdfen/filesets/MC/MC00/MC0015/MC0015.3/zm030a13.000bewk9  
failed. Skipping it. 
Could you please let me know the reason of the failure or where to check  
it?  
 
Robert Illingworth reponse: 
These errors normally indicate some sort of dcache failure (rather than  
SAM). This particular file seems to be accessible now, however, so I don't  
know why it was complaining earlier. 
 
 
</pre>
<!-- Author: Barry -->


<!-- Date: Thu Aug 16 15:58:38 2007 -->
<pre>Additional response to Oscar from Adam Lyon: 
 Exception: Was not able to execute :dccp dcap://cdfdca3.fnal.gov:25158/pnfs/fnal.gov/usr/cdfen/filesets/MC/MC00/MC0015/MC0015.3/zm030a32.000eewk9 
indicates a dCache problem - for some reason dcap is giving an error 
when SAM is calling it. So this is not a SAM problem per se (though to 
users, any data handling looks like a SAM problem). 
</pre>
<!-- Author: Barry -->


<!-- Date: Thu Aug 16 16:30:11 2007 -->
<pre>Final(?) resolution from Oscar: 
I've found out that the problem seems to be the kerberos ticket. After doing kinit the command you sent me worked fine. I was lucky somebody told me about it. 
I don't understand why if the problem is the kerberos, it is not reported like that. I would recommend to change it. 
 
</pre>
<!-- Author: Barry -->


<!-- Date: Mon Aug 20 08:23:50 2007 -->
<pre>Message sent to enstore-admin: 
   On the CDF data handling monitor page http://cdfsam-prd.fnal.gov/~sam/DHAtAGlance/ 
there are problems with CDF-LT03:  Bad Movers/Bad Tape. 
</pre>
<!-- Author: Barry -->


<!-- Date: Mon Aug 20 08:26:35 2007 -->
<pre>I just received the following e-mail: 
    There are errors on 2 of the 4 LSMs in the SL8500 robot. I have  
paused the CD and CDF LTO3 enstore libs. I will place a service call. 
 
 
  George Szmuksta 
  ssa 
  
  
  
</pre>
<!-- Author: Barry -->


<!-- Date: Mon Aug 20 10:39:44 2007 -->
<pre>Another issue: 
CDF DH plots showed large backlog in file restores this morning. It 
turned out that a very large number (hundreds) restores has been 
scheduled to the pools on one node (fcdfdata316). The pools have 
exhausted the file descriptors and all restores were waiting. I've 
restarted the pools on this node, the restores in the backlog have been 
rescheduled to the other nodes and now things seem to go back to normal. 
 
Vladimir. 
</pre>
<!-- Author: Barry -->


<!-- Date: Mon Aug 20 12:26:00 2007 -->
<pre>ns_int nameservice died at fcdfora1: status 0   message 
 
Hello all, 
 
This message is just FYI. 
 
The message included below had been broadcasting  
hourly since around August 10. Since I was finally  
given login access to fcdfora1 this morning I've  
been able to fix it. The problem is that when we  
migrated the SAM services to the new machines and  
decommissioned the old ones, the int nameserver got  
lost. The solution was to stop the ns_int  
nameserver, delete the "log" file that held all of  
the encoded SAM machine references, and then restart  
the nameserver. I then had to go and restart all of  
the int DbServers so that they weren't lost. 
 
Hopefully everything will be running again smoothly  
within an hour once all the stations re-register  
themselves. 
 
Regards, 
Angela 
 
  
</pre>
<!-- Author: Barry -->


<!-- Date: Mon Aug 20 12:33:25 2007 -->
<pre>   The SL8500 is back up and running. One of the arms had to be replaced. 
 
 
   George Szmuksta 
   ssa 
  
  
  
</pre>
<!-- Author: Barry -->


<!-- Date: Mon Aug 20 16:28:48 2007 -->
<pre>Message sent to enstore-admin: 
    On the CDF data handling monitor page http://cdfsam-prd.fnal.gov/~sam/DHAtAGlance/ 
there are problems with CDF-LT03:  Bad Movers/Bad Tape. 
The bad mover is LT03_29. 
</pre>
<!-- Author: Barry -->