<!-- Title: 04:11:30  Tue Mar 13 2007 -->
<!-- Crewchief: G.Compostella -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Start of new shift</pre>


<!-- Date: Tue Mar 13 11:47:45 2007 -->
<pre>No particular problems today. 
 
As Doug previously reported 
plots at 
http://cdfsam-prd.fnal.gov/~sam/DHAtAGlance/upload_rates.html 
http://cdfsam-prd.fnal.gov/~sam/DHAtAGlance/farm_rates.html 
http://cdfsam-prd.fnal.gov/~sam/DHAtAGlance/data_rates.html 
 
have not been updating since the early Sunday morning, and they're still not updating. 
</pre>
<!-- Author: G. Compostella -->


<!-- Date: Wed Mar 14 12:05:32 2007 -->
<pre>No particular problems today. 
 
Tried to give Erik Brubaker some help to make sam get dataset command working, but it didn't work. 
Problem is: 
trying to use sam get dataset on a machine outside fnal without sam_cp installed but with dccp only.</pre>
<!-- Author: G. Compostella -->


<!-- Date: Thu Mar 15 05:32:34 2007 -->
<pre>Mailed cdf_mc_ops about errors in upload to sam. 
Mailed enstore-admin about problems in tape access: 
++++++++++++++ 
Thu Mar 15 05:15:10 CDT 2007 
 
IA2312            200.00GB   (NOTALLOWED 0810-1727 none              )   TST-9940B        cdf.none.none                        
IA2934             63.76GB   (NOTALLOWED 0124-1051 readonly 1112-1445)   CDF-9940B        cdf.mjkim.cpio_odc                   File 9 is bad, need to copy 
IA5057            200.00GB   (NOTALLOWED 0110-1316 none     1227-1645)   CDF-9940B        cdf.jphysr.cpio_odc                  Broken Leader on a blank tape 
IAB837            400.00GB   (NOTALLOWED 0314-1110 none              )   CDF-LTO3         cdf.none.none                        
 
Thu Mar 15 05:15:10 CDT 2007 
++++++++++++++</pre>
<!-- Author: G. Compostella -->


<!-- Date: Thu Mar 15 08:58:27 2007 -->
<pre>Currently systems are down: 
Today is the "big downtime" day starting with enstore work at 7am and the db work at 8am. 
All services are expected to be fully recovered by the end of the working day (Krzysztof will drop an email about this).</pre>
<!-- Author: G. Compostella -->


<!-- Date: Thu Mar 15 11:55:59 2007 -->
<pre>Database back online: 
 
The O/S and oracle patch installations are complete.   
The database is up and available for use. 
 Diana 
 
Diana Bonham wrote: 
&gt; we will be installing Solaris O/S patches on fcdfora4, and  
&gt; oracle security patches on cdfofpr2 database this Thursday,  
&gt; March 15th, 8am-12noon.   The host and database will be  
&gt; unavailable during this time period. 
&gt; 
&gt; Diana </pre>
<!-- Author: G. Compostella -->


<!-- Date: Fri Mar 16 03:06:11 2007 -->
<pre>From Krzysztof: status of todays (March 15th) SAM part of the downtime 
============================= 
An update: 
 
Based on small scale tests and monitoring, the CDF DH related  
systems: 
enstore, production dCache, offline data base and SAM are  
operational. 
 
Krzysztof 
 
Krzysztof Genser wrote: 
&gt; Here is the current status of the SAM related work: 
&gt; 
&gt; .) deployed 3 new sam dbserver nodes cdfsam12-14 replacing  
&gt; cdfsam18,19,21 (thanks to run2-sys for preparing the new nodes) 
&gt; 
&gt; .) upgraded dbservers to v8_0_7 sam_db_srv_pkg 
&gt; 
&gt; .) configured station and multiplexing dbservers to run on  
&gt; cdfsam01 
&gt; and the multiplexer serving dbservers to to run on 
&gt; *srv1* cdfsam01 
&gt; *srv2* cdfsam12 
&gt; *srv3* cdfsam13 
&gt; *srv4* cdfsam14 
&gt; respectively 
&gt; 
&gt; .) decommissioned now unused cat_prd & raw_prd dbservers 
&gt; 
&gt; .) introduced the "or" predicate limit of 20 for the dimension  
&gt; queries (thanks to the farm team for modify the code to fully  
&gt; use the "in" syntax) 
&gt; 
&gt; .) restarted all the dbservers after the data base related  
&gt; patches were applied by the database support group (thanks) 
&gt; 
&gt; .) restarted the cdf-caf & cdf-fncdfsrv0 stations 
&gt; 
&gt; We are still waiting for the enstore and CAF groups to finish  
&gt; their work to proceed with system tests. 
&gt; 
&gt; Krzysztof</pre>
<!-- Author: G. Compostella -->


<!-- Date: Fri Mar 16 03:10:39 2007 -->
<pre>Plots at  
http://cdfsam-prd.fnal.gov/~sam/DHAtAGlance/upload_rates.html  
http://cdfsam-prd.fnal.gov/~sam/DHAtAGlance/farm_rates.html  
http://cdfsam-prd.fnal.gov/~sam/DHAtAGlance/data_rates.html  
 
are now updating correctly, thanks to Dehong and Doug for taking care of this.</pre>
<!-- Author: G. Compostella -->


<!-- Date: Sat Mar 17 04:04:40 2007 -->
<pre>Couple of words from Wayne Baisley about tape Access: 
=============== 
Just to clarify things ... 
We have a page (http://www-cdfen.fnal.gov/enstore/tape_inventory/NOACCESS) 
that we check daily for tape problems.  It looks like CDF's pages are simply copies of ours.  Tapes that show up there will be marked either NOACCESS or NOTALLOWED.  Tapes get marked NOACCESS by the Enstore system whenever there's a problem.  We try to deal with NOACCESS tapes within 24 hours, excluding weekends and holidays. 
If a tape has more than a transient problem, we will mark it NOTALLOWED and usually add a comment about the issue.  Tapes may be NOTALLOWED for quite some time, depending on the nature of the problem. 
The webpage also shows how much tape is remaining, the date and time (but not year) of when the tape went into its current state, its readonly state and when that was set, the library name, and the file family. 
The one tape in the list that obviously needs to be checked on is IA2934 since it has data on it.  It really shouldn't have waited so long.  BTW, the last file on that tape was written on Nov 12, 2005. 
Regards, 
Wayne 
=============== 
</pre>
<!-- Author: G. Compostella -->


<!-- Date: Sun Mar 18 03:45:54 2007 -->
<pre>Notified cdf_mc_oper of errors in uploading to sam with fcdfdata134.</pre>
<!-- Author: G. Compostella -->


<!-- Date: Sun Mar 18 09:48:09 2007 -->
<pre>Warning in Raw Data Logging: 
Sun Mar 18 09:40:41 2007 1174228841 Warning: 
------- -- User 'stager' may be over-quota 
Sun Mar 18 08:15:44 2007 1174223744 Action: Experts have been notified, please wait 
</pre>
<!-- Author: G.Compostella -->


<!-- Date: Sun Mar 18 10:34:28 2007 -->
<pre>Checked  
https://www-cdfonline.fnal.gov/daq/cslmon/stager/disk_quotas.html 
 
Looks like user 'stager' is still under quota, even if log area occupation on disk has been increasing slowly in the past week.</pre>
<!-- Author: G. Compostella -->


<!-- Date: Mon Mar 19 04:49:36 2007 -->
<pre>Notified cdf_mc_ops that a couple of datasets need to be registered. 
 
Notified enstore-admin of a mover in trouble: 9940B24.</pre>
<!-- Author: G. Compostella -->