<!-- Title: 09:58:43  Tue Jun 10 2008 -->
<!-- Crewchief: John Yon (Tu->Th) -->
<!-- Op1: Mike Albrow (Fr-->Mon) -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Continuing Problems 
 
fcdfdata320 (4) offline (since months ago) 
 
fcdfdata154 (4 read pools) offline  (since before last week) 
fcdfdata157 (3 write pools) offline (since before last week) 
 
9940B22 offline ("replace system disk"--since before last week) 
 
Otherwise, things seems to be okay</pre>


<!-- Date: Tue Jun 10 15:54:32 2008 -->
<pre>Date:  Tue, 10 Jun 2008 15:04:13 -0500 
&gt; From:  Angela Bellavance &lt;bellavan@fnal.gov&gt; 
&gt; To:  CDF Data Handling Operations &lt;cdfdh_oper@fnal.gov&gt; 
&gt; Subject:  Restarting SAM DbServers 
&gt; Hello DH people, 
&gt;  
&gt; We've gotten our new DB accounts, so I will be  
&gt; restarting the DbServers, one at a time, to pick up  
&gt; the new access method. So, don't be surprised if a  
&gt; DbServer disappears off of Sam-At-A-Glance for a while. 
&gt;  
&gt; Regards, 
&gt; Angela 
&gt;  
 
--------------- 
indeed, at 15:50 lots of DB servers are not responding-- 
will update this note when Angela is done.</pre>
<!-- Author: John Yoh -->
<!-- Comment: Wed Jun 11 08;57;16 comment by...Angela Bellavance -->
<pre>I messed up when restarting the DbServers and so a bunch of jobs on the CAF station and the farm station crashed. My apologies for that. The switchover was complete and things were running again within about an hour.

- Angela</pre>


<!-- Date: Wed Jun 11 15:39:43 2008 -->
<pre>fcdfdata135-1 went offline around 14:00; Angela is working on it.</pre>
<!-- Author: John Yoh -->
<!-- Comment: Wed Jun 11 16;52;25 comment by...John Yoh -->
<pre>fcdfdata135-1 back online 15:30 (fixed by Angela)</pre>
<!-- Date: Wed Jun 11 16:02:22 2008 -->
<pre>LT03_29 has a red button, and is in the Bad Mover list. 
(this apparently happened around 14:00) 
 
Also tape IAB386 is marked "bad tape" 
 
IAB386              0.76GB   (NOACCESS   0611-1352 full     0518-2251)   CDF-LTO3         cdf.xbhd0j.cpio_odc          
 
email sent to enstore_admin and cdf_dh_help</pre>
<!-- Author: John Yoh -->
<!-- Comment: Wed Jun 11 16;49;02 comment by...John Yoh -->
<pre>
John,

That volume has an unreadable label. I have ordered a new one and will 
get it replaced as soon as it comes in. I will get the mover back online 
shortly.

Thanks for catching this,
Stanley (Hicks, 16:09)</pre>


<!-- Comment: Wed Jun 11 17;10;26 comment by...John Yoh -->
<pre>LT03_29 back online at 17:00 (Thanks, Stanley).</pre>

<!-- Comment: Thu Jun 12 10;50;14 comment by...John Yoh -->
<pre>IAB386 not being available is causing a large number (about 70) of DCache  "difference between requested and queued" plot  --(example:
1	Volume IAB386 is marked NOACCESS. : (SRC: None) (DST: /export/data4/read-pool-4/data/001700000000000000B45C20)

Please do not submit jobs that require IAB386 until it becomes available (please keep checking this elog to find out when it becomes available)  </pre>

<!-- Comment: Thu Jun 12 11;49;44 comment by...John Yoh -->
<pre>Stanley Hicks get the new label on IAB386--and it's back online. The Tape error has disappeared, as is the large number of "difference between requested and queued"--so, all these problems appears okay now.</pre>


<!-- Date: Thu Jun 12 16:11:59 2008 -->
<pre>Mid-week summary (Mike Albrow is taking over 
tomorrow Friday for the rest of this week's shift) 
 
(1) Continuing Problems (to be ignored ??) 
 
fcdfdata320 (4) offline (since months ago)  
 
fcdfdata154 (4 read pools) offline  (since before last week)  
fcdfdata157 (3 write pools) offline (since before last week)  
 
9940B22 offline ("replace system disk"--since before last week)  
 
(2) Tues June 10 15:00  
  Angela restarted the DBServers using new DBaccounts-- 
  It turned out many job were hung, and needed to be 
  restarted later. 
 
(3) Wed June 11, 14:00  
  fcdfdata135-1 went offline ; Angela fixed it around 15:30 
 
(4) Wed June 11, 14:00 
  LT02_29 bad mover--fixed 17:00 
  IAB386 had a bad label--replaced Thur June 12 11:50 
	by Stanley Hicks. 
    While IAB386 was unavailable, there were lots of jobs hung 
	trying to access it (up to 70) which showed up at 
	DCache "difference between requested and queued"</pre>
<!-- Author: John Yoh -->


<!-- Date: Fri Jun 13 11:10:50 2008 -->
<pre>Mike Albrow takes over from John Yoh as CDF SAM shifter</pre>
<!-- Author: Mike -->


<!-- Date: Fri Jun 13 15:05:03 2008 -->
<pre>Fri Jun 13 14:55:45 2008 1213386945 Warning: 
b0csl21 -- Clmeta may be dead 
Fri Jun 13 11:30:43 2008 1213374643 Action:  
Experts have been notified, please wait</pre>
<!-- Author: Mike -->
<!-- Comment: Fri Jun 13 16;39;35 comment by...Angela Bellavance -->
<pre>This will continue for a while. There is a problem with the log file timestamps not getting updated. I have opened a helpdesk ticket. Reference JIRA ticket CDFDH-120.

- Angela</pre>
<!-- Date: Fri Jun 13 15:06:24 2008 -->
<pre>Number of files to restore growing: 320/5</pre>
<!-- Author: Mike -->


<!-- Date: Fri Jun 13 16:38:12 2008 -->
<pre>Corrupted files reported on dCache pool r-fcdfdata097-1. Files have been removed. Reference JIRA ticket CDFDH-129. 
 
- Angela</pre>
<!-- Author: Angela Bellavance -->



<!-- Date: Fri Jun 13 19:25:23 2008 -->
<pre>Database response time 23, has been high for a couple of hours but I think this is not unusual. 
Mike</pre>
<!-- Author: Mike -->


<!-- Date: Sun Jun 15 08:18:17 2008 -->
<pre>r-fcdfdata146-1Domain to 4Domain offline, 4 read pools. 
(in addition to old r-fcdfdata154).</pre>
<!-- Author: Mike -->
<!-- Comment: Sun Jun 15 14;24;09 comment by...Angela Bellavance -->
<pre>Email sent to cdf_dcache_help about all 8 down pools.

- Angela</pre>

<!-- Comment: Mon Jun 16 09;28;58 comment by...Angela Bellavance -->
<pre>cdf_dcache_help isn't working yet. Forwarded message to dcache-admin.

Also, fcdfdata154 is not pingable, so sent email to helpdesk about it.

- Angela</pre>


<!-- Date: Mon Jun 16 19:23:49 2008 -->
<pre>Total number of files to restore very large:1966 
/3 difference. 
Sens email to dcache-admin.</pre>
<!-- Author: Mike -->


<!-- Date: Tue Jun 17 08:05:52 2008 -->
<pre>End shift, ongoing issues: 
fcdfdata320 (4) offline (since months ago)   
fcdfdata154 (4 read pools) offline  (since before last week)  
Not pingable, email sent. 
r-fcdfdata146-1 r-fcdfdata146-1,2,3,4Domain OFFLINE since Sunday. 
Email sent to cdf_dcache_help about all 8 down pools. 
9940B22 offline ("replace system disk"--since before last week) 
Total number of files to restore very large:1966/3 difference.   
Sent email to dcache-admin. 
b0csl21 -- Clsam & Clmeta may be dead Experts have been notified. 
This will continue for a while. There is a problem with the log file timestamps not getting updated. Opened a helpdesk ticket.  
</pre>
<!-- Author: Mike -->