<!-- Title: 06:37:27  Tue Jun  3 2008 -->
<!-- Crewchief: Barry Blumenfeld -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>New shift</pre>


<!-- Date: Tue Jun 3 07:11:51 2008 -->
<pre>Known problems: 
 
4 read pools on fcdfdata154 are offline. 
3 write pools on fcdfdata157 are offline. 
 
2 bad movers: 9940B22, 9940B26</pre>
<!-- Author: Barry -->


<!-- Date: Tue Jun 3 10:11:42 2008 -->
<pre>Bad Tape IAB455 causing Bad Mover LTO3_31 
 
alive : ERROR - ('1', "mount IAB455 failed: ('ERROR', 13, ['ACSSA&gt; mount IAB455 0,3,1,2', 'Mount: Mount failed, Unreadable label.', 'ACSSA&gt; logoff', ''], '', 'MOUNT 13: mount IAB455 0,3,1,2 =&gt; 0,Mount: Mount failed, Unreadable label.')")</pre>
<!-- Author: Barry -->


<!-- Date: Tue Jun 3 10:34:26 2008 -->
<pre>Response from Stanley Hicks: 
 
I have ordered a new label for this volume and put the drive back into production. 
</pre>
<!-- Author: Barry -->


<!-- Date: Tue Jun 3 11:13:48 2008 -->
<pre>Queues growing on dCache ReadOnlyPools: 
r-fcdfdata138-1 
r-fcdfdata138-3 
r-fcdfdata316-1 
 
Pools moved to TestPools to drain. 
 
- Angela</pre>
<!-- Author: Angela Bellavance -->
<!-- Comment: Tue Jun  3 11;53;50 comment by...Angela Bellavance -->
<pre>r-fcdfdata138-3 has drained - moved back to ReadOnlyPools.
New pools moved to TestPools:
r-fcdfdata145-3
r-fcdfdata146-2
r-fcdfdata149-1</pre>

<!-- Comment: Tue Jun  3 12;09;41 comment by...Angela Bellavance -->
<pre>r-fcdfdata316-1 back in nominal service.</pre>


<!-- Comment: Tue Jun  3 13;43;11 comment by...Angela Bellavance -->
<pre>r-fcdfdata138-1
r-fcdfdata145-3
r-fcdfdata149-1
r-fcdfdata146-2
back in service.

r-fcdfdata146-4
r-fcdfdata147-1
moved to TestPools to drain.</pre>
<!-- Comment: Tue Jun  3 14;17;18 comment by...Angela Bellavance -->
<pre>r-fcdfdata146-4 back in service.</pre>
<!-- Comment: Tue Jun  3 16;15;14 comment by...Angela Bellavance -->
<pre>r-fcdfdata147-1 back in service
r-fcdfdata317-1 removed for draining</pre>
<!-- Comment: Wed Jun  4 09;10;53 comment by...Angela Bellavance -->
<pre>r-fcdfdata317-1 back in service. All pools now look okay, except for the known separate problems with r-fcdfdata135-1 and fcdfdata320.

- Angela</pre>
<!-- Date: Tue Jun 3 13:25:58 2008 -->
<pre>User has processes that appear stuck: 
 
Several of the processes that I am running to retrieve data from the 
data handling system onto dukpc32.fnal.gov and dukpcdb.fnal.gov--among 
them, the two that appear below--are stuck. 
 
http://cdfsam-prd.fnal.gov/sam_user_pyapi/cgi/dumpSam.fcgi?project=psubb_20080602174147&station=cdf-caf 
 
http://cdfsam-prd.fnal.gov/sam_user_pyapi/cgi/dumpSam.fcgi?project=psubb_20080602174233&station=cdf-caf 
 
Can you investigate why this is happening? My kerberos principal is 
&lt;psubb&gt;. Thank you in advance for your help. 
 
Prashant Subbarao 
</pre>
<!-- Author: Barry -->
<!-- Comment: Tue Jun  3 14;17;54 comment by...Angela Bellavance -->
<pre>JIRA ticket created - CDFDH-106</pre>


<!-- Date: Tue Jun 3 15:07:24 2008 -->
<pre>From Tim Messer: 
 
I will be taking a few cdfen movers down for a short time this afternoon  
to upgrade them to SL4 and SDE.  These will be taken down one at a time  
to minimize disruption: 
 
cdfenmvr122a (LTO3_22) 
cdfenmvr123a (LTO3_23) 
cdfenmvr124a (LTO3_24) 
cdfenmvr125a (LTO3_25) 
cdfenmvr126a (LTO3_26) 
 
</pre>
<!-- Author: Barry -->



<!-- Date: Wed Jun 4 07:55:31 2008 -->
<pre>On SAM at a Glance, cdf-fzkka has a yellow ball. 
I know there were network issues to FZK this morning from 
other sources. 
I won't worry about it now.</pre>
<!-- Author: Barry -->


<!-- Date: Wed Jun 4 08:37:04 2008 -->
<pre>cdf-fzkka is now green.</pre>
<!-- Author: Barry -->



<!-- Date: Wed Jun 4 15:21:19 2008 -->
<pre>From Tim Messer: 
 
I will be taking a few cdfen movers down for a short time this afternoon  
to upgrade them to SL4 and SDE.  These will be taken down one at a time  
to minimize disruption: 
 
cdfenmvr127a (LTO3_27) 
cdfenmvr128a (LTO3_28) 
cdfenmvr129a (LTO3_29) 
cdfenmvr130a (LTO3_30) 
cdfenmvr131a (LTO3_31) 
</pre>
<!-- Author: Barry -->


<!-- Date: Thu Jun 5 08:06:19 2008 -->
<pre>From the automated check_file_size_and_CRC job (7:17 am EDT): 
 
Filesize mismatches 
 
diff in MiB          # 
----------- ---------- 
          0    3454160 
          1         11 
</pre>
<!-- Author: Barry -->


<!-- Date: Thu Jun 5 12:04:38 2008 -->
<pre>From Randy: 
 
What is causing these errors? 
 
I can change the SAM values to match the Enstore. 
But, it would be ggod to stop the problem at the source. 
 
Filesize mismatches 
 
       FILE_SIZE FILE_SIZE_IN_BYTES    FILE_ID FILE_NAME 
---------------- ------------------ ---------- ------------------------------------------------------------------------ 
   1,686,435,961      1,686,435,840   24528994 /pnfs/cdfen/filesets/MC/MC02/MC0273/MC0273.1/jd033972.0001hgp6 
   1,761,955,966      1,761,955,840   24528966 /pnfs/cdfen/filesets/MC/MC02/MC0273/MC0273.1/jd0358d6.0002hgp5 
   1,545,691,730      1,545,692,160   24528961 /pnfs/cdfen/filesets/MC/MC02/MC0273/MC0273.1/jd035622.0001hgp5 
   1,692,744,687      1,692,744,704   24528959 /pnfs/cdfen/filesets/MC/MC02/MC0273/MC0273.1/jd0339fc.0002hgp5 
   1,306,708,615      1,306,708,992   24528956 /pnfs/cdfen/filesets/MC/MC02/MC0273/MC0273.1/jd033972.0001hgp5 
   1,832,296,592      1,832,296,448   24528953 /pnfs/cdfen/filesets/MC/MC02/MC0273/MC0273.1/jd0333f7.0001hgp5 
   1,313,651,272      1,313,651,712   24528407 /pnfs/cdfen/filesets/MC/MC02/MC0273/MC0273.1/jd0358dd.0001hgp4 
   1,547,873,288      1,547,873,280   24527510 /pnfs/cdfen/filesets/MC/MC02/MC0273/MC0273.1/jd035dbc.0001hgp3 
   1,189,736,501      1,189,736,448   24527501 /pnfs/cdfen/filesets/MC/MC02/MC0273/MC0273.1/jd035957.0002hgp3 
   1,639,796,008      1,639,795,712   24527495 /pnfs/cdfen/filesets/MC/MC02/MC0273/MC0273.1/jd035622.0001hgp3 
   1,819,398,901      1,819,399,168   24527488 /pnfs/cdfen/filesets/MC/MC02/MC0273/MC0273.1/jd0333f7.0001hgp3 
 
11 rows selected. 
 
CRC mismatches 
 
FILE_CRC                         CRC_VALUE                    FILE_ID 
-------------------------------- ------------------------- ---------- 
FILE_NAME 
------------------------------------------------------------------------ 
1637886521                       4151781881L                 24490325 
/pnfs/cdfen/filesets/MC/MC02/MC0276/MC0276.0/wf03abea.0001ewkg 
 
2893350833                       2576549793L                 24517812 
/pnfs/cdfen/filesets/MC/MC02/MC0277/MC0277.7/wf039f22.0035ewki 
</pre>
<!-- Author: Barry -->


<!-- Date: Thu Jun 5 16:32:55 2008 -->
<pre>I notice IAB455 is no longer listed as a Bad Tape. 
Perhaps it got it's new label.</pre>
<!-- Author: Barry -->


<!-- Date: Fri Jun 6 07:50:08 2008 -->
<pre>A couple more file size mismatches: 
Filesize mismatches 
 
diff in MiB          # 
----------- ---------- 
          0    3456088 
          1         13 
</pre>
<!-- Author: Barry -->


<!-- Date: Fri Jun 6 11:12:46 2008 -->
<pre>Drive 9940B26 has been returned to us. 
 
- Angela</pre>
<!-- Author: Angela Bellavance -->


<!-- Date: Fri Jun 6 13:29:36 2008 -->
<pre>As of:  Fri Jun  6 13:23:02 2008 
The following doors are not responding: 
DOOR02   dcap://fcdfrdc3.fnal.gov:22127  
</pre>
<!-- Author: Barry -->


<!-- Date: Fri Jun 6 13:36:04 2008 -->
<pre>My posting to the CDF_DISKPOOL_ADMIN list was rejected, but the problem seems to have cleared itself. 
All diskpool doors are now OK. 
</pre>
<!-- Author: Barry -->


<!-- Date: Fri Jun 6 13:46:23 2008 -->
<pre>Now it has 
 
As of:  Fri Jun  6 13:43:00 2008 
The following doors are not responding: 
DOOR01   dcap://fcdfrdc3.fnal.gov:22126</pre>
<!-- Author: Barry -->


<!-- Date: Fri Jun 6 14:17:55 2008 -->
<pre>Now the diskpool doors are clear again. 
However, the response time plots have been largely saturated since 5:00 AM.</pre>
<!-- Author: Barry -->


<!-- Date: Fri Jun 6 14:55:01 2008 -->
<pre>Cell ofline.  The other cells are there from this machine. 
 
r-fcdfdata097-1 r-fcdfdata097-1Domain OFFLINE </pre>
<!-- Author: Barry -->


<!-- Date: Sat Jun 7 07:57:53 2008 -->
<pre>From Angela: 
 
The pool refuses to come back up. I have asked that the helpdesk ticket (116784 ) be reopened. 
</pre>
<!-- Author: Barry -->


<!-- Date: Sat Jun 7 08:19:14 2008 -->
<pre>Bad Mover: 
 
LTO3_04.mover alive : OFFLINE cdfenmvr104a 2008-Jun-07  </pre>
<!-- Author: Barry -->


<!-- Date: Mon Jun 9 08:12:19 2008 -->
<pre>File size mismatches are huge: 
 
Filesize mismatches 
 
diff in MiB             # 
----------- ---------- 
          0    3462985 
          1            14 
          5             2 
          7             1 
         11             1 
         14             1 
         18             2 
         22             1 
         28             2 
         42             2 
         44             1 
 
diff in MiB             # 
----------- ---------- 
         47             2 
         49             2 
         53             2 
         76             1 
         77             3 
         79             1 
         81             1 
         87             1 
         89             2 
         91             1 
         98             1 
 
diff in MiB             # 
----------- ---------- 
105             1 
108             1 
113             1 
118             1 
123             1 
125             1 
130             1 
131             1 
149             1 
153             1 
161             1 
 
diff in MiB             # 
----------- ---------- 
165             1 
193             1 
201             1 
219             1 
269             1 
271             1 
287             1 
315             1 
351             1 
407             1 
416             1 
 
diff in MiB             # 
----------- ---------- 
435             1 
534             1 
620             1 
663             1 
715             1 
836             1 
840             1 
858             1 
884             1 
908             1 
       1150             1 
 
55 rows selected. 
 
CRC mismatches 
 
  COUNT(*) 
---------- 
60 
  
  
  
</pre>
<!-- Author: Barry -->


<!-- Date: Mon Jun 9 09:48:59 2008 -->
<pre>As of:  Mon Jun  9 09:46:16 2008 
The following doors are not responding: 
DOOR03   dcap://fcdfrdc3.fnal.gov:22128 
</pre>
<!-- Author: Barry -->


<!-- Date: Mon Jun 9 09:59:51 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-73&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-73  width='300' align='top'></a></td><td>It now says all diskpool doors are responding, but this looks odd.</td></tr></table>
<!-- Author: Barry -->


<!-- Date: Mon Jun 9 10:21:28 2008 -->
<pre>Clsam and clmeta lost heartbeat on b0csl21. Restarted both and all is okay now. 
 
- Angela</pre>
<!-- Author: Angela Bellavance -->


<!-- Date: Mon Jun 9 11:18:14 2008 -->
<pre>Cell r-fcdfdata097-1 is working again.</pre>
<!-- Author: Barry -->


<!-- Date: Mon Jun 9 13:19:03 2008 -->
<pre>Clsam and clmeta heartbeat on b0csl21 died again. Restarted. 
 
- Angela</pre>
<!-- Author: Angela Bellavance -->


<!-- Date: Mon Jun 9 16:16:59 2008 -->
<pre>Mover LTO3_04 is back.</pre>
<!-- Author: Barry -->


<!-- Date: Mon Jun 9 17:21:27 2008 -->
<pre>From Alex Kulyavtsev about r-fcdfdata097-1 
 
now all three pools on node 097 are disabled with message "Repository  
got lost". 
I'm restarting pools</pre>
<!-- Author: Barry -->