<!-- Title: 08:52:16  Tue Feb 12 2008 -->
<!-- Crewchief: JKY (John K Yoh) -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Rick's summary from this morning (04:05) email 
 
o Known and continuing problems 
  fcdfdata320 offline 
 
o Issues added after last meeting 
  - Issue 3555 : when user requests 0 files, jobs don't run properly since 
    the consumer process thinks it is done.  A user is likely to do this 
    with a buggy script that sets the number of files in an environment  
    variable.  If the C++ interface realizes it is using SAMDH then it 
    should exit and clearly state that zero files were requested. 
 
  - Issue 3557: Project Master Polling Failure; Project master gets 
    a failure in polling a consumer process. The user program aborts 
    prematurely. 
 
o Pool draining 
  - r-fcdfdata138-1 moved by Angela to testpool to train, then returned 
    to regular service 
   
o TNS operation timeout 
  - had one but recovered 
 
o Bad movers and Tapes (fixed without shifter intervention, Thank you  
                        Enstore group) 
  - LT03-04,23,27,30 tape IAD 419 
  - LT03-05,26 
  - LT03_29, IAD988 
 
o Enstore Downtime as noted by David (and seemed to go smoothly, thanks,  
  again) 
 
o Alarm due to CRC checking error; Quite a long discussion about 
  levels of checking and enstore/dcache/sam.  Defer to experts 
  for final discription in Dave Berg's report. 
 
o User Problem: Chris Neu could not run a top stntuple dataset. Turned 
  out a snapshot had not been taken. He took a snapshot. 
  Two comments: 
     1.  Taking a snapshot of a dataset after defining it should be 
         part of the procedure since the dataset definition can be changed 
         until the snapshot is taken.  This is something to watch for since 
         a user may take a standard definition and append _0 or A or something 
         and if two users happen to do this at the same time, it can 
         lead to confusion as to who gets there first. 
 
     2.  If a sam dataset is used to submit to caf, is it true that the 
         dataset is verified and that works only if a snapshot is taken? 
         Then if a project is started does the snapshot get taken by SAM? 
         How does the first snapshot get taken? This is related to a later 
         query by Ray Culbertson concerning who takes snapshots. 
 
o Diskpool node was down. Was reported to dcache incorrectly by me, then sent 
  to cdf diskpool but no action was taken until a CDF user (Lina Galtieri) 
  could not access data.  Then it was corrected. 
 
o Upload issues: 
  - User (Craig Group) forgot to define dataset, was contacted and corrected 
    problem and thanked us. 
  - User (Sasha Golossanov) had a couple of failures in loading pskt3w. 
    First, no such file or directory mentioned. Rob Harr had not seen this 
    before but instructed Sasha that after a failure, files were put back 
    in a safe area for retry.  Secondly, there was an inability to get 
    metadata for the same file. This problem remains. 
 
o User Problem: User unable to access bottom stntuple.  The problem was that 
  the stntuple uses SAM as the catalog but not to deliver URLs. That it 
  forms in its code by querying the SAM database directly for the file 
  location and builds the dcache URL.  The user had a very very old copy 
  of the code. I recognized this problem in June 2006 and Ray Culbertson 
  patched the code.   For future reference, the stntuple user sees 
  files listed with something like: 
dcap://cdfdca3.fnal.gov:25159/pnfs/fnal.gov/usr/cdfen/filesets/NB/NB00/NB0003/NB0003.5/xi035993.0196bhdi 
 
 xbhdii.002 1307495424.000 
 61973 219539 3066869 219539 3798018 0.000 0.000 
 
dcap://cdfdca3.fnal.gov:25159/pnfs/fnal.gov/usr//cdfcaf-backbone:dcap://cdfdca3.fnal.gov:25164/pnfs/fn/xi035993.0196bhdi 
xbhdii.002 1307495424.000 61973 219539  3066869 219539 3798018 0.000 0.000 
 
 
  The first line is fine, it is the second line, with cdfcaf-backbone that  
  is the problem. 
 
  The code is Stntuple/loop/TStnCatalogServer.cc and was fixed in 
  2006/06/22.  It is difficult to track stntuple as the tag name is kept 
  the same and the code is changed in CVS.  But this was rev 1.34 of the 
  file.  One should check user code for this. 
 
  A user has to have very old code and I am very surprised that this code  
  works at all for many other reasons!  This may never come up again. 
 
o Pools down: r-cdfdata135-1 through 4 (turned out that admins said the 
                                        node was not pingable and hardware 
                                        folks worked on it) 
              r-cdfdata111-1 through 4 
  This goes to dcache-auto so I only note this.  Angela sent mail. 
  Is that needed?  DCache group magically seems to fix this.  (Thankyou) 
 
o Angela is getting a new maillist deployed for the sam test jobs 
 
o CSL error came up but it seemed a communication issue. But discovered 
  the name of the node is not that on SAG, have to use b0gateway or b0doorway 
  to log in and I am not in k5login.  What should we be doing on shift with 
  CSL? 
 
</pre>


<!-- Date: Tue Feb 12 09:15:57 2008 -->
<pre>(1) Error onUpload Server fcdfdata321 (may be a continueing error 
since a similar error existed yesterday afternoon) 
 
08:55:07 20080212 1202828107 
Unable to get the metadata structure for file /data/top3-131/submitted/pw02eae8.0001skt3 --- [Errno 2] No such file or directory: /data/top3-131/submitted/pw02eae8.0001skt3 
 
 
(2) DCache fcdfdata111 offline since Monday 2/11 15:25 
 
(3) DCache difference between requested and queue rises 
to about 40 exactly at the same time as (2) and stay around 
40 (52/39) 
 
(4) Enstore Tapes in trouble IAD988 (Note : this already 
occurred Monday 1327) 
 
Tue Feb 12 09:00:15 CST 2008 
 
IA1853              0.23GB   (NOTALLOWED 0201-0820 full     1227-2115)   CDF-9940B        cdf.xbmu0i.cpio_odc                  Vol stuck in drive DBT2MV - unable to dismount 
IA2312            200.00GB   (NOTALLOWED 0810-1727 none              )   TST-9940B        cdf.none.none                        
IA2934             63.76GB   (NOTALLOWED 0124-1051 readonly 1112-1445)   CDF-9940B        cdf.mjkim.cpio_odc                   File 9 is bad, need to copy 
IA5057            200.00GB   (NOTALLOWED 0110-1316 none     1227-1645)   CDF-9940B        cdf.none.none                        Broken Leader on a blank tape 
IA6978            181.47GB   (NOTALLOWED 1024-1728 none     0619-1350)   CDF-9940B        cdf.sgi2test.cpio_odc                
IAB837            400.00GB   (NOTALLOWED 0314-1110 none              )   CDF-LTO3         cdf.none.none                        
IAB973            400.00GB   (NOTALLOWED 0721-1151 none              )   CDF-LTO3         cdf.hbhd0j.cpio_odc                  9 consecutive mount failures 
IAC226            189.05GB   (NOTALLOWED 0622-1405 none              )   CDF-9940B        cdf.cphysr.cpio_odc                  To be replaced 06/22/07 
IAD988              0.44GB   (NOACCESS   0208-1938 full     0122-0411)   CDF-LTO3         cdf.gphysr.cpio_odc                  
 
Tue Feb 12 09:00:15 CST 2008</pre>
<!-- Author: JKY -->
<!-- Comment: Tue Feb 12 12;50;35 comment by...JKY -->
<pre>1030 : cdfdata111 problem (associated with the 40 extra 
difference delivered/queue issue) has been resolved-- 
and have turned green--fcdfdata320 is still inactive 
---------- 
solution: ling@fnal.gov sent this solution:  

System is back up. Faulty memory has been replaced. 

</pre>
<!-- Date: Tue Feb 12 09:35:26 2008 -->
<pre>A last bit of unfinished business regarding the problem with the bntuple.  
 
It turned out that the code was old and the bug had been fixed in June 2006. 
 
See notes in the summary for the location of the file in case this happens again.</pre>
<!-- Author: r -->


<!-- Date: Tue Feb 12 11:21:51 2008 -->
<pre>fcdfdata111 problem (associated with the 40 extra 
difference delivered/queue issue) has been resolved-- 
and have turned green--fcdfdata320 is still inactive 
---------- 
solution: ling@fnal.gov sent this solution:  
 
System is back up. Faulty memory has been replaced. 
</pre>
<!-- Author: JKY -->



<!-- Date: Tue Feb 12 12:52:05 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-42&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-42  width='300' align='top'></a></td><td>difference request/queued</td></tr></table>
<!-- Author: JKY -->


<!-- Date: Tue Feb 12 13:56:19 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-43&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-43  width='300' align='top'></a></td><td>fcdfdata111 being offline--fixed</td></tr></table>
<!-- Author: JKY -->


<!-- Date: Tue Feb 12 14:01:50 2008 -->
<pre>9940B27.mover is offline for a test by GS</pre>
<!-- Author: JKY -->


<!-- Date: Wed Feb 13 09:10:41 2008 -->
<pre>Status : 2/13/08 09:00  
 
Data Handling at a glance 
 
  No change from yesterday afternoon 
 
  remaining problems 
   
  (1) Upload server fcdfdata321--1 failed job/1 error 
 
09:00:07 20080213 1202914807 
Unable to get the metadata structure for file /data/top3-131/submitted/pw02eae8.0001skt3  
     --- [Errno 2] No such file or directory: /data/top3-131/submitted/pw02eae8.0001skt3 
 
  (2) DCache 
   No of cells inactive --4  
r-fcdfdata320-1 
r-fcdfdata320-2 
r-fcdfdata320-3 
r-fcdfdata320-4 
 
 (3) Enstore 
 
CDF-9940B --1 bad mover 
9940B27 (being used for tape test by GS--since yesterday PM) 
 
CDF-LT03 --1 Bad tape IAD988 
   
IAD988              0.44GB   (NOACCESS   0208-1938 full     0122-0411)   CDF-LTO3         cdf.gphysr.cpio_odc                  
 
 NO OTHER rED PROBLEMS 
 
----------------- 
sam AT A GLANCE 
 
All clickable SAMStations and SAM DbServers have green button 
(except for None)</pre>
<!-- Author: JKY -->


<!-- Date: Thu Feb 14 11:23:10 2008 -->
<pre>No change (No news is good news)  
 
All files on IA1853 (the tape that was 
"Vol stuck in drive DBT2MV - unable to dismount") 
have been marked bad. Tape was sent to  
vendor for recovery--no estimate.</pre>
<!-- Author: JKY -->


<!-- Date: Thu Feb 14 14:57:48 2008 -->
<pre>In DH-0at-a-glance 
 
Enstore CDF-9940B --new bad tape showed up as Yellow 
  --IA 6978.  
Also, Number of "stage read" in the Read Pending row 
 turned yellow (at 42). 
 
CDF-LT03 still have the old Bad Tape IAD988 (been there 
for 2 days--so far, no info from the experts that we 
report the problems to). 
 
Number of "stage read" in the Read Pending row turn 
red at 15 
 
Will keep looking at it and act if it gets worse.   
</pre>
<!-- Author: JKY -->


<!-- Date: Fri Feb 15 09:46:55 2008 -->
<pre>Status : 2/14/08 09:40  
 
Data Handling at a glance 
 
  New RED items--changed from yesterday afternoon 
 
   New Problems 
 
  (A) One DCache Diskpool door not opening (probably happened 
	on 2/14 around 1830)--email sent to cdf_diskpool_admin 
The following doors are not responding: 
DOOR02   dcap://fcdfrdc3.fnal.gov:22127 
 
BTW--there also appeared to be a sharp spike in the Number of 
	files to restore just before 1830 --leading to a  
	short spike of 100 in the "difference between restorehandler 
	and queueInfo" plot --could there be a connection ?? 
 
 
  (B) Enstore --new bad mover LT03_26.mover 
 
alive : ERROR - ('1', "mount IAB313 failed: ('ERROR', 13, ['ACSSA&gt; mount  
IAB313 0,1,1,3', 'Mount: Mount failed, Unreadable label.',  
'ACSSA&gt; logoff', ''], '', 'MOUNT 13: mount IAB313 0,1,1,3 =&gt; 0, 
Mount: Mount failed, Unreadable label.')") 
 
as well as a new bad tape IAB313 
 
  Will keep it in mind and report if the status stay red.  
 
----------------------------------------------------- 
  Old remaining problems 
   
  (1) Upload server fcdfdata321--1 failed job/1 error 
 
09:00:07 20080213 1202914807 
Unable to get the metadata structure for file /data/top3-131/submitted/pw02eae8.0001skt3  
     --- [Errno 2] No such file or directory: /data/top3-131/submitted/pw02eae8.0001skt3 
 
  (2) DCache 
   No of cells inactive --4  
r-fcdfdata320-1 
r-fcdfdata320-2 
r-fcdfdata320-3 
r-fcdfdata320-4 
 
 (3) Enstore 
 
CDF-9940B --1 bad mover 
9940B27 (being used for tape test by GS--since yesterday PM)l 
 
CDF-LT03 --1 Bad tape IAD988 
   
IAD988              0.44GB   (NOACCESS   0208-1938 full     0122-0411)   CDF-LTO3         cdf.gphysr.cpio_odc                 
 NO OTHER RED PROBLEMS 
 
----------------- 
sam AT A GLANCE 
 
All clickable SAMStations and SAM DbServers have green button 
(except for None)</pre>
<!-- Author: JKY -->
<!-- Comment: Fri Feb 15 11;41;14 comment by...JKY -->
<pre>DOOR02 is now back up (Problem (A) above.
See plot </pre>


<!-- Date: Fri Feb 15 11:42:49 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-44&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-44  width='300' align='top'></a></td><td>History of Door02 
down at 2/14 1825 
back up at 2/15 1120</td></tr></table>
<!-- Author: JKY -->


<!-- Date: Fri Feb 15 15:20:39 2008 -->
<pre>fcdfdata135 is offline since around 14:45 
email sent to helpdesk and run2-sys</pre>
<!-- Author: JKY -->


<!-- Date: Fri Feb 15 15:26:10 2008 -->
<pre>Enstore CDF LT03 traffic is extremely busy for the last 
3 hours--the Ongoing transfer is pinned at 14 (one 
mover LT03_26 is down--no response from enstore-admin yet). 
The number of pending transfers peaked at 450 15 min. 
ago and is still above 400 --highest since about 
10 days ago. </pre>
<!-- Author: JKY -->


<!-- Date: Fri Feb 15 15:33:44 2008 -->
<pre>Status of "bad" tape IAD988 --from Stanley Hicks 
 
John, 
  I have some news on the "lost" tape. This volume was actually in its home cell in the library, but wasn't being read. The SUN tech and I finally got it out, but it didn't come out of the database and subsequently won't go back into the library right now. (It wouldn't even accept it physically into the library.) We now have to delete this volume from the library's db and then re-insert it into the robot. I don't think we will be able to do all this prior to the weekend. 
  I have given the tape to David and he will be looking into getting it back into production as soon as possible. 
 
Thanks for all your patience in this, 
Stanley </pre>
<!-- Author: JKY -->


<!-- Date: Fri Feb 15 15:59:33 2008 -->
<pre>LT03_26 fixed; IAB313 status to be determined early next week 
 
From David Berg : 
John, 
 
The mover is is fine. I will have to eject and inspect IAB313 to see if 
it has a bad label. If so, we will replace the label early next week. 
 
Thanks, 
David</pre>
<!-- Author: JKY -->


<!-- Date: Fri Feb 15 17:08:25 2008 -->
<pre>Status : 2/14/08 17:00  
 
Data Handling at a glance 
----------------------------------------------------------- 
   New Problems--not yet resolved 
 
  None, except heavy traffic on LT03 (see item (xx) below) 
         causing some reds on LT03 pending reads.  
------------------------------------------------------ 
Incidents within the last 24 hours 
 
  (i) One DCache Diskpool door not opening (probably happened 
	on 2/14 around 1830)--email sent to cdf_diskpool_admin 
The following doors are not responding: 
DOOR02   dcap://fcdfrdc3.fnal.gov:22127 
---email sent to cdf_diskpooladmin 0932 
---problem fixed around 1115 
 
  (ii) Enstore --bad mover LT03_26.mover 
 
alive : ERROR - ('1', "mount IAB313 failed: ('ERROR', 13, ['ACSSA&gt; mount  
IAB313 0,1,1,3', 'Mount: Mount failed, Unreadable label.',  
'ACSSA&gt; logoff', ''], '', 'MOUNT 13: mount IAB313 0,1,1,3 =&gt; 0, 
Mount: Mount failed, Unreadable label.')") 
 
--- new bad tape IAB313 (see error message above) 
---reported to enstore-admin 1235 
---1547 --LT03_26 is fixed by David Berg--they will find out 
	early next week whether IAB313 is okay or not 
 
  (iii) SAMDbServer.caf_srv4_prd:SAMDbServer (Using: samdbs@cdfofprd) 
                                                      cdfsam14.fnal.gov:46717  
   was yellow for a short time--but is now green 
   srv2, 3, 4 (cdfsam 12, 13,14) were all restarted today 2/15  
       between 2/15/08 1234 to 1245 
 
 
  (iv) fcdfdata135 is offline since 1450 
   --reported to helpdesk and run2-sys at 1519 
   --fixed around 1612 
 
  (xx) (not a problem --but we should be aware) 
  Heavy traffic on Enstore LT03--with pending transfers peaked at 450 around 1510 
  is now down to around 250 --this was the heaviest traffic in LT03 since 
  10 days ago. Note that this resulted in many red entries in the Enstore 
	CDF-LT02 tables--such as pending  General read, and stage read-- 
	these are presumably not serious and will go away when the demand 
	is reduced.  
 
--------------------------------------------------------------- 
  Old remaining problems 
   
  (1) Upload server fcdfdata321--1 failed job/1 error 
 
09:00:07 20080213 1202914807 
Unable to get the metadata structure for file /data/top3-131/submitted/pw02eae8.0001skt3  
     --- [Errno 2] No such file or directory: /data/top3-131/submitted/pw02eae8.0001skt3 
 
  (2) DCache 
   No of cells inactive --4  
r-fcdfdata320-1 
r-fcdfdata320-2 
r-fcdfdata320-3 
r-fcdfdata320-4 
 
 (3) Enstore 
 
CDF-9940B --1 bad mover 
9940B27 (being used for tape test by GS--since 2/13 PM)l 
 
IA1853 (stuck in drive--sent to vendor for recovery--all files marked bad) 
   
IAD988              0.44GB   (NOACCESS   0208-1938 full     0122-0411)   CDF-LTO3         cdf.gphysr.cpio_odc                 
 NO OTHER RED PROBLEMS 
--email from Stanley Hicks re IAD988 2/15/08 1526--John, 
  I have some news on the "lost" tape. This volume was actually in its home  
cell in the library, but wasn't being read. The SUN tech and I finally got  
it out, but it didn't come out of the database and subsequently won't go  
back into the library right now. (It wouldn't even accept it physically  
into the library.) We now have to delete this volume from the library's  
db and then re-insert it into the robot. I don't think we will be able  
to do all this prior to the weekend. 
  I have given the tape to David and he will be looking into getting it  
back into production as soon as possible. 
 
Thanks for all your patience in this, 
Stanley  
 
----------------- 
sam AT A GLANCE 
 
All clickable SAMStations and SAM DbServers have green button 
(except for None)</pre>
<!-- Author: JKY -->


<!-- Date: Sat Feb 16 10:28:47 2008 -->
<pre>No new problems</pre>
<!-- Author: JKY -->


<!-- Date: Sun Feb 17 12:05:16 2008 -->
<pre>It's eight bells, and all's quiet. 
 
One additional 9940B11.mover can not updating 
status--but since the demand is low--sent  
email to enstore-admin to look into it tomorrow.  
</pre>
<!-- Author: JKY -->


<!-- Date: Mon Feb 18 11:15:54 2008 -->
<pre>Some Enstore issues--including summary of unresolved issues 
(email sent ot enstore-admin 
 
List of enstore issues unresolved : 
 
(1) 940B11.mover	dead	cdfenmvr11a	 
     2008-Feb-18 10:59:25	2008-Feb-17 12:22:55 
 
(9940B11 has been down since Sunday AM--I've already send 
an email then --please let me know the status) 
 
(2) 	yelball.gif	CDF-9940B.library_manager 
 
is in yellow status for the last several hours; what is 
the problem ?? 
 
------------------------------------------------- 
 
(Please keep me updated on the following tapes--I enclose the  
previous responses).  
 
(3) IAB313              1.23GB   (NOACCESS   0215-0815 full     
     0424-0246)   CDF-LTO3         cdf.cdfpewk.cpio_odc        
 
John, 
 
The mover is is fine. I will have to eject and inspect IAB313 to see if 
it has a bad label. If so, we will replace the label early next week. 
 
Thanks, 
David Berg (2/15/08 1547) 
 
 
 
(4)  IAD988              0.44GB   (NOACCESS   0208-1938 full      
0122-0411)   CDF-LTO3         cdf.gphysr.cpio_odc    
 
John, 
  I have some news on the "lost" tape. This volume was actually in its home cell in the library, but wasn't being read. The SUN tech and I finally got it out, but it didn't come out of the database and subsequently won't go back into the library right now. (It wouldn't even accept it physically into the library.) We now have to delete this volume from the library's db and then re-insert it into the robot. I don't think we will be able to do all this prior to the weekend. 
  I have given the tape to David and he will be looking into getting it back into production as soon as possible. 
 
Thanks for all your patience in this, 
Stanley Hicks  (2/15/08 1526) 
 
(5) IA6978 --if the vendor responded and this 'mangled" tape, let me know  
 
      </pre>
<!-- Author: JKY -->
<!-- Comment: Mon Feb 18 14;14;24 comment by...JKY -->
<pre>update on problems (1) and (2)

(1) 	9940B11.mover
(NIC failure, opened helpdesk ticket)

(2) 	greenball.gif	CDF-9940B.library_manager
(they apparantly fixed what was making it yellow--
or the problem went away)

No update on the 3 tape problems as yet</pre>


<!-- Date: Mon Feb 18 16:04:02 2008 -->
<pre>Some of the Dcache plots hae gaps--happened this morning 
around 2:45-3:35 AM, and also earlier (e.g., Thursday 8PM). 
emailed dcache-admin to inquire.(see plots below)</pre>
<!-- Author: JKY -->


<!-- Date: Mon Feb 18 16:05:03 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-45&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-45  width='300' align='top'></a></td><td>gaps in Dcache monitoring plots</td></tr></table>
<!-- Author: JKy -->


<!-- Date: Mon Feb 18 16:05:41 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-47&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-47  width='300' align='top'></a></td><td>gap</td></tr></table>
<!-- Author: JKY -->


<!-- Date: Mon Feb 18 16:06:26 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-48&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-48  width='300' align='top'></a></td><td>more on gap</td></tr></table>
<!-- Author: JKY -->


<!-- Date: Mon Feb 18 16:49:46 2008 -->
<pre>Update in tapes IAD988 and IAB313 
 
    ...IAD988              0.44GB   (NOACCESS   0208-1938 full      
           0122-0411)   CDF-LTO3         cdf.gphysr.cpio_odc    
           latest update : IAD988  This volume has been  
           deleted from the library  
           database and put back into the system. You should be  
           able to access your data now. [Stanley Hicks,  
           2/18 1611] 
  
 
    ...IAB313              1.23GB   (NOACCESS   0215-0815 full      
        0424-0246)   CDF-LTO3         cdf.cdfpewk.cpio_odc   
  
         latest update : IAB313  This tape has been removed  
         from the library and a new  
         label will be printed and put on it for re-insertion 
         into the library - most probably  
         tomorrow.[Stanley Hicks, 2/18 1611] 
</pre>
<!-- Author: JKY -->


<!-- Date: Tue Feb 19 09:07:33 2008 -->
<pre> 
Subject: CDF Sam Shift Report 
From: John Yoh &lt;johny@fnal.gov&gt; 
Date: Mon,  Feb 19, 2008 0900 
To: sam-design@fnal.gov 
 
o Known and continuing problems 
  fcdfdata320 offline 
 
o Problems that were fixed during the last week 
   ...fcdfdata111 down 2/11/08 1525 to 2/12/08 1100 
   ,,,fcdfdata135 offline since 2/14 1450-fixed around 1612  
 
o One DCache Diskpool door not opening (happened  
	on 2/14 around 1830)--email sent to cdf_diskpool_admin  
    The following doors are not responding:  
     DOOR02   dcap://fcdfrdc3.fnal.gov:22127  
    back up at 2/15 1120 
 
o Issues added after last meeting 
  ..IA1853 (see below) --tape stuck in drive--sent to vendor for recovery 
	about 150 files on this tape (mostly muo files) have been marked 
	bad on the DB 
 
  ..gaps in dcache monitoring plots --see elog--entry Mon feb 18 16:06 
	gap Monday 2/18 0245-0335, and Thursday 2/14 around 8PM. 
 
  ..problem with CNAF SAM jobs 
     (AC++ executable is crashing, cnaf group needs to debug  
      it as one can not even get the core files from their worker nodes) 
 
  [email from users with failing jobs --from Vataga, etc.starting 2/13/08 
  [Lots of follow-up email from Donatella, Bob illingworth, Doug B, K Genser, 
	Thomas Kuhr, etc.] 
  I enclose here a snippet of the latest email from Doug today 2/18 at 0559 
  [Dear all, 
 
    Here is an update from my work on CNAF over the weekend. 
 
     I used the ldd command to see what libraries are used by AC++Dump on a CNAF 
     worker node and on a Fermigrid worker node.  I also checked my desktop 
     at FNAL as it uses a similar Kernel (32 bit). I could not see any significant 
     differences. (there was one difference on the Fermigrid worker nodes 
      library -  linux-gate.so.1 =&gt;  (0xffffe000) ) 
 
     I have put the output from both jobs in the directory /cdf/home/benjamin/CNAF for 
     anyone to look at. In both cases I run offline version 6.1.4 and dataset bpel0i. 
 
      Here is a snippet of the log file that includes error that is returned when my Job 
      runs at CNAF. ........... 
 
    Though Elena is still complaining about failing jobs at 0803] 
 
 
----------------------------------------------------------------------   
o Bad movers and Tapes  
  Fixed -- 
    ...LT03_26  down Friday 2/15 afternoon for a couple of hours 
	due to a stuck tape IAB313 (see below) 
    ...IAD988              0.44GB   (NOACCESS   0208-1938 full      
           0122-0411)   CDF-LTO3         cdf.gphysr.cpio_odc    
           latest update : IAD988  This volume has been deleted from the library  
           database and put back into the system. You should be  
           able to access your data now. [Stanley Hicks, 2/18 1611] 
 
   ----------------------------------------------- 
  Outstanding unresolved issues (to be followed up)-- 
    ...9940B11.mover (down since Sunday AM)-- 
             (NIC failure, opened helpdesk ticket) 
    ...IAB313              1.23GB   (NOACCESS   0215-0815 full      
        0424-0246)   CDF-LTO3         cdf.cdfpewk.cpio_odc   
  
         latest update : IAB313  This tape has been removed from the library and a new  
         label will be printed and put on it for re-insertion into the  
         library - most probably tomorrow.[Stanley Hicks, 2/18 1611] 
      
   ...IA1853              0.23GB   (NOTALLOWED 0201-0820 full     1227-2115)   CDF-9940B        cdf.xbmu0i.cpio_odc    
                       Vol stuck in drive DBT2MV - unable to dismount 
       Tape sent to vendor for recovery--no estimate 
  -------------------------------------------- 
  To be noted : 
    ...9940B27.mover offline for tape test by GS since early last week 
 
 
John Yoh, johny@fnaol.gov, x4774, </pre>
<!-- Author: JKY -->
<!-- Comment: Wed Feb 20 14;14;06 comment by...JKY -->
<pre>David Berg just send me email regarding IAB313

John,

IAB313 - label replaced, tape is available

 - David
</pre>