<!-- Title: 13:45:00  Tue Feb 20 2007 -->
<!-- Crewchief: Rick St. Denis -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>On shift in GMT -- All looks well this evening, although the tape queue is a bit long.</pre>


<!-- Date: Wed Feb 21 04:36:08 2007 -->
<pre>Tape queue looks fine -- as does all else.</pre>
<!-- Author: Rick -->


<!-- Date: Wed Feb 21 10:52:36 2007 -->
<pre>Request from Sam (Harper) to SAM: 
 
Date: Wed, 21 Feb 2007 16:37:10 +0000 
From: Sam Harper &lt;S.Harper1@physics.ox.ac.uk&gt; 
To: Raymond Culbertson &lt;rlc@fnal.gov&gt;, cdfdh_oper@fnal.gov 
Cc: harper@fnal.gov 
Subject: delete 10 
 
Hi, 
 
In the process of cleaning up the concatenated files, these files are found 
to be bad. Please could you delete them at your earilest convience. 
 
gh030090.0317jt2a 
gi031d98.0001jt2a 
gi031d98.0240jt2a 
gi0322ec.0022jt2a 
gi031d57.0372jt1a 
gi031d98.0076jt1a 
gi031d98.0144jt1a 
gi031d98.026ajt1a 
gi031d98.0001jt3a 
gi031d98.022cjt3a 
 
Thanks, 
Sam 
 
</pre>
<!-- Author: Rick -->
<!-- Comment: Wed Feb 21 11;15;27 comment by...R -->
<pre>Date: Wed, 21 Feb 2007 10:57:34 -0600
From: Randolph J. Herber &lt;herber@fnal.gov&gt;
To: S.Harper1@physics.ox.ac.uk, rlc@fnal.gov, cdfdh_oper@fnal.gov
Cc: harper@fnal.gov
Subject: Re: delete 10

The SAM removals have been done.
</pre>
<!-- Comment: Wed Feb 21 11;16;43 comment by...r -->
<pre>Date: Wed, 21 Feb 2007 11:09:24 -0600
From: Raymond Culbertson &lt;rlc@fnal.gov&gt;
To: Randolph J. Herber &lt;herber@fnal.gov&gt;
Cc: S.Harper1@physics.ox.ac.uk, cdfdh_oper@fnal.gov, harper@fnal.gov
Subject: Re: delete 10

Hi,
    The last couple of deletes it seems you have not done the enstore
delete.
when I thought you did do that before.  I happen to have the locations 
for these so I could rm them, but I see some of them have other locations,
I don't know how to delete those.  Could you?  Or someone could show me 
how to 1) find files in enstore efficeinty without "sam locate" and 2)
delete
non-pnfs sam locations.  Thanks.
                                             Ray

 Ray Culbertson  CDF/FNAL  rlc@fnal.gov  630-840-6744  Trailers 170F (MS318)


</pre>
<!-- Comment: Wed Feb 21 15;47;18 comment by...r -->
<pre>Date: Wed, 21 Feb 2007 12:04:09 -0600
From: Randolph J. Herber &lt;herber@fnal.gov&gt;
To: rlc@fnal.gov
Cc: S.Harper1@physics.ox.ac.uk, harper@fnal.gov, cdfdh_oper@fnal.gov
Subject: Re: delete 10

No, generally I can not delete the files because I am not user 'sam'.

E.g.,

$ chkpath /pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.6/gi031d98.022cjt3a
      2 drwxr-xr-x   23 root     root         4096 Oct  4 09:45 /
 426924 drwxr-xr-x    5 root     root         4096 Dec  6  2005 /pnfs
16781408 drwxrwxrwx    1 root     root          512 Sep 12 13:41 /pnfs/cdfen
16959360 drwxrwxr-x    1 cdf      cdfodhgd      512 Nov 22 16:09
/pnfs/cdfen/filesets
419434592 drwxrwxr-x    1 sam      cdf           512 Feb  2  2006
/pnfs/cdfen/filesets/NS
419434672 drwxr-xr-x    1 sam      ods           512 Feb 19 16:54
/pnfs/cdfen/filesets/NS/NS00
421096872 drwxr-xr-x    1 sam      9626          512 Sep  4 12:00
/pnfs/cdfen/filesets/NS/NS00/NS0028
421127512 drwxrwxr-x    1 sam      9626          512 Sep  3 19:43
/pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.6
421129728 -rw-r--r--    1 sam      cdf      1231134997 Sep  3 17:10
/pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.6/gi031d98.022cjt3a

Dehong can do these.

I do not believe that either Dehong or I can do anything about the
off-site locations.

Randolph J. Herber, herber@fnal.gov, +1 630 840 2966, CD/CDFTF PK-149F,
Mail Stop 318, Fermilab, Kirk & Pine Rds., PO Box 500, Batavia, IL
60510-0500,
USA.  (Speaking for myself and not for US, US DOE, FNAL nor URA.)  (Product,
trade, or service marks herein belong to their respective owners.)
</pre>
<!-- Comment: Wed Feb 21 15;47;48 comment by...r -->
<pre>Hi Ray,

I found the last 2
&gt; &gt; | gi031d98.0001jt3a
&gt; &gt; | gi031d98.022cjt3a
in /pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.6, and deleted them.
Do you see more?

Best regards,
Dehong

</pre>
<!-- Comment: Wed Feb 21 15;48;16 comment by...r -->
<pre>Date: Wed, 21 Feb 2007 13:09:09 -0600
From: Randolph J. Herber &lt;herber@fnal.gov&gt;
To: rlc@fnal.gov, dhzhang@fnal.gov
Cc: S.Harper1@physics.ox.ac.uk, harper@fnal.gov, cdfdh_oper@fnal.gov
Subject: Re: delete 10

/pnfs/cdfen/filesets/NS/NS00/NS0027/NS0027.7/gh030090.0317jt2a was not
deleted.
/pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.8/gi031d98.0001jt2a was not
deleted.
/pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.8/gi031d98.0240jt2a was not
deleted.
/pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.7/gi0322ec.0022jt2a was not
deleted.
/pnfs/cdfen/filesets/NS/NS00/NS0027/NS0027.7/gi031d57.0372jt1a was not
deleted.
/pnfs/cdfen/filesets/NS/NS00/NS0027/NS0027.7/gi031d98.0076jt1a was not
deleted.
/pnfs/cdfen/filesets/NS/NS00/NS0027/NS0027.7/gi031d98.0144jt1a was not
deleted.
/pnfs/cdfen/filesets/NS/NS00/NS0027/NS0027.7/gi031d98.026ajt1a was not
deleted.
/pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.6/gi031d98.0001jt3a was not
deleted.
/pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.6/gi031d98.022cjt3a was not
deleted.


Randolph J. Herber, herber@fnal.gov, +1 630 840 2966, CD/CDFTF PK-149F,
Mail Stop 318, Fermilab, Kirk & Pine Rds., PO Box 500, Batavia, IL
60510-0500,
USA.  (Speaking for myself and not for US, US DOE, FNAL nor URA.)  (Product,
trade, or service marks herein belong to their respective owners.)

</pre>
<!-- Comment: Wed Feb 21 15;48;44 comment by...r -->
<pre>
  The following shows all the others are still there. Clearly we need to 
get organized on this!  Maybe Dehong can delete them in enstore first, 
so he can sam locate them, then Randy can delete them in SAM.
Thanks.
                                     Ray

ls -lh /pnfs/cdfen/filesets/NS/NS00/NS0027/NS0027.7/gh030090.0317jt2a
ls -lh /pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.8/gi031d98.0001jt2a
ls -lh /pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.8/gi031d98.0240jt2a
ls -lh /pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.7/gi0322ec.0022jt2a
ls -lh /pnfs/cdfen/filesets/NS/NS00/NS0027/NS0027.7/gi031d57.0372jt1a
ls -lh /pnfs/cdfen/filesets/NS/NS00/NS0027/NS0027.7/gi031d98.0076jt1a
ls -lh /pnfs/cdfen/filesets/NS/NS00/NS0027/NS0027.7/gi031d98.0144jt1a
ls -lh /pnfs/cdfen/filesets/NS/NS00/NS0027/NS0027.7/gi031d98.026ajt1a
ls -lh /pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.6/gi031d98.0001jt3a
ls -lh /pnfs/cdfen/filesets/NS/NS00/NS0028/NS0028.6/gi031d98.022cjt3a

 Ray Culbertson  CDF/FNAL  rlc@fnal.gov  630-840-6744  Trailers 170F (MS318)


</pre>
<!-- Comment: Wed Feb 21 15;49;13 comment by...r -->
<pre>Date: Wed, 21 Feb 2007 13:17:14 -0600 (CST)
From: Dehong_Zhang &lt;dhzhang@fnal.gov&gt;
To: Raymond Culbertson &lt;rlc@fnal.gov&gt;
Cc: Randolph J. Herber &lt;herber@fnal.gov&gt;, cdfdh_oper@fnal.gov,
    harper@fnal.gov
Subject: Re: delete 10

Hi Ray,

All the 10 pnfs entries are gone now.

Best regards,
Dehong




</pre>
<!-- Comment: Wed Feb 21 15;49;48 comment by...r -->
<pre>Date: Wed, 21 Feb 2007 13:29:57 -0600
From: Angela Bellavance &lt;bellavan@fnal.gov&gt;
To: Raymond Culbertson &lt;rlc@fnal.gov&gt;
Cc: Dehong_Zhang &lt;dhzhang@fnal.gov&gt;, Randolph J. Herber &lt;herber@fnal.gov&gt;,
    cdfdh_oper@fnal.gov, harper@fnal.gov
Subject: Re: delete 10

Any SAM shifter should be able to do what Dehong did since they log in to
various machines as "sam". Including "cdfsam_help" in your "To" email should
include the shifter.

- Angela

</pre>
<!-- Comment: Wed Feb 21 15;50;20 comment by...r -->
<pre>Date: Wed, 21 Feb 2007 14:44:45 -0600
From: Krzysztof Genser &lt;genser@fnal.gov&gt;
To: Angela Bellavance &lt;bellavan@fnal.gov&gt;
Cc: Raymond Culbertson &lt;rlc@fnal.gov&gt;, Dehong_Zhang &lt;dhzhang@fnal.gov&gt;,
    Randolph J. Herber &lt;herber@fnal.gov&gt;, cdfdh_oper@fnal.gov,
    harper@fnal.gov
Subject: Re: delete 10

Small correction to the second sentence:
"cdfsam_help" was set up to let "power users" give advice to people from
their specific physics group. cdfdh_oper was/is the list read by the
shifters.
BTW: The procedure written/used by Randolph when run as sam on a cdfsam..
node with properly mounted /pnfs will remove the files from pnfs.
Krzysztof
</pre>
<!-- Date: Wed Feb 21 10:58:33 2007 -->
<pre>All looks good</pre>
<!-- Author: Rick -->




<!-- Date: Wed Feb 21 11:50:41 2007 -->
<pre>While looking at why a test job hung noted that it wrote: 
 
02/21/07 05:54:58   sam_cpp_api.ConsumerProcess 29235: ConsumerProcess::getNextFile(): Requesting next file for CPID 3525795 
02/21/07 05:54:58   sam_cpp_api.ConsumerProcessInterface_impl 29235: ConsumerProcessInterface_impl::getNextFile(): Method called 
02/21/07 05:54:58   sam_cpp_api.ConsumerProcessInterface_impl 29235: ConsumerProcessInterface_impl::getNextFile(): No files available, waiting... 
 
then noted several core files on cdfsam02: 
 
-rw-------    1 sam      ods       1986560 Feb 11 16:21 core.7346 
-rw-------    1 sam      ods       1986560 Feb 12 13:03 core.21936 
-rw-------    1 sam      ods       1978368 Feb 12 20:51 core.7695 
-rw-------    1 sam      ods       2420736 Feb 21 05:14 core.21509 
-rw-------    1 sam      ods       6336512 Feb 21 05:16 core.24087 
-rw-------    1 sam      ods      14041088 Feb 21 05:20 core.24181 
-rw-------    1 sam      ods       2236416 Feb 21 05:40 core.19443 
-rw-------    1 sam      ods       2277376 Feb 21 06:01 core.23732 
 
one of them corresponding to the job: 
 
[sam@cdfsam02 station__cdfsam02__cdfcafstation_prd__cdf-caf]$ gdb -c core.23732 
GNU gdb Red Hat Linux (6.1post-1.20040607.52rh) 
Copyright 2004 Free Software Foundation, Inc. 
GDB is free software, covered by the GNU General Public License, and you are 
welcome to change it and/or distribute copies of it under certain conditions. 
Type "show copying" to see the conditions. 
There is absolutely no warranty for GDB.  Type "show warranty" for details. 
This GDB was configured as "i386-redhat-linux-gnu". 
Core was generated by `pmaster --name=genser_cdf-caf_dhtestbhmu0hlt195000_222055_2022 --log-file=stati'. 
Program terminated with signal 6, Aborted. 
#0  0xb6fbecdf in ?? () 
 
searching for the project yielded: 
 
02/21/07 06:01:37 cdf-caf.genser_cdf-caf_dhtestbhmu0hlt195000_222055_2022.GPM 23732: Internal: Database Error:  max # extents (10000) reached in index SAMPRD.CONF_PK 
 
more cases like that found: 
 
02/21/07 05:16:08 cdf-caf.jdeng_cdf-caf_gjt40i_221664_10486.GPM 24087: Internal: Database Error:  max # extents (10000) reached in index SAMPRD.CONF_PK 
02/21/07 05:20:50 cdf-caf.mhartz_cdf-caf_mhartz_hbhd0d_15_221854_17885.GPM 24181: Internal: Database Error:  max # extents (10000) reached in index SAMPRD.CONF_PK 
... 
02/21/07 06:10:29 cdf-caf.cdf-caf_hauser_xbhd0d_20070220.142131.GPM 22235: Internal: Database Error:  max # extents (10000) reached in index SAMPRD.CONF_PK 
02/21/07 06:10:29 cdf-caf.cdf-caf_hauser_xbhd0d_20070220.142131.GPM 22235: Internal project master exception raised in getNextFileViaUdpCallbackInternal: Database Error:  max # extents (10000) reached in index SAMPRD.CONF_PK 
 
from the dba's: 
 
http://www-css.fnal.gov/dsg/external/cdfdbmtgs/2007/Feb21_2007.htm 
cdfofpr2 
      ORA-1632: max # extents 10000 reached in index SAMPRD.CONF_PK 
      Wed Feb 21 05:08:39 2007      --&gt; doubled extents  --&gt;  looks like it ate up 4 extents between 1:41am and 5:01am.  
        
</pre>
<!-- Author: Krzysztof -->









<!-- Date: Thu Feb 22 02:01:33 2007 -->
<pre>All looks ok - SAG, DHAG, SAMTV 
The only oddity is the test jobs seem not to have entries since 13:41 on the 21st. Did Krzysztof stop them?</pre>
<!-- Author: r -->


<!-- Date: Thu Feb 22 16:59:01 2007 -->
<pre>cdfcafstation_prd station died at cdf-caf: status 134 
Thu, 22 Feb 2007 12:44:32 -0600 
https://plone3.fnal.gov/SAMGrid/tracking/2607 
</pre>
<!-- Author: Krzysztof -->


<!-- Date: Sat Feb 24 06:44:57 2007 -->
<pre>Warning on CSL -- indication that experts notified: 
Sat Feb 24 06:35:48 2007 1172320548 Warning: 
b0csl21 -- Files remain in .open for too long 
b0csl22 -- Files remain in .open for too long 
b0csl23 -- Files remain in .open for too long 
b0csl23 -- Csl_fetch encountered problems 
b0csl24 -- Files remain in .open for too long 
b0csl24 -- Csl_fetch encountered problems 
b0csl25 -- Files remain in .open for too long 
b0csl25 -- Csl_fetch encountered problems 
b0csl26 -- Files remain in .open for too long 
b0csl26 -- Csl_fetch encountered problems 
b0csl27 -- Files remain in .open for too long 
b0csl27 -- Csl_fetch encountered problems 
b0csl28 -- Files remain in .open for too long 
b0csl28 -- Csl_fetch encountered problems 
b0csl29 -- Files remain in .open for too long 
b0csl29 -- Csl_fetch encountered problems</pre>
<!-- Author: r -->


<!-- Date: Sun Feb 25 17:15:24 2007 -->
<pre>All looks good</pre>
<!-- Author: r -->


<!-- Date: Mon Feb 26 03:58:01 2007 -->
<pre>                                                                                                     These seem a bit full.... 
 
 
Mon Feb 26 03:48:01 CST 2007: Beginning nightly purgeLogs.sh script on cdfsam01.fnal.gov, disk 
space at the moment: 
Filesystem            Size  Used Avail Use% Mounted on 
/dev/sdc1              68G   56G  8.7G  87% /home 
                                                                                                      
                                                                                                      
Mon Feb 26 03:48:06 CST 2007: Completed nightly purgeLogs.sh script on cdfsam01.fnal.gov, disk 
space now: 
Filesystem            Size  Used Avail Use% Mounted on 
/dev/sdc1              68G   53G   12G  83% /home 
                                                                                                      
                                                                                                      
</pre>
<!-- Author: r -->


<!-- Date: Tue Feb 27 05:09:52 2007 -->
<pre> User had problems with dataset bhelkd. Seems like dcache was not giving 
the file but I could stand corrected. Did a dccp just fine on one file so 
I suspect it should just be retried.  I am trying to submit a test job to 
check 
 
&gt; Sam2DiskCacheInterface_samapi::getNextFile-&gt; Request next file from 
SAM(R) on Mon Feb 26 08:09:01 2007 
&gt; 
&gt; Sam2DiskCacheInterface_samapi::getNextFile-&gt; Got file 
&gt; 
dcap://cdfdca2.fnal.gov:25154/pnfs/fnal.gov/usr/cdfen/filesets/NT/NT00/NT0003/NT0003.7/ 
&gt; bd028e7f.019bhelk from SAM(R) on Mon Feb 26 08:09:02 2007 
&gt; 
&gt; Sam2DiskCacheInterface_samapi::getNextFile-&gt; The check of file 
&gt; 
dcap://cdfdca2.fnal.gov:25154/pnfs/fnal.gov/usr/cdfen/filesets/NT/NT00/NT0003/ 
&gt; NT0003.7/bd028e7f.019bhelk failed. Skipping it. 
&gt; Sam2DiskCacheInterface::releaseFile-&gt; release file 
dcap://cdfdca2.fnal.gov:25154/pnfs/fnal.gov/usr/cdfen/filesets/NT/ 
&gt; NT00/NT0003/NT0003.7/bd0 
&gt; 28e7f.019bhelk with status 'not processed'. Potential data loss. 
&gt; Sam2DiskCacheInterface_samapi::releaseFile: releasing 
&gt; 
dcap://cdfdca2.fnal.gov:25154/pnfs/fnal.gov/usr/cdfen/filesets/NT/NT00/NT0003/NT0003.7/ 
 
&gt; bd028e7f.019bhelk with status 'NOT processed' succeeded with consumer 
process id: 3555465 from SAM(R) on Mon Feb 26 
&gt; 08:21:03 2007 
&gt; 
&gt; Sam2DiskCacheInterface_samapi::getNextFile-&gt; Request next file from 
SAM(R) on Mon Feb 26 08:21:03 2007 
&gt; 
</pre>
<!-- Author: r -->


<!-- Date: Wed Feb 28 09:05:10 2007 -->
<pre>SAM upload problem: Unable to get the metadata structure for file /export/data2/cdfmc/submitted/kc0228e8.0001hgp1 --- Unable to get the description for dataset khgp1c, please register it 
The affected group is informed 
two dCache doors: door27 and door29 are not responding, dcache admin team is working on it. door27 was alright after restart, door29 not, might be a malformed SQL query which causes the trouble. 
Problem with some files in the datasets: nhgp1c or 
&gt; pagan-test_hbb-single_file-1 or pagan-test_hbb-single_file-2, tested and getting errors connected to CdfTauCollection, CdfTau::IdAlgorithm =&gt; might be a corrupted file or some software not being in sync, cause not known</pre>
<!-- Author: Valeria -->