<!-- Title: 10:15:42  Wed Sep 12 2007 -->
<!-- Crewchief: stdenis -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Day after start of shift: logbook down on first day</pre>


<!-- Date: Wed Sep 12 10:17:06 2007 -->
<pre>Could not make entries on Sept 11: upgrade of server. 
Things that were observed/reported follow</pre>
<!-- Author: r -->


<!-- Date: Wed Sep 12 10:18:31 2007 -->
<pre>The section of DH-at-a-glance having to do with the upload server is something that I usually ignore since the errors are almost always due to a failure of some MC producer to get the metadata syntax right and eventually they realize that there is a problem  when the users are missing files so they get fixed at some level. 
 
Krzysztof asked me to contact Rob Harr and Rob responded as follows: 
                                                                                 
Many of the files are top group, so I contacted their MC rep.  He'll point 
out the problem to the owner (Yen-Chu).  There are only 2 things we can do 
with the files: 
1.  Move them off the SAM machines to a location where the owner can look at 
them, or 
2.  delete them. 
                                                                                 
I asked that he let us know which action to take. 
                                                                                 
The solution is to having these files pile up on the SAM machines is to 
automatically copy them back to the source machine and push the problem back 
to the owner.  I will implement this, but first want to eliminate the use of 
fcdfdata134 as an intermediate machine for the files.  Moving files back to 
fcdfdata134 doesn't solve the problem, since it is not the original buffer 
location used by the owner. 
                                                                                 
 
</pre>
<!-- Author: r -->


<!-- Date: Wed Sep 12 10:24:12 2007 -->
<pre>09/11/07 01:22:00 cdf-fncdfsrv0.SM.FileUncachedInDBTimer 17501: Warning, will ignore CachedFileNotFound 
exception for CFID 24115562: 
Cached file with id '24115562' not found. 
 
This problem went away. There were a number of exchanges: 
 
Ray said: 
"                                                                                                                    
  This was probably caused by the failure of Elena and me to coordinate on 
the deletion of duplicate files.  Even though we agreed to coordinate 
on this, I have been running on e stream, and I suspect Elena deleted files 
I was requesting.  Please clean up the station, and feel free to stop any 
ntupling jobs necessary ("stntuple..." or "topntuple...").  Currently 
no e stream ntupling jobs are running according to caf and I won't 
submit any more until clear." 
 
 
I wrote back explaining that this was discussed at the sam meeting: 
 
   This was discussed at the sam meeting.  At present, any time a file is deleted from the database, all stations referencing the file in cache need to be restarted.  The script you use of course removes the information on caching from the db, so it is hit and miss to find these. 
  Randolph has been given a command that will do this more elegantly but he needs to update the script to handle this. 
 
   The bottom line is that the farm station needs a restart.  Don't know if Thomas already did this but I thought we might want to coordinate this. 
 
 
Ray then asked for further explanation: 
 
  I don't quite understand the practical implication of phrases like 
"stations referencing the file in cache".  I'm not sure if you 
are discussing cleaning up from a bad reference or how to 
delete a file. 
    In the last offline meeting, we thought it was sufficient to check that 
no running jobs are requesting a file before deleting it.   We then delete 
with "rm" in /pnfs and "sam undeclare". 
   Is this sufficient, or are you saying we need  a new procedure 
to delete files?  Thanks. 
 
To which I added further clarification 
 
 
   I am talking about deleting a file. 
 
 At the sam meeting Robert pointed out that  
samadmin remove station file 
 has to be executed. 
 So definitely, a new procedure is needed and Randolph promised to add this to his script. 
 
 The station keeps track of files and if it has one in its cache then it needs to be told to clean up its bookkeeping. It can be done with the command above or, it will do so whenever the station is (re)started. 
 
 Robert did not get specific of how this would manifest itself other than 
 "Sam stations are not designed to have files taken out from under them". 
 
I gather that a bad reference is when the station thinks it has a file to keep track of in cache and indeed, may physically have the file, but that the db no longer has the file. 
 
</pre>
<!-- Author: r -->
<!-- Comment: Wed Sep 12 10;30;06 comment by...r -->
<pre>station_prd station died at cdf-fncdfsrv0: status 134

this probably means the machine was restarted  Tue, 11 Sep 2007 13:07:57 -0500
</pre>
<!-- Comment: Wed Sep 12 10;31;34 comment by...r -->
<pre>Ray closed this thread with

    Thank you for clarifying.  We have not run this command at all
as far as I know and it looks like a user can't.   For now, I would like to proceed
by simply trying to avoid referring to a deleted file.  I believe I am not
running on any deleted files, but I am recovering other corrections in
those troubled datasets now.   Please watch the sam station in case
I made a mistake, or let me know how I should do it.  Thanks.
                                Ray



However, there is a pending issue of the release of a new delete script from Randolph</pre>
<!-- Comment: Wed Sep 12 10;32;22 comment by...r -->
<pre>Comment from Randolph

The actual problem is the file removal program only cleans up the data
base and the PNFS directory.  The PNFS directory cleanup has the indirect
effect of deleting the file from the tape later.  The cached copies of
files are NOT REMOVED from station caches by the program.  There is a
samadmin remove station file for that purpose; but, as of yet, the file
removal command does not issue the command.  After the file removal
command has executed, it is difficult to issue those commands as the
necessary data is gone from the database.
</pre>
<!-- Date: Wed Sep 12 10:26:48 2007 -->
<pre>There was a warning on DH-AAG that there were only 85 9940B tapes. Sent mail to enstore-admin and they said there were none allocated to CDF.  Krzysztof pointed out that this warning is spurious as we don't use these tapes any longer.  (Privately he thought the last ones were used for Period 10 processing). So he pointed out that this warning should be removed.</pre>
<!-- Author: r -->


<!-- Date: Wed Sep 12 10:27:23 2007 -->
<pre>                                                                                                                    
Mailed enstore-admin: 
 
There is a  bad tape  IAD276 (NOACCESS) 
</pre>
<!-- Author: r -->


<!-- Date: Wed Sep 12 10:28:58 2007 -->
<pre> These cells are offline  
&gt; r-fcdfdata138-1 
&gt; r-fcdfdata138-2 
&gt; r-fcdfdata138-3 
&gt; r-fcdfdata138-4 
 
From sept 4: 
There is a replacement motherboard on order for fcdfdata138. 
                                                                                                                    
The machine is down until the motherboard can be replaced. 
 
 
So these will be in the red until the replacement is made. 
</pre>
<!-- Author: r -->
<!-- Comment: Thu Sep 13 14;33;12 comment by...r -->
<pre>                                                                                                                    Thu, 13 Sep 2007 14:20:40 -0500
From: Etta Burns
fcdfdata138 is up.  It did have it's motherboard replaced.

And the cells offline error is gone.</pre>
<!-- Comment: Thu Sep 13 15;28;22 comment by...r -->
<pre>The machine is in the TestPools group and dcache/java may be upgraded on it to the software version all other
nodes were upgraded to during the downtime, hyperthreding checked, etc...
We can discuss it offline if needed, thanks,
</pre>
<!-- Date: Wed Sep 12 10:33:13 2007 -->
<pre>                                                                                                                   (sent at 1am, 12-sep-07) 
 
Can you  please investigate the following bad movers 
 
9940B21 
                                                                                                                    
LTO3_19 (which still has the bad tape IAD276) 
</pre>
<!-- Author: r -->


<!-- Date: Thu Sep 13 10:43:46 2007 -->
<pre>Error last night that the db access time was slow. 
(could not access this log file from home last night to log error) 
 
Sent mail to krzysztof who said that this has been asked of dbas and they respond it is sam.</pre>
<!-- Author: r -->


<!-- Date: Thu Sep 13 11:07:13 2007 -->
<pre>flood of mail messages this morning.  
 
Subject: ALERT from cdf-caf.SM 
                                                                                                                     
09/13/07 09:02:56 cdf-caf.SM.StationRequestFactory 7580: Cannot create request for file 5151737 
Does it have a valid location? 
                                                                                                                     
This was due to Donatella submitting a job to cdf-caf that had a dataset accessible only at cnaf. 
 
  I have traced a dataset whose files flooded my mailbox with errors 
starting at 8:38 to a project you ran called 
09/13/07 08:37:32 cdf-caf.lucchesi_cdf-caf_skbh01_239690_27164.GPM 27884: Notified the Station Master OK 
                                                                                                                     
I got messages, and still get messages like: 
                                                                                                                     
I took one and did a sam get metatdata and get 
[stdenis@ppecdf42 stdenis]$ sam get metadata --fileId=5495964 | grep 
fileName 
             'fileName' : 's102bdbb.01c1kbh0', 
[stdenis@ppecdf42 stdenis]$ sam get metadata --fileId=5495964 | grep 
dataset 
               'params' : Params({'cdf' : 
CaseInsensitiveDictionary({'dataset' : 'skbh01'})}), 
[stdenis@ppecdf42 stdenis]$ 
                                                                                                                     
                                                                                                                     
[stdenis@ppecdf42 stdenis]$ sam locate s102bdbb.01c1kbh0 
['cdfsam.cnaf.infn.it:/storage/gpfs_cdf3/durable' 
'cdfsam.cnaf.infn.it:/storage/gpfs_cdf7/sam/cache21//boo'] 
                                                                                                                     
                                                                                                                     
I gather that you did not intend for the job to run on caf  
 
here is what the sam station log says (over and over) 
09/13/07 08:38:33 cdf-caf.lucchesi_cdf-caf_skbh01_239690_27164.GPM 27884: 
Umers interested in file 4922713: (602611) 
09/13/07 08:38:33 cdf-caf.lucchesi_cdf-caf_skbh01_239690_27164.GPM 27884: 
No waiting processes 
09/13/07 08:38:33 cdf-caf.SM.FileReceiverProjectProxy 7580: Project 608707 
will not be able to use file s1027b47.0599kbh0 
09/13/07 08:38:33 cdf-caf.SM.FileReceiverProjectProxy 7580: Trying to give 
file s1027ab0.0056kbh0 to project 608707 
09/13/07 08:38:33 cdf-caf.lucchesi_cdf-caf_skbh01_239690_27164.GPM 27884: 
s1027ab0.0056kbh0 did not arrive anywhere, reason: Delivery Error 
 
The project is stopped but the errors keep coming. 
                                                                                                                     
</pre>
<!-- Author: r -->
<!-- Comment: Thu Sep 13 11;08;43 comment by...r -->
<pre>Gabriele Compestella sent email that was related to the above.
&gt; &gt; &gt; Dear All,
&gt; &gt; &gt;
&gt; &gt; &gt; We are experiencing a strange problem when trying to run AC++ jobs on
&gt; &gt; &gt; SAM
&gt; &gt; &gt; datasets at CNAF, hope you may have some suggestions on how
&gt; &gt; &gt; to deal with it.
&gt; &gt; &gt; Maybe it's a simple problem, but we haven't been able to debug it, I
&gt; &gt; &gt; think we
&gt; &gt; &gt; are missing something...
&gt; &gt; &gt;
&gt; &gt; &gt; Here's what happens: if we run a simple root ntuple squeezer on a SAM
&gt; &gt; &gt; dataset, accessing files through TSam Root interface, everything
&gt; &gt; &gt; works
&gt; &gt; &gt; fine as expected.
&gt; &gt; &gt; Instead, if we try running an AC++Dump job with a DHinput module
&gt; &gt; &gt; configured to talk to SAM, the job core dumps: we tried setting up
&gt; &gt; &gt; different cdfsoft versions and different sam queues, submitting as
&gt; &gt; &gt; different users, but everytime we have the same failure.
&gt; &gt; &gt; We tried to submit exactly the same job on the GroupCAF and there it
&gt; &gt; &gt; works as
&gt; &gt; &gt; it should.
&gt; &gt; &gt;
&gt; &gt; &gt; Some more info:
&gt; &gt; &gt; Looking in the trace file of the Sam station at CNAF everything seems
&gt; &gt; &gt; ok:
&gt; &gt; &gt; project and file delivery started correctly;
&gt; &gt; &gt; debugging the core given by the AC++Dump we get something like:
&gt; &gt; &gt; +++++++++++++++
&gt; &gt; &gt; #  17 0x4388e701 in GetUserName() () at Utility.cpp:115
&gt; &gt; &gt; #  18 0x43698887 in ConfigManager (this=0xe107268) at
&gt; &gt; &gt; #  ConfigManager.hpp:256
&gt; &gt; &gt; #  19 0x43698361 in sam::ConfigManager::getInstance() () at
&gt; &gt; &gt; ConfigManager.cpp:43
&gt; &gt; &gt; #20 0x4367feda in sam::DBServerProxy::getProxy(std::string const&,
&gt; &gt; &gt; bool)
&gt; &gt; &gt; (this=0xe104420,
&gt; &gt; &gt;    servantType=@0x43718858, waitForServer=false) at
&gt; &gt; &gt;    servantType=@0x43718858, waitForServer=false) at
&gt; &gt; &gt; DBServerProxy.cpp:95
&gt; &gt; &gt; [...]
&gt; &gt; &gt; at line 17 we have the crash, before that only standard libraries.
&gt; &gt; &gt; +++++++++++++++
&gt; &gt; &gt; ...maybe a problem in user mapping?
&gt; &gt; &gt;
&gt; &gt; &gt;
&gt; &gt; &gt; ...and in the summary email we get this error:
&gt; &gt; &gt; +++++++++++++++
&gt; &gt; &gt; These files were either not processed or processed by sections that
&gt; &gt; &gt; failed to
&gt; &gt; &gt; release their last file
&gt; &gt; &gt; =====================================================================================================
&gt; &gt; &gt; Traceback (most recent call last):
&gt; &gt; &gt; File
&gt; &gt; &gt; "/afs/infn.it/project/cdf/cdfsoft/products/sam/v8_1_5/Linux+2/bin/sam",
&gt; &gt; &gt; line 4, in ?
&gt; &gt; &gt; sys.exit(Sam.main(sys.argv))
&gt; &gt; &gt; File "sam_user_pyapi/bin/Sam.py", line 6424, in main
&gt; &gt; &gt; File "sam_common_pylib/SamCommand/CommandInterfaceSuite.py", line
&gt; &gt; &gt; 120, in dispatch
&gt; &gt; &gt; File "sam_common_pylib/SamCommand/CommandInterfaceSuite.py", line
&gt; &gt; &gt; 118, in
&gt; &gt; &gt; dispatchCommand
&gt; &gt; &gt; File "sam_common_pylib/SamCommand/CommandInterface.py", line 61, in
&gt; &gt; &gt; mainDispatch
&gt; &gt; &gt; File
&gt; &gt; &gt; "sam_common_pylib/SamCommand/BlessedCommandInterfacePlaceHolder.py",
&gt; &gt; &gt; line 38, in dispatch
&gt; &gt; &gt; File "sam_common_pylib/SamCommand/SamCommandInterface.py", line 205,
&gt; &gt; &gt; in
&gt; &gt; &gt; cliDispatch
&gt; &gt; &gt; File "sam_common_pylib/SamCommand/CommandInterface.py", line 331, in
&gt; &gt; &gt; cliDispatch
&gt; &gt; &gt; File "sam_common_pylib/SamCommand/CommandInterface.py", line 344, in
&gt; &gt; &gt; _baseClass_cliDispatch
&gt; &gt; &gt; File "sam_user_pyapi/src/samProject.py", line 931, in implementation
&gt; &gt; &gt; UnboundLocalError: local variable 'consumer_id' referenced before
&gt; &gt; &gt; assignment
&gt; &gt; &gt; +++++++++++++++
&gt; &gt; &gt;
&gt; &gt; &gt; We also tried to run the same job on a dedicated machine at CNAF,
&gt; &gt; &gt; setting up
&gt; &gt; &gt; manually a project on the station, but we get the same error.
&gt; &gt; &gt;
&gt; &gt; &gt; Do you have any idea on what we could be doing wrong or where the
&gt; &gt; &gt; misconfiguration of our site could be?
&gt; &gt; &gt;
&gt; &gt; &gt; Thanks a lot for yor help,
&gt; &gt; &gt;
&gt; &gt; &gt; Gabriele
</pre>

<!-- Comment: Thu Sep 13 11;09;27 comment by...r -->
<pre>Hi Donatella
  This we need to look into with sam developers in any case.
                                                                                                                    
  Sam design was not on this list and CAF should follow the second point
as it involves the sam/caf interface.
                                                                                                                    
  You submitted a job to cdf-caf which had no physical location at
fermilab and was only at cnaf. The sam station has no way to transfer this
to fermilab since there is no physical sam cache, but a cataloging of
dcache.  However, the station sends email on every failure to get a file
and this absolutely floods the cdfsam-auto mailing list.
                                                                                                                    
                                                                                                                    
  So first, the question is why is this so important that it should be
sent to the cdfsam-auto mailing list and for every file. It seems that
this should be disabled.
                                                                                                                    
  Second, it is perfectly reasonable for a user to get confused about
sending a job to the wrong caf, even more so for a developer who is
submitting more jobs than a normal user and to more cafs.  It seems to me
that if a dataset is not at all available at a given caf and has no way to
get there, then since the caf knows that the user has submitted the job,
then it should bomb out the job for the user telling them that the
particular dataset is not avialable.  An alternative "real" cache should
be available and there are a number of ways to imagine implementing this:
could copy to diskpool and sam could send the url, could use a fileserver
(although that could be hit too hard, so I think a dcache-based diskpool
that load levels is better), could use the copy function to just copy to
the local node. This assumes for this case, of course, that you can get it
out of cnaf with gridftp!
                                                                                                                    
  In any case, the mail message thing shouldbe solved but the user
issue needs to be addressed.
</pre>


<!-- Date: Thu Sep 13 12:25:55 2007 -->
<pre>From Krzysztof: 
                                                                                                                     
On the cdf-caf station node I have modified 
/home/sam/products/upsdb/sam_bootstrap/sam_bootstrap.env 
file and removed the line containing setting SAM_MAIL_RECIPIENT from it. 
                                                                                                                     
This should disable the potential e-mail floods from the station (and unfortunately all its e-mail) till the 
problem is addressed in a programmatic way. 
https://plone3.fnal.gov/SAMGrid/Wiki/AdministeringAndUsingSAMStation#station_envs 
                                                                                                                     
Since it is an environmental variable, the station had to be restarted to pick it up. 
                                                                                                                     
Thanks to Rick for diagnosing the initial problem with lucchesi_cdf-caf_skbh01_239690_27164 project requesting 
files not present on the station (the station was restarted at that time as well). 
                                                                                                                     
</pre>
<!-- Author: r -->




<!-- Date: Thu Sep 13 15:40:14 2007 -->
<pre>fcdfdata309 is up and functional</pre>
<!-- Author: r -->


<!-- Date: Thu Sep 13 17:04:43 2007 -->
<pre>The cdf-caf SAM station was restarted earlier today due to a problem with a project requesting files located at 
remote institution but not on enstore tapes. 
                                                                                                                     
The restart have caused some clients to fail (if they tried to communicate with the station during the time it 
was being restarted). 
The following errors are usually associated with this type of situation: 
                                                                                                                     
... 
Sam2DiskCacheInterface_samapi::releaseFile-&gt; SAM Exception caught: Unknown exception caught. 
... 
%ERLOG-s Abnormal termination: caused by signal:  6 Framework abortJob() 
... 
                                                                                                                     
Sorry about the trouble the station restart has created. 
</pre>
<!-- Author: r -->


<!-- Date: Fri Sep 14 16:33:57 2007 -->
<pre>All looks fine</pre>
<!-- Author: r -->


<!-- Date: Fri Sep 14 21:47:10 2007 -->
<pre>dcap://fcdfrdc3.fnal.gov:22125 
not responding. 
Sent mail to Alexei</pre>
<!-- Author: r -->
<!-- Comment: Sat Sep 15 08;17;58 comment by...r -->
<pre>Alexei restarted it within an hour of getting this late message</pre>


<!-- Date: Sat Sep 15 12:23:13 2007 -->
<pre>The following tapes are no access: 
 
IAB444              0.45GB   (NOACCESS   0915-1142 full     0522-1800)   CDF-LTO3         cdf.cdfpewk.cpio_odc                 
IAB464              0.15GB   (NOACCESS   0915-0058 full     0526-1024)   CDF-LTO3         cdf.cdfhstn.cpio_odc                 
</pre>
<!-- Author: r -->
<!-- Comment: Mon Sep 17 10;49;58 comment by...r -->
<pre>From Dave Berg (Sunday 16 sep 07, 19:45)
Rick,
                                                                                                                       
These 2 tapes are among a group of 6 that are currently AWOL from the
SL8500, probably because of bad barcodes. The complete list is:
  IAB440
  IAB444
  IAB450
  IAB454
  IAB460
  IAB464
                                                                                                                       
The problem will be investigated in the morning.
</pre>
<!-- Comment: Mon Sep 17 14;29;38 comment by...r -->
<pre>
Subject: HelpDesk ticket 104205
                                                                                                                       
    [ The following text is in the "windows-1252" character set. ]
    [ Your display is set for the "ISO-8859-1" character set.  ]
    [ Some special characters may be displayed incorrectly. ]
                                                                                                                       
Thank you for contacting the Fermilab Computing Division Helpdesk.
                                                                                                                       
To check the status of a ticket visit:
http://helpdesk.fnal.gov
                                                                                                                       
Note: Normal Helpdesk hours are Mon. - Fri. 08:00 to 16:30.
                                                                                                                       
If this issue requires immediate attention you MUST call 630-840-2345.
                                                                                                                       
You can update this ticket by replying to this email and placing an update between the 2 lines below.
                                                                                                                       
&lt;-- # @@@  Enter Update below this line. @@@ # --&gt;
                                                                                                                       
                                                                                                                       
&lt;-- # @@@  Enter Update above this line. @@@ # --&gt;
                                                                                                                       
___________________________________________
Ticket #: 104205
___________________________________________
Short Description: Tapes no-access
Problem Description: The following tapes are "no access". Can you please help?
IAB444              0.45GB   (NOACCESS   0915-1142 full     0522-1800)
CDF-LTO3         cdf.cdfpewk.cpio_odc
IAB464              0.15GB   (NOACCESS   0915-0058 full     0526-1024)
CDF-LTO3         cdf.cdfhstn.cpio_odc
                                                                                                                       
thanks
Rick
(CDF Sam Shifter)
                                                                                                                       
This ticket is assigned to SZMUKSTA, GEORGE of the CD-SF/DMS/DSC/SSA.
</pre>
<!-- Comment: Mon Sep 17 14;31;24 comment by...R -->
<pre>Status as of 14:30, 17 SEP
Mon Sep 17 14:30:08 CDT 2007

IA2312            200.00GB   (NOTALLOWED 0810-1727 none              )   TST-9940B        cdf.none.none                       
IA2934             63.76GB   (NOTALLOWED 0124-1051 readonly 1112-1445)   CDF-9940B        cdf.mjkim.cpio_odc                   File 9 is bad, need to copy
IA5057            200.00GB   (NOTALLOWED 0110-1316 none     1227-1645)   CDF-9940B        cdf.jphysr.cpio_odc                  Broken Leader on a blank tape
IAB837            400.00GB   (NOTALLOWED 0314-1110 none              )   CDF-LTO3         cdf.none.none                       
IAB973            400.00GB   (NOTALLOWED 0721-1151 none              )   CDF-LTO3         cdf.hbhd0j.cpio_odc                  9 consecutive mount failures
IAC226            189.05GB   (NOTALLOWED 0622-1405 none              )   CDF-9940B        cdf.cphysr.cpio_odc                  To be replaced 06/22/07

Mon Sep 17 14:30:08 CDT 2007
</pre>
<!-- Comment: Mon Sep 17 14;33;23 comment by...r -->
<pre>Subject: Help Desk Ticket 104205 Has Been Resolved.
                                                                                                                       
    [ The following text is in the "windows-1252" character set. ]
    [ Your display is set for the "ISO-8859-1" character set.  ]
    [ Some special characters may be displayed incorrectly. ]
                                                                                                                       
We have received a resolution from our support staff. If you find the resolution inadequate in any
way, please contact us.
                                                                                                                       
If we do not hear from you within 14 days, we will close the problem.
___________________________________________________________________
                                                                                                                       
Ticket #: 104205
___________________________________________________________________
                                                                                                                       
Solution: berg@fnal.gov sent this solution:
                                                                                                                       
Rick,
                                                                                                                       
The six tapes have had their barcode labels replaced and are available
again. This includes the two to which you refer here.
</pre>
<!-- Date: Sun Sep 16 15:06:05 2007 -->
<pre>r-fcdfdata145-1 offline 
Sent mail to helpdesk, cdfrun2-sys, cc'd dcache-admin</pre>
<!-- Author: r -->
<!-- Comment: Mon Sep 17 14;27;29 comment by...r -->
<pre>From  Alex Kulyavtsev, 12:38 Monday 17
nothing obvious. I investigated exceptions reported and filed bug report to desy developers, but it looks the same
as other pools.
I restarted the pool.
</pre>
<!-- Date: Mon Sep 17 10:51:00 2007 -->
<pre>Mario Campenelli writes 
 
Ok, I was able to run on these files (up to run number 227782) and 
create beamlines. 
 
This is the beamline processing of jpmm0i,  a project started in April! 
 
Will try to get more files copied for him. 
</pre>
<!-- Author: r -->
<!-- Comment: Mon Sep 17 11;06;25 comment by...r -->
<pre>Checked space:
-bash-2.05b$ /pnfs/diskpool/space bnt
  13258.9 GB allocated
   8466.2 GB used
                                                                                                                       
So looks like 13.2-8.5=4.7TB and we need
                                                                                                                       
Total File Size:    2.36TB
                                                                                                                       
for
 sam translate constraints --dim="cdf.dataset jpmm0i and run_number &gt; 227782"
                                                                                                                       
so I started a copy, log is
                                                                                                                       
cdfmc@fcdfdata153.fnal.gov:/export/data4/cdfmc/svt/testsam/files/filelistgt227782-pass1.log
                                                                                                                       
and the files will go to
/pnfs/diskpool/bnt/svt17sep07
</pre>