<!-- Title: 13:39:35  Thu Feb 22 2007 -->
<!-- Crewchief: Elena Gerchtein -->
<!-- Op1: Marcelo Vogel -->
<!-- Op2: Pasha Murat -->
<!-- Op3: Peter Bednar -->
<!-- Op4: Rick Snider -->
<!-- Op5: etc...... -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>CDF Offline production activities</pre>


<!-- Date: Thu Feb 22 13:40:21 2007 -->
<pre>This is a test of the new log...</pre>
<!-- Author: Ting Miao -->


<!-- Date: Thu Feb 22 16:35:49 2007 -->
<pre>Elena found this useful enstore link that can be used to plot rates for specific fileservers 
 
http://stkenscan1.fnal.gov:8080/tablePlotter/tableProperties?tablename=encp_xfer 
 
For example, to plot tranfer rate: 
Put in full node name (fncdfsrv11.fnal.gov) 
Choose parameter for X axis (date)* 
Choose paramener for Y axis (transfer rate) 
*Choose the time window  
 
If only the Y axis is chosen it plots a histogram 
 
</pre>
<!-- Author: Marcelo -->
<!-- Comment: Fri Feb 23 16;35;06 comment by...Marcelo -->
<pre>Sorry, that should be choose transfer rate as X only for histogram</pre>
<!-- Date: Thu Feb 22 16:47:55 2007 -->
<table><tr><td><a href=/elog/notebooks/offline_production/Thu_Feb_22_2007_164640.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline_production&action=view&page=-4  align='top'></a></td><td>First transfer rates for fncdfsrv11 after service by Ling Ho</td></tr></table>
<!-- Author: Marcelo -->
<!-- Comment: Thu Feb 22 16;50;56 comment by...Marcelo -->
<pre>Rates are in bytes per second, so keep in mind conversion:
1024byte = 1KB</pre>


<!-- Date: Thu Feb 22 17:12:21 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline_production&action=view&page=-5&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline_production&action=view&page=-5  width='300' align='top'></a></td><td>More plots for fncdfsrv11 by Ling Ho</td></tr></table>
<!-- Author: Marcelo -->
<!-- Comment: Thu Feb 22 17;14;40 comment by...Marcelo -->
<pre>Units are MB/s</pre>


<!-- Date: Thu Feb 22 17:40:36 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline_production&action=view&page=-8&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline_production&action=view&page=-8  width='300' align='top'></a></td><td>Rates of srv12 by Ling</td></tr></table>
<!-- Author: Marcelo -->


<!-- Date: Fri Feb 23 16:23:20 2007 -->
<pre>STATUS OF MAINTENANCE OF FILESERVERS 
---------------------------------------------------------------- 
fncdfsrv6 
older raid architecture, did not need reconfiguring controller connections 
---------------------------------------------------------------- 
fncdfsrv9 
-export/ file system rebuilt 
-controller-bus connections reconfigured 
---------------------------------------------------------------- 
fncdfsrv10 
-waiting for new parts (will come in next week) 
-nothing done here yet- currently in use by ntuplers 
--------------------------------------------------------------- 
fncdfsrv11 
-export/ file system rebuilt 
-controller-bus connections reconfigured 
---------------------------------------------------------------- 
fncdfsv12 
-waiting for new parts 
-export/ file system rebuild in process 
-controller-bus connections reconfigured 
---------------------------------------------------------------- 
fncdfsrv13 
-export/ file system rebuild in progress 
-controller-bus connections reconfigured 
----------------------------------------------------------------</pre>
<!-- Author: Marcelo -->



<!-- Date: Fri Feb 23 18:16:49 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline_production&action=view&page=-9&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline_production&action=view&page=-9  width='300' align='top'></a></td><td>Latest plot of rates for srv11 by Ling Ho</td></tr></table>
<!-- Author: Marcelo -->


<!-- Date: Sat Feb 24 10:32:34 2007 -->
<pre>STATUS OF MAINTENANCE OF FILESERVERS  
----------------------------------------------------------------  
fncdfsrv6  
older raid architecture, did not need reconfiguring controller connections  
----------------------------------------------------------------  
fncdfsrv9  
-export/ file system rebuilt  
-controller-bus connections reconfigured  
----------------------------------------------------------------  
fncdfsrv10  
-waiting for new parts (will come in next week)  
-nothing done here yet- currently in use by ntuplers  
---------------------------------------------------------------  
fncdfsrv11  
-export/ file system rebuilt  
-controller-bus connections reconfigured  
----------------------------------------------------------------  
fncdfsv12  
-waiting for new parts  
-export/ file system rebuilt  
-controller-bus connections reconfigured  
----------------------------------------------------------------  
fncdfsrv13  
-export/ file system rebuilt  
-controller-bus connections reconfigured  
----------------------------------------------------------------</pre>
<!-- Author: Marcelo -->


<!-- Date: Mon Feb 26 19:44:11 2007 -->
<pre>STATUS OF MAINTENANCE OF FILESERVERS   
----------------------------------------------------------------   
fncdfsrv6   
older raid architecture, did not need reconfiguring controller connections   
----------------------------------------------------------------   
fncdfsrv9   
-export/ file system rebuilt   
-controller-bus connections reconfigured   
----------------------------------------------------------------   
fncdfsrv10   
-waiting for new parts (will come in this week) 
-export/ file system rebuilt   
-controller-bus connections reconfigured   
---------------------------------------------------------------   
fncdfsrv11   
-export/ file system rebuilt   
-controller-bus connections reconfigured   
----------------------------------------------------------------   
fncdfsv12   
-waiting for new parts   
-export/ file system rebuilt   
-controller-bus connections reconfigured   
----------------------------------------------------------------   
fncdfsrv13   
-export/ file system rebuilt   
-controller-bus connections reconfigured   
----------------------------------------------------------------</pre>
<!-- Author: Marcelo -->


<!-- Date: Fri Mar 2 10:31:53 2007 -->
<pre>Stopped cron jobs for calibration on farm to maximize speed of data 
processing. 
</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Fri Mar 2 11:20:22 2007 -->
<pre>On 02/01/2007 at 20:03 cron was rebuilt by unknown user. 
Concatenators and tape upload stopped working and data processing was disrupted. We disabled cron build in cdfprd/GNUmakefile for 
v&_0 and v7_1 versions and updated all test releases in servers home area.  
 
Elena. The list of updated files is here: 
cdfprd@fncdfsrv12:~/testRel]$ for f in `ls | grep 6.1.4 `; do echo $f; cd $f; cvs -d $CVSROOT co -A cdfprd/GNUmakefile; cd .. ; done 
6.1.4 
6.1.4_ag 
6.1.4_aidan 
U cdfprd/GNUmakefile 
6.1.4_bjk 
U cdfprd/GNUmakefile 
6.1.4_bstn 
6.1.4_bstntuple 
6.1.4_chenyc 
U cdfprd/GNUmakefile 
6.1.4_common 
6.1.4_cphysr 
U cdfprd/GNUmakefile 
6.1.4_dphysr 
U cdfprd/GNUmakefile 
6.1.4_elena 
U cdfprd/GNUmakefile 
6.1.4_ephysr 
U cdfprd/GNUmakefile 
6.1.4_gphysr 
U cdfprd/GNUmakefile 
6.1.4_linacre 
U cdfprd/GNUmakefile 
6.1.4_murat 
6.1.4_mussini 
U cdfprd/GNUmakefile 
6.1.4_prod 
U cdfprd/GNUmakefile 
6.1.4_schsu 
6.1.4_shalhout 
U cdfprd/GNUmakefile 
6.1.4_stntuple 
U cdfprd/GNUmakefile 
6.1.4_stntuple_schsu 
6.1.4_tarball_newCalc 
U cdfprd/GNUmakefile 
6.1.4_tarball_newCalc_maxopt 
U cdfprd/GNUmakefile 
6.1.4_topnt 
U cdfprd/GNUmakefile 
new6.1.4_linacre 
U cdfprd/GNUmakefile 
production_6.1.4 
test_6.1.4 
U cdfprd/GNUmakefile 
 
 
 </pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Fri Mar 2 12:01:08 2007 -->
<pre>Switched STREAM J from farm to caf to balance the load.</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Fri Mar 2 17:16:32 2007 -->
<pre>srv6-stream b 
export/stage has 1.0T  
Changed minimum number to upload from 50 to 20 to liberate disk space. 
 
SamStore.LargerEqual         20</pre>
<!-- Author: Marcelo -->
<!-- Comment: Fri Mar  2 17;34;32 comment by...Marcelo -->
<pre>Changed it back up to 40 after this message; I will decrease it more slowly

2007/03/02 17:20:20 [ CdfStore::StoreFile ] WARNING: number of encp processes = 100
2007/03/02 17:20:20 [ SamStore::StoreFiles ] : WARNING: rc=100
2007/03/02 17:20:20 [ SamStore::__main__ ] : exit, rc=100</pre>


<!-- Date: Fri Mar 2 23:18:44 2007 -->
<pre>b-stream concatenators running again after setting 
SamStore.LargerEqual         20</pre>
<!-- Author: Marcelo -->


<!-- Date: Sat Mar 3 14:03:28 2007 -->
<pre>corrupted file in jrar0i, AC++Dump returned rc=231 
 
reco.ji037d38.0001rar0 
 
%ERLOG-e readNextEvent-pre: TFile pointer is null. File is not open.   
         GlobalLibraryLogger MultiRootInputFile::readNextEvent()  
         03-Mar-2007 03:40:38 CST run = 0 event = 0 
 
moved file from jrar0i to folder ERRORS and deleted lock file</pre>
<!-- Author: Marcelo -->


<!-- Date: Sat Mar 3 23:40:11 2007 -->
<pre>b-stream 
 
Lowered minimum number to upload to 10</pre>
<!-- Author: Marcelo -->


<!-- Date: Sun Mar 4 13:15:47 2007 -->
<pre>switched stream c - to caf, stream j - to farm to balance the load</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Sun Mar 4 15:40:38 2007 -->
<pre>Crashed files in stream j: 
 
ProductionExe_0i_jphysr_228664_228664 
 
(jr037d38.06e5phys 25072886) (jr037d38.06e5phys 25080376) (jr037d38.06e5phys 25072461) (jr037d38.06e5phys 25072962)  
---------------------------------------------- 
ProductionExe_0i_jphysr_228902_228902 
 
(jr037e26.069bphys 0) (jr037e26.03a4phys 0) (jr037e26.0982phys 0) (jr037e26.02f9phys 0) (jr037e26.02f9phys 12181638) (jr037e26.0680phys 23211457) (jr037e26.0680phys 0)  
</pre>
<!-- Author: Marcelo -->


<!-- Date: Sun Mar 4 23:42:34 2007 -->
<pre>Stream C: moved files to ./ERROR until further investigation: 
error message: Found merged input files. 
Removed lock files. 
</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Mon Mar 5 08:02:10 2007 -->
<pre>Manually killed three jobs running for more than two days on fcdfcaf869.fnal.gov 
 
these jobs corresponded to the following projects/files: 
 
ProductionExe_0i_bphysr_228664_228664_20070301_164300 
br037d38.017ephys 
 
ProductionExe_0i_bphysr_228683_228683_20070301_170156 
br037d4b.0499phys 
 
ProductionExe_0i_bphysr_228763_228763_20070302_000243 
br037d9b.0081phys 
 
 
</pre>
<!-- Author: Marcelo -->
<!-- Comment: Fri Mar  9 18;15;06 comment by...Marcelo -->
<pre>helpdesktichet issued for this worker node. Currently blocked by Peter</pre>
<!-- Date: Mon Mar 5 15:34:20 2007 -->
<pre>Marked the following projects as processed 
while switching to caf 
 
DONT FORGET TO MARK IT UNDONE LATER! 
 
/home/cdfprd/Projects/ProductionExe_0i_ephysr/Metalia 
-rw-r--r--  1 cdfprd cdf  78540 Mar  5 00:55 ProductionExe_0i_ephysr_228902_228902 
  -rw-r--r--  1 cdfprd cdf  17325 Mar  5 05:43 ProductionExe_0i_ephysr_228995_228995 
  -rw-r--r--  1 cdfprd cdf  26950 Mar  5 05:56 ProductionExe_0i_ephysr_229040_229040 
  -rw-r--r--  1 cdfprd cdf  43505 Mar  5 07:56 ProductionExe_0i_ephysr_229084_229084 
</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Mon Mar 5 15:39:26 2007 -->
<pre>Files mentioned Mar4, 23:45, were processed twice. 
When stream c was moved fron farm to caf, cronSubmit 
was looking for running projects on caf, not on farm,  
didn't find it on caf and submitted them again. 
 
Files in ./ERROR direcrory were deleted.</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Mon Mar 5 23:48:22 2007 -->
<pre>removed lock stream C #2,  AC++ error.</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Tue Mar 6 19:28:12 2007 -->
<pre>Failed to get running CAF jobs 
no b-stream jobs were running at the time of removing lock file (19:27) 
 
Farm.py: ERROR parsing CafMon output. 
CafMon output=WARNING: Python C API version mismatch for module krb5: 
  This Python has API version 1010, module krb5 has version 1007. 
./bin/Linux2_SL-GCC_3_4/CafMon: line 88: 23344 Segmentation fault      python $FULLCOMMAND $* 
2007/03/06 18:06:39 [ CronSubmit::CheckProjects ] ERROR: rc=-1  Failed to get running CAF jobs.   
2007/03/06 18:06:40 [ cronSubmit.cron ]  finished, rc=255</pre>
<!-- Author: Marcelo -->
<!-- Comment: Tue Mar  6 20;16;27 comment by...Marcelo -->
<pre>stager for stream b full, no new jobs are initiating</pre>
<!-- Date: Tue Mar 6 20:13:58 2007 -->
<pre>stream g 
removed lock file 
 File "sam_common_pylib/SamCorba/SamServerProxyRetryHandler.py", line 266, in handleCall 
InternalException: DbSQLException:  connection lost contact</pre>
<!-- Author: Marcelo -->



<!-- Date: Tue Mar 6 20:33:50 2007 -->
<pre>exceeded tarball quota for stream g jobs 
 
Error: Failed in submission from SUBMITTER to BMGR : You have exceeded your tar ball quota! 
Wait until some of your jobs have finished.</pre>
<!-- Author: Marcelo -->


<!-- Date: Wed Mar 7 13:15:29 2007 -->
<pre>To address run-away python monitoring processes 
 
Peter Bednar provided patched copies of: 
CommModule.py 
TelnetModule.py 
 
I replaced old ones by new ones in: 
/home/cdfcaf/CafCondor/monitor/ 
 
killed all the monitor.py and xml_monitor.py processes 
 
Restarted them by 
 
./startup monitor monitor start 
./startup monitor xml_monitor start 
 
from /home/cdfcaf/CafCondor/ directory 
 
Removed lock files from streams 
 
Thanks Peter for the patches and instructions</pre>
<!-- Author: Marcelo -->


<!-- Date: Wed Mar 7 19:29:01 2007 -->
<pre>e stream 
removed lock file: AC++Dump error 
%ERLOG-e ROOT/TFile:  
        file /export/stage/ProductionExe_0i_ephysr/edil0i/reco.ei038048.032edil0 is truncated at 547844096 bytes: should be 594621607, trying to recover 
 
moved file to ERRORS</pre>
<!-- Author: Marcelo -->
<!-- Comment: Sun Mar 25 20;50;19 comment by...Marcelo -->
<pre>Solution:
file corrupted
sam undeclared file, changed 1 to 0 in metalia summary, deleted file from ERRORS</pre>
<!-- Comment: Sun Mar 25 21;14;31 comment by...Marcelo -->
<pre>deleted .ok file as well</pre>
<!-- Date: Wed Mar 7 20:03:26 2007 -->
<pre>b stream 
removed lock file 
 
2007/03/07 18:33:37 [ MergeSubmit::CheckMergeCandidate ] Examine reco.bi03857a.01c0hmu0 lost reco.bi03857a.01c0hmu0 
2007/03/07 18:33:38 [ MergeSubmit::CheckMergeCandidate ] FAILURE: metadata for ['reco.bi03857a.01c0hmu0'] are not 
 
moved file to ERRORS</pre>
<!-- Author: Marcelo -->
<!-- Comment: Wed Mar  7 20;05;27 comment by...Marcelo -->
<pre>The error message above ends: "are not in SAM"</pre>


<!-- Date: Thu Mar 8 00:30:23 2007 -->
<pre>http://fncdfsrv0.fnal.gov/~cdfprd/df.html 
 
concatenators disk space monitor is now fully functional.</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Thu Mar 8 12:39:34 2007 -->
<pre>b stream 
 
MergeSubmit Error: 
 
2007/03/08 10:33:26 [ MergeSubmit::CheckMergeCandidate ] Examine reco.bi03857a.01c0hel0 lost reco.bi03857a.01c0hel0 
2007/03/08 10:33:29 [ MergeSubmit::CheckMergeCandidate ] FAILURE: metadata for ['reco.bi03857a.01c0hel0'] are not in SAM 
 
removed lock file and moved file to ERRORS 
 
 
 
reco.bi03857a.01c0hel0</pre>
<!-- Author: Marcelo -->


<!-- Date: Thu Mar 8 14:43:01 2007 -->
<pre>Investigated files  
-rw-r--r--    1 cdfprd   cdf      252748796 Mar  7 15:13 reco.bi03857a.01c0hel0 
-rw-r--r--    1 cdfprd   cdf      100618240 Mar  7 15:14 reco.bi03857a.01c0hmu0 
 
deleting files,  
they were not declared to SAM.</pre>
<!-- Author: Elena Gerchtein -->
<!-- Comment: Fri Mar  9 17;13;33 comment by...Marcelo -->
<pre>Checked these two files for daughters. They were produced, concatenated and uploaded. No further action needs to be taken.

$ sam get metadata --file=reco.bi03857a.01c0hel0 |grep daughter
       'Comments' : 'DHInputCount:7334|daughter:bi03857a.01c0hel0'
$ sam get metadata --file=reco.bi03857a.01c0hmu0 |grep daughter
       'Comments' : 'DHInputCount:7334|daughter:bi03857a.01c0hmu0'</pre>
<!-- Date: Thu Mar 8 15:25:57 2007 -->
<pre>UNMARKING THE FOLLOWING FILES: (from Mar 5th) 
 
/home/cdfprd/Projects/ProductionExe_0i_ephysr/Metalia  
-rw-r--r--  1 cdfprd cdf  78540 Mar  5 00:55 ProductionExe_0i_ephysr_228902_228902  
  -rw-r--r--  1 cdfprd cdf  17325 Mar  5 05:43 ProductionExe_0i_ephysr_228995_228995  
  -rw-r--r--  1 cdfprd cdf  26950 Mar  5 05:56 ProductionExe_0i_ephysr_229040_229040  
  -rw-r--r--  1 cdfprd cdf  43505 Mar  5 07:56 ProductionExe_0i_ephysr_229084_229084 </pre>
<!-- Author: Elena Gerchtein -->




<!-- Date: Sat Mar 10 08:17:05 2007 -->
<pre>set max number of jobs for stream g and h to 350</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Sat Mar 10 11:33:20 2007 -->
<pre>stream c  
removed lock file 
 
Traceback (most recent call last): 
  File "/home/cdfprd/testRel/production_6.1.4/bin/Linux2_SL-GCC_3_4/cdfprd/cronSubmit.py", line 852, in ? 
    rc = cs.CheckProjects(cs.fSamStation,1); 
  File "/home/cdfprd/testRel/production_6.1.4/bin/Linux2_SL-GCC_3_4/cdfprd/cronSubmit.py", line 492, in CheckProjects 
    m             = sam.getMetadata(filename=job); 
  File "sam_common_pylib/SamCommand/BlessedCommandInterfacePlaceHolder.py", line 81, in __call__ 
  File "sam_common_pylib/SamCommand/CommandInterface.py", line 251, in __call__ 
  File "sam_common_pylib/SamCommand/SamCommandInterface.py", line 243, in apiWrapper 
  File "sam_user_pyapi/src/samMetadata.py", line 457, in implementation 
  File "sam_common_pylib/SamCorba/SamServerProxy.py", line 257, in _callRemoteMethod 
  File "sam_common_pylib/SamCorba/SamServerProxyRetryHandler.py", line 266, in handleCall 
DataFileNotFound: Datafile with name '210' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Sat Mar 10 16:56:01 2007 -->
<pre>stream j 
removed lock file 
 
DataFileNotFound: Datafile with name 'fi000276.7715arm0' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Sat Mar 10 22:26:27 2007 -->
<pre>Stream c 
removed lock file 
 
2007/03/10 21:25:06 [ cronSubmit.cron ]  /home/cdfprd/testRel/production_6.1.4/bin/Linux2_SL-GCC_3_4/cdfprd/cronSubmit.py --project=ProductionExe_0i_cphysr --doit=1 
/home/cdfsoft/products/sam/v7_7_1/Linux+2/bin/sampy/whrandom.py:38: DeprecationWarning: the whrandom module is deprecated; please use the random module 
Traceback (most recent call last): 
  File "/home/cdfprd/testRel/production_6.1.4/bin/Linux2_SL-GCC_3_4/cdfprd/cronSubmit.py", line 852, in ? 
    rc = cs.CheckProjects(cs.fSamStation,1); 
  File "/home/cdfprd/testRel/production_6.1.4/bin/Linux2_SL-GCC_3_4/cdfprd/cronSubmit.py", line 492, in CheckProjects 
    m             = sam.getMetadata(filename=job); 
  File "sam_common_pylib/SamCommand/BlessedCommandInterfacePlaceHolder.py", line 81, in __call__ 
  File "sam_common_pylib/SamCommand/CommandInterface.py", line 251, in __call__ 
  File "sam_common_pylib/SamCommand/SamCommandInterface.py", line 243, in apiWrapper 
  File "sam_user_pyapi/src/samMetadata.py", line 457, in implementation 
  File "sam_common_pylib/SamCorba/SamServerProxy.py", line 257, in _callRemoteMethod 
  File "sam_common_pylib/SamCorba/SamServerProxyRetryHandler.py", line 266, in handleCall 
DataFileNotFound: Datafile with name 'fi000037.0747arm0' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Mon Mar 12 12:18:43 2007 -->
<pre>c stream 
removed lock file 
DataFileNotFound: Datafile with name '210' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Mon Mar 12 12:23:05 2007 -->
<pre>stream j 
removed lock file 
InternalException: DbSQLException:  connection lost contact</pre>
<!-- Author: Marcelo -->


<!-- Date: Mon Mar 12 13:33:10 2007 -->
<pre>execute ~/scripts/check projects fron cron, 
it is updated every 5 minutes.</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Mon Mar 12 17:38:26 2007 -->
<pre>data base glitch resulted in lock files in streams c, g, h and j 
InternalException: DbSQLException:  connection lost contact 
 
removed lock files</pre>
<!-- Author: Marcelo -->


<!-- Date: Mon Mar 12 19:22:07 2007 -->
<pre>stream h 
 
strange db glitch that has been ocurring lately 
 
Examine reco.hi0386d6.019abck0 lost reco.hi0386d6.019abck0 
2007/03/12 17:11:24 [ MergeSubmit::CheckMergeCandidate ] FAILURE: metadata for ['reco.hi0386d6.019abck0'] are not in SAM 
 
I checked the file's metadata and the file was indeed declared to SAM. I removed lock file.</pre>
<!-- Author: Marcelo -->


<!-- Date: Tue Mar 13 11:19:08 2007 -->
<pre>2007/03/13 02:35:59 [ MergeSubmit::CheckMergeCandidate ] FAILURE: metadata for ['reco.xi0384dd.0085bmm0', 'reco.xi0384dd.0105bmm0'] are not in SAM 
 
2007/03/13 07:03:43 [ MergeSubmit::CheckMergeCandidate ] FAILURE: metadata for ['reco.ji0384dd.0048bmm0', 'reco.ji0384dd.006dbmm0'] are not in SAM 
 
not in sam indeed, moving files with ok files to ./ERROR 
deleting locks #1 and #2</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Tue Mar 13 14:16:40 2007 -->
<pre>stream j 
removed lock file 
DataFileNotFound: Datafile with name '210' not found</pre>
<!-- Author: Marcelo -->


<!-- Date: Tue Mar 13 14:23:13 2007 -->
<pre>This error needs some careful attention. Files are declared to SAM; I removed locks. 
 
2007/03/13 13:04:18 [ MergeSubmit::CheckMergeCandidate ] Examine reco.ci038837.025adip0 lost reco.ci038837.025adip0 
2007/03/13 13:05:19 [ MergeSubmit::CheckMergeCandidate ] Examine reco.ci03885a.0022dip0 lost reco.ci03885a.0022dip0 
2007/03/13 13:05:55 [ MergeSubmit::CheckMergeCandidate ] FAILURE: metadata for ['reco.ci038837.025adip0', 'reco.ci03885a.0022dip0'] are not in SAM 
</pre>
<!-- Author: Marcelo -->


<!-- Date: Tue Mar 13 14:28:44 2007 -->
<pre>files were declared to SAM. I removed locks 
 
2007/03/13 13:04:13 [ MergeSubmit::CheckMergeCandidate ] Examine reco.hi03870b.0a33bhd0 lost reco.hi03870b.0a33bhd0 
2007/03/13 13:05:14 [ MergeSubmit::CheckMergeCandidate ] Examine reco.hi03870b.0a3cbhd0 lost reco.hi03870b.0a3cbhd0 
2007/03/13 13:05:41 [ MergeSubmit::CheckMergeCandidate ] FAILURE: metadata for ['reco.hi03870b.0a33bhd0', 'reco.hi03870b.0a3cbhd0'] are not in SAM 
2007/03/13 13:33:16 [ MergeSubmit::InitProject ] self.fMaxSegments = 10</pre>
<!-- Author: Marcelo -->


<!-- Date: Tue Mar 13 20:27:43 2007 -->
<pre>Found already merged file. I moved it to ERRORS (with ok file) and removed lock 
 
2007/03/13 17:03:48 Found merged input files 
   input	==	output 
   reco.ei038488.0929met0	==	ei038488.082fmet0 
2007/03/13 17:33:10 [ MergeSubmit::InitProject ] self.fMaxSegments = 10</pre>
<!-- Author: Marcelo -->
<!-- Comment: Sun Mar 25 20;53;46 comment by...Marcelo -->
<pre>Solution:
file is on tape merged into ei038488.082fmet0
deleted duplicate</pre>
<!-- Date: Tue Mar 13 23:19:18 2007 -->
<pre>stream j concatenators stuck around 20:00 
 
see jphysr_0i_01 line 60320: moved these files to ./ERROR, 
there were no ok files. 
 
and jphysr_0i_03 line 25230: moved files to ./ERROR 
 
removing lock files #1 and #3</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Wed Mar 14 00:36:24 2007 -->
<pre>stream b: both concatenators stopped for lack of disk space. I lowered minimum space to concatenate from 300 to 200. 
 
MergeSubmit.OutputDir        fcp://$MergeSubmit.Host:$MergeSubmit.Dir:200</pre>
<!-- Author: Marcelo -->


<!-- Date: Wed Mar 14 09:27:38 2007 -->
<pre>/export/stage/ProductionExe_0i_hphysr/mergeLogs/hphysr_0i_01: 
 
removing temporary file (ac++dump segmentation fault) 
 -rw-r--r--  1 cdfprd cdf 1902674574 Mar 14 02:17 AA_31755_temp_0 
 
removing lock #1</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Thu Mar 15 13:47:42 2007 -->
<pre>switched to optimized root for production. 
Modified: mergeSubmit.cron, ~/testRel/production_6.1.4/bin/*/AC++Dump</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Thu Mar 15 18:23:50 2007 -->
<pre>Initiating concatenation and upload after shutdown</pre>
<!-- Author: Marcelo -->


<!-- Date: Thu Mar 15 20:12:07 2007 -->
<pre>Project configuration tiki unavailable 
 
http://www-cdf.fnal.gov/tiki/tiki-index.php?page=ProductionFarm.ProjectConfiguration</pre>
<!-- Author: Marcelo -->


<!-- Date: Fri Mar 16 03:00:37 2007 -->
<pre>samStore stuck since March 12 on one file.  
 
bi038723.00e4hel0  bi038723.00e4hel0.par 
 
2007/03/12 05:08:56 [ SamStore::StoreFiles ] : ERROR: get reco parent failed: reco.bi038723.00e4hel0 reco.bi038723.00edhel0 reco.bi038723.00f7hel0 reco.bi038723.0101hel0 reco.bi038723.010ahel0 reco.bi038723.0112hel0 
2007/03/12 05:08:56 [ SamStore::__main__ ] : exit, rc=-48 
 
Moved them to ERRORS</pre>
<!-- Author: Marcelo -->


<!-- Date: Fri Mar 16 12:34:38 2007 -->
<pre>stream h 
AC++Dump error 
DHInputModule_2.cc: releasing file 
%ERLOG-e ROOT/TFile:  
         Abnormal case while reading the header key.  The TFile (/export/stage/ProductionExe_0i_hphysr/hbck0i/reco.hi038836.044bbck0) is probably corrupted. 
 
Moved file to ERRORS</pre>
<!-- Author: Marcelo -->


<!-- Date: Fri Mar 16 14:11:09 2007 -->
<pre>farm project status monitoring script 
http://fncdfsrv0.fnal.gov/~cdfprd/projects.txt 
 
is now html: 
http://fncdfsrv0.fnal.gov/~cdfprd/projects.html 
 
Please, update your bookmarks 
 
 
</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Fri Mar 16 15:47:37 2007 -->
<pre>h stream: 
removed lock from cronsubmit 
 
KeyError: 'Project.SamStation'</pre>
<!-- Author: Marcelo -->
<!-- Comment: Fri Mar 16 15;49;30 comment by...Marcelo -->
<pre>someone beat me to it. I didn't find the lock</pre>


<!-- Date: Sat Mar 17 07:49:44 2007 -->
<pre>stream c 
 
DataFileNotFound: Datafile with name 'fi000278.2572arm0' not found. 
 
DataFileNotFound: Datafile with name 'fi000278.2304arm0' not found. 
removed CronSubmit locks</pre>
<!-- Author: Marcelo -->


<!-- Date: Sat Mar 17 20:10:39 2007 -->
<pre>stream c 
 
removed lock file from CronSubmit 
 
DataFileNotFound: Datafile with name 'fi000278.8209arm0' not found. 
 
</pre>
<!-- Author: Marcelo -->


<!-- Date: Sun Mar 18 15:56:08 2007 -->
<pre>server10 - removed concat. lock #2, 
removed output file, IO error.</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Sun Mar 18 18:38:36 2007 -->
<pre>b stream  
 
removed cronsubmit lock. 
 
DataFileNotFound: Datafile with name 'fi000279.5003arm0' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Tue Mar 20 01:00:56 2007 -->
<pre>stream E: lowering the min. number of files for samStore to 30</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Tue Mar 20 13:36:45 2007 -->
<pre>stream c 
Lower minimum number to upload to 30</pre>
<!-- Author: Marcelo -->


<!-- Date: Tue Mar 20 13:54:19 2007 -->
<pre>stream j 
AC++Dump error 
 
%ERLOG-e ROOT/TFile:  
        file /export/stage/ProductionExe_0i_jphysr/jrar0i/reco.ji03870b.0047rar0 is truncated at 292175872 bytes: should be 949385311, trying to recover  
 
Moved file reco.ji03870b.0047rar0 to ERRORS</pre>
<!-- Author: Marcelo -->


<!-- Date: Tue Mar 20 16:46:12 2007 -->
<pre>srv10 
removing lock #1 (root io error) 
removing temporary file.</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Wed Mar 21 14:37:24 2007 -->
<pre>Commented out CronSubmit for stream C</pre>
<!-- Author: Marcelo -->


<!-- Date: Wed Mar 21 16:17:26 2007 -->
<pre>stream c 
killed project   
 
CafMon --farm caf kill 2836892 
 
Project was submitted ~40 times</pre>
<!-- Author: Marcelo -->
<!-- Comment: Wed Mar 21 16;45;14 comment by...Marcelo -->
<pre>job failed to produce a single reco file: reco.ci03827a.06cdphb0

cdfprd@fncdfsrv0:~]$ sam get file descendants --fileName=cr03827a.06cdphys |grep reco
      'fileName' : 'reco.ci03827a.06cdpjj0',
      'fileName' : 'reco.ci03827a.06cddip0',
      'fileName' : 'reco.gi03827a.06cddif0',
      'fileName' : 'reco.ci03827a.06cdphc0',
      'fileName' : 'reco.ci03827a.06cdph10',</pre>
<!-- Date: Wed Mar 21 16:23:16 2007 -->
<pre>STREAM H: 
 
FROM LOG FILE: 
 
2007/03/21 15:39:02 [ SamStore::StoreFiles ] : self.fDoit=1 Execute: self.store.StoreCdfFile(dataset=hbhd0i,file=/export/stage/ProductionExe_0i_hphysr/mergeOut/hi038836.0573bhd0,parents=hr038836.0573phys) 
Error in &lt;TFile::ReadKeys&gt;: Abnormal case while reading the header key.  The TFile (/export/stage/ProductionExe_0i_hphysr/mergeOut/hi038836.0573bhd0) is probably corrupted. 
terminate called after throwing an instance of 'std::bad_alloc' 
  what():  St9bad_alloc 
Traceback (most recent call last): 
  File "/home/cdfprd/testRel/production_6.1.4/bin/Linux2_SL-GCC_3_4/cdfprd/SamStore.py", line 826, in ? 
    rc = store.Run() 
  File "/home/cdfprd/testRel/production_6.1.4/bin/Linux2_SL-GCC_3_4/cdfprd/SamStore.py", line 623, in Run 
    rc = self.StoreFiles(); 
  File "/home/cdfprd/testRel/production_6.1.4/bin/Linux2_SL-GCC_3_4/cdfprd/SamStore.py", line 367, in StoreFiles 
    rc = self.store.StoreFile(Dataset=dsid,File=fn,Parents=parents); 
  File "/home/cdfprd/testRel/production_6.1.4/bin/Linux2_SL-GCC_3_4/cdfprd/CdfStore.py", line 520, in StoreFile 
    rc = self.GetFileInfo(File) 
  File "/home/cdfprd/testRel/production_6.1.4/bin/Linux2_SL-GCC_3_4/cdfprd/CdfStore.py", line 343, in GetFileInfo 
    (key, value) = string.split (line, ':', 1) 
ValueError: need more than 1 value to unpack 
 
 
mergeOut/hi038836.0573bhd0 
tryed AC++Dump, get the same diagnostics. 
 
 
Moving file to ./ERRORS: 
mv /export/stage/ProductionExe_0i_hphysr/mergeOut/hi038836.0573bhd0 /export/stage/ProductionExe_0i_hphysr/ERRORS</pre>
<!-- Author: Elena Gerchtein -->



<!-- Date: Wed Mar 21 17:09:22 2007 -->
<pre>stream H - continuing recovery. 
 
# from Marcelo: 
sam undeclare reco.hi038836.0573bhd0 
 
Moving files to ./ERRORS: 
mv /export/stage/ProductionExe_0i_hphysr/mergeOut/hi038836.0573bhd0 /export/stage/ProductionExe_0i_hphysr/ERRORS 
cdfprd@fncdfsrv10:/export/stage/ProductionExe_0i_hphysr/mergeOut]$ mv ./hi038836.0573bhd0.par ../ERRORS/ 
 
# mark this file undone on Metalia pages: 
rm home/cdfprd/Projects/ProductionExe_0i_hphysr/Metalia/ProductionExe_0i_hphysr_231478_231478 
</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Thu Mar 22 14:05:50 2007 -->
<pre>stream B 
removed lock  
DataFileNotFound: Datafile with name 'fi000038.7744arm0' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Thu Mar 22 14:08:09 2007 -->
<pre>Stream G 
removed lock 
DataFileNotFound: Datafile with name 'fi000284.0359arm0' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Thu Mar 22 23:39:07 2007 -->
<pre>Stream G 
removed lock 
 
DataFileNotFound: Datafile with name 'fi000284.2764arm0' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Fri Mar 23 15:12:37 2007 -->
<pre>Stream E 
removed lock 
DataFileNotFound: Datafile with name 'fi000284.1870arm0' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Fri Mar 23 15:28:01 2007 -->
<pre>Stream E 
all concatenators stopped due to lack of disk space 
momentarely set minimum number to upload to 5</pre>
<!-- Author: Marcelo -->


<!-- Date: Fri Mar 23 19:05:49 2007 -->
<pre>Stream B: Recovery 
 
file in ERRORS  
bi038723.00e4hel0  bi038723.00e4hel0.par 
 
-AC++Dump failed to open file reco.bi038723.0112hel0 
but returned rc=0 and par file was created including it 
-SamStore refused to upload file because of inconsistency between merged files and par file. 
-file reco.bi038723.0112hel0 was later merged and uploaded separately. 
 
Solution: 
I could have edited par file and deleted reference to reco.bi038723.0112hel0, however I chose to delete merged file, sam undeclare its contensts and edit metalia project summary. 
for reprocessing. 
sam undeclare files: 
reco.bi038723.00e4hel0  
reco.bi038723.00edhel0  
reco.bi038723.00f7hel0  
reco.bi038723.0101hel0  
reco.bi038723.010ahel0 
 
 
 
</pre>
<!-- Author: Marcelo -->


<!-- Date: Fri Mar 23 19:12:05 2007 -->
<pre>Stream H 
removed lock 
DataFileNotFound: Datafile with name 'fi000039.1210arm0' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Sat Mar 24 12:28:40 2007 -->
<pre>Stream E 
removed lock 
DataFileNotFound: Datafile with name 'fi000284.4520arm0' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Sat Mar 24 12:52:09 2007 -->
<pre>Stream B 
Putting stream B in dry-out mode.  
 
MergeSubmit.MinNFilesToMerge 1 
SamStore.LargerEqual         0 
 
One last job running attempting to produce a file for the 2nd time. 
ProductionExe_0i_bphysr_232812_232812 
br038d6c.02c4phys 
 
</pre>
<!-- Author: Marcelo -->
<!-- Comment: Sat Mar 24 13;55;12 comment by...Marcelo -->
<pre>Correction: two jobs running (last two) reconstructing single files.

ProductionExe_0i_bphysr_232812_232812 
br038d6c.02c4phys 

ProductionExe_0i_bphysr_233035_233035
br038e4b.01cephys

will keep stream B in dry-out mode</pre>


<!-- Date: Sun Mar 25 16:12:50 2007 -->
<pre>Stream H 
removed merge lock 
moved offending file to ERRORS 
reco.xi038e76.0083bhd0 
 
R__unzip: error in header 
%ERLOG-e ROOT/TBasket:  
         fNbytes = -447190541, fKeylen = 13854, fObjlen = 215718573, noutot = 0, nout=0, nin=3763544, nbuf=14589941  
          GlobalLibraryLogger TBasket::ReadBasketBuffers()  
         25-Mar-2007 06:42:51 CDT run = 233078 event = 1821563 
%ERLOG-s Abnormal termination: caused by signal:  11 Framework abortJob()  
         25-Mar-2007 06:42:51 CDT run = 233078 event = 1821563</pre>
<!-- Author: Marcelo -->
<!-- Comment: Sun Mar 25 21;17;26 comment by...Marcelo -->
<pre>moved .reco.xi038e76.0083bhd0.ok to ERRORS</pre>
<!-- Date: Sun Mar 25 16:47:58 2007 -->
<pre>Stream J 
removed cronsubmit lock 
 
DataFileNotFound: Datafile with name 'fi000288.0992arm0' not found.</pre>
<!-- Author: Marcelo -->






<!-- Date: Sun Mar 25 23:55:23 2007 -->
<pre>Stream E 
removed lock 
DataFileNotFound: Datafile with name 'fi000039.4999arm0' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Mon Mar 26 11:01:08 2007 -->
<pre>Stream H  
Putting stream B in dry-out mode.   
 
MergeSubmit.MinNFilesToMerge 0  
SamStore.LargerEqual         0 </pre>
<!-- Author: Marcelo -->
<!-- Comment: Mon Mar 26 11;01;56 comment by...Marcelo -->
<pre>Correction: stream H, not B above</pre>


<!-- Date: Tue Mar 27 22:01:16 2007 -->
<pre>Stream J 
 
set 
SamStore.LargerEqual   40</pre>
<!-- Author: Marcelo -->


<!-- Date: Thu Mar 29 13:19:16 2007 -->
<pre>Stream J 
Removed cronsubmit lock 
DataFileNotFound: Datafile with name 'fi000290.0379arm0' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Thu Mar 29 13:25:50 2007 -->
<pre>Stream H 
Removed cronsubmit lock 
 
DataFileNotFound: Datafile with name 'fi000289.9358arm0' not found.</pre>
<!-- Author: Marcelo -->


<!-- Date: Mon Apr 2 14:45:37 2007 -->
<pre>Streams J and E 
set 
SamStore.LargerEqual         50</pre>
<!-- Author: Marcelo -->