<!-- Title: 12:51:09  Sat Dec 30 2006 -->
<!-- Crewchief: Rick St. Denis -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Emergency sam shifter as Valeria has no kerberos password anymore</pre>


<!-- Date: Sat Dec 30 12:52:11 2006 -->
<pre>Rob Harr reports that there are files to be uploaded but they are not being uploaded.  
 
I have requested more information on where and what</pre>
<!-- Author: Rick -->
<!-- Comment: Sat Dec 30 14;55;01 comment by...Rick -->
<pre>Hi Rick and Valeria,
                                                                                                                             
Thanks for looking.  Rick, files are uploaded from specific directories on
fcdfdata133 and fcdfdata134.  The upload proceeds automatically as long as the
data file and corresponding .ok file exist.  The directories are
/export/dataN/cdfmc/to_tape/
where N = 1 to 8.  As a specific example,
fcdfdata133:/export/data1/cdfmc/to_tape/   has about 120 files awaiting upload
(but I believe there were more a couple of hours ago).
                                                                                                                             
Now when I look more closely at the DHAtAGlance page, I see thousands of
failures.  I don't see an obvious reason for the failures -- but I am not
familiar with what should be checked.  This certainly looks odd ... either
thousands of bad MC files, or something broken in the SAM upload, or ...
                                                                                                                             
--- So I sent this to sam on call

HI
  The sam uploads have thousands of errors according to DH at a glance:
                                                                                                                             
Upload Server       New / Submitted / Queued / Failed / Done        Errors          Rate To Enstore         Uploaded Last
Hour / Day / Week / Month / Year
fcdfdata133         549 / 0 / 0 / 2209 / 0          1       0.0             0 / 438 / 6161 / 18974 / 46592
fcdfdata134         891 / 0 / 0 / 1288 / 0          1       0.0             0 / 0 / 7872 / 15895 / 113013
                                                                                                                             
When I click on the Errors I get:
                                                                                                                             
for 133
14:45:07 20061230 1167511507
The previous session is still running, exit !!!
                                                                                                                             
                                                                                                                             
for 134
                                                                                                                             
14:45:03 20061230 1167511503
The previous session is still running, exit !!!
                                                                                                                             
The mystery is what this "previous session" is and what produced it. If
something is hung, what is it doing and how do I check?
                                                                                                                             
                                                                                                                             
So the problem is twofold. There seems to be a hang -- and the second
problem is that we have no idea what to check to fix it so we can see the
root cause.
                                                                                                                             
Rick
</pre>

<!-- Comment: Sat Dec 30 16;12;17 comment by...Rick -->
<pre>Rob responds:

I hope som-oncall is helpful.   My guess is that an instance of the sam uploader is running, but something got corrupted
so that all the files "fail" (one of the grabber checks?).  Maybe stopping and restarting will cure it.
                                                                                                                             
I'm doing top on fcdfdata133 and I see getFileInfo (one of the grabbers) and sampy (I assume this is part of the
uploader) running.
                                                                                                                             
I'm not sure, but I think the main driver script is ~cdfmc/upload/upload.sh.  It appears in the crontab ("crontab -l"
command), and runs every 15 minutes.  But maybe the running instance has gone rogue and needs to be killed?
                                                                                                                             
</pre>

<!-- Comment: Sat Dec 30 16;13;51 comment by...Rick -->
<pre>I cannot do anything in cdfmc since I am not in the k5login for cdfmc --
I am in the k5 login for sam and of course that is not where the crontab runs.
So I did not have a hope of seeing the crontab!  Can you send me the
output of crontab -l?  I need to at least see if this is in the sam
universe where failures occur.  Meanwhile I will try to follow the hint.
</pre>

<!-- Comment: Sat Dec 30 16;16;46 comment by...Rick -->
<pre>Following the hint, I find  fcdfdata133:~cdfmc/upload/upload.sh
runs 
cd /home/cdfmc/upload
 
./upload_fss.py

In upload_fss.py

I find 
if os.path.isfile( PID_File ):
...snip
    else:
        msg = "The previous session is still running, exit !!!"
        

so this is definitly the origin if the issue.

and this me
</pre>

<!-- Comment: Sat Dec 30 16;34;13 comment by...Rick -->
<pre>The jobs run crontabs with no trace files or logs -- no way to know what is going wrong.  The PID lock is in place -- but why?

I have phoned the primar pager.  First try. Left a message.</pre>

<!-- Comment: Sat Dec 30 16;55;32 comment by...Rick -->
<pre>I found the root cause:
16:41:19  [ERROR] Unable to get the metadata structure for file
'/export/data3/cdfmc/submitted/z2032a98.0001topt' --- Failed to import the
ok file, please check its content
                                                                                                                             
                                                                                                                             
                                                                                                                             
Here is how I found it:
                                                                                                                             
In ~cdfmc/upload:
 more upload_config.py
                                                                                                                             
This is more sam-like: there is often a "config" py file.
                                                                                                                             
There you find
PID_File  = "/export/data8/cdfmc/upload.pid"
ERR_File  = "/export/data8/cdfmc/upload.err"
HeartBeat = "/export/data8/cdfmc/upload.HeartBeat"
LOG_Root  = "/export/data8/cdfmc/upload_logs"
                                                                                                                             
BUF_LIST  = [ "/export/data1/cdfmc/", "/export/data2/cdfmc/",
              "/export/data3/cdfmc/", "/export/data4/cdfmc/",
              "/export/data5/cdfmc/", "/export/data6/cdfmc/",
              "/export/data7/cdfmc/" ]
                                                                                                                             
This is very handy since I was able to find the logs and indeed, again,
this is more what I am used to there is
                                                                                                                             
 /export/data8/cdfmc/upload_logs/2006/12/upload_fss_20061230.log
                                                                                                                             
It wall went pear-shaped at 11:17 and the log file is full of messages of
this kind. This is the reason it is hung.
                                                                                                                             
11:17:01  [ERROR] Unable to get the metadata structure for file
'/export/data1/cdfmc//submitted/dw023695.0001top2' --- Failed to import
the ok file, please check its content
                                                                                                                             
-bash-3.00$ find /export/data1/cdfmc/ -name "*dw023695.0001top2*"
/export/data1/cdfmc/failed/dw023695.0001top2
/export/data1/cdfmc/failed/dw023695.0001top2.ok
-bash-3.00$ more /export/data1/cdfmc/failed/dw023695.0001top2.ok
-bash-3.00$
                                                                                                                             
The file is empty.
                                                                                                                             
                                                                                                                             
</pre>

<!-- Comment: Sun Dec 31 13;24;34 comment by...Rick -->
<pre>Hi Rick,
thanks for taking the shift over.
                                                                                                                             
Looking at the problem, it looks like someone changed the upload_util script
Dec 29 11:19 upload_util.py
adding the line:
        raise RuntimeError, "Failed to import the ok file, please check its content"
and a first failure happened:
/export/data8/cdfmc/upload_logs/2006/12/upload_fss_20061229.log
12:33:12  [ERROR] Unable to get the metadata structure for file '/export/data1/cdfmc//submitted/uw031c2b.0002top2' ---
Failed to import the ok file, please check its content
but those ok files are empty, so they can not be wrong.
I am going to restore the old file. I hope it will fix thing for now.
I'll check it tomorrow and go from there.
There is still some space left on the node, so I hope it can wait that long.
Krzysztof
</pre>

<!-- Comment: Sun Dec 31 13;25;15 comment by...Rick -->
<pre>Date: Sat, 30 Dec 2006 23:59:01 -0600 (CST)
From: Dehong_Zhang &lt;dhzhang@fnal.gov&gt;
To: Krzysztof Genser &lt;genser@fnal.gov&gt;
Cc: Richard St. Denis &lt;r.stdenis@physics.gla.ac.uk&gt;, Robert Harr &lt;harr@hep6.physics.wayne.edu&gt;,
    Valeria Bartsch &lt;bartsch@fnal.gov&gt;, cdfdh_oper@fnal.gov, Andrew Ivanov &lt;andrew@fnal.gov&gt;,
    Andreas Warburton &lt;awarburt@hep.physics.mcgill.ca&gt;
Subject: Re: SAM uploads from 133/134
                                                                                                                             
Hi Krzysztof,
                                                                                                                             
Thanks for restoring the system, sorry for the trouble!  I was hoping to
add more protection!  Will check it.
                                                                                                                             
Best regards,
Dehong
                                                                                                                             
</pre>

<!-- Comment: Sun Dec 31 13;26;06 comment by...Rick  -->
<pre>Date: Sun, 31 Dec 2006 01:06:32 -0600
From: Andrew Ivanov &lt;andrew@fnal.gov&gt;
To: Krzysztof Genser &lt;genser@fnal.gov&gt;
Cc: Richard St. Denis &lt;r.stdenis@physics.gla.ac.uk&gt;, Robert Harr &lt;harr@hep6.physics.wayne.edu&gt;,
    Valeria Bartsch &lt;bartsch@fnal.gov&gt;, cdfdh_oper@fnal.gov, Andreas Warburton &lt;awarburt@hep.physics.mcgill.ca&gt;
Subject: Re: SAM uploads from 133/134
                                                                                                                             
                                                                                                                             
      Krzystof,
                                                                                                                             
  Thanks a lot!  That was it, and we are back to 50MB/sec SAM upload.
                                                                                                                             
Cheers,
Andrew</pre>


<!-- Date: Sun Dec 31 13:26:59 2006 -->
<pre>Date: Sun, 31 Dec 2006 06:47:40 -0600 (CST) 
From: HangMonitor@fnal.gov 
To: cdfsam-auto@fnal.gov 
Subject: serverName 'SAMDbServer.test_prd:SAMDbServer' was killed by the HangMonitor 
                                                                                                                              
The serverName 'SAMDbServer.test_prd:SAMDbServer' was killed because it appeared to be hung. 
It should restart automatically. 
                                                                                                                              
                                                                                                                              
</pre>
<!-- Author: Rick -->


<!-- Date: Sun Dec 31 13:27:23 2006 -->
<pre>                                                                                                                              
Date: Sun, 31 Dec 2006 07:10:51 -0600 (CST) 
From: HangMonitor@fnal.gov 
To: cdfsam-auto@fnal.gov 
Subject: serverName 'SAMDbServer.test_prd:SAMDbServer' was killed by the HangMonitor 
                                                                                                                              
The serverName 'SAMDbServer.test_prd:SAMDbServer' was killed because it appeared to be hung. 
It should restart automatically. 
                                                                                                                              
</pre>
<!-- Author: Rick -->


<!-- Date: Sun Dec 31 13:27:59 2006 -->
<pre>Date: Sun, 31 Dec 2006 08:48:00 -0600 (CST) 
From: HangMonitor@fnal.gov 
To: cdfsam-auto@fnal.gov 
Subject: serverName 'SAMDbServer.cdfcafstation_prd:SAMDbServer' was killed by the HangMonitor 
                                                                                                                              
The serverName 'SAMDbServer.cdfcafstation_prd:SAMDbServer' was killed because it appeared to be hung. 
It should restart automatically. 
                                                                                                                              
</pre>
<!-- Author: Rick -->


<!-- Date: Sun Dec 31 13:28:22 2006 -->
<pre>Date: Sun, 31 Dec 2006 09:14:01 -0600 (CST) 
From: HangMonitor@fnal.gov 
To: cdfsam-auto@fnal.gov 
Subject: serverName 'SAMDbServer.test_prd:SAMDbServer' was killed by the HangMonitor 
                                                                                                                              
The serverName 'SAMDbServer.test_prd:SAMDbServer' was killed because it appeared to be hung. 
It should restart automatically. 
                                                                                                                              
                                                                                                                              
</pre>
<!-- Author: Rick -->


<!-- Date: Sun Dec 31 13:28:52 2006 -->
<pre>                                                                                                                              
Date: Sun, 31 Dec 2006 09:32:04 -0600 
From: Krzysztof Genser &lt;genser@fnal.gov&gt; 
To: CDF DH Group &lt;cdfdh_oper@fnal.gov&gt; 
Cc: cdfdb-support@fnal.gov 
Subject: offline db slow [Fwd: serverName 'SAMDbServer.test_prd:SAMDbServer' was killed by the HangMonitor] 
                                                                                                                              
It looks like there may be some problem with the offline db. 
the response time got much worse at about 8am 
http://cdfsam-prd.fnal.gov/~sam/DHAtAGlance/db_response.html 
I restarted dbservers on cdfsam01,18 
Not sure if it will help for long though, 
Krzysztof 
</pre>
<!-- Author: Rick -->
<!-- Comment: Sun Dec 31 13;30;40 comment by...Rick -->
<pre>                                                                                                                             
Date: Sun, 31 Dec 2006 10:51:30 -0600
From: Nelly Stanfield &lt;nelly@fnal.gov&gt;
To: Krzysztof Genser &lt;genser@fnal.gov&gt;, CDF DH Group &lt;cdfdh_oper@fnal.gov&gt;
Cc: cdfdb-support@fnal.gov
Subject: Re: offline db slow [Fwd: serverName 'SAMDbServer.test_prd:SAMDbServer' was killed by the HangMonitor]
                                                                                                                             
Kryzstof,
                                                                                                                             
I am starting to investigating as I see some of our  monitoring tools also has connectivity problems at ~6:30
intermittantly to until  8:30.    And I see at least 7 errors: 'WARNING: inbound connection timed out (ORA-3136)' which
generally mean a user is using an old client and/or connections are timing out.
                                                                                                                             
We have had issues all week-end on the cdfonline with replication.  It seems the cdfofpr2 box is too busy to receive
messages and it halts and cdfonprd has an issue holding them.       Once again we are stuck with replication sync'd up to
only 06:15 this a.m.  so I will need to manually cleanup and restart it.
                                                                                                                             
I see  connected to cdfofpr2:
                                                                                                                             
119 SAMDBS      cdfsam01, cdfsam18, cdfsamra1, cdfsam19 ...
                                                                                                                             
2 CDF_READER  stager which are the top 2 users right now from fcdflnx9
                                                                                                                             
 PID USERNAME THR PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
12223 oracle    11   0    0    0K    0K cpu/6  239:42 13.07% oracle
15199 oracle    11   0    0    0K    0K cpu/5  236:59 13.01% oracle
                                                                                                                             
Box is idle !20
Load average is 6
                                                                                                                             
More after I investigate further.  I will restart replication as well.
                                                                                                                             
Nelly
</pre>
<!-- Date: Sun Dec 31 13:29:23 2006 -->
<pre>Date: Sun, 31 Dec 2006 09:35:26 -0600 (CST) 
From: HangMonitor@fnal.gov 
To: cdfsam-auto@fnal.gov 
Subject: serverName 'SAMDbServer.cdfsamstore_prd:SAMDbServer' was killed by the HangMonitor 
                                                                                                                              
The serverName 'SAMDbServer.cdfsamstore_prd:SAMDbServer' was killed because it appeared to be hung. 
It should restart automatically. 
                                                                                                                              
</pre>
<!-- Author: Rick -->


<!-- Date: Sun Dec 31 13:29:44 2006 -->
<pre>Date: Sun, 31 Dec 2006 09:54:58 -0600 (CST) 
From: HangMonitor@fnal.gov 
To: cdfsam-auto@fnal.gov 
Subject: serverName 'SAMDbServer.test_prd:SAMDbServer' was killed by the HangMonitor 
                                                                                                                              
The serverName 'SAMDbServer.test_prd:SAMDbServer' was killed because it appeared to be hung. 
It should restart automatically. 
                                                                                                                              
</pre>
<!-- Author: Rick -->



<!-- Date: Sun Dec 31 13:31:26 2006 -->
<pre>                                                                                                                              
Date: Sun, 31 Dec 2006 12:36:01 -0600 
From: Nelly Stanfield &lt;nelly@fnal.gov&gt; 
To: Krzysztof Genser &lt;genser@fnal.gov&gt;, CDF DH Group &lt;cdfdh_oper@fnal.gov&gt; 
Cc: cdfdb-support@fnal.gov 
Subject: Re: offline db slow [Fwd: serverName 'SAMDbServer.test_prd:SAMDbServer' was killed by the HangMonitor] 
                                                                                                                              
Reponse on fcdfora4 is slow.  I have replication going again and it should be caught in in the next hour 
                                                                                                                              
Top 3 users 
                                                                                                                              
12223          491   30100 CDF_READER   stager       fcdflnx9.fnal.gov sqlplu 
s@fcdflnx9.fnal.gov (TNS V1-V3) 
                                                                                                                              
15199          425   46701 CDF_READER   stager       fcdflnx9.fnal.gov sqlplu 
s@fcdflnx9.fnal.gov (TNS V1-V3) 
                                                                                                                              
28929          350   30462 SAMREAD      rlc          ncdf164.fnal.gov bin/Li 
nux2_SL-GCC_3_4-maxopt/stnmaker_pr 
                                                                                                                              
Can someone investigate the stager acct connection? 
                                                                                                                              
Nelly 
</pre>
<!-- Author: Rick  -->
<!-- Comment: Sun Dec 31 14;46;44 comment by...Rick -->
<pre>Date: Sun, 31 Dec 2006 13:41:10 -0600
From: Krzysztof Genser &lt;genser@fnal.gov&gt;
To: Nelly Stanfield &lt;nelly@fnal.gov&gt;
Cc: cdfdb-support@fnal.gov, Raymond Culbertson &lt;rlc@fnal.gov&gt;, CDF DH Group &lt;cdfdh_oper@fnal.gov&gt;
Subject: Re: Fw: offline db slow [Fwd: serverName 'SAMDbServer.test_prd:SAMDbServer' was killed by the HangMonitor]
                                                                                                                             
I killed two stager processes. They were daily jobs set up by Randolph. They were not known to have cause problems so
far.
Krzysztof
</pre>

<!-- Comment: Sun Dec 31 15;42;35 comment by...Rick -->
<pre>Date: Sun, 31 Dec 2006 15:04:48 -0600
From: Krzysztof Genser &lt;genser@fnal.gov&gt;
To: Nelly Stanfield &lt;nelly@fnal.gov&gt;
Cc: oem-admin@fnal.gov, Raymond Culbertson &lt;rlc@fnal.gov&gt;, CDF DH Group &lt;cdfdh_oper@fnal.gov&gt;
Subject: Re: Fw: offline db slow [Fwd: serverName 'SAMDbServer.test_prd:SAMDbServer' was killed by the HangMonitor]
                                                                                                                             
Hi Nelly,
I do not see any sql stager processes on fcdflnx9, so I think you can kill those on fcdfora4. We can discuss the details
on Tuesday. There is nothing what critically depends on those queries. I had commented out some of the stager cron jobs
which would run tomorrow.
Thanks for dealing with it all.
Krzysztof
</pre>


<!-- Comment: Sun Dec 31 15;43;30 comment by...Rick  -->
<pre>Date: Sun, 31 Dec 2006 15:19:57 -0600
From: Nelly Stanfield &lt;nelly@fnal.gov&gt;
To: Krzysztof Genser &lt;genser@fnal.gov&gt;
Cc: oem-admin@fnal.gov, Raymond Culbertson &lt;rlc@fnal.gov&gt;, CDF DH Group &lt;cdfdh_oper@fnal.gov&gt;
Subject: Re: Fw: offline db slow [Fwd: serverName 'SAMDbServer.test_prd:SAMDbServer' was killed by the HangMonitor]
                                                                                                                             
hi krzysztof,
                                                                                                                             
great, thanks for getting back to me.
                                                                                                                             
i have killed the 2 processes and the load average has drastically reduced. i hope this is the ticket and we will
definitely continue our investiagtion tuesday.    i hope this helps settle things out til then.
                                                                                                                             
signing off
nelly
                                                                                                                             
USERNAME                       PROGRAM
------------------------------ ------------------------------------------------
MACHINE                                                          STATUS
---------------------------------------------------------------- --------
      425      46701 stager
CDF_READER                     sqlplus@fcdflnx9.fnal.gov (TNS V1-V3)
fcdflnx9.fnal.gov                                                ACTIVE
                                                                                                                             
      491      30100 stager
CDF_READER                     sqlplus@fcdflnx9.fnal.gov (TNS V1-V3)
fcdflnx9.fnal.gov                                                ACTIVE
                                                                                                                             
                                                                                                                             
SQL&gt; alter system kill session '425,46701';
                                                                                                                             
System altered.
                                                                                                                             
SQL&gt; alter system kill session '491,30100';
                                                                                                                             
System altered.
</pre>
<!-- Comment: Sun Dec 31 18;04;09 comment by...Rick -->
<pre>Probably a consequence of getting killed....

Date: Sun, 31 Dec 2006 17:22:32 -0600
From: Randolph Herber &lt;herber@fcdflnx9.fnal.gov&gt;
To: cdfsam-auto@fnal.gov, herber@fnal.gov
Subject: Files added to Enstore 12/31/06-17:22 (in previous 2 hours)
                                                                                                                             
ERROR:
ORA-03113: end-of-file on communication channel
</pre>
<!-- Date: Sun Dec 31 15:43:01 2006 -->
<pre>Date: Sun, 31 Dec 2006 15:08:07 -0600 (CST) 
From: HangMonitor@fnal.gov 
To: cdfsam-auto@fnal.gov 
Subject: serverName 'SAMDbServer.station_prd2:SAMDbServer' was killed by the HangMonitor 
                                                                                                                              
The serverName 'SAMDbServer.station_prd2:SAMDbServer' was killed because it appeared to be hung. 
It should restart automatically. 
                                                                                                                              
</pre>
<!-- Author: Rick -->



<!-- Date: Sun Dec 31 17:32:08 2006 -->
<pre>Date: Sun, 31 Dec 2006 17:08:06 -0600 (CST) 
From: HangMonitor@fnal.gov 
To: cdfsam-auto@fnal.gov 
Subject: serverName 'SAMDbServer.test_prd:SAMDbServer' was killed by the HangMonitor 
                                                                                                                              
The serverName 'SAMDbServer.test_prd:SAMDbServer' was killed because it appeared to be hung. 
It should restart automatically. 
                                                                                                                              
</pre>
<!-- Author: Rick  -->


<!-- Date: Sun Dec 31 17:32:36 2006 -->
<pre>Date: Sun, 31 Dec 2006 17:22:04 -0600 (CST) 
From: HangMonitor@fnal.gov 
To: cdfsam-auto@fnal.gov 
Subject: serverName 'SAMDbServer.cdfcafstation_prd:SAMDbServer' was killed by the HangMonitor 
                                                                                                                              
The serverName 'SAMDbServer.cdfcafstation_prd:SAMDbServer' was killed because it appeared to be hung. 
It should restart automatically. 
                                                                                                                              
</pre>
<!-- Author: Rick  -->



<!-- Date: Sun Dec 31 18:05:23 2006 -->
<pre>                                                                                                                              
Date: Sun, 31 Dec 2006 17:31:17 -0600 (CST) 
From: HangMonitor@fnal.gov 
To: cdfsam-auto@fnal.gov 
Subject: serverName 'SAMDbServer.test_prd:SAMDbServer' was killed by the HangMonitor 
                                                                                                                             
The serverName 'SAMDbServer.test_prd:SAMDbServer' was killed because it appeared to be hung. 
It should restart automatically. 
                                                                                                                             
                                                                                                                             
    [ Note: This message contains email list management information ] 
                                                                                                                             
</pre>
<!-- Author: Rick -->


<!-- Date: Mon Jan 1 08:25:25 2007 -->
<pre>The db server SAMDbServer.cdfcafstation_prd:SAMDbServer  
was not responding and was dead. This affected file delivery 
at around 01:00 this morning. File delivery has resumed. 
One should watch this db server for failure again. 
 
 </pre>
<!-- Author: Doug Benjamin -->


<!-- Date: Mon Jan 1 10:49:32 2007 -->
<pre>A rash of problems as the new year hits 
o  Communication failures from cdf-caf station to DB 
o cdfcafstation_prd and farm1station_prd starting automatically often 
o cdfsamstore_prd also restarted automatically 
o Dehong logged in to restart rawdata_prd and test_prd 
o communication failures in the database all night long. 
o Memory errors on fcdfdata133 found by Doug Benjamin 
I would think there are two separate issues 
1. Oracle Database server issue 
2. Hardware on fcdfdata133 (unrelated) 
 
Will page sam on call. 
                                                                                                                             
01/01/07 01:09:46 cdf-caf.SM.FileUncachedInDBTimer 2329: Cannot uncache file in DB: 
CORBA::SystemException caught in FileUncachedInDBTimer::doIt(): 
System exception `COMM_FAILURE' 
Completed: no 
Minor code: 1330577411 (recv() returned zero) 
</pre>
<!-- Author: Rick -->
<!-- Comment: Mon Jan  1 10;51;20 comment by...Rick -->
<pre>Failure in communication seems limited to cdf-caf so unless this is the only one making requests to the oracle server, this seems to be another problem.  So I would not look at the oracle server just now. </pre>


<!-- Date: Mon Jan 1 10:55:52 2007 -->
<pre>Seems that online is not collapsing so wont bother to page. 
However I do note that cdf_raw_prd is not responding. Seems to have worked for a while after Dehong started it but now is not running.</pre>
<!-- Author: Rick -->
<!-- Comment: Mon Jan  1 10;59;21 comment by...Rick -->
<pre>Can't log in so can't restart it. Sent mail to cdfdh_oper</pre>


<!-- Date: Mon Jan 1 18:14:43 2007 -->
<pre>Kicked DB Servers since the samstore server was down and Rob emailed that he was worried about the lack of transfers. 
 
Looking at the fss log I see a number of errors that Rob should have a 
look at: datasets not registered, file not recongized so the grabber 
cannot run etc. 
                                                                                                                           
But the last error is : 
                                                                                                                           
17:59:13  [ERROR] Unable to store file 
'/export/data6/cdfmc/submitted/xp02fcdd.0001top7' --- TRANSIENT; 
CORBA.TRAN\SIENT(omniORB.TRANSIENT_CallTimedout, CORBA.COMPLETED_MAYBE) 
18:00:06  [ERROR] The previous session is still running, exit !!! 
18:04:19  [ERROR] Unable to get destination for file 
'/export/data6/cdfmc/submitted/xp02fe4a.0001top7' --- TRANSIE\NT; 
CORBA.TRANSIENT(omniORB.TRANSIENT_CallTimedout, CORBA.COMPLETED_MAYBE) 
                                                                                                                           
and I see 
SAMDbServer.cdfsamstore_prd:SAMDbServer 
is dead, so I will restart. 
 
 
I Did a KickDBServers on cdfsam01</pre>
<!-- Author: Rick -->


<!-- Date: Tue Jan 2 10:51:58 2007 -->
<pre>Mark Schmitz is requesting that fcdfdata133 be rebooted  
to see if the memory allocation errors go away. 
 
I have edited the crontab command to stop the sam upload 
portion of the machine usage ahead of a reboot.</pre>
<!-- Author: Doug Benjamin -->
<!-- Comment: Tue Jan  2 15;04;16 comment by...Doug B -->
<pre>fcdfdata133 has been rebooted and Dehong Zhang restarted sam and cdfmc crontab's. </pre>


<!-- Date: Sat Jan 6 06:18:54 2007 -->
<pre>One last check on ACTIVE processes on cdfofpr2~fcdfora4. 
  
Sat Jan  6 00:04:21 CST 2007 
 
select username,program from v$session where status = 'ACTIVE'  and osuser != 'oracle' 
SQL&gt; / 
  
USERNAME                       PROGRAM 
------------------------------ ------------------------------------------------ 
SAMDBS                         python@cdfsam18.fnal.gov (TNS V1-V3) 
SAMDBS                         python@cdfsam01.fnal.gov (TNS V1-V3) 
SAMDBS                         python@cdfsam01.fnal.gov (TNS V1-V3) 
SAMDBS                         python@cdfsam19.fnal.gov (TNS V1-V3) 
SAMDBS                         python@cdfsam18.fnal.gov (TNS V1-V3) 
CDF_READER                     /fbsng/condor_local/execute/dir_22449/work/bin/L 
SAMDBS                         python@cdfsam19.fnal.gov (TNS V1-V3) 
SAMDBS                         python@cdfsam18.fnal.gov (TNS V1-V3) 
SAMDBS                         python@cdfsam19.fnal.gov (TNS V1-V3) 
SAMDBS                         python@cdfsam01.fnal.gov (TNS V1-V3) 
CDF_READER                     sqlplus@fncdfsrv0.fnal.gov (TNS V1-V3) 
CDF_READER                     /fbsng/condor_local/execute/dir_6044/work/bin/Li 
CDF_READER                     /fbsng/condor_local/execute/dir_18568/work/bin/L 
SAMDBS                         python@cdfsam19.fnal.gov (TNS V1-V3) 
CSLACE                         sqlplus@b0gateway.fnal.gov (TNS V1-V3) 
SAMDBS                         python@cdfsam19.fnal.gov (TNS V1-V3) 
CDF_READER                     sqlplus@cdfsamweb.fnal.gov (TNS V1-V3) 
SAMDBS                         python@cdfsam19.fnal.gov (TNS V1-V3) 
SAMDBS                         python@cdfsam19.fnal.gov (TNS V1-V3) 
CDF_READER                     /fbsng/condor_local/execute/dir_30751/work/bin/L 
SAMDBS                         python@cdfsam01.fnal.gov (TNS V1-V3) 
SAMDBS                         python@cdfsam19.fnal.gov (TNS V1-V3) 
CDF_READER                     sqlplus@fcdflnx9.fnal.gov (TNS V1-V3) 
CDF_READER                     sqlplus@cdfsamweb.fnal.gov (TNS V1-V3) 
CDF_READER                     /fbsng/condor_local/execute/dir_18578/work/bin/L 
FILECATALOG_READUSER           /fbsng/condor_local/execute/dir_29071/work/bin/L 
CDF_READER                     /fbsng/condor_local/execute/dir_16465/work/bin/L 
SAMDBS                         python@cdfsam19.fnal.gov (TNS V1-V3) 
CDF_READER                     /fbsng/condor_local/execute/dir_9229/work/bin/Li 
SAMDBS                         python@cdfsam19.fnal.gov (TNS V1-V3) 
CDF_READER                     /fbsng/condor_local/execute/dir_2028/work/bin/Li 
FILECATALOG_READUSER           /fbsng/condor_local/execute/dir_10567/work/bin/L 
SAMDBS                         python@cdfsam19.fnal.gov (TNS V1-V3) 
SAMDBS                         python@cdfsam19.fnal.gov (TNS V1-V3) 
CDF_READER                     ./bin/Linux2_SL-GCC_3_4-maxopt/stnmaker_prod.exe 
SAMDBS                         python@cdfsam18.fnal.gov (TNS V1-V3) 
SAMDBS                         python@cdfsam18.fnal.gov (TNS V1-V3) 
CDF_READER                     /fbsng/condor_local/execute/dir_31544/work/bin/L 
CDF_READER                     /fbsng/condor_local/execute/dir_25227/work/bin/L 
SAMDBS                         python@cdfsam18.fnal.gov (TNS V1-V3) 
CDF_READER                     perl@fncdfsrv0.fnal.gov (TNS V1-V3) 
CDF_READER                     /local/stage1/condor/execute/dir_6144/work/bin/L 
SAMDBS                         python@cdfsam19.fnal.gov (TNS V1-V3) 
</pre>
<!-- Author: Doug Benjamin -->


<!-- Date: Sat Jan 6 06:19:50 2007 -->
<pre>I restarted the cafstation dbserver and the files are being 
delivered again.  File delivery stopped at 01:00 this morning.</pre>
<!-- Author: Doug Benjamin -->


<!-- Date: Sat Jan 6 07:19:57 2007 -->
<pre>I am restarting the db servers with the largest number of 
connections. 
 
restarted 
 
dbs dbs_farmstation_prd    v7_7_0 
dbs dbs_farm1station_prd   v7_7_0 
 
 
dbs dbs_farm1_srv1_prd v7_7_0 
dbs dbs_farm1_srv2_prd v7_7_0 
dbs dbs_farm1_srv3_prd v7_7_0 
dbs dbs_farm1_srv4_prd v7_7_0 
dbs dbs_farm1monitoring_prd v7_7_0 
dbs dbs_farm2_prd v7_7_0 
dbs dbs_farm2monitoring_prd v7_7_0 
dbs dbs_farm1_prd v7_7_0 
dbs dbs_test_prd v8_0_4 
 
I was hoping that bouncing the db servers would release some 
of the load on the database machine.  
 
With all of the dbservers on cdfsam18 restarting the load spiked 
and took several minutes to relax. I will monitor the database time plots to see if there is any improvement. 
</pre>
<!-- Author: Doug Benjamin -->


<!-- Date: Sat Jan 6 21:28:00 2007 -->
<pre>Hello, 
 
 Things are not normal with SAM.  The raw data logger db server has been down since 
20:30 and the hang monitor first killed it at ~ 18:15. I can not log into the machine and 
restart the db server. Various db servers are showing long times for the simple sam locate 
command. 
 
   Doug Benjamin 
 
</pre>
<!-- Author: Doug  -->


<!-- Date: Sat Jan 6 21:28:46 2007 -->
<pre>SAM sifters are not in the cdfsamraw1 sam .k5login so we can 
restart the db server</pre>
<!-- Author: Doug -->


<!-- Date: Sun Jan 7 07:29:22 2007 -->
<pre>Dear all, 
 
  I just restarted the farm1station db server. The dh at a glance plots showed 
no response from the db server. It is now responding. 
 
  Doug 
 
PS I can not restart the other "dead" db server rawdata because I am not 
in the sam account .k5login account on cdfsamraw1 
 
</pre>
<!-- Author: Doug Benjamin -->


<!-- Date: Sun Jan 7 13:52:22 2007 -->
<pre>Dear all, 
 
   cdfcafstation_prd db server on node cdfsam19 had to be restarted 
yet again. This db server running v7_7_0 is for the main caf sam station 
and normally does not have this many problems!!  The instability of this 
db server is cause problems for all CDF SAM jobs running on the CAF. 
This also had a negative effect on the production jobs running on the 
caf. Would some expert please look at the log/trace files to understand why 
we are experiencing such problems. 
 
   Regards, 
 
   Doug Benjamin 
 
  (SAM shifter) 
</pre>
<!-- Author: Doug Benjamin -->


<!-- Date: Sun Jan 7 13:52:24 2007 -->
<pre>Dear all, 
 
   cdfcafstation_prd db server on node cdfsam19 had to be restarted 
yet again. This db server running v7_7_0 is for the main caf sam station 
and normally does not have this many problems!!  The instability of this 
db server is cause problems for all CDF SAM jobs running on the CAF. 
This also had a negative effect on the production jobs running on the 
caf. Would some expert please look at the log/trace files to understand why 
we are experiencing such problems. 
 
   Regards, 
 
   Doug Benjamin 
 
  (SAM shifter) 
</pre>
<!-- Author: Doug Benjamin -->


<!-- Date: Sun Jan 7 20:34:14 2007 -->
<pre>I restarted farm1station_prd db server 
Current SAM_BOOTSTRAP_ENV is /home/sam/products/upsdb/sam_bootstrap/sam_bootstrap.env 
RESTART: stopping 
#dpbdbs dbs_farm1station_prd v7_7_0 
RESTART: starting 
dbs dbs_farm1station_prd v7_7_0 
sam_bootstrap&gt;run.sh start dbs dbs_farm1station_prd v7_7_0   
</pre>
<!-- Author: Doug B -->


<!-- Date: Tue Jan 9 15:19:41 2007 -->
<pre>Hi, 
 
  After noticing that the cdf caf sam station has been stopped for almost one 1 hour 
(according to the dh at a glance plots). I restarted it.  There was not unusual timing 
problems with the main database though. 
 
http://cdfsam-prd.fnal.gov/~sam/sam_dbs_info.html 
 
The trace file shows this error about the time the db server "locked up" 
 
&lt;HangMonitor 09-Jan-2007 13:58:42 (CST)&gt; Ping failed with inconclusive exception omniORB.CORBA.COMM_FAILURE CORBA.COMM_FAILURE(omniORB.COMM_FAILURE_WaitingForReply, CORBA.COMPLETED_MAYBE) 
&lt;HangMonitor 09-Jan-2007 13:58:42 (CST)&gt; Server is assumed to be alive. 
 
  Cheers, 
 
   Doug B 
 
</pre>
<!-- Author: Doug B -->


<!-- Date: Fri Jan 12 11:08:52 2007 -->
<pre>Node fcdfdata154 is offline in dCache. 
It is not pingable. I will send e-mail to the 
system administrators to restart the node</pre>
<!-- Author: Doug Benjamin -->