<!-- Title: 08:58:11  Tue Mar 11 2003 -->
<!-- Crewchief: New Day -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>database activity</pre>


<!-- Date: Tue Mar 11 08:58:32 2003 -->
<pre>rerunning rman last nights failed the cron job for cleanup 
failed so rman immediately failed due to space</pre>
<!-- Author: ns -->


<!-- Date: Tue Mar 11 09:01:07 2003 -->
<pre>Clean up for MCS will not continue. 
 
I do not have additional undo or temp space. 
 
Still looking to get additional space on  
 
/dev/vx/dsk/oradg/tempdatvol 
                     5242880 1864356 3167372    38%    /dbs24 
/dev/vx/dsk/oradg/rollbkvol 
                     3145728 3144487    1170   100%    /dbs25 
</pre>
<!-- Author: ns -->
<!-- Comment: Tue Mar 11 11;45;44 comment by...js -->
<pre>ok, it's going pretty slow, since some of the filesystems I'm dealing with are 68gb +</pre>
<!-- Date: Tue Mar 11 09:31:00 2003 -->
<pre>Many local=no on box  Still trying to understand 
what daspc0 connections are? why the connecitons below 
do not get terminated! my guess is they are not dead connections 
as oracle knows them so they are not cleaned up! 
 
They came on the machine when archiver hung 4:51 and 4:56... 
 
It;s been 12 hours think things are fine but I will try 
to learn who is daspc0. 
 
Explanation of cdffarms2 below the local=no connections which 
will be killed off today.  
 
Only people in oracle's .k5login are nelly, diana, julie, anil, johnw, alansill.  
 
 
  oracle 27154     1  0 16:47:42 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27883     1  0 16:50:50 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27324     1  0 16:49:15 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 26650     1  0 16:46:01 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 28537     1  0 16:52:38 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27887     1  0 16:50:51 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27127     1  0 16:47:36 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27332     1  0 16:49:16 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27901     1  0 16:50:58 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 18034     1  0   Feb 27 ?       14:40 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 26632     1  0 16:45:56 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27133     1  0 16:47:38 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27893     1  0 16:50:55 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27358     1  0 16:49:22 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 28525     1  0 16:52:33 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 28539     1  0 16:52:39 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27117     1  0 16:47:33 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27891     1  0 16:50:54 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 26662     1  0 16:46:05 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27346     1  0 16:49:19 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27895     1  0 16:50:56 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27889     1  0 16:50:53 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27356     1  0 16:49:21 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27152     1  0 16:47:41 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 18026     1  0   Feb 27 ?        0:06 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27142     1  0 16:47:40 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 28531     1  0 16:52:36 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 28521     1  0 16:52:32 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 17595     1  0   Feb 27 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 18032     1  0   Feb 27 ?        3:34 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27131     1  0 16:47:37 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27125     1  0 16:47:35 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27899     1  0 16:50:57 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 22532     1  0 20:50:07 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 26638     1  0 16:45:57 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 28501     1  0 16:52:29 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27348     1  0 16:49:20 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27139     1  0 16:47:39 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 26652     1  0 16:46:02 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle  2510     1  0 19:30:33 ?        0:02 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 26626     1  0 16:45:55 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 26658     1  0 16:46:04 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27318     1  0 16:49:12 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 17593     1  0   Feb 27 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 17597     1  0   Feb 27 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle  2416     1  9 19:30:01 ?       30:22 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 28515     1  0 16:52:31 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 26654     1  0 16:46:03 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 18028     1  0   Feb 27 ?        3:05 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 28535     1  0 16:52:37 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 22981     1  5 20:51:15 ?        0:02 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27158     1  0 16:47:44 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27340     1  0 16:49:17 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 26640     1  0 16:45:58 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 17091     1  0   Feb 27 ?        8:43 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27322     1  0 16:49:14 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 27907     1  0 16:50:59 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 26644     1  0 16:45:59 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 17599     1  0   Feb 27 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 
  oracle 28505     1  0 16:52:30 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=B 
 
 
 
REGARDING CDFFARMS 2 
 
From cdffarm2 (I think you are asking about this machine,although you  
wrote 'cdfarms2')  we are  connecting to database from : 
 
1, daemon "dispatcher",which starts jobs processing the data  on the farm.  
This daemon is connecting to database to find out level3 tag. It executes only this 
one query: 
""" 
select l3_tcls.tcl_tag from runconfigurations, runsets, valid_tag_sets, 
l3_tcls where runconfigurations.rs_name = runsets.name and  
runsets.l3tagset = 
valid_tag_sets.l3tagset and valid_tag_sets.tcl_tag = l3_tcls.tcl_tag and 
runconfigurations.runnumber="+str(run) 
""" 
 
But last job with dispatcher was started on: 
""" 
Mar 08 00:19:27 2003: Submitted job 66588 
""" 
and so,the last connection should be before this time. 
 
 
2, cronjob 'progress.csh' which executes at 1.00 in night,and is used to  
create graph on our web page and I haven't seen any problem with graph... 
 
3, cronjob "good_run_list_check" 
This cronjob is running every hour(starts at whole hour): 
from crontab: 
""" 
0 0-23/1 * * * /home/cdfprod0/good_runs/good_run_list_check.csh 
""" 
 
and checks the status of good runs in database and compares with our  
internal database.  
 
So, I don't know about nothig what could be connecting between 4:51 pm  
and 4:57. 
 
 
About machine 'daspc0' I don't know anything. 
 
Roman 
 
we have one more daemon at cdffarm2 that "checks" the health of the offline production database about once in a minute or so. It does so by connecting to the database and closing the connection right after it gets it. No activity whatsoever is performed by this daemon. This daemon checks the conditions on the farm for all of the sub components. 
 
Miro 
 
</pre>
<!-- Author: nstanfield  -->
<!-- Comment: Tue Mar 11 09;55;12 comment by...ns  -->
<pre>processes left remotely logged in db.. replication 
and sam all processes that came on 4:51 to 4:57 from
cdffarms2 and daspc0  killed manually oracle will
not clean them up since it is a strange dead connection

still trying to locate daspc0

  oracle 27901     1  0 16:50:58 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
  oracle 18034     1  4   Feb 27 ?       27:07 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
  oracle 18026     1  0   Feb 27 ?        0:06 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
  oracle 17595     1  0   Feb 27 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
  oracle 18032     1  0   Feb 27 ?        3:36 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
  oracle 17593     1  0   Feb 27 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
  oracle 17597     1  0   Feb 27 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
  oracle 18028     1  0   Feb 27 ?        3:05 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
  oracle 17091     1  0   Feb 27 ?        9:08 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
  oracle 17599     1  0   Feb 27 ?        0:00 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))
  oracle  5243     1  6 09:10:01 ?       22:37 oraclecdfofprd (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ)))</pre>

<!-- Comment: Tue Mar 11 10;23;21 comment by...ns -->
<pre>There are many many connections in the oracle's
listener file  that reflect the info below:


10-MAR-2003 00:03:00 *
(CONNECT_DATA=(SID=cdfofprd)(GLOBAL_NAME=cdfofprd.fnal.gov)(CID=(PROGRAM=)(HOST=cdffarm2.fnal.gov)(USER=cdfprod0))) *
(ADDRESS=(PROTOCOL=tcp)(HOST=131.225.238.3)(PORT=34717)) * establish * cdfofprd * 0

There are several during this period.. our trouble
period.. that did not connect to the db but tried
to.

10-MAR-2003 16:51:01 *
(CONNECT_DATA=(SID=cdfofprd)(GLOBAL_NAME=cdfofprd.fnal.gov)(CID=(PROGRAM=)(HOST=cdffarm2.fnal.gov)(USER=cdfprod0))) *
(ADDRESS=(PROTOCOL=tcp)(HOST=131.225.238.3)(PORT=55623)) * establish * cdfofprd * 0

11-MAR-2003 10:09:43 *
(CONNECT_DATA=(SID=cdfofprd)(GLOBAL_NAME=cdfofprd.fnal.gov)(CID=(PROGRAM=)(HOST=cdffarm2.fnal.gov)(USER=cdfprod0))) *
(ADDRESS=(PROTOCOL=tcp)(HOST=131.225.238.3)(PORT=49127)) * establish * cdfofprd * 0

Why are they trying to connect if there is no
FARMS activity?

</pre>
<!-- Comment: Tue Mar 11 12;16;38 comment by...ns -->
<pre>Hi All

We have been investigating the interruption
on cdfofprd between 4:51 pm and 4:56 yesterday.

We have been investigating daspc0 and it's
location.  I was just in Anil's office
and unplugged his Linux box from the
network while Richard did a ping.  It
seems this machine is Anil's linux desktop.

We had connections from this box to
fcdfora1 yesterday when we had the
issue with the /cdf/dbs_bkup area showing
100% interrupting connections to the
database between 4:51 and 4:56.  We also
had many many connections from cdffarms2.

We have resolved the issues.  Anil
did cleanup that mount point, and we 
understand  the connections
to the machines from daspc0 and cdffarm2
are legitimate.  Although the cdffarm2
was not running it did attempt to connect?

</pre>
<!-- Date: Tue Mar 11 10:08:22 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace SYSTEM is 47.59% full.  103776.0 KBytes of free space remain. Tablespace RBS is 51.58% full.  942070.0 KBytes of free space remain. Tablespace LRG_RBS is 23.44% full.  5.01757e+06 KBytes of free space remain. Tablespace USERDATA_01 is 59.23% full.  1.97198e+06 KBytes of free space remain. Tablespace USERINDEX_01 is 35.25% full.  5.03879e+06 KBytes of free space remain. Tablespace SYSTEM_REP_DATA is 72.20% full.  193240.0 KBytes of free space remain. Tablespace TOOLMAN is 72.05% full.  1.37378e+06 KBytes of free space remain. Tablespace CALIB_PRD_REP_DATA is 69.52% full.  1.29221e+07 KBytes of free space remain. Tablespace CALIB_PRD_REP_INDEX is 54.08% full.  2.96585e+07 KBytes of free space remain. Tablespace REPADMIN_PRD_DATA is 24.74% full.  77064.0 KBytes of free space remain. Tablespace HDWDB_PRD_REP_DATA is 42.38% full.  557608.0 KBytes of free space remain. Tablespace RUNDB_PRD_REP_DATA is 66.07% full.  2.81458e+06 KBytes of free space remain. Tablespace RUNDB_PRD_RE
CRITICAL 
Mar 11 2003 10:07:46 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 10:21:03 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Mar 11 2003 10:20:46 AM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Tue Mar 11 10:30:23 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : STARTED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 10:30:23 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : COMPLETED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 10:41:30 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Failed Jobs 
Event test "Failed Jobs" is cleared 
CLEARED 
Mar 11 2003 10:40:45 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 10:41:30 2003 -->
<pre>
Target Name : fcdfora1.fnal.gov 
Target Type : Node 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Data Gatherer UpDown 
Event test "Data Gatherer UpDown" is cleared 
CLEARED 
Mar 11 2003 10:40:45 AM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Tue Mar 11 11:48:06 2003 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=7022 sec, SCN: 0x0000.4af0ed17):. 
WARNING 
Mar 11 2003 11:47:44 AM 



</pre>
<!-- Author: oem-admin (email) -->




<!-- Date: Tue Mar 11 12:34:50 2003 -->
<pre>Discussions on fcdfora2/1 replacements  
HI, 
 
CPU load is growing on fcdflnx1 to the point that it is between 25% and  
50% loaded at all times, on average, sometimes for hours between 50%  
and 75%.  Under these conditions I cannot expect to shut off cdfrep01  
and have fcdfora1 pick up the load without overloading. 
 
We can deal with short term anticipated growth in user load by adding  
fcdfora3 to the mix (and continuing to try to improve the efficiency of  
CDF code, our only ace-in-the-hole), but in general we knew at the  
outset that we would have to replace fcdfora1 as we would eventually  
reach this condition. 
 
Having 2 replicas to load balance and take the bulk of the user load  
when we get streams running will be reassuring (and I hope will improve  
and simplify the replication picture), but we should have amore  
substantial system with its equivalent dev/int system behind this.   
Also SAM usage will be growing and right now the SAM db servers connect  
only to cdfofprd, something that will be more important in the future. 
 
Hope this helps clear up some of the issues... 
 
Thanks, 
Alan 
 
On Monday, March 10, 2003, at 04:27  PM, Nelly Stanfield wrote: 
 
&gt; Hi Alan 
&gt; 
&gt; Trying to multitask here.  Julie has been 
&gt; asking about the status for the fcdfora1/2 
&gt; replacement.  What is CDF timeline replacement 
&gt; for these machines?  As you can see 
&gt; our plan is to understand fcdfora3 utilization 
&gt; then make a recommendation.  I want to make 
&gt; sure I am not holding CDF up? 
&gt; 
&gt; Thanks 
&gt; Nelly 
&gt; ----- Original Message ----- 
&gt; From: "Nelly Stanfield" &lt;nelly@fnal.gov&gt; 
&gt; To: &lt;css-dsg@fnal.gov&gt; 
&gt; Sent: Monday, March 10, 2003 4:23 PM 
&gt; Subject: replacing fcdfora1/2 
&gt; 
&gt; 
&gt;&gt; Julie 
&gt;&gt; 
&gt;&gt; To update you on fcdfora1/2 replacements. Richard 
&gt;&gt; is working with us since he understands 
&gt;&gt; fcdfora1, cdfrep01 utilization.  We gave 
&gt;&gt; him spec's on RAC which we will not be pursuing. 
&gt;&gt; We half thought we would buy something that 
&gt;&gt; was flexible enough to grow into RAC but have 
&gt;&gt; decided against it. 
&gt;&gt; 
&gt;&gt; We need (ANIL AND I) to start up our stream 
&gt;&gt; integration testing on FCDFORA3 this will 
&gt;&gt; allow us to see how this linux box performs 
&gt;&gt; along with Richard watching i/o, writes, etc. 
&gt;&gt; 
&gt;&gt; We know for sure we will not use IDE disks 
&gt;&gt; on the replacements even though fcdfora3 
&gt;&gt; is IDE at this time.  Testing this box 
&gt;&gt; will give us another metric. 
&gt;&gt; 
&gt;&gt; Then I need to understand from CDF what 
&gt;&gt; the schedule is for the replacements as 
&gt;&gt; this will definitely affect Richard's purchase. 
&gt;&gt; 
&gt;&gt; Below is the requisition Fagan filled out 
&gt;&gt; for the d0ora1 replacement.  Nice machine! 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; D0 replacement for d0ora1 
&gt;&gt; 
&gt;&gt; Request: CD89432 Status: In MMS (See the Lab's MMS Sytem ) 
&gt;&gt;       Fiscal Year: 2003 Request Date:  02/13/2003 
&gt;&gt;       Requester Badge: 08244N FAGAN, DAVID J 
&gt;&gt;       Requisition For: CD/D0A 
&gt;&gt;       Suggested Vendor:  Best Source Need by Date: 06/30/2003 Sole  
&gt;&gt; Source? N 
&gt;&gt;       Deliver To: FCC PREP 
&gt;&gt;      Choose Person 
&gt;&gt;       Short Project Description  Sun production server 
&gt;&gt;      In Warranty? N Urgent? N 
&gt;&gt;       Blanket Order?N 
&gt;&gt;       Change Order? N 
&gt;&gt;       Line  Quantity  Description  Price  Budget 
&gt;&gt;       Code  Cost 
&gt;&gt;       Element  Line 
&gt;&gt;       Type  Category  Div Project  Dept Project  Capitalize  WBS#   
&gt;&gt; Line Total 
&gt;&gt;       1 1 EACH Sun V880,8x900Mhz,16GB memory,12-73GB fibre Hdd,3  
&gt;&gt; add'l Fibre Channel optical Gigabit Ethernets's, 4 Fibre 
&gt; Channel 
&gt;&gt; optical controllers.Includes Trade-in credit(4-16x300 Origin  
&gt;&gt; 2000's,w/ 3 Meta Routers & 8 Cards(represents min $40K credit) 
&gt;&gt; 43700.00 ORP 41 Goods-Receivables COMPUTER PERIPHERALS  
&gt;&gt; D0-RUN2-ANALYSIS-CPU  SELF CD89432  $ 43,700 
&gt;&gt; 
&gt; 
&gt; 
</pre>
<!-- Author: ns  -->


<!-- Date: Tue Mar 11 12:53:57 2003 -->
<pre>we were living on 1 72G disk with 4G to spare.. now NOTHING 
is to spare! 
 
 
space 
 
Filesystem            kbytes    used   avail capacity  Mounted on 
/dev/vx/dsk/oradg/backupvol 
                     71303168 70613696  684152   100%    /backup</pre>
<!-- Author: nstanfield  -->

<!-- Date: Tue Mar 11 15:28:06 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 3736 ORA-12008: error in snapshot refresh path ORA-01555: snapshot too old: rollback segment number 72 with name "_SYSSMU72$" too small ORA-02063: preceding line from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 ORA-06512: at line 1. 
WARNING 
Mar 11 2003 03:27:26 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 15:28:06 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 3736 ORA-12008: error in snapshot refresh path ORA-01555: snapshot too old: rollback segment number 72 with name "_SYSSMU72$" too small ORA-02063: preceding line from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 ORA-06512: at line 1. 
WARNING 
Mar 11 2003 03:27:26 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 15:39:20 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace SYSTEM is 47.59% full.  103776.0 KBytes of free space remain. Tablespace RBS is 51.58% full.  942070.0 KBytes of free space remain. Tablespace USERDATA_01 is 59.23% full.  1.97198e+06 KBytes of free space remain. Tablespace USERINDEX_01 is 35.25% full.  5.03879e+06 KBytes of free space remain. Tablespace SYSTEM_REP_DATA is 72.20% full.  193240.0 KBytes of free space remain. Tablespace TOOLMAN is 72.05% full.  1.37378e+06 KBytes of free space remain. Tablespace CALIB_PRD_REP_DATA is 69.54% full.  1.2917e+07 KBytes of free space remain. Tablespace CALIB_PRD_REP_INDEX is 54.10% full.  2.96472e+07 KBytes of free space remain. Tablespace REPADMIN_PRD_DATA is 24.74% full.  77064.0 KBytes of free space remain. Tablespace HDWDB_PRD_REP_DATA is 42.38% full.  557608.0 KBytes of free space remain. Tablespace RUNDB_PRD_REP_DATA is 66.07% full.  2.81458e+06 KBytes of free space remain. Tablespace RUNDB_PRD_REP_INDEX is 79.36% full.  1.6283e+06 KBytes of free space remain. Tablespace SU
CRITICAL 
Mar 11 2003 03:38:56 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 15:41:13 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Failed Jobs 
Event test "Failed Jobs" is cleared 
CLEARED 
Mar 11 2003 03:40:50 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 15:41:32 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Mar 11 2003 03:41:15 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 15:49:33 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace SYSTEM is 47.59% full.  103776.0 KBytes of free space remain. Tablespace RBS is 51.58% full.  942070.0 KBytes of free space remain. Tablespace USERDATA_01 is 59.23% full.  1.97198e+06 KBytes of free space remain. Tablespace USERINDEX_01 is 35.25% full.  5.03879e+06 KBytes of free space remain. Tablespace SYSTEM_REP_DATA is 72.20% full.  193240.0 KBytes of free space remain. Tablespace TOOLMAN is 72.05% full.  1.37378e+06 KBytes of free space remain. Tablespace CALIB_PRD_REP_DATA is 69.54% full.  1.2917e+07 KBytes of free space remain. Tablespace CALIB_PRD_REP_INDEX is 54.10% full.  2.96472e+07 KBytes of free space remain. Tablespace REPADMIN_PRD_DATA is 24.74% full.  77064.0 KBytes of free space remain. Tablespace HDWDB_PRD_REP_DATA is 42.38% full.  557608.0 KBytes of free space remain. Tablespace RUNDB_PRD_REP_DATA is 66.07% full.  2.81458e+06 KBytes of free space remain. Tablespace RUNDB_PRD_REP_INDEX is 79.36% full.  1.6283e+06 KBytes of free space remain. Tablespace SU
CRITICAL 
Mar 11 2003 03:49:05 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Tue Mar 11 15;57;57 comment by...ns -->
<pre>Dmitri

I had changed the threshold on this alert
because we were toubleshooting alerts.

I will reset to it's original value.

The snapshot to old is a REAL alert
and our replication for mcs is failing
due to limited space on onprd!

We expected this alert yesterday too
but did not receive it, thus the
work on OEM/alerts.

Seems the alerts are alerting NOW!
</pre>

<!-- Date: Tue Mar 11 16:05:36 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace EVENTS_MAXVALUE is 91.85% full.  55032.0 KBytes of free space remain. Tablespace RUNDB_PRD_REP_INDEX is 79.36% full.  1.6283e+06 KBytes of free space remain. Tablespace EVENTS_IDX_P04 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_IDX_P05 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_IDX_P01 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_P01 is 80.01% full.  155032.0 KBytes of free space remain. Tablespace EVENTS_IDX_P03 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_IDX_P02 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_IDX_MAXVALUE is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_P02 is 80.01% full.  155032.0 KBytes of free space remain. Tablespace EVENTS_P03 is 80.01% full.  155032.0 KBytes of free space remain. Tablespace EVENTS_P04 is 80.01% full.  155032.0 KBytes of free space remain. Tablespace EVENTS_P05 is 80.01% full
CRITICAL 
Mar 11 2003 04:04:15 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 16:24:58 2003 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=392 sec, SCN: 0x0000.4afeb900):. 
WARNING 
Mar 11 2003 04:22:44 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 16:53:23 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 3736 ORA-12008: error in snapshot refresh path ORA-01555: snapshot too old: rollback segment number 70 with name "_SYSSMU70$" too small ORA-02063: preceding line from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 ORA-06512: at line 1. 
WARNING 
Mar 11 2003 04:52:42 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Tue Mar 11 17:14:15 2003 -->
<pre>Dropped the refresh group for Slow Control ie. MCS due to not enough space in undo tablespace. Refresh of Slow Control is erroring out with the following error message : 
 
error in snapshot refresh path ORA-01555: snapshot too old: rollback segment number 70 with name "_SYSSMU70$" too small ORA-02063: preceding line from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT".  
 
It tried couple of times with the same error. Waiting for more 
space to be allocated to the mount point where undo tablespace resides. Hopefully reboot of b0dau35 will allow sysadmin  
to have more space allocated to that area.  
 
Once we will have more space for undo on cdfonprd, we will kick off the refresh for Slow Control.  
 
Further cleanup of MCS on-hold until this batch is replicated to  
cdfofprd.  
 
Replication for other schema will continue to run. </pre>
<!-- Author: akumar -->


<!-- Date: Tue Mar 11 17:26:44 2003 -->
<pre>Frank/Jeff 
 
Originally I wanted to manually stop 
the database since we were closely monitoring 
some replication that had been going on. 
 
We have stopped the replication of MCS refresh 
groups and will restart it once we get 
additional space.  All other replication 
is uninterrupted. 
 
Stating that!  You can simply reboot the 
machine when you are ready and let the 
rc scripts handle the shutdown/restart of the 
database. 
 
 
 
 
</pre>
<!-- Author: ns -->

<!-- Date: Tue Mar 11 17:33:20 2003 -->
<pre>
Target Name : b0dau35.fnal.gov 
Target Type : Node 
Node Name : b0dau35.fnal.gov 
Status : NODE UNREACHABLE 

Node UpDown 
 VNI-4009 : Cannot contact agent on the node. Agent may be down or network communication to the node has failed.  
NODE UNREACHABLE 
Mar 11 2003 05:30:38 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 17:33:20 2003 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : NODE UNREACHABLE 

Node UpDown 
 VNI-4009 : Cannot contact agent on the node. Agent may be down or network communication to the node has failed.  
NODE UNREACHABLE 
Mar 11 2003 05:30:38 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 17:33:20 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 3898 ORA-02068: following severe error from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 ORA-06512: at line 1 ORA-12012: error on auto execute of job 3378 ORA-02068: following severe error from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 ORA-06512: at line 1 ORA-12012: error on auto execute of job 3885 ORA-02068: following severe error from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 .... 
WARNING 
Mar 11 2003 05:32:50 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 17:39:33 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 3378 ORA-12535: TNS:operation timed out ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 ORA-06512: at line 1. 
WARNING 
Mar 11 2003 05:37:51 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 17:39:48 2003 -->
<pre>
Target Name : b0dau35.fnal.gov 
Target Type : Node 
Node Name : b0dau35.fnal.gov 
Status : CLEARED 

Node UpDown 
Event test "Node UpDown" is cleared 
CLEARED 
Mar 11 2003 05:39:30 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 17:39:48 2003 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CLEARED 

Node UpDown 
Event test "Node UpDown" is cleared 
CLEARED 
Mar 11 2003 05:39:30 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 17:41:37 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CRITICAL 

Failed Jobs 
Job 3378 has failed 2 times. Job 3898 has failed 2 times. Job 3885 has failed 2 times. Job 3894 has failed 2 times. 
CRITICAL 
Mar 11 2003 05:40:52 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 17:43:42 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 3885 ORA-12541: TNS:no listener ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 ORA-06512: at line 1 ORA-12012: error on auto execute of job 3898 ORA-12541: TNS:no listener ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 ORA-06512: at line 1 ORA-12012: error on auto execute of job 3894 ORA-12541: TNS:no listener ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 .... 
WARNING 
Mar 11 2003 05:42:52 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Mar 11 18:41:41 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Failed Jobs 
Event test "Failed Jobs" is cleared 
CLEARED 
Mar 11 2003 06:40:53 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Mar 12 00:03:39 2003 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Mar 12 2003 12:02:44 AM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Wed Mar 12 13;45;20 comment by...ns -->
<pre>when we bounced the db we took out parameters set
while turning on functional indexes.. we believe these
in ORACLE 9 do not have to be set to use functional indexes
we do believe it affects explain plan and optimizer on
tables totally unrelated to functional indexes 

more later </pre>