<!-- Title: 08:03:18  Mon Apr  4 2005 -->
<!-- Crewchief: New day -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>database activity</pre>

<!-- Date: Mon Apr 4 08:56:45 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace MCS_INDEX is 82.28% full.  16025432.0 KBytes of free space remain. Tablespace MCS_DATA is 80.06% full.  4002744.0 KBytes of free space remain. Tablespace RUNDB_DATA is 77.02% full.  5150560.0 KBytes of free space remain. Tablespace RUNDB_INDEX is 76.29% full.  7443152.0 KBytes of free space remain. 
CRITICAL 
Apr 04 2005 08:56:19 AM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Mon Apr  4 12;04;39 comment by...ns  -->
<pre>no action - monitoring hope to move off soon  </pre>

<!-- Date: Mon Apr 4 13:34:30 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-02050: transaction 5.38.1479080 rolled back, some remote DBs may be in-doubt ORA-02068: following severe error from CDFOFPR1 ORA-03113: end-of-file on communication channel. 
WARNING 
Apr 04 2005 01:34:00 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Apr 4 14:00:04 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 629 
WARNING 
Apr 04 2005 01:59:43 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Apr 4 14:09:57 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 916.999999999999999999999999999999999998 
CRITICAL 
Apr 04 2005 02:09:44 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Apr 4 15:59:50 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 821 
WARNING 
Apr 04 2005 03:59:45 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Apr 4 16:04:51 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 585 
CLEARED 
Apr 04 2005 04:04:38 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Apr 4 17:02:21 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Process Limit 
The process usage is reaching its limit.  Current value: 167, limit: 200. 
WARNING 
Apr 04 2005 05:01:22 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Mon Apr 4 17:10:20 2005 -->
<pre>killed some tomcat (user) connections from b0web01 on  
the client side, but they seem to be sticky... 
</pre>
<!-- Author: W.Badgett -->
<!-- Comment: Mon Apr  4 17;22;38 comment by...rjetton  -->
<pre>similarly we are killing some toolman sessions not in the db yet connected the the listener  </pre>

<!-- Comment: Mon Apr  4 18;01;26 comment by...nstanfield/rjetton  -->
<pre>ORACLE USER: RUN_USER     
MACHINE:     b0web01 
OSUSER:      TOMCAT
PROGRAM:   /cdf/code-Linux-2.0.34/products/jdk/1.4.

&gt; grep tomcat users.15_57 | wc -l
      22
&gt; grep tomcat users.16_02 | wc -l
      25
&gt; grep tomcat users.16_17 | wc -l
      31
&gt; grep tomcat users.17_02 | wc -l
      53
&gt; grep tomcat users.17_12 | wc -l
      55
&gt; grep tomcat users.17_17 | wc -l 
      55
&gt;  grep tomcat users.17_23 | wc -l 
      39
&gt; grep tomcat users.17_27 | wc -l
      14
&gt; grep tomcat users.17_32 | wc -l
       3
&gt; grep tomcat users.17_42 | wc -l
       1

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
we had many toolman in the listener yet not in the db..
richard killed these off thus the additional mail
of terminated every5min

Your "cron" job on b0dau35
. bin/bootstrap_toolman_cron; every5min

produced the following output:

Terminated



</pre>
<!-- Comment: Mon Apr  4 18;08;23 comment by...ns  -->
<pre>queries failed 

Mon Apr  4 17:22:35 2005
SELECT BB.TEVSTORE,MAX(D.CREATETIME),MAX(D.UPDATETIME) FROM  (SELECT TEVSTORE,MIN(FRONTENDTIME) T0,MAX(FRONTENDTIME) T1 FROM BEAM_CONDITIONS_FULL B   WHERE TEVSTORE&gt;600 AND B.B0ILUM&gt;=0.1 AND B.B0Q5&gt;=1500 AND B.ERING&gt;=900   GROUP BY TEVSTORE) BB,DOWNTIME_HISTORY D  WHERE D.DOWNCLINK BETWEEN T0 AND T1 GROUP BY BB.TEVSTORE
Mon Apr  4 17:25:01 2005
ORA-01555 caused by SQL statement below (Query Duration=5022 sec, SCN: 0x050e.bc0afcbc):
Mon Apr  4 17:25:01 2005
SELECT BB.TEVSTORE,MAX(D.CREATETIME),MAX(D.UPDATETIME) FROM  (SELECT TEVSTORE,MIN(FRONTENDTIME) T0,MAX(FRONTENDTIME) T1 FROM BEAM_CONDITIONS_FULL B   WHERE TEVSTORE&gt;600 AND B.B0ILUM&gt;=0.1 AND B.B0Q5&gt;=1500 AND B.ERING&gt;=900   GROUP BY TEVSTORE) BB,DOWNTIME_HISTORY D  WHERE D.DOWNCLINK BETWEEN T0 AND T1 GROUP BY BB.TEVSTORE</pre>
<!-- Date: Mon Apr 4 17:24:18 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=5874 sec, SCN: 0x050e.bc096033):. 
WARNING 
Apr 04 2005 05:24:09 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Apr 4 17:28:53 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CLEARED 

Process Limit 
Event test "Process Limit" is cleared 
CLEARED 
Apr 04 2005 05:28:46 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Apr 4 17:37:31 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=5022 sec, SCN: 0x050e.bc0afcbc):. 
WARNING 
Apr 04 2005 05:29:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Apr 4 17:37:31 2005 -->
<pre>
</pre>
<!-- Author: oem-admin (email) -->




<!-- Date: Mon Apr 4 19:09:01 2005 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Alert 
The following errors are in the ALERT file : ORA-00600: internal error code, arguments: [12333], [7], [6], [99], [], [], [], []. 
CRITICAL 
Apr 04 2005 07:08:48 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Apr 4 19:34:38 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-02050: transaction 1.26.2334415 rolled back, some remote DBs may be in-doubt ORA-02068: following severe error from CDFOFPR1 ORA-03113: end-of-file on communication channel. 
WARNING 
Apr 04 2005 07:34:09 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Mon Apr  4 20;13;47 comment by...ns  -->
<pre>
SQL&gt; select * from dba_2pc_pending;

no rows selected

</pre>


<!-- Date: Mon Apr 4 20:19:08 2005 -->
<pre>51761352 SAMREAD            cdfopr cdfofpr1 
fcdflnx3.fnal.gov pts/14                         ./CalibExe@fcdflnx3.fnal.gov (TNS V1-V3) 
13397     131.225.240.8 
tcp 04-APR-05         50       6922 
 
 
----- Original Message -----  
From: &lt;oem-admin@fnal.gov&gt; 
To: "CDFOFLINEMGR" &lt;cdfofoem@fnal.gov&gt;; "CDFOFLINEMGR" &lt;oem-admin@fnal.gov&gt;; "CDFOFLINEMGR" &lt;cdf_elog@b0dau30.fnal.gov&gt; 
Sent: Monday, April 04, 2005 7:08 PM 
Subject: EVENT cdfofprd_alert_log CRITICAL 
 
 
&gt; Name : cdfofprd_alert_log  
&gt; Target Name : cdfofpr1.fnal.gov  
&gt; Target Type : Database  
&gt; Node Name : fcdfora4.fnal.gov  
&gt; Status : CRITICAL  
&gt;  
&gt; Alert  
&gt; The following errors are in the ALERT file : ORA-00600: internal error code, arguments: [12333], [7], [6], [99], [], [], [], [].  
&gt; CRITICAL  
&gt; Apr 04 2005 07:08:48 PM  
&gt;</pre>
<!-- Author: nstanfield -->


<!-- Date: Mon Apr 4 22:10:00 2005 -->
<pre>load averages:  9.04, 9.66 ,  8.73  
 
we collect info on users using large amts of temp  
 
oracleuser:  run_user 
osuser:      cdfdaq    
machine:     b0dap85a         
program:    runedit.Downtime  --&gt;    has 23 connections  
and using lrge amts of sort ... 
 
RMAN is measurably slower going to bzora1; 40% slower for some extended stretches.   
 
REPLICATION  on fcdfora4 looks timely enough. 
 
DITTO for replication jobs on fcdflnx1.   
 
 
 
 
  </pre>
<!-- Author: nstanfield -->
<!-- Comment: Mon Apr  4 22;34;54 comment by...ns  -->
<pre>top cpu users

lot of these come in -- take cpu  -- then depart 

&gt; runscript cdfonprd show_users | grep 27214
27214          197    6222 RUN_USER     cdfdaq       b0dap85a             runedit.DownTime -a

&gt; runscript cdfonprd show_users | grep 1878
1878            26   53614 RUN_USER     cdfdaq       b0dap85a             runedit.DownTime -a

27172  is our capture process   he comes & goes as a top user

  PID USERNAME THR PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
27172 oracle     1  20    0 1361M 1325M run     36.1H 12.94% oracle


i see the repadmin_prd com in take CPU finish his job & leave 


rcp  - richard jobs getting kicked down alot </pre>

<!-- Comment: Tue Apr  5 06;59;53 comment by...ns  -->
<pre>rman took 8 hours to copy to bzora1 vs 6
due to load & cpu activity 



i can see we got back some cpu after 2am
 
and the load average was reasonable about at 2:30 a.m. 

 
&gt; sar
 
SunOS b0dau35 5.7 Generic_106541-19 sun4u    04/05/05
 
00:00:01    %usr    %sys    %wio   %idle
01:00:01      47      37      12       4
02:00:01      64      23       9       5
03:00:01      38      13      12      37
04:00:01      24       9      15      52
05:00:01      23       9      15      53
06:00:01      24      10      15      51
 
Average       37      17      13      34    
 
sys.00_05:last pid: 22974;  load averages:  7.15,  7.09,  6.89    00:05:25
sys.00_20:last pid: 26842;  load averages:  6.23,  6.49,  6.48    00:20:11
sys.00_35:last pid:   469;  load averages:  6.78,  6.07,  6.32    00:35:17
sys.00_50:last pid:  3984;  load averages:  7.96,  6.96,  6.50    00:50:11
sys.01_05:last pid:  7429;  load averages:  6.99,  7.05,  6.94    01:05:09
sys.01_20:last pid: 11444;  load averages:  9.42,  8.27,  7.79    01:20:17
sys.01_35:last pid: 14834;  load averages:  7.18,  7.17,  7.31    01:35:14
sys.01_50:last pid: 18265;  load averages:  8.19,  8.20,  7.77    01:50:14
sys.02_05:last pid: 22162;  load averages:  7.81,  7.82,  7.76    02:05:08
sys.02_20:last pid: 26178;  load averages:  6.85,  7.75,  7.64    02:20:07
sys.02_35:last pid: 29671;  load averages:  1.96,  2.02,  4.03    02:35:05
sys.02_50:last pid:  3126;  load averages:  1.50,  1.43,  2.38    02:50:03
sys.03_05:last pid:  6926;  load averages:  1.42,  1.62,  1.91    03:05:08
sys.03_20:last pid: 10719;  load averages:  2.11,  1.98,  1.91    03:20:06
sys.03_35:last pid: 14544;  load averages:  1.32,  1.48,  1.70    03:35:06
sys.03_50:last pid: 18040;  load averages:  1.70,  1.49,  1.57    03:50:10
sys.04_05:last pid: 21820;  load averages:  1.14,  1.50,  1.56    04:05:05
sys.04_20:last pid: 25470;  load averages:  0.99,  1.38,  1.52    04:20:03
sys.04_35:last pid: 28992;  load averages:  2.49,  1.97,  1.72    04:35:05
sys.04_50:last pid:  2810;  load averages:  0.68,  1.49,  1.75    04:50:05
sys.05_05:last pid:  6472;  load averages:  1.32,  1.27,  1.40    05:05:07
sys.05_20:last pid: 10269;  load averages:  1.46,  1.58,  1.58    05:20:05
sys.05_35:last pid: 13831;  load averages:  1.55,  1.48,  1.53    05:35:10
sys.05_50:last pid: 17236;  load averages:  1.57,  1.42,  1.52    05:50:05
sys.06_05:last pid: 21569;  load averages:  2.34,  2.61,  2.21    06:05:05
sys.06_20:last pid: 25599;  load averages:  0.70,  1.51,  1.79    06:20:04
sys.06_35:last pid: 29538;  load averages:  1.36,  2.49,  2.48    06:35:05
sys.06_50:last pid:  3224;  load averages:  1.12,  1.35,  1.69    06:50:04
&gt;
</pre>
<!-- Comment: Tue Apr  5 07;02;26 comment by...nstanfield  -->
<pre>NO MORE LONG QUERIES USING TEMP AFTER 1am..........</pre>
<!-- Comment: Tue Apr  5 07;27;40 comment by...nstanfield  -->
<pre>oracle user: run_user
os user:   cdfdaq
machine:    b0dap85a
program:   runedit.Downtime 

Staring April 4  at 14_47 pm.  our user reports
show that this user is adding + adding + adding
connections and does not clear till 2:30 this a.m. which
seems to coincide with the LOAD average 

Can someone please look into this.      


14_42   3 connections 
14_47   4 connections
15_02   5 connections
15_42   8 connections
15_57   11 connections
16_57   21 connections
........
......... 
21_57   21 connections
22_57   21 connections
23_57   18 connections
00_02   15 connectios
00_32   13 connections
1_32    10 connections
2_22    4  connections
4_32     1  connection
6_12     1  connection
6_17     1 connection



</pre>
<!-- Date: Tue Apr 5 00:03:54 2005 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Apr 05 2005 12:03:48 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Apr 5 00:04:24 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Apr 05 2005 12:04:12 AM 



</pre>
<!-- Author: oem-admin (email) -->