<!-- Title: 08:48:23  Tue Jun 15 2004 -->
<!-- Crewchief: New day -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>database activity</pre>


<!-- Date: Tue Jun 15 08:48:34 2004 -->
<pre>start refresh of run user in cdfondev </pre>
<!-- Author: ns  -->

<!-- Date: Tue Jun 15 09:03:16 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CLEARED 

Maximum Extents 
Event test "Maximum Extents" is cleared 
CLEARED 
Jun 15 2004 09:02:58 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 15:05:21 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Database UpDown 
The database instance is down. 
CRITICAL 
Jun 15 2004 03:04:59 PM 
Node UpDown 
Event test "Node UpDown" is cleared 
CLEARED 
May 28 2004 12:47:41 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 15:17:09 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : NODE UNREACHABLE 

Node UpDown 
 VNI-4009 : Cannot contact agent on the node. Agent may be down or network communication to the node has failed.  
NODE UNREACHABLE 
Jun 15 2004 03:15:40 PM 
Database UpDown 
The database instance is down. 
CRITICAL 
Jun 15 2004 03:04:59 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 15:19:14 2004 -->
<pre>
Target Name : fcdfora4.fnal.gov 
Target Type : Node 
Node Name : fcdfora4.fnal.gov 
Status : NODE UNREACHABLE 

Node UpDown 
 VNI-4009 : Cannot contact agent on the node. Agent may be down or network communication to the node has failed.  
NODE UNREACHABLE 
Jun 15 2004 03:15:40 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 15:31:36 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Node UpDown 
Event test "Node UpDown" is cleared 
CLEARED 
Jun 15 2004 03:31:15 PM 
Database UpDown 
The database instance is down. 
CRITICAL 
Jun 15 2004 03:04:59 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 15:31:36 2004 -->
<pre>
Target Name : fcdfora4.fnal.gov 
Target Type : Node 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Node UpDown 
Event test "Node UpDown" is cleared 
CLEARED 
Jun 15 2004 03:31:15 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 15:34:01 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Tablespace Full 
Event test "Tablespace Full" is cleared 
CLEARED 
Jun 15 2004 03:33:40 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 15:34:01 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Jun 15 2004 03:33:48 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 15:34:20 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Maximum Extents 
Event test "Maximum Extents" is cleared 
CLEARED 
Jun 15 2004 03:34:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 15:35:18 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Database UpDown 
Event test "Database UpDown" is cleared 
CLEARED 
Jun 15 2004 03:34:59 PM 
Node UpDown 
Event test "Node UpDown" is cleared 
CLEARED 
Jun 15 2004 03:31:15 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 15:44:22 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Jun 15 2004 03:43:56 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Tue Jun 15 16:36:46 2004 -->
<pre>On Tue, 15 Jun 2004, Nelly Stanfield wrote: 
... 
&gt; WOW......   b0dau35 
&gt; 
&gt; last pid:  8228;  load averages:  5.29,  5.11,  4.58 
&gt; 16:07:04 
&gt; 150 processes: 141 sleeping, 7 running, 2 on cpu 
&gt; CPU states:  2.4% idle, 54.4% user, 28.3% kernel, 14.9% iowait,  0.0% swap 
&gt; Memory: 1792M real, 26M free, 1476M swap in use, 1862M swap free 
&gt; 
&gt;    PID USERNAME THR PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND 
&gt;   5175 stager     1   4    2 3080K 1600K run      6:50 16.07% ssh 
&gt;     82 oracle     1  59    0 1188M 1160M cpu/4   11:59 12.95% oracle 
&gt;   5174 stager     1  26    2 2944K 1488K sleep    2:45  6.41% rsync 
&gt;  27138 stager     1  38    2 2872K 1144K run     82:08  3.78% rcp 
&gt;  10009 oracle   258  58    0 1202M 1146M run    122:44  2.17% oracle 
 
This is a 4 processor system, so a load average of 5 should not be a big deal. 
 
The rsync/ssh arch file copy to data012 was using most of one CPU. 
The rcp of rman to fcdfdata012 was using about 1/6 of a CPU. 
 
The system is running about 50% idle now. 
 
Here is the recent sar data : 
(I see nothing particular change at 13:00 when the nightly rman copies start. ) 
 
b0dau35 $ sar 
 
SunOS b0dau35 5.7 Generic_106541-19 sun4u    06/15/04 
 
00:00:00    %usr    %sys    %wio   %idle 
... 
12:00:00      11       4       4      81 
12:20:00      11       4       3      82 
12:40:00      12       5       4      80 
13:00:00      15       5      12      68 
13:20:01      18      20       3      59 
13:40:00      13      19       3      65 
14:00:00      17      21       4      57 
14:20:00      21      23      11      45 
14:40:01      18      25       5      51 
15:00:01      15      20       4      61 
15:20:01      16      20       4      60 
15:40:01      21      22      11      46 
16:00:01      39      27      21      13 
16:20:01      39      27      18      16 
 
Average       18      15      10      57 
 
Something seems to have put a big load on the system from 15:40 to 16:00 . 
I do not see any obvious candidates among the presently running processes. 
</pre>
<!-- Author: ns -->

<!-- Date: Tue Jun 15 16:58:56 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 8193 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 2049 (blocksize=512) ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 6145 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 1 (blocksize=512) ORA-16038: log 2 sequence# 291767 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 2 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_02A.log' ORA-16038: log 2 sequence# 291767 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 2 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_02A.log
WARNING 
Jun 15 2004 04:58:17 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:03:44 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 2049 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 1 (blocksize=512) ORA-16038: log 2 sequence# 291767 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 2 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_02A.log' ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 6145 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 1 (blocksize=512) ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc" ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc" ORA-27044: unable to writ
WARNING 
Jun 15 2004 05:03:16 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:09:18 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 2049 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 1 (blocksize=512) ORA-16038: log 2 sequence# 291767 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 2 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_02A.log' ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 6145 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 1 (blocksize=512) ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc" ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc" ORA-27044: unable to writ
WARNING 
Jun 15 2004 05:08:16 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:09:18 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 480 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1 ORA-12012: error on auto execute of job 378 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1 ORA-12012: error on auto execute of job 506 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 .... 
WARNING 
Jun 15 2004 05:08:48 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:13:26 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 6145 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 1 (blocksize=512) ORA-16038: log 3 sequence# 291768 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 3 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_03A.log' ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 2049 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 1 (blocksize=512) ORA-16038: log 2 sequence# 291767 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 2 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_02A.log' O
WARNING 
Jun 15 2004 05:13:16 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:14:06 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 521 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1. 
WARNING 
Jun 15 2004 05:13:48 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:18:23 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 6145 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 1 (blocksize=512) ORA-16038: log 2 sequence# 291767 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 2 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_02A.log' ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 2049 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 1 (blocksize=512) ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc" ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc" ORA-27044: unable to writ
WARNING 
Jun 15 2004 05:18:16 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:19:39 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 521 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1. 
WARNING 
Jun 15 2004 05:18:49 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:23:47 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 2049 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 1 (blocksize=512) ORA-16038: log 2 sequence# 291767 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 2 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_02A.log' ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 4097 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 1 (blocksize=512) ORA-16038: log 2 sequence# 291767 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 2 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_02A.log' O
WARNING 
Jun 15 2004 05:23:16 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:24:02 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 480 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1 ORA-12012: error on auto execute of job 378 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1 ORA-12012: error on auto execute of job 506 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 .... 
WARNING 
Jun 15 2004 05:23:48 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:25:12 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Failed Jobs 
Job 378 has failed 2 times. Job 506 has failed 2 times. Job 480 has failed 2 times. Job 521 has failed 5 times. 
CRITICAL 
Jun 15 2004 05:24:54 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:28:29 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 2049 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 1 (blocksize=512) ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc" ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc" ORA-27044: unable to write the header block of file ORA-16038: log 2 sequence# 291767 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 2 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_02A.log' ORA-16038: log 3 sequence# 291768 cannot be archived ORA-19504: failed to create file "" ORA-00312: online log 3 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_03A.log' ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc" OR
WARNING 
Jun 15 2004 05:28:16 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:33:33 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 4097 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 1 (blocksize=512) ORA-16038: log 3 sequence# 291768 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 3 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_03A.log' ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc" ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc" ORA-27044: unable to write the header block of file ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 6145 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc",
WARNING 
Jun 15 2004 05:33:16 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:38:31 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 6145 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 1 (blocksize=512) ORA-16038: log 2 sequence# 291767 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 2 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_02A.log' ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 6145 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 1 (blocksize=512) ORA-16038: log 3 sequence# 291768 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 3 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_03A.log' O
WARNING 
Jun 15 2004 05:38:17 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:39:03 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 480 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1 ORA-12012: error on auto execute of job 378 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1 ORA-12012: error on auto execute of job 506 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 .... 
WARNING 
Jun 15 2004 05:38:48 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:43:58 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 6145 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 1 (blocksize=512) ORA-16038: log 3 sequence# 291768 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 3 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_03A.log' ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc" ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc" ORA-27044: unable to write the header block of file ORA-16038: log 3 sequence# 291768 cannot be archived ORA-19504: failed to create file "" ORA-00312: online log 3 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_03A.log' ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc" OR
WARNING 
Jun 15 2004 05:43:16 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:48:39 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 6145 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 1 (blocksize=512) ORA-16038: log 2 sequence# 291767 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 2 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_02A.log' ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 2049 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291768.arc", blockno 1 (blocksize=512) ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc" ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc" ORA-27044: unable to writ
WARNING 
Jun 15 2004 05:48:17 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:53:31 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 2049 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc", blockno 1 (blocksize=512) ORA-16038: log 2 sequence# 291767 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 2 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_02A.log' ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc" ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc" ORA-27044: unable to write the header block of file ORA-16038: log 2 sequence# 291767 cannot be archived ORA-19504: failed to create file "" ORA-00312: online log 2 thread 1: '/dbs26/oradata/cdfonprd/redocdfonprd1_02A.log' ORA-19504: failed to create file "/dbs8/oradata/cdfonprd/arch/arch1_prd_291767.arc" OR
WARNING 
Jun 15 2004 05:53:17 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 17:54:21 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 480 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1 ORA-12012: error on auto execute of job 378 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1 ORA-12012: error on auto execute of job 506 ORA-00257: archiver error. Connect internal only, until freed. ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 .... 
WARNING 
Jun 15 2004 05:53:48 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 18:25:56 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Failed Jobs 
Event test "Failed Jobs" is cleared 
CLEARED 
Jun 15 2004 06:24:54 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 20:18:47 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=4687 sec, SCN: 0x050e.12045d6b):. 
WARNING 
Jun 15 2004 08:18:18 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Tue Jun 15 21;23;04 comment by...ns  -->
<pre>query that failed has to do with replication slow controls up'd undo from  3600 to 7200
if this doesn't solve it i will talk to anil in the morning

Tue Jun 15 20:17:52 2004

SELECT /*+ */
"A2"."SENSH_ID","A2"."SENST_ID","A2"."VALUE","A2"."SENS_ID","A2".

HISTYP_ID" FROM "MCS_PRD"."SENSOR_HISTORIES" "A2", (SELECT /*+ */ DISTINCT
"A3"

"SENSH_ID" "SENSH_ID" FROM "MCS_PRD"."MLOG$_SENSOR_HISTORIES" "A3" WHERE
"A3"."

NAPTIME$$"&gt;:1 AND "A3"."DMLTYPE$$"&lt;&gt;'D') "A1" WHERE
"A2"."SENSH_ID"="A1"."SENSH

ID"
</pre>
<!-- Date: Tue Jun 15 20:20:17 2004 -->
<pre>archives started at a high rate at  15:29 through 18:07 
 
Returned to it's normal activity ...   cleanup 2 days of archives. 
 
-rw-rw----   1 oracle   dba      20971008 Jun 15 18:07 arch1_prd_291871.arc 
-rw-rw----   1 oracle   dba      20971008 Jun 15 18:07 arch1_prd_291872.arc 
-rw-rw----   1 oracle   dba      20971008 Jun 15 18:07 arch1_prd_291873.arc 
-rw-rw----   1 oracle   dba      20971008 Jun 15 18:07 arch1_prd_291874.arc 
-rw-rw----   1 oracle   dba      20971008 Jun 15 18:07 arch1_prd_291875.arc 
-rw-rw----   1 oracle   dba      20971008 Jun 15 18:07 arch1_prd_291876.arc 
-rw-rw----   1 oracle   dba      20971008 Jun 15 18:07 arch1_prd_291877.arc 
-rw-rw----   1 oracle   dba      20971008 Jun 15 18:08 arch1_prd_291878.arc 
-rw-rw----   1 oracle   dba      20971008 Jun 15 18:10 arch1_prd_291879.arc 
-rw-rw----   1 oracle   dba      20971008 Jun 15 18:12 arch1_prd_291880.arc 
-rw-rw----   1 oracle   dba      20971008 Jun 15 18:13 arch1_prd_291881.arc 
-rw-rw----   1 oracle   dba      20970496 Jun 15 18:17 arch1_prd_291882.arc 
-rw-rw----   1 oracle   dba      20971008 Jun 15 18:19 arch1_prd_291883.arc 
... 
 
1.  fcdfora4 was down for memory replacement 3:15 to 3:45 or so 
replication jobs were fine when checked excluding 1 repadmin_prd process hanging in there either it's catching up or a dead connection... will investigate 
 
2.  will run a archive or 2 thru logminer to see db activity 
 
last time this occured it was due to run schema cleanup... 
we are no aware of any db clean up activity....we have a log of users logged on during this period & none looked suspect at first glance 
 
</pre>
<!-- Author: nstanfield/jtrumbo -->

<!-- Date: Tue Jun 15 20:34:28 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 376 ORA-12008: error in materialized view refresh path ORA-01555: snapshot too old: rollback segment number 4 with name "_SYSSMU4$" too small ORA-02063: preceding line from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1. 
WARNING 
Jun 15 2004 08:33:48 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Tue Jun 15 21;25;40 comment by...ns  -->
<pre>FAILED   MCS GROUP 
Ended: 06/15/04 20:31:02
** Total transactions............. 5
** - successful........ 4
** - failed............ 1

SUCCESSFUL NEXT ROUND 

* ***************************************** **

REPADMIN_PRD.RUNDB_PRD_REP_G1 job&lt;521&gt;
Started: 06/15/04 20:49:44

** SUCCESSFULLY PROCESSED ***

Ended: 06/15/04 20:50:02

* Total transactions............. 5
* - successful........ 5
* - failed............ 0

* ***************************************** **


SET UNDO TO 5200 ...vs leaving it a 7200 or back to 3600 </pre>
<!-- Date: Tue Jun 15 21:25:07 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Failed Jobs 
Job 376 has failed 1 times. 
CRITICAL 
Jun 15 2004 09:24:54 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Tue Jun 15 22:49:09 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=4932 sec, SCN: 0x050e.121b0870):. 
WARNING 
Jun 15 2004 10:48:16 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 23:04:38 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 376 ORA-12008: error in materialized view refresh path ORA-01555: snapshot too old: rollback segment number 5 with name "_SYSSMU5$" too small ORA-02063: preceding line from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1. 
WARNING 
Jun 15 2004 11:03:48 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 15 23:25:22 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Failed Jobs 
Job 376 has failed 2 times. 
CRITICAL 
Jun 15 2004 11:24:54 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 00:03:48 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Jun 16 2004 12:03:16 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 00:04:52 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Jun 16 2004 12:03:48 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 01:09:00 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=4632 sec, SCN: 0x050e.122cd1c6):. 
WARNING 
Jun 16 2004 01:08:19 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 01:25:34 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Failed Jobs 
Job 376 has failed 3 times. 
CRITICAL 
Jun 16 2004 01:24:54 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 01:29:42 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 376 ORA-12008: error in materialized view refresh path ORA-01555: snapshot too old: rollback segment number 5 with name "_SYSSMU5$" too small ORA-02063: preceding line from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1. 
WARNING 
Jun 16 2004 01:28:48 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 02:09:02 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 12289 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 6145 (blocksize=512) ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/d
WARNING 
Jun 16 2004 02:08:49 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 02:13:59 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 12289 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 6145 (blocksize=512) ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not ar
WARNING 
Jun 16 2004 02:13:48 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 02:19:04 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 12289 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 6145 (blocksize=512) ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-0
WARNING 
Jun 16 2004 02:18:49 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 02:24:25 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 12289 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 6145 (blocksize=512) ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not ar
WARNING 
Jun 16 2004 02:23:48 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 02:29:32 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 12289 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 6145 (blocksize=512) ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 12289 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 6145 (blocksize=512) ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-195
WARNING 
Jun 16 2004 02:28:48 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 02:33:53 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 12289 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 6145 (blocksize=512) ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not ar
WARNING 
Jun 16 2004 02:33:48 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 02:38:54 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 12289 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 6145 (blocksize=512) ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 12289 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 6145 (blocksize=512) ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-160
WARNING 
Jun 16 2004 02:38:48 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 02:44:14 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16014: log 1 sequence# 35867 not archived, no available destinations ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 12289 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-1
WARNING 
Jun 16 2004 02:43:48 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 06:01:17 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 12289 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 6145 (blocksize=512) ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-16038: log 1 sequence# 35867 cannot be archived ORA-19502: write error on file "", blockno  (blocksize=) ORA-00312: online log 1 thread 1: '/cdf/dbs01/oradata/cdfofpr1/redo1.log' ORA-19502: write error on file "/cdf/dbs03/oradata/cdfofpr1/arch/arch_1_35867.arc", blockno 12289 (blocksize=512) 
WARNING 
Jun 16 2004 06:00:52 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 06:04:22 2004 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=15758 sec, SCN: 0x050e.12408f21):. 
WARNING 
Jun 16 2004 06:03:16 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 06:24:12 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 376 ORA-12008: error in materialized view refresh path ORA-01555: snapshot too old: rollback segment number 4 with name "_SYSSMU4$" too small ORA-02063: preceding line from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 794 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 851 ORA-06512: at "SYS.DBMS_IREFRESH", line 683 ORA-06512: at "SYS.DBMS_REFRESH", line 195 ORA-06512: at line 1. 
WARNING 
Jun 16 2004 06:23:48 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 16 06:25:10 2004 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Failed Jobs 
Job 376 has failed 4 times. 
CRITICAL 
Jun 16 2004 06:24:54 AM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Wed Jun 16 07:43:56 2004 -->
<pre>checked cdfonprd first thing this a.m. and cdfofpr1 was halted.. 
from 2:07 to 5:50...   i cleaned up the cdfofpr2 archives... 
there was NOT a lot of archive being created... similar to cdfonprd 
issue... we just filled up .....  we had cdfofpr2 archives still there... 
along with cdfofpr1 ---   with pulling out of streams we have been 
cleaning up are deployment 
 
still need to investigate failed jobs on cdfofpr1..cdfrep01 jobs cleared  
</pre>
<!-- Author: ns  -->