<!-- Title: 11:39:00  Fri Feb 21 2003 -->
<!-- Crewchief: New Day -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>database activity</pre>


<!-- Date: Fri Feb 21 11:39:08 2003 -->
<pre>I will be bringing down both instances 
cdfofdev and cdfofint at noon 
today. 
 
This is necessary to patch oracle 
with some security patches and to 
prepare to patch fcdfora1 for next 
week. 
 
The samdbservers will have to be restarted. 
 
I will send out email when this is 
complete expected downtime 1 hour. 
 
</pre>
<!-- Author: nstanfield  -->
<!-- Comment: Fri Feb 21 13;37;29 comment by...ns -->
<pre>completed by 12:30 
</pre>
<!-- Date: Fri Feb 21 13:37:14 2003 -->
<pre>Cleanup of MCS data. 
 
I have tried rebuilding indexes, moving them to LMT 
and neither regained the space appropriately -- it does 
regain the space but they extents are too many..the 
best way to get back space and reduce no. of extents 
is the exp/import.  This is the standard way of doing it 
but the test I performed were to see if we could reduce 
downtime.   
 
I will start an import of tables NOW with the deleted 
records and record the time.  Then we can schedule 
this for PRD.</pre>
<!-- Author: nstanfield  -->
<!-- Comment: Fri Feb 21 13;41;55 comment by...ns -->
<pre>The following tablespaces in cdfonint require coalescing:

Tablespace Tablespace                     Coalescable
    Number Name                               Extents
---------- ------------------------------ -----------
        18 MLOG_DATA                               11
        34 MCS_DATA                                39
        35 MCS_INDEX                              155
beginning coalescing of cdfonint
after the coalesce for cdfonint
</pre>


<!-- Date: Fri Feb 21 14:53:35 2003 -->
<pre>Name : cdfofprd_max_extents 
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : WARNING 
 
Maximum Extents 
Segment LITVINSE.RR_RS_FK_I in tablespace USERINDEX_01 can only allocate 2 additional extents. 
WARNING 
 
DMITRI will handle.</pre>
<!-- Author: nstanfield  -->
<!-- Comment: Fri Feb 21 15;00;17 comment by...Dmitry Litvintsev -->
<pre>executed :
alter index LITVINSE.RR_RS_FK_I storage (maxextents 8);</pre>

<!-- Date: Fri Feb 21 16:37:54 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1650: unable to extend rollback segment SYSTEM_01 by 1280 in         tablespace SYSTEM. 
WARNING 
Feb 21 2003 04:37:29 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Feb 21 16:49:29 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1650: unable to extend rollback segment SYSTEM_01 by 1280 in         tablespace SYSTEM  ORA-1650: unable to extend rollback segment SYSTEM_01 by 1280 in         tablespace SYSTEM. 
WARNING 
Feb 21 2003 04:47:31 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Feb 21 18:21:45 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace EVENTS_MAXVALUE is 91.85% full.  55032.0 KBytes of free space remain. Tablespace SYSTEM is 85.42% full.  23040.0 KBytes of free space remain. Tablespace EVENTS_IDX_P04 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_IDX_P05 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_IDX_P01 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_P01 is 80.01% full.  155032.0 KBytes of free space remain. Tablespace EVENTS_IDX_P03 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_IDX_P02 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_IDX_MAXVALUE is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_P02 is 80.01% full.  155032.0 KBytes of free space remain. Tablespace EVENTS_P03 is 80.01% full.  155032.0 KBytes of free space remain. Tablespace EVENTS_P04 is 80.01% full.  155032.0 KBytes of free space remain. Tablespace EVENTS_P05 is 80.01% full.  155032.0 KBytes of free space remain.
CRITICAL 
Feb 21 2003 06:21:13 PM 




</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Feb 21 20:20:42 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Maximum Extents 
Event test "Maximum Extents" is cleared 
CLEARED 
Feb 21 2003 08:20:31 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Feb 22 00:03:03 2003 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Feb 22 2003 12:02:44 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Feb 22 04:22:49 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace EVENTS_MAXVALUE is 91.85% full.  55032.0 KBytes of free space remain. Tablespace EVENTS_IDX_P04 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_IDX_P05 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_IDX_P01 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_P01 is 80.01% full.  155032.0 KBytes of free space remain. Tablespace EVENTS_IDX_P03 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_IDX_P02 is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_IDX_MAXVALUE is 80.04% full.  36792.0 KBytes of free space remain. Tablespace EVENTS_P02 is 80.01% full.  155032.0 KBytes of free space remain. Tablespace EVENTS_P03 is 80.01% full.  155032.0 KBytes of free space remain. Tablespace EVENTS_P04 is 80.01% full.  155032.0 KBytes of free space remain. Tablespace EVENTS_P05 is 80.01% full.  155032.0 KBytes of free space remain. 
CRITICAL 
Feb 22 2003 04:21:18 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Feb 22 10:28:04 2003 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 37986 ORA-00001: unique constraint (HDWDB_PRD.CU_PK) violated ORA-06512: at "HDWDB_PRD.CARDUPDATES18", line 24 ORA-04088: error during execution of trigger 'HDWDB_PRD.CARDUPDATES18' ORA-06512: at "HDWDB_PRD.HDWDB", line 648 ORA-06512: at line 1 ORA-12012: error on auto execute of job 37987 ORA-00001: unique constraint (HDWDB_PRD.CU_PK) violated ORA-06512: at "HDWDB_PRD.CARDUPDATES18", line 24 ORA-04088: error during execution of trigger 'HDWDB_PRD.CARDUPDATES18' ORA-06512: at "HDWDB_PRD.HDWDB", line 648 ORA-06512: at line 1 ORA-12012: error on auto execute of job 37988 ORA-00001: unique constraint (HDWDB_PRD.CU_PK) violated ORA-06512: at "HDWDB_PRD.CARDUPDATES18", line 24 ORA-04088: error during execution of trigger 'HDWDB_PRD.CARDUPDATES18' ORA-06512: at "HDWDB_PRD.HDWDB", line 648 ORA-06512: at line 1. 
WARNING 
Feb 22 2003 10:27:44 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Feb 22 10:32:13 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : STARTED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Feb 22 10:32:13 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : COMPLETED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 23 00:04:39 2003 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Feb 23 2003 12:02:44 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 23 10:36:52 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : STARTED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 23 10:36:52 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : COMPLETED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 23 15:33:02 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 3879 ORA-12008: error in snapshot refresh path ORA-01555: snapshot too old: rollback segment number 67 with name "_SYSSMU67$" too small ORA-02063: preceding line from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 ORA-06512: at line 1. 
WARNING 
Feb 23 2003 03:32:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 23 15:33:22 2003 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=1 sec, SCN: 0x0000.484b1f52):. 
WARNING 
Feb 23 2003 03:32:44 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 23 15:47:53 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Feb 23 2003 03:47:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 23 16:23:08 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Failed Jobs 
Event test "Failed Jobs" is cleared 
CLEARED 
Feb 23 2003 04:22:44 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 23 19:19:23 2003 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=13272 sec, SCN: 0x0000.4850dbb8):. 
WARNING 
Feb 23 2003 07:17:44 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 23 19:19:23 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 3879 ORA-12008: error in snapshot refresh path ORA-01555: snapshot too old: rollback segment number 65 with name "_SYSSMU65$" too small ORA-02063: preceding line from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 ORA-06512: at line 1. 
WARNING 
Feb 23 2003 07:17:55 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 23 19:23:32 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Failed Jobs 
Event test "Failed Jobs" is cleared 
CLEARED 
Feb 23 2003 07:22:48 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 23 19:27:40 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Feb 23 2003 07:27:12 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 23 19:48:25 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Feb 23 2003 07:47:13 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 23 21:23:17 2003 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=1046057003 sec, SCN: 0x0000.485bb145): ORA-01555 caused by SQL statement below (Query Duration=1046057003 sec, SCN: 0x0000.485bb145):. 
WARNING 
Feb 23 2003 09:22:44 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 24 00:04:23 2003 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Feb 24 2003 12:02:45 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 24 09:04:20 2003 -->
<pre>
Target Name : cdfonprd 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=0 sec, SCN: 0x0000.48591763):. 
WARNING 
Feb 24 2003 09:02:44 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 24 09:06:18 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 3879 ORA-12008: error in snapshot refresh path ORA-01555: snapshot too old: rollback segment number 70 with name "_SYSSMU70$" too small ORA-02063: preceding line from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 ORA-06512: at line 1. 
WARNING 
Feb 24 2003 09:05:45 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 24 09:08:28 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Feb 24 2003 09:07:53 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 24 09:23:46 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Failed Jobs 
Event test "Failed Jobs" is cleared 
CLEARED 
Feb 24 2003 09:23:02 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 24 09:29:06 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Feb 24 2003 09:27:54 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 24 10:31:26 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : STARTED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 24 10:31:26 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : COMPLETED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 24 10:56:57 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-00028: your session has been killed ORA-02063: preceding line from CDFONPRD ORA-12008: error in snapshot refresh path ORA-00028: your session has been killed. 
WARNING 
Feb 24 2003 10:56:07 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 24 11:09:23 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Feb 24 2003 11:07:59 AM 



</pre>
<!-- Author: oem-admin (email) -->