<!-- Title: 13:06:39  Mon Dec 19 2005 -->
<!-- Crewchief: New day -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>database activity </pre>


<!-- Date: Mon Dec 19 13:07:00 2005 -->
<pre>proceeding with password protecting listener on all cdf boxes. 
 
no interruption expected.</pre>
<!-- Author: nstanfield  -->

<!-- Date: Mon Dec 19 13:14:09 2005 -->
<pre>
Name : cdfofpr2_tablespaces 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace USERTEMP_01 is 85.19% full.  2426920.0 KBytes of free space remain. 
CRITICAL 
Dec 19 2005 01:14:02 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Dec 19 13:19:20 2005 -->
<pre>
Name : cdfofpr2_tablespaces 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Tablespace Full 
Event test "Tablespace Full" is cleared 
CLEARED 
Dec 19 2005 01:19:02 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Dec 19 13:22:50 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
Dec 19 2005 01:22:30 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Mon Dec 19 16:16:16 2005 -->
<pre>Schema cut version v1_3y for RUNDB application has been  
run on cdfonprd;  it has successfully been copied to cdfofprd. 
The cut adds columns E1...E8 to STORE_SUMMARY table.</pre>
<!-- Author: W.Badgett -->

<!-- Date: Mon Dec 19 22:36:08 2005 -->
<pre>
Name : cdfofpr2_long_transaction 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: oracle  27 1 oracle@fcdfora4 (P001) fcdfora4 insert into "CALIB_PRD"."COTCHANT0" p("CID","G_CHAN","I_CHAN","OCCUPANCY","OFFSET","OFFSETRMS","WIDTH","WIDTHRMS")values(:1,:2,:3,:4,:5,:6,:7,:8) 5.08333333333333333333333333333333333392 
CRITICAL 
Dec 19 2005 10:34:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Dec 20 00:03:06 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Dec 20 2005 12:02:30 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Dec 20 00:10:16 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Dec 20 2005 12:09:49 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Dec 20 06:58:00 2005 -->
<pre>
Name : cdfofpr2_multiple_extents 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Multiple Extents 
Segment HERBER.EN_FILES in tablespace USERDATA_01 is comprised of 2057 extents. Segment SAMPRD.PROJECT_FILES in tablespace OPERATIONAL_01 is comprised of 978 extents. Segment SAMPRD.CONSUMED_FILES in tablespace CONSUMED_FILES_P01 is comprised of 2259 extents. Segment SAMPRD.DFLOC_DFLS_FK_I in tablespace PARTITIONED_IDX_01 is comprised of 1650 extents. Segment SAMPRD.DFLOC_FI_FK_I in tablespace PARTITIONED_IDX_01 is comprised of 2797 extents. Segment SAMPRD.DFLOC_VOL_FK_I in tablespace PARTITIONED_IDX_01 is comprised of 1080 extents. Segment HERBER.FILE_UK in tablespace USERDATA_01 is comprised of 1326 extents. Segment SAMPRD.PF1_PK in tablespace OPERATIONAL_IDX_01 is comprised of 736 extents. Segment SAMPRD.PF1_UQ1_UK in tablespace OPERATIONAL_IDX_01 is comprised of 1041 extents. Segment SAMPRD.PF1_FI_FK_I in tablespace OPERATIONAL_IDX_01 is comprised of 1253 extents. Segment SAMPRD.CONF_C_FK_I in tablespace PARTITIONED_IDX_01 is comprised of 5170 extents. Segment SAMPRD.CONF_PF1_FK_
CRITICAL 
Dec 20 2005 06:57:14 AM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Tue Dec 20 08:31:37 2005 -->
<pre>cdfofdev profile deafault changed  regarding failed logins</pre>
<!-- Author: bs  -->

<!-- Date: Tue Dec 20 14:42:46 2005 -->
<pre>
Name : cdfonprd_tablespaces 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace MCS_INDEX is 85.65% full.  18629512.0 KBytes of free space remain. 
CRITICAL 
Dec 20 2005 02:40:44 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 00:02:54 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Dec 21 2005 12:02:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 00:11:11 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Dec 21 2005 12:09:49 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 04:04:22 2005 -->
<pre>
Name : cdfonprd_capture_msg_spill_alert  
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 35811 
CRITICAL 
Dec 21 2005 04:03:51 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 04:16:58 2005 -->
<pre>
Name : cdfofpr2_apply_latency_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 712 
WARNING 
Dec 21 2005 04:12:42 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 04:19:46 2005 -->
<pre>
Name : cdfofpr2_apply_latency_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 916.000000000000000000000000000000000004 
CRITICAL 
Dec 21 2005 04:17:42 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 04:30:10 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 14551 
CRITICAL 
Dec 21 2005 04:29:59 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 04:51:36 2005 -->
<pre>
Name : cdfofpr2_long_transaction 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: oracle  27 1 oracle@fcdfora4 (P001) fcdfora4 insert into "CALIB_PRD"."COTCHANT0" p("CID","G_CHAN","I_CHAN","OCCUPANCY","OFFSET","OFFSETRMS","WIDTH","WIDTHRMS")values(:1,:2,:3,:4,:5,:6,:7,:8) 6.91666666666666666666666666666666666704 
CRITICAL 
Dec 21 2005 04:38:27 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 04:51:36 2005 -->
<pre>
Name : cdfofpr2_long_transaction_count 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 1 
WARNING 
Dec 21 2005 04:39:26 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 04:51:36 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 21264 
CRITICAL 
Dec 21 2005 04:42:57 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 04:51:36 2005 -->
<pre>
Name : cdfofpr2_long_transaction_count 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 0 
CLEARED 
Dec 21 2005 04:48:44 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 04:51:36 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 1861.999999999999999999999999999999999998 
CRITICAL 
Dec 21 2005 04:49:40 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 05:00:04 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 25 
CLEARED 
Dec 21 2005 04:57:57 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 05:06:03 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 2640.999999999999999999999999999999999997 
CRITICAL 
Dec 21 2005 05:04:41 AM 



From oracle@bzora1.fnal.gov  Wed Dec 21 05:04:45 2005
Received: from mailgw1.fnal.gov (mailgw1.fnal.gov [131.225.111.11])
	by b0dau30.fnal.gov (SGI-8.9.3/8.9.3) with ESMTP id FAA29967
	for <cdf_elog@b0dau30.fnal.gov>; Wed, 21 Dec 2005 05:04:45 -0600 (CST)
Received: from mailav1.fnal.gov (mailav1.fnal.gov [131.225.111.18])
 by mailgw1.fnal.gov
 (iPlanet Messaging Server 5.2 HotFix 2.06 (built Mar 28 2005))
 with SMTP id <0IRU00KRGHFM6F@mailgw1.fnal.gov> for cdf_elog@b0dau30.fnal.gov;
 Wed, 21 Dec 2005 05:04:45 -0600 (CST)
Received: from mailgw2.fnal.gov ([131.225.111.12])
 by mailav1.fnal.gov (SAVSMTP 3.1.7.47) with SMTP id M2005122105044427369 for
 <cdf_elog@b0dau30.fnal.gov>; Wed, 21 Dec 2005 05:04:44 -0600
Received: from conversion-daemon.mailgw2.fnal.gov by mailgw2.fnal.gov
 (iPlanet Messaging Server 5.2 HotFix 2.06 (built Mar 28 2005))
 id <0IRU00E01HDH22@mailgw2.fnal.gov>
 (original mail from oracle@bzora1.fnal.gov) for cdf_elog@b0dau30.fnal.gov;
 Wed, 21 Dec 2005 05:04:45 -0600 (CST)
Received: from bzora1.fnal.gov (bzora1.fnal.gov [131.225.236.156])
 by mailgw2.fnal.gov
 (iPlanet Messaging Server 5.2 HotFix 2.06 (built Mar 28 2005))
 with ESMTP id <0IRU004YNHFW23@mailgw2.fnal.gov> for cdf_elog@b0dau30.fnal.gov;
 Wed, 21 Dec 2005 05:04:45 -0600 (CST)
Received: (from oracle@localhost)	by bzora1.fnal.gov (8.11.6p2/8.11.6)
 id jBLB4ig29874	for cdf_elog@b0dau30.fnal.gov; Wed,
 21 Dec 2005 05:04:44 -0600 (CST)
Date: Wed, 21 Dec 2005 05:04:44 -0600 (CST)
From: "oracle@bzora1" <oracle@fnal.gov>
To: cdf_elog@b0dau30.fnal.gov
Message-id: <200512211104.jBLB4ig29874@bzora1.fnal.gov>
Content-type: text

Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=209148">209148</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_3_07 [2,725,597] 
<!-- Author: Salvador x2080 -->


<!-- Date: Wed Dec 21 05:14:20 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 27 
CLEARED 
Dec 21 2005 05:12:44 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 05:30:06 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 1365 
CRITICAL 
Dec 21 2005 05:27:44 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 05:57:49 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 4390.000000000000000000000000000000000001 
CRITICAL 
Dec 21 2005 05:39:41 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 05:57:49 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 4897.999999999999999999999999999999999999 
CRITICAL 
Dec 21 2005 05:49:40 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 05:58:05 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
Dec 21 2005 05:57:44 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 06:05:46 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 5652.999999999999999999999999999999999996 
CRITICAL 
Dec 21 2005 06:04:40 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 06:45:20 2005 -->
<pre>
Name : cdfofpr2_stream_heartbeat_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 7269.999999999999999999999999999999999998 
WARNING 
Dec 21 2005 06:26:10 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 06:45:20 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 7214.999999999999999999999999999999999996 
CRITICAL 
Dec 21 2005 06:39:40 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 07:02:15 2005 -->
<pre>
Name : cdfofpr2_multiple_extents 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Multiple Extents 
Segment SAMPRD.PROJECT_FILES in tablespace OPERATIONAL_01 is comprised of 979 extents. Segment SAMPRD.CONSUMED_FILES in tablespace CONSUMED_FILES_P01 is comprised of 2260 extents. Segment SAMPRD.DFLOC_DFLS_FK_I in tablespace PARTITIONED_IDX_01 is comprised of 1650 extents. Segment SAMPRD.DFLOC_FI_FK_I in tablespace PARTITIONED_IDX_01 is comprised of 2799 extents. Segment SAMPRD.DFLOC_VOL_FK_I in tablespace PARTITIONED_IDX_01 is comprised of 1080 extents. Segment HERBER.FILE_UK in tablespace USERDATA_01 is comprised of 1328 extents. Segment SAMPRD.PF1_PK in tablespace OPERATIONAL_IDX_01 is comprised of 737 extents. Segment SAMPRD.PF1_UQ1_UK in tablespace OPERATIONAL_IDX_01 is comprised of 1042 extents. Segment SAMPRD.PF1_FI_FK_I in tablespace OPERATIONAL_IDX_01 is comprised of 1254 extents. Segment SAMPRD.CONF_C_FK_I in tablespace PARTITIONED_IDX_01 is comprised of 5175 extents. Segment SAMPRD.CONF_PF1_FK_I in tablespace PARTITIONED_IDX_01 is comprised of 4991 extents. Segment SAMPRD.
CRITICAL 
Dec 21 2005 06:56:17 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 07:06:24 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 8456.999999999999999999999999999999999996 
CRITICAL 
Dec 21 2005 07:04:40 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 07:25:07 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 9548.000000000000000000000000000000000004 
CRITICAL 
Dec 21 2005 07:24:40 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 07:58:24 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 2480 
CRITICAL 
Dec 21 2005 07:57:44 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 08:06:41 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 11497 
CRITICAL 
Dec 21 2005 08:04:40 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 08:28:38 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 16 
CLEARED 
Dec 21 2005 08:27:44 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 08:56:18 2005 -->
<pre>
Name : cdfofpr2_stream_heartbeat_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 14431 
CRITICAL 
Dec 21 2005 08:55:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 09:30:53 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 15577 
CRITICAL 
Dec 21 2005 09:29:40 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 10:01:30 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 17113 
CRITICAL 
Dec 21 2005 09:59:40 AM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Wed Dec 21 10:36:44 2005 -->
<pre>We have a latency both on CDFOFPR2 & CDFSTRM1.   We are currently caught up to 5:10 a.m.  Current rate is taking 20 minutes to make up 10 minutes of data. 
 
We have DOUBLE THE AMOUNT OF ARCHIVES WE GENERALLY EXPECT! 
 
 
bzora1&gt; ls -rlt | grep 'Dec 19'  | wc -l 
     197 
 
bzora1&gt; ls -rlt | grep 'Dec 20' | wc -l 
     325 
 
bzora1&gt; ls -rlt | grep 'Dec 21' | wc -l 
     498 
 
 
Long transaction  -- yet not the cause simply lots of updating 
going on?!     
 
 
Anybody understand what activity is going on?!  
 
I found 1 large transaction but started at 4:08 altho't it did 
complete at 4:48  
User Defined SQL Test  
Current Value: oracle  27 1 oracle@fcdfora4 (P001) fcdfora4 insert into "CALIB_PRD"."COTCHANT0" p("CID","G_CHAN","I_CHAN","OCCUPANCY","OFFSET","OFFSETRMS","WIDTH","WIDTHRMS")values(:1,:2,:3,:4,:5,:6,:7,:8) 6.91666666666666666666666666666666666704  
CRITICAL  
Dec 21 2005 04:38:27 AM  
 
 
</pre>
<!-- Author: nstanfied  -->
<!-- Comment: Wed Dec 21 10;52;23 comment by...ns  -->
<pre>

bzora1&gt;  runscript cdfonprd show_users | grep RUN_USER | wc -l
      34

bzora1&gt;  runscript cdfonprd show_users | grep CDFSCICO | wc -l
      27

bzora1&gt; runscript cdfonprd show_users  | grep HDWDB_PRD | wc -l
       5
</pre>

<!-- Comment: Wed Dec 21 13;51;19 comment by...w.badgett -->
<pre>We had a problem before with the COTCHANT0 table; the Stage0 (?) 
program was filling it up, I believe;    wasn't this fixed?</pre>
<!-- Date: Wed Dec 21 10:55:11 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 19877 
CRITICAL 
Dec 21 2005 10:54:40 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 10:58:56 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=1135184214 sec, SCN: 0x056f.4db370c4): ORA-01555 caused by SQL statement below (Query Duration=1135184217 sec, SCN: 0x056f.4db370c4): ORA-01555 caused by SQL statement below (Query Duration=1135184217 sec, SCN: 0x056f.4db370c4): ORA-01555 caused by SQL statement below (Query Duration=1135184220 sec, SCN: 0x056f.4db370c4): ORA-01555 caused by SQL statement below (Query Duration=1135184226 sec, SCN: 0x056f.4db370c4): ORA-01555 caused by SQL statement below (Query Duration=1135184226 sec, SCN: 0x056f.4db370c4):. 
WARNING 
Dec 21 2005 10:57:31 AM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Wed Dec 21 11:00:49 2005 -->
<pre>1368                  oracle@bzora1 (PMON) ACTIVE    12/06/2005 09:25 
1370                  oracle@bzora1 (DBW0) ACTIVE    12/06/2005 09:25 
1378                  oracle@bzora1 (SMON) ACTIVE    12/06/2005 09:25 
1374                  oracle@bzora1 (LGWR) ACTIVE    12/06/2005 09:25 
1376                  oracle@bzora1 (CKPT) ACTIVE    12/06/2005 09:25 
1380                  oracle@bzora1 (RECO) ACTIVE    12/06/2005 09:25 
1382                  oracle@bzora1 (CJQ0) ACTIVE    12/06/2005 09:25 
1384                  oracle@bzora1 (QMN0) ACTIVE    12/06/2005 09:25 
1386                  oracle@bzora1 (ARC0) ACTIVE    12/06/2005 09:25 
1388                  oracle@bzora1 (ARC1) ACTIVE    12/06/2005 09:25 
1442       RUN_USER   resusr.ResClient -cc ACTIVE    12/06/2005 09:25 
2087       RUN_USER   ListenToAlerts@b0ser ACTIVE    12/12/2005 16:43 
2121       RUN_USER   ListenToAlerts@b0ser ACTIVE    12/12/2005 16:43 
5476       RUN_USER   resmgr.ResMgr resmgr ACTIVE    12/14/2005 16:42 
6642       RUN_USER   rc.vxmon.VxMonSumFer ACTIVE    12/14/2005 16:45 
1563       RUN_USER   runedit.RunStatusEdi ACTIVE    12/18/2005 02:43 
24091      RUN_USER   rc.GUI               ACTIVE    12/18/2005 15:49 
21055      RUN_USER   rc.vxmon.ScalerMonit ACTIVE    12/20/2005 01:41 
21246      RUN_USER   rc.vxmon.ScalerMonit ACTIVE    12/20/2005 01:42 
27039      RUN_USER   inhibit.InhibitDispl ACTIVE    12/20/2005 20:30 
18136      RUN_USER   rc.GUI               ACTIVE    12/21/2005 06:17 
21254      RUN_USER   rc.GUI               ACTIVE    12/21/2005 10:21 
135        RUN_USER   rundb.DownTimeCopy   ACTIVE    12/21/2005 10:55 
148        RUN_USER   rc.GUI               ACTIVE    12/21/2005 10:55 
698                                        ACTIVE    12/21/2005 10:59 
702                                        ACTIVE    12/21/2005 10:59 
705                   oracle@bzora1 (CP01) ACTIVE    12/21/2005 10:59 
1397                  oracle@bzora1 (P000) ACTIVE    12/21/2005 10:59 
1399                  oracle@bzora1 (P001) ACTIVE    12/21/2005 10:59 
29810                 oracle@bzora1 (P002) ACTIVE    12/21/2005 10:59 
700                                        ACTIVE    12/21/2005 10:59 
26366      STRMADMIN_                      ACTIVE    12/21/2005 10:59 
1446       DBREADER   sqlplus@bzora1 (TNS  ACTIVE    12/21/2005 11:00 
</pre>
<!-- Author: ns  -->
<!-- Comment: Wed Dec 21 13;30;20 comment by...w.badgett -->
<pre>doesn't look unusual...</pre>
<!-- Date: Wed Dec 21 11:20:09 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 20719 
CRITICAL 
Dec 21 2005 11:19:40 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 11:35:11 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 21503 
CRITICAL 
Dec 21 2005 11:34:40 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 12:30:30 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 24327 
CRITICAL 
Dec 21 2005 12:29:40 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 12:45:00 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 25119 
CRITICAL 
Dec 21 2005 12:44:40 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 12:51:12 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 547941 ORA-26672: timeout occurred while stopping STREAMS process CDFONPRD_CAPTURE ORA-06512: at "SYS.DBMS_LOGREP_UTIL", line 111 ORA-06512: at "SYS.DBMS_CAPTURE_ADM", line 59 ORA-06512: at "FLOWC.FLOWC_RUN", line 124 ORA-06512: at line 1. 
WARNING 
Dec 21 2005 12:49:49 PM 



</pre>
<!-- Author: oem-admin (email) -->




<!-- Date: Wed Dec 21 14:11:21 2005 -->
<pre>
Name : cdfonprd_max_extents 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Maximum Extents 
Segment MCS_PRD.SENSH_SENS_FK_I in tablespace MCS_INDEX can only allocate 1 additional extents. 
WARNING 
Dec 21 2005 02:11:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Dec 21 14:27:05 2005 -->
<pre>
Name : cdfonprd_max_extents 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Maximum Extents 
Event test "Maximum Extents" is cleared 
CLEARED 
Dec 21 2005 02:26:07 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Wed Dec 21 16:58:28 2005 -->
<pre>replicated data caught up to  7:25 a.m.   Dec 21, 2005 
9 hours behind!  
 
</pre>
<!-- Author: ns  -->


<!-- Date: Wed Dec 21 21:24:46 2005 -->
<P> I believe I have found & fixed the problem with the DownTimeCopy application running on b0server00. The DOWNTIME_HISTORY database table contains basic entries copied from the AcNet SyBase database table called APPDB.JWANG.CDF_DOWNTIME, plus extra information derived by the DownTimeCopy program. The replication proceeds with the help of a <A HREF=/java/cdfdb/servlet/DownTimeHistory?DOWNCLINK=0> special entry</A> in DOWNTIME_HISTORY table that keeps track of the time of the last replication from AcNet/SyBase to CDF/Oracle. Somehow this entry was accidentally deleted, forcing the replication to repeat update commands ad nauseum. Who accidentally deleted this entry? I really don't like to play the blame game, let's just let the so-called <I>CDF Arrow of Blame</I> keep spinning. <P>
<!-- Author: W.Badgett -->


<!-- Date: Wed Dec 21 21:25:28 2005 -->
<P> I believe I have found & fixed the problem with the DownTimeCopy application running on b0server00. The DOWNTIME_HISTORY database table contains basic entries copied from the AcNet SyBase database table called APPDB.JWANG.CDF_DOWNTIME, plus extra information derived by the DownTimeCopy program. The replication proceeds with the help of a <A HREF=/java/cdfdb/servlet/DownTimeHistory?DOWNCLINK=0> special entry</A> in DOWNTIME_HISTORY table that keeps track of the time of the last replication from AcNet/SyBase to CDF/Oracle. Somehow this entry was accidentally deleted, forcing the replication to repeat update commands ad nauseum. Who accidentally deleted this entry? I really don't like to play the blame game, let's just let the so-called <I>CDF Arrow of Blame</I> keep spinning. <P> 
<!-- Author: W.Badgett -->
<!-- Comment: Wed Dec 21 21;26;25 comment by...wb -->
<pre>Click on the green ball to read this entry...</pre>

<!-- Date: Thu Dec 22 00:04:24 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Dec 22 2005 12:02:30 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Dec 22 00:25:05 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Dec 22 2005 12:09:49 AM 



</pre>
<!-- Author: oem-admin (email) -->