<!-- Title: 08:54:05  Tue May 10 2005 -->
<!-- Crewchief: New day -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>database activity</pre>

<!-- Date: Tue May 10 10:48:26 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 5768 
CRITICAL 
May 10 2005 10:48:10 AM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Tue May 10 11;12;35 comment by...akumar  -->
<pre>Dropped the user samprd on cdfstrm1 instance.</pre>
<!-- Date: Tue May 10 10:48:26 2005 -->
<pre>Dropping SAMPRD on replica cdfstrm1 database. Since SAMPRD is not replicated. So SAMPRD on cdfstrm1 is out of sync. So SAMPRD@cdfstrm1 is useless. </pre>
<!-- Author: akumar -->


<!-- Date: Tue May 10 11:56:11 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 271 
CLEARED 
May 10 2005 11:56:01 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 12:06:32 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 693 
WARNING 
May 10 2005 12:06:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 12:11:17 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 970.000000000000000000000000000000000004 
CRITICAL 
May 10 2005 12:11:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 12:21:11 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 772 
WARNING 
May 10 2005 12:21:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 12:31:09 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 1182.000000000000000000000000000000000004 
CRITICAL 
May 10 2005 12:31:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 13:06:20 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 1672.000000000000000000000000000000000004 
CRITICAL 
May 10 2005 01:06:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 13:16:09 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 1906.000000000000000000000000000000000001 
CRITICAL 
May 10 2005 01:16:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 13:31:10 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 2759.999999999999999999999999999999999996 
CRITICAL 
May 10 2005 01:31:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 13:51:04 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 2897.000000000000000000000000000000000001 
CRITICAL 
May 10 2005 01:51:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 14:06:25 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 3231.999999999999999999999999999999999999 
CRITICAL 
May 10 2005 02:06:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 14:16:11 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 3808.999999999999999999999999999999999996 
CRITICAL 
May 10 2005 02:16:01 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Tue May 10 14:30:04 2005 -->
<pre>hi, 
I recompiled HDWDB PACKAGE on offline instances 
 
cdfofpr2 
cdfstrm1 
 
Anil checked that the HDWDB package is now VALID. 
 
Please try again your query/program and let us know if now it's OK. 
 
Thanks, 
Donatella 
</pre>
<!-- Author: dt -->

<!-- Date: Tue May 10 14:41:06 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 4267.000000000000000000000000000000000002 
CRITICAL 
May 10 2005 02:41:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 14:52:30 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 4721 
CRITICAL 
May 10 2005 02:51:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 15:01:09 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 5278.000000000000000000000000000000000003 
CRITICAL 
May 10 2005 03:01:02 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 15:16:45 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 5215 
CRITICAL 
May 10 2005 03:16:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 15:27:06 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 5602.000000000000000000000000000000000003 
CRITICAL 
May 10 2005 03:26:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 15:35:23 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 03:35:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 15:40:30 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 03:40:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 15:41:35 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 6307.000000000000000000000000000000000001 
CRITICAL 
May 10 2005 03:41:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 15:51:22 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 6427.000000000000000000000000000000000002 
CRITICAL 
May 10 2005 03:51:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 16:02:42 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 6963.000000000000000000000000000000000002 
CRITICAL 
May 10 2005 04:01:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 16:10:59 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 04:10:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 16:15:14 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 04:15:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 16:17:11 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 7453.000000000000000000000000000000000002 
CRITICAL 
May 10 2005 04:16:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 16:26:05 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 8017.999999999999999999999999999999999998 
CRITICAL 
May 10 2005 04:26:01 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Tue May 10 16;41;18 comment by...akumar -->
<pre>I guess , there is more activity on cdfofpr2 today as compared previous 
couple of days.
We already got 180 archive logs today Vs average of 100 archive logs for 
whole day in last couple of days.


&lt;fcdfora4&gt; ls -lsrt | grep 'May  6' | wc -l
      91
&lt;fcdfora4&gt; ls -lsrt | grep 'May  7' | wc -l
     105
&lt;fcdfora4&gt; ls -lsrt | grep 'May  8' | wc -l
     100
&lt;fcdfora4&gt; ls -lsrt | grep 'May  9' | wc -l
     115
&lt;fcdfora4&gt; ls -lsrt | grep 'May 10' | wc -l
     180

We have to monitor this archive load.

</pre>

<!-- Date: Tue May 10 16:46:08 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 8015 
CRITICAL 
May 10 2005 04:46:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 16:54:02 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Alert 
The following errors are in the ALERT file : ORA-00600: internal error code, arguments: [kngoSize], [1], [], [], [], [], [], []. 
CRITICAL 
May 10 2005 04:53:35 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Tue May 10 17;22;50 comment by...akuamr -->
<pre>Verified the streams process on cdfofpr2 . Nothing broken. Capture is 
disabled
because of flowc threshold.

SQL&gt; @check

Session altered.

Streams Heart Beat Status

DESCRIPTION                                                  UPDATE_DATE
------------------------------------------------------------ -----------------
STREAM STATUS                                                05/10/05 
17:00:00

Streams Apply Status

APPLY_NAME                     STATUS
------------------------------ --------
CDFOFPR2_APPLY                 ENABLED

Number of Apply Errors

  COUNT(*)
----------
         0

Streams Capture Status

CAPTURE_NAME                   STATUS
------------------------------ --------
CDFOFPR2_CAPTURE               DISABLED

Streams Propagation Status

QNAME                            FAILURES    LATENCY
------------------------------ ---------- ----------
CAP_STREAMS_QUEUE                       0          5

</pre>
<!-- Date: Tue May 10 17:01:12 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 8735.999999999999999999999999999999999999 
CRITICAL 
May 10 2005 05:01:01 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Tue May 10 17;24;31 comment by...akumar  -->
<pre>I guess , there is more activity on cdfofpr2 today as compared previous 
couple of days.
We already got 180 archive logs today Vs average of 100 archive logs for 
whole day in last couple of days.


&lt;fcdfora4&gt; ls -lsrt | grep 'May  6' | wc -l
      91
&lt;fcdfora4&gt; ls -lsrt | grep 'May  7' | wc -l
     105
&lt;fcdfora4&gt; ls -lsrt | grep 'May  8' | wc -l
     100
&lt;fcdfora4&gt; ls -lsrt | grep 'May  9' | wc -l
     115
&lt;fcdfora4&gt; ls -lsrt | grep 'May 10' | wc -l
     180

We have to monitor this archive load.

</pre>
<!-- Date: Tue May 10 17:11:10 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 9208.000000000000000000000000000000000002 
CRITICAL 
May 10 2005 05:11:01 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Tue May 10 17;23;27 comment by...akumar  -->
<pre>Stopped the capture on cdfofpr2 since there is lot of spillage of messages.
I will monitor. Once the spillage will be cleared then capture can be 
started.

Thanks,

Anil.</pre>
<!-- Date: Tue May 10 17:20:57 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 05:20:08 PM 



</pre>
<!-- Author: oem-admin (email) -->





<!-- Date: Tue May 10 18:03:21 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 22 
CLEARED 
May 10 2005 06:03:11 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 18:03:50 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Alert 
The following errors are in the ALERT file : ORA-00600: internal error code, arguments: [kpotcpop1], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kpotcpop1], [], [], [], [], [], [], []. 
CRITICAL 
May 10 2005 06:03:35 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 18:50:15 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 06:50:09 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 18:51:30 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 10596 
CRITICAL 
May 10 2005 06:51:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 19:03:25 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 8903 
CRITICAL 
May 10 2005 07:03:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 19:05:29 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 07:05:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 19:21:17 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 07:20:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 19:21:17 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 8156.999999999999999999999999999999999998 
CRITICAL 
May 10 2005 07:21:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 19:30:19 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 07:30:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 19:36:32 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 8275.999999999999999999999999999999999997 
CRITICAL 
May 10 2005 07:36:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 19:40:40 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 07:40:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 19:50:14 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 07:50:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 19:58:31 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 24 
CLEARED 
May 10 2005 07:58:11 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 20:05:31 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 08:05:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 20:06:14 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 8671 
CRITICAL 
May 10 2005 08:06:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 20:13:16 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 3167 
CRITICAL 
May 10 2005 08:13:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 20:35:12 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 08:35:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 20:55:13 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 08:55:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 20:56:26 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 9205.999999999999999999999999999999999998 
CRITICAL 
May 10 2005 08:56:02 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 20:58:30 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 24 
CLEARED 
May 10 2005 08:58:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:03:30 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 2363 
CRITICAL 
May 10 2005 09:03:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:06:13 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 09:05:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:10:21 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 09:10:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:13:52 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 24 
CLEARED 
May 10 2005 09:13:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:15:56 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 09:15:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:25:23 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 09:25:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:31:11 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 9504 
CRITICAL 
May 10 2005 09:31:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:38:54 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 525 
CRITICAL 
May 10 2005 09:38:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:40:18 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 09:40:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:46:12 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 09:45:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:46:12 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 9034.000000000000000000000000000000000001 
CRITICAL 
May 10 2005 09:46:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:48:16 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 13 
CLEARED 
May 10 2005 09:48:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:50:20 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 09:50:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:55:35 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 09:55:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 21:56:19 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 8530.999999999999999999999999999999999998 
CRITICAL 
May 10 2005 09:56:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:03:23 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 2247 
CRITICAL 
May 10 2005 10:03:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:05:53 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 10:05:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:08:15 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 7 
CLEARED 
May 10 2005 10:08:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:11:01 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 10:10:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:11:16 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 7973.000000000000000000000000000000000001 
CRITICAL 
May 10 2005 10:11:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:15:25 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 10:15:09 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:20:33 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 10:20:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:21:21 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 7635.999999999999999999999999999999999997 
CRITICAL 
May 10 2005 10:21:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:23:26 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 833 
CRITICAL 
May 10 2005 10:23:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:25:30 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 10:25:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:31:42 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 7361.000000000000000000000000000000000004 
CRITICAL 
May 10 2005 10:31:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:33:46 2005 -->
<pre>
Name : cdfofpr2_streams_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 10:33:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:40:16 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 0 
CRITICAL 
May 10 2005 10:40:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:45:15 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 10:45:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:46:03 2005 -->
<pre>
Name : cdfofpr2_capture_errors  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 10 2005 10:30:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:46:13 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 3790.999999999999999999999999999999999999 
CRITICAL 
May 10 2005 10:46:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 10 22:51:05 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 0 
CLEARED 
May 10 2005 10:51:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed May 11 00:01:54 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
May 11 2005 12:01:32 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed May 11 00:03:59 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
May 11 2005 12:03:35 AM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Wed May 11 13:46:31 2005 -->
<pre>on May 10, 2005  on cdfofprd, we had 280 archives vs ~100/day other days. 
 
 
&lt;fcdfora4&gt; ls -lsrt | grep 'May  6' | wc -l 
       0 
&lt;fcdfora4&gt; ls -lsrt | grep 'May  7' | wc -l 
      89 
&lt;fcdfora4&gt; ls -lsrt | grep 'May  8' | wc -l 
     100 
&lt;fcdfora4&gt; ls -lsrt | grep 'May  9' | wc -l 
     115 
&lt;fcdfora4&gt; ls -lsrt | grep 'May 10' | wc -l 
     280 
 
 
Also around 12 large transactions were detected. 
 
Tue May 10 10:11:35 2005 
CP01: large txn detected (30108 LCRs), xid: 0x000a.023.00009dde 
 
Tue May 10 10:21:35 2005 
CP01: large txn detected (50784 LCRs), xid: 0x0006.023.00009da8 
 
Tue May 10 10:41:35 2005 
CP01: large txn detected (14647 LCRs), xid: 0x0001.026.000139b4 
 
Tue May 10 11:11:35 2005 
CP01: large txn detected (21025 LCRs), xid: 0x0009.010.00009ef1 
 
Tue May 10 11:31:36 2005 
CP01: large txn detected (28201 LCRs), xid: 0x0002.000.00009efc 
 
Tue May 10 14:25:45 2005 
CP01: large txn detected (53388 LCRs), xid: 0x0003.006.0000a012 
 
Tue May 10 15:10:01 2005 
CP01: large txn detected (17535 LCRs), xid: 0x0004.018.0000a199 
 
Tue May 10 19:28:05 2005 
CP01: large txn detected (26536 LCRs), xid: 0x0004.002.0000a463 
 
Tue May 10 19:45:15 2005 
CP01: large txn detected (53854 LCRs), xid: 0x0008.023.0000a509 
 
Tue May 10 20:21:52 2005 
CP01: large txn detected (85562 LCRs), xid: 0x0009.020.0000a53f 
 
Tue May 10 21:01:33 2005 
CP01: large txn detected (34840 LCRs), xid: 0x0005.016.0000a4dd 
 
 
If you know any exceptional activity on cdfofprd yesterday, please let's  
know. 
 
Also in future if you are planning to do some exceptional activity , please  
let's know 
in advance sothat we can monitoir the resources. 
 
Yesterday's load on cdfofprd caused around 2 hours of latency on streams  
from cdfofprd 
to cdfstrm1.  Large transactions are the suspect of this latency. We know  
the timings 
when these transactions were detected and we are investigation which 
applicatio(s) caused these large transactions on cdfofprd database. 
</pre>
<!-- Author: akumar  -->