<!-- Title: 07:49:48  Mon Jun 27 2005 -->
<!-- Crewchief: New day  -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>database activity</pre>

<!-- Date: Mon Jun 27 07:56:55 2005 -->
<pre>
Name : cdfonprd_max_extents 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Maximum Extents 
Event test "Maximum Extents" is cleared 
CLEARED 
Jun 27 2005 07:56:10 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 09:32:51 2005 -->
<pre>
Name : cdfofpr2_apply_latency_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 228393.9999999999999999999999999999999996 
CRITICAL 
Jun 27 2005 09:31:58 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 09:36:12 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 227047.0000000000000000000000000000000003 
CRITICAL 
Jun 27 2005 09:35:50 AM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Mon Jun 27 09:41:52 2005 -->
<pre>issue with mt pts on b0dau36 ..   starting with a reboot to see if we can clear up the issue i.e. missing mt pts..... 
 
more later...........</pre>
<!-- Author: nstanfield  -->
<!-- Comment: Mon Jun 27 10;46;27 comment by...nstanfield  -->
<pre>maurine is troubleshooting the missing mt pts ... veritas is new to us & she will follow-up with sun ......   in summary the db's are down till further notice......</pre>
<!-- Comment: Mon Jun 27 16;05;26 comment by...nstanfield  -->
<pre>maurine has worked with sun & resolved the veritas issue.
the databases are up & we are currently backing up cdfondev..cdfonint will run at it's std time this evening </pre>
<!-- Comment: Mon Jun 27 17;02;23 comment by...mmihalek -->
<pre>the power glitch on thursday night caused veritas to disable certain mount 
points. sun was finally able to diagnose the problem and the 4 mount points in 
question have been re-enabled, re-started, and the file systems checked. once 
this was done, they all mounted ok.

the db has been started and checked. nelly is running rman backups.

we feel the rmans should be verified when completed to make sure there are no 
other problems, they be saved onto tape, and that they should run several more 
days.

with the holiday and vacations the end of this week and early next week, we 
would like to postpone the move of b0dau36 until the week of july 11.

maurine
</pre>
<!-- Date: Mon Jun 27 10:13:38 2005 -->
<pre>Over the weekend , we had couple of long duration transaction  from acnet monitor API. 
 
Here are details : 
 
Start time           End Time           Duration 
06/24/05 4:36PM      06/25/05 12:16AM   ~7.5 Hours 
06/25/05 4:30AM      06/25/05 06:41AM   ~2 Hrs 11 Min 
06/25/05 10:25PM     06/26/05 1:31AM    ~3 Hrs 
06/26/05 2:14AM      06/26/05 2:46AM    ~30 Min 
 
Afterthat no long running has been reported. It will be great help if API owner can troubleshoot this. 
 
Now we are dealing with tuning of streams to clear-up all of these long duration transactions. 
 
Thanks, 
 
Anil. 
 
 
----- Original Message -----  
From: &lt;badgett@fnal.gov&gt; 
To: &lt;cdf-rc-support@fnal.gov&gt;; &lt;cdfdb-support@fnal.gov&gt; 
Sent: Sunday, June 26, 2005 3:44 PM 
Subject: store_summary AcnetMonitor problems 
 
 
&gt; All, 
&gt; 
&gt; I've tried a couple work-arounds to avoid the store_summary 
&gt; update hang-up that is happening to the AcnetMonitor 
&gt; program on the database server side, but the problem is still 
&gt; happenning.     It would be a big help if you happen to see this 
&gt; problem occur, before restarting AcnetMonitor, could you please 
&gt; run the scripts: 
&gt; 
&gt;     setup dbatools 
&gt;     runscript cdfonprd show_locks_dml_ddl 
&gt;     runscript cdfonprd show_locks 
&gt;     runscript cdfonprd show_locks_statement 
&gt;     runscript cdfonprd show_rollback_locks 
&gt; 
&gt; and put the output in the database eLog ? 
&gt; 
&gt; Thanks, 
&gt; Bill 
&gt;  
</pre>
<!-- Author: akumar -->
<!-- Comment: Mon Jun 27 13;49;02 comment by...akumar -->
<pre>These long duration transactions caused the data replication latency from 
cdf on-line
prd to cdfofprd and then in turn to cdf replica.

On cdfofprd , we are synced up until 06/25/05 19:35PM
On cdf replica , we are synced up until 06/25/05 15:25PM

It is estimated that by midnight we should in  total sync on cdfoffline 
databases.

</pre>
<!-- Date: Mon Jun 27 10:16:34 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 24 
CLEARED 
Jun 27 2005 10:16:10 AM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Mon Jun 27 10:50:08 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 202229.0000000000000000000000000000000004 
CRITICAL 
Jun 27 2005 10:49:53 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 11:05:57 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 22239 
CRITICAL 
Jun 27 2005 11:05:07 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 11:15:00 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-07445: exception encountered: core dump [0000000100E6E684] [SIGSEGV] [Address not mapped to object] [0x0000004F0] [] [] ORA-07445: exception encountered: core dump [0000000100E6E684] [SIGSEGV] [Address not mapped to object] [0x0000004F0] [] []. 
WARNING 
Jun 27 2005 11:14:44 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 11:20:09 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value:  
CLEARED 
Jun 27 2005 11:19:52 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 11:29:29 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 7470 
CRITICAL 
Jun 27 2005 11:29:10 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 11:43:31 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 418 
CLEARED 
Jun 27 2005 11:42:34 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 11:51:06 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 21 
CLEARED 
Jun 27 2005 11:50:13 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 11:53:11 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 842 
WARNING 
Jun 27 2005 11:52:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 11:57:25 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 481 
CLEARED 
Jun 27 2005 11:57:20 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 12:06:02 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 612 
WARNING 
Jun 27 2005 12:05:13 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 12:10:10 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 179186.0000000000000000000000000000000003 
CRITICAL 
Jun 27 2005 12:09:52 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 12:13:18 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 457 
CLEARED 
Jun 27 2005 12:12:59 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 12:29:04 2005 -->
<pre>
Name : cdfonprd_capture_msg_spill_alert  
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 16 
CLEARED 
Jun 27 2005 12:28:52 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 12:30:14 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 30 
CLEARED 
Jun 27 2005 12:29:34 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 12:39:33 2005 -->
<pre>
Name : cdfonprd_capture_msg_spill_alert  
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 36188 
CRITICAL 
Jun 27 2005 12:38:52 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 12:54:03 2005 -->
<pre>
Name : cdfonprd_capture_msg_spill_alert  
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 28 
CLEARED 
Jun 27 2005 12:53:52 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 13:05:24 2005 -->
<pre>
Name : cdfonprd_capture_msg_spill_alert  
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 2840 
CRITICAL 
Jun 27 2005 01:03:52 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 13:11:36 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 1576.999999999999999999999999999999999999 
CRITICAL 
Jun 27 2005 01:09:55 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 13:30:08 2005 -->
<pre>
Name : cdfonprd_capture_msg_spill_alert  
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 5 
CLEARED 
Jun 27 2005 01:28:52 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 13:30:08 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 153605.0000000000000000000000000000000002 
CRITICAL 
Jun 27 2005 01:29:53 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 13:40:29 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 1412.000000000000000000000000000000000001 
CRITICAL 
Jun 27 2005 01:39:28 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Mon Jun 27 14:04:00 2005 -->
<pre>
Name : cdfonprd_capture_msg_spill_alert  
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 46650 
CRITICAL 
Jun 27 2005 02:03:53 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 14:05:18 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 135250.0000000000000000000000000000000001 
CRITICAL 
Jun 27 2005 02:04:51 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 14:06:22 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 1486.000000000000000000000000000000000002 
CRITICAL 
Jun 27 2005 02:05:55 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 14:14:22 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 1722.999999999999999999999999999999999997 
CRITICAL 
Jun 27 2005 02:14:11 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 14:14:22 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 3965 
CRITICAL 
Jun 27 2005 02:14:11 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 14:17:43 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 41959 
CRITICAL 
Jun 27 2005 02:17:11 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 14:26:00 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=3954 sec, SCN: 0x056e.c546bf93):. 
WARNING 
Jun 27 2005 02:25:36 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 14:40:28 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 126946.0000000000000000000000000000000002 
CRITICAL 
Jun 27 2005 02:39:54 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 14:50:49 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 122236.0000000000000000000000000000000001 
CRITICAL 
Jun 27 2005 02:49:55 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 15:00:09 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 119153.0000000000000000000000000000000002 
CRITICAL 
Jun 27 2005 02:59:54 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 15:05:56 2005 -->
<pre>
Name : cdfofpr2_tablespaces 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace USERTEMP_01 is 87.30% full.  2080240.0 KBytes of free space remain. 
CRITICAL 
Jun 27 2005 03:05:25 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 15:15:39 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 116726 
CRITICAL 
Jun 27 2005 03:14:51 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 15:21:51 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
Jun 27 2005 03:21:02 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 15:30:59 2005 -->
<pre>
Name : cdfofpr2_tablespaces 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Tablespace Full 
Event test "Tablespace Full" is cleared 
CLEARED 
Jun 27 2005 03:30:11 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 15:30:59 2005 -->
<pre>
Name : cdfofpr2_tablespaces 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Tablespace Full 
Event test "Tablespace Full" is cleared 
CLEARED 
Jun 27 2005 03:30:11 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Mon Jun 27 16:09:58 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 100958 
CRITICAL 
Jun 27 2005 04:09:51 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 16:09:58 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 100958 
CRITICAL 
Jun 27 2005 04:09:51 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 16:30:40 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 94946.0000000000000000000000000000000003 
CRITICAL 
Jun 27 2005 04:29:54 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 16:55:06 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 88889 
CRITICAL 
Jun 27 2005 04:54:59 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Mon Jun 27 17:10:19 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value:  
CLEARED 
Jun 27 2005 05:09:59 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 17:15:22 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 80198 
CRITICAL 
Jun 27 2005 05:15:04 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 17:17:26 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 4617 
CRITICAL 
Jun 27 2005 05:17:19 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 17:25:43 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 76448 
CRITICAL 
Jun 27 2005 05:25:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 17:35:25 2005 -->
<pre>
Name : cdfonprd_long_transaction 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: plujan PLUJAN 67 29326 sqlplus@b0dap72.fnal.gov (TNS V1-V3) b0dap72.fnal.gov BEGIN DBMS_APPLICATION_INFO.SET_MODULE(:1,NULL); END; 7.51666666666666666666666666666666666608 
CRITICAL 
Jun 27 2005 05:34:52 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Mon Jun 27 17;55;43 comment by...nstanfield  -->
<pre>file located on b0dap30.fnal.gov/tmp/long_trans_june27.lst

SQL&gt; @show_locks_dml_ddl.sql

OS/Oracle            SID/Serial#  Type Hold       Object                         Program                                          
-------------------- ------------ ---- ---------- ------------------------------ -----------------------------------              
Wait Time (minutes)                                                                                                               
-------------------                                                                                                               
plujan:PLUJAN        67/29326     DML  Row-X (SX) SYS.SNAP$                      sqlplus@b0dap72.fnal.gov (TNS V1-V3              
                                                                                 )                                                
                 25                                                                                                               
                                                                                                                                  
plujan:PLUJAN        67/29326     DML  Row-X (SX) SYS.JOB$                       sqlplus@b0dap72.fnal.gov (TNS V1-V3              
                                                                                 )                                                
                 25                                                                                                               
                                                                                                                                  
oracle:              19/26232     DML  Row-X (SX) STRMADMIN_PRD.AQ$_CAP_STREAM_Q oracle@bzora1 (CP01)                             
                                                  UEUE_TABLE_P                                                                    
                  0                                                                                                               
                                                                                                                                  
plujan:PLUJAN        67/29326     DML  Row-X (SX) RUNDB_PRD.RUN_STATUS           sqlplus@b0dap72.fnal.gov (TNS V1-V3              
                                                                                 )                                                
                 25                                                                                                               
                                                                                                                                  
plujan:PLUJAN        67/29326     DML  Row-X (SX) RUNDB_PRD.RUN_STATUS_HISTORY   sqlplus@b0dap72.fnal.gov (TNS V1-V3              
                                                                                 )                                                
                 25                                                                                                               
                                                                                                                                  
plujan:PLUJAN        67/29326     DML  Row-X (SX) RUNDB_PRD.RUN_STATUS_UPDATES   sqlplus@b0dap72.fnal.gov (TNS V1-V3              
                                                                                 )                                                
                 25                                                                                                               
                                                                                                                                  
plujan:PLUJAN        67/29326     DML  Row-X (SX) SYS.DBMS_ALERT_INFO            sqlplus@b0dap72.fnal.gov (TNS V1-V3              
                                                                                 )                                                
                 25                     
7 rows selected.


SQL&gt; @show_locks
SQL&gt; @show_locks_statement

USERNAME                              SID OWNER                                                                                   
------------------------------ ---------- ----------------------------------------------------------------                        
Object                         LOCKWAIT SQL_TEXT                                                                                  
------------------------------ -------- ----------------------------------------------------------------                          
PLUJAN                                 64 RUNDB_PRD                                                                               
RUN_STATUS                              BEGIN DBMS_APPLICATION_INFO.SET_MODULE(:1,NULL); END;                                     
                                                                                                                                  
PLUJAN                                 64 PLUJAN                                                                                  
RUN_STATUS                              BEGIN DBMS_APPLICATION_INFO.SET_MODULE(:1,NULL); END;                                     
                                                                                                                                  
PLUJAN                                 67 RUNDB_PRD                                                                               
RUN_STATUS_HISTORY                      BEGIN DBMS_APPLICATION_INFO.SET_MODULE(:1,NULL); END;                                     
                                                                                                                                  
PLUJAN                                 67 RUNDB_PRD                                                                               
RUN_STATUS_UPDATES                      BEGIN DBMS_APPLICATION_INFO.SET_MODULE(:1,NULL); END;                                     
                                                                                                                                  
PLUJAN                                 64 PUBLIC                                                                                  
RUN_STATUS                              BEGIN DBMS_APPLICATION_INFO.SET_MODULE(:1,NULL); END;                                     
                                                                                                                                  
RUN_USER                               78 RUNDB_PRD                                                                               
RUN_STATUS                              BEGIN RUNDB.SENSOR_SUMMARY_JOB(200056); END;                                              
                                                                                                                                  
RUN_USER                               40 RUNDB_PRD                                                                               
RUN_STATUS                              0056                                                                                      
                                                                                                                                  
RUN_USER                               40 RUNDB_PRD                                                                               
RUN_STATUS                              UPDATE RUNCONFIGURATIONS SET TSEVENTS=7992065 WHERE RUNNUMBER=20                          
                                                                                                                                  

8 rows selected.


SQL&gt; @show_rollback_locks
 18  ;

OS User  DB User  Schema   Object Name          Type RBS   # of Records                                                           
-------- -------- -------- -------------------- ---- ----- ------------                                                           
plujan   PLUJAN   SYS      SNAP$                TABL _SYSS 525                                                                    
                                                E                                                                                 
                                                                                                                                  
plujan   PLUJAN   SYS      JOB$                 TABL _SYSS 525                                                                    
                                                E                                                                                 
                                                                                                                                  
plujan   PLUJAN   RUNDB_PR RUN_STATUS           TABL _SYSS 525                                                                    
                                                E                                                                                 
                                                                                                                                  
plujan   PLUJAN   RUNDB_PR RUN_STATUS_UPDATES   TABL _SYSS 525                                                                    
                                                E                                                                                 
                                                                                                                                  
plujan   PLUJAN   RUNDB_PR RUN_STATUS_HISTORY   TABL _SYSS 525                                                                    
                                                E                                                                                 
                                                                                                                                  
plujan   PLUJAN   SYS      DBMS_ALERT_INFO      TABL _SYSS 525                                                                    
                                                E                                                                                 
                                                                                                                                  

6 rows selected.</pre>
<!-- Date: Mon Jun 27 17:36:57 2005 -->
<pre>
Name : cdfonprd_long_transaction_count 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 1 
WARNING 
Jun 27 2005 05:36:11 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Mon Jun 27 17:37:41 2005 -->
<pre>Randy deleted table that was associated   TSLTZ  column  so we can proceed with dbtimezone change...   
 
&gt; |SQL&gt; select u.name || '.' || o.name || '.' || c.name TSLTZcolumn 
&gt; |from sys.obj$ o, sys.col$ c, sys.user$ u 
&gt; |where c.type# = 231 
&gt; |and o.obj# = c.obj# 
&gt; |and u.user# = o.owner#; 
&gt; |2 3 4 5 
&gt; |TSLTZCOLUMN 
&gt; |-------------------------------------------------------------------------------- 
&gt; |HERBER.TIMES.LOCAL 
 
 
Need to re-schedule a bounce of cdfofpr2 to re-deploy US/Central  
dbtimezone  --&gt; this was not completed last bounce due to Randy's table referencing  TSLTZ  column  ...he has now deleted that table & we can proceed.   
 
Did this check on cdfonprd & we are ok there so can also 
proceed in scheduling this dbtimezone on cdfonprd as well.   
 
Again, STILL PENDING ON CDFONPRD...........already done on cdfondev & cdfonint </pre>
<!-- Author: ns  -->

<!-- Date: Mon Jun 27 17:38:17 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 5265 
CRITICAL 
Jun 27 2005 05:37:41 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 17:40:22 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 67779 
CRITICAL 
Jun 27 2005 05:40:02 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Mon Jun 27 18:05:21 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 52939 
CRITICAL 
Jun 27 2005 06:04:56 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 18:22:32 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 5876.000000000000000000000000000000000004 
CRITICAL 
Jun 27 2005 06:22:07 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 18:26:02 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 48460 
CRITICAL 
Jun 27 2005 06:24:58 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 18:31:55 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 5976.999999999999999999999999999999999996 
CRITICAL 
Jun 27 2005 06:31:44 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 18:45:14 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 38180 
CRITICAL 
Jun 27 2005 06:44:57 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 19:10:45 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 32533 
CRITICAL 
Jun 27 2005 07:10:06 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 19:16:58 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value:  
CLEARED 
Jun 27 2005 07:16:12 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 19:30:29 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 7585.999999999999999999999999999999999998 
CRITICAL 
Jun 27 2005 07:29:47 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 19:40:11 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 16251 
CRITICAL 
Jun 27 2005 07:39:51 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 19:41:13 2005 -->
<pre>
Name : cdfonprd_long_transaction_count 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 0 
CLEARED 
Jun 27 2005 07:41:08 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 19:42:54 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 8070.000000000000000000000000000000000002 
CRITICAL 
Jun 27 2005 07:42:47 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 19:57:04 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 8780.000000000000000000000000000000000003 
CRITICAL 
Jun 27 2005 07:56:50 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 20:00:26 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 8282.000000000000000000000000000000000002 
CRITICAL 
Jun 27 2005 07:59:54 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 20:05:01 2005 -->
<pre>
Name : cdfofpr2_stream_heartbeat_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 11985 
WARNING 
Jun 27 2005 08:04:46 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 20:06:39 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 8869.999999999999999999999999999999999997 
CRITICAL 
Jun 27 2005 08:06:26 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 20:14:56 2005 -->
<pre>
Name : cdfofpr2_stream_heartbeat_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 6285.000000000000000000000000000000000004 
CLEARED 
Jun 27 2005 08:14:46 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 20:25:17 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: .9999999999999999999999999999999999999996 
CLEARED 
Jun 27 2005 08:24:52 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 20:27:21 2005 -->
<pre>
Name : cdfofpr2_apply_latency_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 0 
CLEARED 
Jun 27 2005 08:26:58 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 20:29:03 2005 -->
<pre>
Name : cdfonprd_capture_msg_spill_alert  
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 32 
CLEARED 
Jun 27 2005 08:28:53 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 20:31:29 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 28 
CLEARED 
Jun 27 2005 08:31:04 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 20:38:53 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 7 ORA-26672: timeout occurred while stopping STREAMS process CDFOFPR2_CAPTURE ORA-06512: at "SYS.DBMS_LOGREP_UTIL", line 111 ORA-06512: at "SYS.DBMS_CAPTURE_ADM", line 59 ORA-06512: at "FLOWC.FLOWC_RUN", line 124 ORA-06512: at line 1. 
WARNING 
Jun 27 2005 08:38:35 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Jun 27 22:39:29 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Alert 
The following errors are in the ALERT file : ORA-04030: out of process memory when trying to allocate  bytes (,) ORA-02063: preceding line from CDFSTRM1 ORA-00600: internal error code, arguments: [kpotcpop1], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kpotcpop1], [], [], [], [], [], [], []. 
CRITICAL 
Jun 27 2005 10:39:24 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 00:05:00 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Jun 28 2005 12:04:32 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 00:10:37 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Jun 28 2005 12:09:59 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 02:17:19 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 14586 
CRITICAL 
Jun 28 2005 02:16:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 02:31:54 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 26 
CLEARED 
Jun 28 2005 02:31:34 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 03:09:10 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=3973 sec, SCN: 0x056e.c61ea3d2):. 
WARNING 
Jun 28 2005 03:08:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 03:28:39 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 7 ORA-26666: cannot alter STREAMS process CDFOFPR2_CAPTURE ORA-06512: at "SYS.DBMS_LOGREP_UTIL", line 111 ORA-06512: at "SYS.DBMS_CAPTURE_ADM", line 30 ORA-06512: at "FLOWC.FLOWC_RUN", line 142 ORA-06512: at line 1. 
WARNING 
Jun 28 2005 03:28:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 03:44:22 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 9 
CLEARED 
Jun 28 2005 03:44:11 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 04:08:54 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Alert 
The following errors are in the ALERT file : ORA-00600: internal error code, arguments: [kghfrempty:ds], [0xFFFFFFFF7BB97410], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [0000000101DB136C] [SIGSEGV] [Address not mapped to object] [0x760077007852D0] [] [] ORA-00600: internal error code, arguments: [kghfrempty:ds], [0xFFFFFFFF7BB97410], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [0000000101DB136C] [SIGSEGV] [Address not mapped to object] [0x760077007852D0] [] [] ORA-07445: exception encountered: core dump [0000000101DB136C] [SIGSEGV] [Address not mapped to object] [0x760077007852D0] [] [] ORA-00600: internal error code, arguments: [kghfrempty:ds], [0xFFFFFFFF7BB97410], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [0000000101DB136C] [SIGSEGV] [Address not mapped to object] [0x760077007852D0] [] [] ORA-07445: exception encountered: core dump [0000000101DB136C] [SIGSEGV] [Address not mapped to object] [0x760077007852D0]
CRITICAL 
Jun 28 2005 04:08:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 04:12:05 2005 -->
<pre>
Name : cdfofpr2_apply_abort 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 1 
CRITICAL 
Jun 28 2005 04:11:14 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 04:41:03 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

Alert 
The following errors are in the ALERT file : ORA-00600: internal error code, arguments: [kope2ucoll700], [], [], [], [], [], [], [] ORA-04030: out of process memory when trying to allocate 16396 bytes (koh-kghu sessi,kolcalm coll) ORA-04030: out of process memory when trying to allocate 16396 bytes (koh-kghu sessi,kod objects) ORA-04030: out of process memory when trying to allocate 16396 bytes (koh-kghu sessi,kod objects) ORA-04030: out of process memory when trying to allocate 16396 bytes (koh-kghu sessi,kod objects) ORA-04030: out of process memory when trying to allocate 16396 bytes (koh-kghu sessi,kod objects) ORA-04030: out of process memory when trying to allocate 16396 bytes (koh-kghu sessi,kod objects) ORA-04030: out of process memory when trying to allocate 16396 bytes (koh-kghu sessi,kod objects) ORA-04030: out of process memory when trying to allocate 16396 bytes (koh-kghu sessi,kod objects) ORA-04030: out of process memory when trying to allocate 16396 bytes (koh-kghu se
CRITICAL 
Jun 28 2005 04:39:59 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 04:55:25 2005 -->
<pre>
Name : cdfonprd_long_transaction 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: cdfdaq RUN_USER 87 23744 rc.acnet.AcnetMonitor b0dap85a INSERT INTO BEAM_CONDITIONS_FULL (CREATETIME,FRONTENDTIME,OUTTMP,IBEAM_A,STCKRT,TEVSTORE,ERING,IBEAM,CLDRST,STRDUR,FWDEL,B0Q5,B0ILUM,B0LLUM,B0TLUM,B0TLIV,B0PLOS,B0ALOS,BQUAKE,LUMLIF,LOSTP,LOSTPB,LOSTPH,SVRAD0,SVRAD1,SVRAD2,SVRAD3,SVBLA1,SVBLA2,SVBLB1,SVBLB2,FBIPNG,FBIANG,FBIPWG,FBIAWG,TPOT1M,TPOT2M,TPOT3M,RFSUM,RFSUMA,CDFDN,B0XIP,B0YIP,B0SIGX,B0SIGY,B0DX,B0DY,B0XTD,B0YTD,B0SXTD,B0SYTD,B0DXTD,B0DYTD,B0PHSM,B0PHN,B0PHT,B0PHS,B0PHB,B0AHSM,B0AHN,B0AHT,B0AHS,B0AHB,B0RAT1,B0RAT2,B0RAT3,B0RAT4,B0RAT5,B0RAT6,B0RAT7,B0PBSM,B0PBN,B0PBT,B0PBS,B0PBB,B0ABSM,B0ABN,B0ABT,B0ABS,B0ABB,B0PAGC,B0AAGC,B0PLB,B0ALB,AGISUM,AGIGI1,AGIGI2,AGIGI3,SLPAG1,SLPAG2,SLPAG3,SLPAH,SBDMS,SBDPWS,SBDPIS,SBDPES,PLBRIS,SBDAWS,SBDAIS,SBDAES,E1TC3R,E1EVUH,E0LTOT,E0LABT,L1COLI,SVTDSX,SVTDSY,SVTDDX,SVTDDY,SVTGVL,SVTGX,SVTGXE,SVTGY,SVTGYE,SVTGWX,SVGWXE,SVTGWY,SVGWYE,SVTDX,SVTDXE,SVTDY,SVTDYE) VALUES (SYSDATE,1119952094,72.420846,131.95405,9.155818,4235,979.
CRITICAL 
Jun 28 2005 04:54:59 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 04:57:29 2005 -->
<pre>
Name : cdfonprd_long_transaction_count 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 1 
WARNING 
Jun 28 2005 04:56:09 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 05:06:20 2005 -->
<pre>
Name : cdfonprd_long_transaction_count 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 0 
CLEARED 
Jun 28 2005 05:06:09 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 05:09:07 2005 -->
<pre>
Name : cdfonprd_capture_msg_spill_alert  
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 3640 
CRITICAL 
Jun 28 2005 05:08:54 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 06:10:04 2005 -->
<pre>
Name : cdfofpr2_stream_heartbeat_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 7485.999999999999999999999999999999999998 
WARNING 
Jun 28 2005 06:09:46 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 06:16:45 2005 -->
<pre>
Name : cdfofpr2_apply_abort 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 0 
CLEARED 
Jun 28 2005 06:16:17 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 06:17:09 2005 -->
<pre>
Name : cdfofpr2_apply_latency_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 7500.999999999999999999999999999999999997 
CRITICAL 
Jun 28 2005 06:16:58 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 06:20:24 2005 -->
<pre>
Name : cdfofpr2_stream_heartbeat_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 7186 
CLEARED 
Jun 28 2005 06:19:46 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 06:20:24 2005 -->
<pre>
Name : cdfonprd_capture_latency_alert 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 233 
CLEARED 
Jun 28 2005 06:19:51 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 06:37:28 2005 -->
<pre>
Name : cdfofpr2_apply_latency_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: .9999999999999999999999999999999999999996 
CLEARED 
Jun 28 2005 06:36:59 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 06:39:01 2005 -->
<pre>
Name : cdfonprd_capture_msg_spill_alert  
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 13 
CLEARED 
Jun 28 2005 06:38:52 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 10:36:12 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: .9999999999999999999999999999999999999996 
CLEARED 
Jun 28 2005 10:36:01 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 14:16:46 2005 -->
<pre>
Name : cdfofpr2_long_transaction 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: swhite SAMREAD 128 45069 tora.exe FERMI\CD-90474 SELECT TO_CHAR(SYSDATE) FROM sys.DUAL 6.3 
CRITICAL 
Jun 28 2005 02:16:09 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 14:18:51 2005 -->
<pre>
Name : cdfofpr2_long_transaction_count 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 1 
WARNING 
Jun 28 2005 02:18:09 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 15:18:45 2005 -->
<pre>
Name : cdfofpr2_long_transaction_count 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 0 
CLEARED 
Jun 28 2005 03:18:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Jun 28 18:25:33 2005 -->
<pre>
Name : cdfonprd_long_transaction 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: cdf-cap-solenoid ICICLE 102 58618 icicle.exe FERMI\SOLENOID2 UPDATE SENSORS SET LAST_MEASURE= 0,LAST_ERROR= 0,LAST_STATUS= 'ON',SENST_ID= 1839971 WHERE SENS_ID= 14407 7.41666666666666666666666666666666666672 
CRITICAL 
Jun 28 2005 06:24:42 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 29 00:04:17 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Jun 29 2005 12:03:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 29 00:10:29 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Jun 29 2005 12:09:50 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Jun 29 03:19:35 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=4440 sec, SCN: 0x056e.c6633608):. 
WARNING 
Jun 29 2005 03:18:35 AM 



</pre>
<!-- Author: oem-admin (email) -->