<!-- Title: 07:46:38  Thu May 26 2005 -->
<!-- Crewchief: New Day  -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>database activity</pre>

<!-- Date: Thu May 26 08:00:46 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 47 
CLEARED 
May 26 2005 08:00:38 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 08:31:31 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 33198 
CRITICAL 
May 26 2005 08:30:38 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 09:00:45 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 27 
CLEARED 
May 26 2005 09:00:38 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 12:16:06 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 752 
CRITICAL 
May 26 2005 12:15:38 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 12:30:43 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 26 
CLEARED 
May 26 2005 12:30:38 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 13:02:17 2005 -->
<pre>
Name : cdfofpr2_db_down 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Database UpDown 
The database instance is down. 
CRITICAL 
May 26 2005 01:02:06 PM 
Node UpDown 
Event test "Node UpDown" is cleared 
CLEARED 
Apr 29 2005 01:19:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 13:03:40 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-02068: following severe error from CDFSTRM1 ORA-01034: ORACLE not available ORA-27101: shared memory realm does not exist. 
WARNING 
May 26 2005 01:03:35 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 13:04:37 2005 -->
<pre>
Name : cdfofpr2_propagation_hung  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 2 
CRITICAL 
May 26 2005 01:04:17 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 13:06:17 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 92359.0000000000000000000000000000000002 
CRITICAL 
May 26 2005 01:06:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 13:07:06 2005 -->
<pre>
Name : cdfofpr2_apply_latency_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 2 
CLEARED 
May 26 2005 01:06:59 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 13:07:26 2005 -->
<pre>
Name : cdfofpr2_db_down 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Database UpDown 
Event test "Database UpDown" is cleared 
CLEARED 
May 26 2005 01:07:06 PM 
Node UpDown 
Event test "Node UpDown" is cleared 
CLEARED 
Apr 29 2005 01:19:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 13:08:46 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-02068: following severe error from CDFSTRM1 ORA-01033: ORACLE initialization or shutdown in progress. 
WARNING 
May 26 2005 01:08:35 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 13:09:30 2005 -->
<pre>
Name : cdfofpr2_propagation_hung  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 0 
CLEARED 
May 26 2005 01:09:17 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 13:11:46 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-02068: following severe error from CDFOFPR2 ORA-03114: not connected to ORACLE ORA-02068: following severe error from CDFOFPR2 ORA-03113: end-of-file on communication channel ORA-02068: following severe error from CDFOFPR2 ORA-03113: end-of-file on communication channel ORA-02068: following severe error from CDFOFPR2 ORA-01034: ORACLE not available ORA-27101: shared memory realm does not exist. 
WARNING 
May 26 2005 01:11:31 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Thu May 26 13:12:08 2005 -->
<pre>Dear Colleagues, 
 
I was authorised by Petar to make this announcement. 
 
An erroneous update query generated huge load on replication 
process resulting in replica DB not being updated. 
 
After careful investigation of the situation DBAs think it is 
best to bounc the source - offline production database. 
 
Bouncing means - start/stop of the DB. Interruption is 10 minutes. 
 
DBAs want to do it on a.s.a.p. basis, that means 1:00 pm today. 
 
General users are not affected. 
 
Affected parties: 
 
1) Farms 
2) data logging 
3) SAM 
 
  I talked with Suen (that mean Farms) and they have low activity so 
low impact. 
 
(3) will reconnect so not affected 
 
(2) is not affected (physcal data going to tape). Cataloguing 
_may_ be affected but can be easily recovered (thanks to Art scripts) 
 
And in the end - I was the one who made the query , I was updating 
luminosity values of files which became out of sync and made a mistake. 
 
I am sorry fo inconvenience it has caused. 
 
regards, 
Dmitri Litvintsev 
 
</pre>
<!-- Author: ns -->


<!-- Date: Thu May 26 13:12:22 2005 -->
<pre>Sent: Thursday, May 26, 2005 1:11 PM 
Subject: Re: immenent bouncing of offline production database at 1:00 pm (10 min interruption) - COMPLETED 
 
 
&gt; This is to inform you that the bouncing was made. Things are going 
&gt; back to normal on replica now (except FILECATALOG which will be out 
&gt; of sync for  little more, I will keep you posted) 
&gt;  
&gt; Dmitri Litvintsev</pre>
<!-- Author: ns  -->

<!-- Date: Thu May 26 13:14:19 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 0 
CLEARED 
May 26 2005 01:14:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 13:29:27 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 9801 
CRITICAL 
May 26 2005 01:29:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 14:46:15 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 57742 
CRITICAL 
May 26 2005 02:46:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 15:11:14 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 51279 
CRITICAL 
May 26 2005 03:11:02 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 15:36:33 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 47235 
CRITICAL 
May 26 2005 03:36:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 15:46:06 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 43881 
CRITICAL 
May 26 2005 03:46:01 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Thu May 26 15:48:47 2005 -->
<pre>We are synced-up on cdfstrm1 except FILECATALOG upto 22:35 yesterday. 
It may take another 2.5 hours to sync-up. 
 
Then tommorrow morning, we are planning to resinstantiate FILECATALOG  application from cdfofpr2 to cdfstrm1. 
 
In the evening Dmitri is planning to correct the data for FILECATALOG. 
Reviewed the script. There is commit after every 1000 files.  It will 
not have any replication load since FILECATALOG is except from streaming 
until tommorrow morning. 
 
Petar, 
 
Please schedule the root cause. 
 
</pre>
<!-- Author: akumar -->

<!-- Date: Thu May 26 16:01:19 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 41935 
CRITICAL 
May 26 2005 04:01:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 16:36:36 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 40041 
CRITICAL 
May 26 2005 04:36:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 16:44:53 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 27 
CLEARED 
May 26 2005 04:44:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 16:53:57 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12012: error on auto execute of job 7 ORA-26672: timeout occurred while stopping STREAMS process CDFOFPR2_CAPTURE ORA-06512: at "SYS.DBMS_LOGREP_UTIL", line 111 ORA-06512: at "SYS.DBMS_CAPTURE_ADM", line 59 ORA-06512: at "FLOWC.FLOWC_RUN", line 124 ORA-06512: at line 1. 
WARNING 
May 26 2005 04:53:36 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 16:59:23 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 2023 
CRITICAL 
May 26 2005 04:59:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 17:16:08 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 13959 
CRITICAL 
May 26 2005 05:16:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 17:26:11 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 11818 
CRITICAL 
May 26 2005 05:26:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 17:29:24 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 20 
CLEARED 
May 26 2005 05:29:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 17:46:10 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 4432 
CRITICAL 
May 26 2005 05:46:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 17:56:18 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 2235.000000000000000000000000000000000004 
CRITICAL 
May 26 2005 05:56:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu May 26 18:06:26 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 0 
CLEARED 
May 26 2005 06:06:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri May 27 00:02:36 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
May 27 2005 12:01:32 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri May 27 00:03:43 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
May 27 2005 12:03:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri May 27 12:04:00 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-07445: exception encountered: core dump [0000000101DAC2D4] [SIGBUS] [Invalid address alignment] [0xFFFFFFFFDC096FEC] [] [] ORA-12805: parallel query server died unexpectedly. 
WARNING 
May 27 2005 12:03:35 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Fri May 27 16;16;17 comment by...akumar -->
<pre>Cron job started the streams replication processes. </pre>


<!-- Date: Fri May 27 16:19:16 2005 -->
<pre>FILECATALOG schema was not reinstantiated from cdfofpr2 to cdfstrm1 since Dmitri Litvinse ( FILECATALOG application owner) 's data correction script for FILECATALOG script is still running. 
 
FILECATALOG application will be reinstantiated on Tuesday 05/31/05. This has been discussed with Dmitri. </pre>
<!-- Author: akumar -->

<!-- Date: Sat May 28 00:02:14 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
May 28 2005 12:01:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat May 28 00:04:18 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
May 28 2005 12:03:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat May 28 11:02:08 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

Alert 
The following errors are in the ALERT file : ORA-00600: internal error code, arguments: [kohfrm505], [], [], [], [], [], [], [] ORA-04030: out of process memory when trying to allocate 16396 bytes (koh-kghu sessi,kolcpcon indlds) ORA-04030: out of process memory when trying to allocate  bytes (,) ORA-02063: preceding line from CDFOFPR2 ORA-00600: internal error code, arguments: [kpotcpop1], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kpotcpop1], [], [], [], [], [], [], []. 
CRITICAL 
May 28 2005 11:01:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun May 29 00:01:59 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
May 29 2005 12:01:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun May 29 00:05:44 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
May 29 2005 12:03:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun May 29 11:22:31 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-07445: exception encountered: core dump [kghbshrt()+72] [SIGBUS] [Object specific hardware error] [0xED567FC0] [] [] ORA-07445: exception encountered: core dump [kghalf()+1252] [SIGBUS] [Object specific hardware error] [0xFEB7703C] [] []. 
WARNING 
May 29 2005 11:21:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun May 29 21:37:59 2005 -->
<pre>
Name : cdfonprd_tablespaces 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace USERTEMP_01 is 97.74% full.  184976.0 KBytes of free space remain. Tablespace MCS_INDEX is 88.42% full.  10726920.0 KBytes of free space remain. Tablespace MCS_DATA is 77.99% full.  4858264.0 KBytes of free space remain. 
CRITICAL 
May 29 2005 09:37:39 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Tue May 31 09;51;52 comment by...ns  -->
<pre>added 8G to mc_index </pre>
<!-- Date: Sun May 29 21:42:07 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 29 2005 09:41:31 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun May 29 21:51:57 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 29 2005 09:51:32 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun May 29 23:01:48 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 29 2005 11:01:31 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun May 29 23:32:50 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 29 2005 11:31:31 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 00:01:47 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 12:01:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 00:03:51 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
May 30 2005 12:03:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 00:31:45 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 12:31:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 01:01:48 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 01:01:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 02:01:47 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 02:01:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 02:31:44 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 02:31:32 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 02:37:57 2005 -->
<pre>
Name : cdfonprd_tablespaces 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace MCS_INDEX is 88.42% full.  10726920.0 KBytes of free space remain. Tablespace MCS_DATA is 77.99% full.  4858264.0 KBytes of free space remain. 
CRITICAL 
May 30 2005 02:37:39 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 03:31:44 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 03:31:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 03:37:57 2005 -->
<pre>
Name : cdfonprd_tablespaces 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace USERTEMP_01 is 90.72% full.  760208.0 KBytes of free space remain. Tablespace MCS_INDEX is 88.42% full.  10726920.0 KBytes of free space remain. Tablespace MCS_DATA is 77.99% full.  4858264.0 KBytes of free space remain. 
CRITICAL 
May 30 2005 03:37:39 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 03:51:53 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 03:51:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 05:01:45 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01  ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 05:01:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 06:01:45 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 06:01:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 06:31:45 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 06:31:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 06:37:57 2005 -->
<pre>
Name : cdfonprd_tablespaces 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace MCS_INDEX is 88.42% full.  10726920.0 KBytes of free space remain. Tablespace MCS_DATA is 77.99% full.  4858264.0 KBytes of free space remain. 
CRITICAL 
May 30 2005 06:37:39 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:19:46 2005 -->
<pre>
Name : bzora1_node_down 
Target Name : bzora1.fnal.gov 
Target Type : Node 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Node UpDown 
Event test "Node UpDown" is cleared 
CLEARED 
May 30 2005 09:18:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:19:46 2005 -->
<pre>
Name : cdfofpr2_db_down 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Node UpDown 
Event test "Node UpDown" is cleared 
CLEARED 
May 30 2005 09:18:35 AM 
Database UpDown 
Event test "Database UpDown" is cleared 
CLEARED 
May 26 2005 01:07:06 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:19:46 2005 -->
<pre>
Name : cdfonprd_db_down 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Node UpDown 
Event test "Node UpDown" is cleared 
CLEARED 
May 30 2005 09:18:35 AM 
Database UpDown 
Event test "Database UpDown" is cleared 
CLEARED 
May 05 2005 10:33:08 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:25:59 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12545: Connect failed because target host or object does not exist ORA-12545: Connect failed because target host or object does not exist. 
WARNING 
May 30 2005 07:23:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:25:59 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12545: Connect failed because target host or object does not exist ORA-12545: Connect failed because target host or object does not exist. 
WARNING 
May 30 2005 07:28:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:25:59 2005 -->
<pre>
Name : cdfofpr2_propagation_hung  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 2 
CRITICAL 
May 30 2005 07:24:17 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:25:59 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12545: Connect failed because target host or object does not exist. 
WARNING 
May 30 2005 07:33:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:25:59 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12545: Connect failed because target host or object does not exist. 
WARNING 
May 30 2005 07:58:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:25:59 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12545: Connect failed because target host or object does not exist. 
WARNING 
May 30 2005 07:43:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:25:59 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-12545: Connect failed because target host or object does not exist. 
WARNING 
May 30 2005 08:28:36 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:34:15 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 08:01:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:34:15 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 07:31:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:34:15 2005 -->
<pre>
Name : cdfonprd_tablespaces 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace USERTEMP_01 is 88.68% full.  927128.0 KBytes of free space remain. Tablespace MCS_INDEX is 88.42% full.  10726920.0 KBytes of free space remain. Tablespace MCS_DATA is 78.10% full.  4832984.0 KBytes of free space remain. 
CRITICAL 
May 30 2005 07:37:39 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:34:15 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 09:21:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 09:34:15 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01  ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 09:01:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 10:32:13 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 10:31:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 11:32:13 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 11:31:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 12:02:05 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 12:01:31 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 13:02:09 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 01:01:31 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 13:31:39 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 01:31:31 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 13:38:14 2005 -->
<pre>
Name : cdfonprd_tablespaces 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace MCS_INDEX is 88.43% full.  10718280.0 KBytes of free space remain. Tablespace MCS_DATA is 78.10% full.  4832984.0 KBytes of free space remain. 
CRITICAL 
May 30 2005 01:37:39 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 18:13:40 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01  ORA-01555 caused by SQL statement below (Query Duration=6117 sec, SCN: 0x056e.befb1509):. 
WARNING 
May 30 2005 05:21:31 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 19:31:37 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 07:31:31 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Mon May 30 20;01;53 comment by...ns  -->
<pre>many queries by   run_user tomcat on b0web01  </pre>
<!-- Date: Mon May 30 19:37:51 2005 -->
<pre>
Name : cdfonprd_tablespaces 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace USERTEMP_01 is 88.65% full.  929696.0 KBytes of free space remain. Tablespace MCS_INDEX is 88.43% full.  10718280.0 KBytes of free space remain. Tablespace MCS_DATA is 78.10% full.  4832984.0 KBytes of free space remain. 
CRITICAL 
May 30 2005 07:37:39 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 20:01:53 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 08:01:31 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Mon May 30 20:31:50 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 08:31:31 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Mon May 30 20:50:24 2005 -->
<pre>ISSUE:  several old client(s) accessing cdfstrm1. 
 
Alert   The following errors are in the ALERT file : ORA-12571: TNS:packet writer failure ORA-12571: TNS:packet writer failure.  
WARNING  May 30 2005 07:26:28 AM 
 
1. 
 
176733344 CDF_READER   cafuser5 cdfstrm1 cabinet-2-2-14.local TRGSim++@cabinet-2-2-14.local (TNS V1-V3) 16261     198.202.74.107 tcp 30-MAY-05 107  17858 
 
2. 
 
176733366 CDF_READER  cafuser cdfstrm1 cluster54.knu.ac.kr 
TRGSim++@cluster54.knu.ac.kr (TNS V1-V3) 
32265     155.230.20.84 tcp 30-MAY-05         87      38885 
 
3. 
 
176757680 CDF_READER   cafuser6 cdfstrm1 cabinet-3-3-10.local   TRGSim++@cabinet-3-3-10.local (TNS V1-V3) 11111     198.202.74.107 tcp 30-MAY-05        127        715</pre>
<!-- Author: ns  -->

<!-- Date: Mon May 30 21:01:39 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01  ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 09:01:31 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Mon May 30 22;00;41 comment by...ns  -->
<pre>many queries by   run_user tomcat on b0web01  </pre>
<!-- Comment: Mon May 30 22;01;16 comment by...ns -->
<pre>MANY MANY QUERIES    using up USERTEMP_01 on cdfonrpd
STARTING SUNDAY  May 29   at19:08
through NOW    --&gt;  not sure if queries are not tuned
but they are definitely asking for a large sorts which are
not completing?!

RUNUSER TOMACT on B0WEB01  ?   anyone know
what this activity is?


 --------- ------------------------------
RUN_USER     tomcat          b0web01                43   46896 USERTEMP_01
  SORT      SELECT PS.RC_RUNNUMBER, PS.PHY SICSTABLENAME, 
PS.LUMINOSITY,PS.CMUP_E, PS.CMX_E, RSO.RUNNUMBER, RC.RUNNUMBER FROM 
PHYSMON_SUMMARY PS, RUN_STATUS RSO,
RUNCONFIGURATIONS RC WHERE PS.RC_RUNNUMBER=RSO.RUNNUMBER ANDPS.DATATYPE=1 
AND PS.RC_RUNNUMBER&gt;=176000 AND PS.GOODRUN=1AND PS.LUMINOSITY&gt;=100 ORDER BY 
PS.RC_RUNNUMBER DESC


RUN_USER     tomcat          b0web01                82   13440 USERTEMP_01
  SORT      SELECT PS.*, RSO.RUNNUMBER, RSO.RUNCONTROL_STATUS, 
RSO.SHIFTCREW_STATUS, RSO.SVX_STATUS, RC.RUNNUMBER, RC.B0INTEGRATEDLIVELUMI 
FROM PHYSMON_SUMMARY PS , RUN_STATUS RSO, RUNCONFIGURATIONS RC WHERE 
PS.RC_RUNNUMBER=RSO.RUNNUMBER AND RC.RUNNUMBER=PS.RC_RUNNUMBER AND 
PS.DATATYPE=1 AND PS.RC_RUNNUMBER&gt;=0 AND PS.RC_RUNNUMBER&lt;=9999999 ORDER BY 
PS.RC_RUNNUMBER DESC

RUN_USER     tomcat          b0web01               101   37785 USERTEMP_01
  SORT      SELECT PS.RC_RUNNUMBER, PS.PHYSICSTABLENAME, 
PS.LUMINOSITY,PS.ND, PS.ND_ERR, RSO.RUNNUMBER, RC.RUNNUMBER, RSO.SVX_STATUS 
FROM PHYSMON_SUMMARY PS, RUN
_STATUS RSO, RUNCONFIGURATIONSRC WHERE PS.RC_RUNNUMBER=RSO.RUNNUMBER AND 
PS.DATATYPE=1 AND PS.RC_RUNNUMBER&gt;=176000 ANDPS.GOODRUN=1 AND PS.LUMINOSITY
 &gt;=400 ORDER BY PS.RC_RUNNUMBER   DESC

</pre>
<!-- Date: Mon May 30 21:19:37 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Alert 
The following errors are in the ALERT file : ORA-00600: internal error code, arguments: [kghGetHpSz1], [0xFFFFFFFF7C3C4070], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [0000000101DB136C] [SIGSEGV] [Address not mapped to object] [0x76007700784478] [] [] ORA-00600: internal error code, arguments: [kghGetHpSz1], [0xFFFFFFFF7C3C4070], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [0000000101DB136C] [SIGSEGV] [Address not mapped to object] [0x76007700784478] [] [] ORA-07445: exception encountered: core dump [0000000101DB136C] [SIGSEGV] [Address not mapped to object] [0x76007700784478] [] [] ORA-00600: internal error code, arguments: [kghGetHpSz1], [0xFFFFFFFF7C3C4070], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [0000000101DB136C] [SIGSEGV] [Address not mapped to object] [0x76007700784478] [] [] ORA-07445: exception encountered: core dump [0000000101DB136C] [SIGSEGV] [Address not mapped to object] [0x76007700784478] [] []
CRITICAL 
May 30 2005 09:18:35 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 21:22:21 2005 -->
<pre>
Name : cdfofpr2_apply_latency_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 2 
CLEARED 
May 30 2005 09:21:58 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 21:32:02 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 09:31:31 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 21:34:47 2005 -->
<pre>
Name : cdfofpr2_propagation_hung  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 0 
CLEARED 
May 30 2005 09:34:17 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 21:41:23 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 237023.0000000000000000000000000000000001 
CRITICAL 
May 30 2005 09:41:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 21:51:20 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 232899.9999999999999999999999999999999998 
CRITICAL 
May 30 2005 09:51:01 PM 



</pre>
<!-- Author: oem-admin (email) -->




<!-- Date: Mon May 30 22:03:04 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 10:01:31 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Mon May 30 22;21;54 comment by...ns -->
<pre>NOT SURE  what this activity is ...   does not affect data taking yet the queries are failing! 
 



RUN_USER     tomcat          b0web01                31   14592 USERTEMP_01 
  SORT      SELECT PS.RC_RUNNUMBER, PS.PHYSICSTABLENAME, PS.LUMINOSITY,PS.ND, PS.ND_ERR,
RSO.RUNNUMBER, RC.RUNNUMBER, RSO.SVX_STATUS FROM PHYSMON_SUMMARY PS, RUN_STATUS RSO,
RUNCONFIGURATIONSRC WHERE PS.RC_RUNNUMBER=RSO.RUNNUMBER AND PS.DATATYPE=1 AND
PS.RC_RUNNUMBER&gt;=176000 ANDPS.GOODRUN=1 AND PS.LUMINOSITY &gt;=400 ORDER BY PS.RC_RUNNUMBERDESC 


RUN_USER     tomcat          b0web01                47   11125 USERTEMP_01 
  SORT      SELECT PS.RC_RUNNUMBER, PS.PHYSICSTABLENAME, PS.LUMINOSITY, PS.ND, PS.ND_ERR,
RSO.RUNNUMBER, RC.RUNNUMBER, RSO.SVX_STATUS FROM PHYSMON_SUMMARY PS, RUN_STATUS RSO,
RUNCONFIGURATIONSRC WHERE PS.RC_RUNNUMBER=RSO.RUNNUMBER AND PS.DATATYPE=1 AND
PS.RC_RUNNUMBER&gt;=176000 ANDPS.GOODRUN=1 AND PS.LUMINOSITY&gt;=400 ORDER BY PS.RC_RUNNUMBER 
</pre>
<!-- Date: Mon May 30 22:11:22 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value:  
CLEARED 
May 30 2005 10:11:01 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Mon May 30 22:21:26 2005 -->
<pre>NOT SURE  what this activity is ...   does not affect data taking yet the queries are failing!   
 
 
RUN_USER     tomcat          b0web01                31   14592 USERTEMP_01 
  SORT      SELECT PS.RC_RUNNUMBER, PS.PHYSICSTABLENAME, PS.LUMINOSITY,PS.ND, PS.ND_ERR, RSO.RUNNUMBER, RC.RUNNUMBER, RSO.SVX_STATUS FROM PHYSMON_SUMMARY PS, RUN_STATUS RSO, RUNCONFIGURATIONSRC WHERE PS.RC_RUNNUMBER=RSO.RUNNUMBER AND PS.DATATYPE=1 AND PS.RC_RUNNUMBER&gt;=176000 ANDPS.GOODRUN=1 AND PS.LUMINOSITY &gt;=400 ORDER BY PS.RC_RUNNUMBERDESC 
 
RUN_USER     tomcat          b0web01                47   11125 USERTEMP_01 
  SORT      SELECT PS.RC_RUNNUMBER, PS.PHYSICSTABLENAME, PS.LUMINOSITY, PS.ND, PS.ND_ERR, RSO.RUNNUMBER, RC.RUNNUMBER, RSO.SVX_STATUS FROM PHYSMON_SUMMARY PS, RUN_STATUS RSO, RUNCONFIGURATIONSRC WHERE PS.RC_RUNNUMBER=RSO.RUNNUMBER AND PS.DATATYPE=1 AND PS.RC_RUNNUMBER&gt;=176000 ANDPS.GOODRUN=1 AND PS.LUMINOSITY&gt;=400 ORDER BY PS.RC_RUNNUMBER 
</pre>
<!-- Author: ns  -->


<!-- Date: Mon May 30 22:27:28 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 219480.9999999999999999999999999999999998 
CRITICAL 
May 30 2005 10:26:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 22:31:37 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01  ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01  ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 10:31:31 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 22:36:12 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 217700.0000000000000000000000000000000004 
CRITICAL 
May 30 2005 10:36:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 23:03:07 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01  ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 11:01:31 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 23:17:36 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 206429.9999999999999999999999999999999999 
CRITICAL 
May 30 2005 11:16:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 23:26:08 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 204412.9999999999999999999999999999999997 
CRITICAL 
May 30 2005 11:26:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 23:32:21 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-1652: unable to extend temp segment by 321 in tablespace                 USERTEMP_01. 
WARNING 
May 30 2005 11:31:32 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon May 30 23:52:56 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 198678.0000000000000000000000000000000001 
CRITICAL 
May 30 2005 11:51:01 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 00:04:03 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
May 31 2005 12:01:31 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 00:04:03 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
May 31 2005 12:03:35 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 00:31:12 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 225 
CRITICAL 
May 31 2005 12:29:10 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 00:37:25 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 192618 
CRITICAL 
May 31 2005 12:36:01 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 00:39:29 2005 -->
<pre>
Name : cdfonprd_tablespaces 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

Tablespace Full 
Tablespace MCS_INDEX is 88.43% full.  10718280.0 KBytes of free space remain. Tablespace MCS_DATA is 78.10% full.  4832984.0 KBytes of free space remain. 
CRITICAL 
May 31 2005 12:37:39 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 01:00:11 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 1 
CLEARED 
May 31 2005 12:59:10 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 01:36:10 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 186705 
CRITICAL 
May 31 2005 01:36:01 AM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Tue May 31 01:37:11 2005 -->
<pre>Switched the default db user of tomcat@b0web01 to  
cdf_reader, instead of run_user; cdf_reader should have  
no write privs, as far as i know.</pre>
<!-- Author: W.Badgett -->

<!-- Date: Tue May 31 02:14:42 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value:  
CLEARED 
May 31 2005 02:06:01 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 02:14:42 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 567 
CRITICAL 
May 31 2005 02:14:14 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 03:25:21 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: 23 
CLEARED 
May 31 2005 02:59:12 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 03:58:54 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Alert 
The following errors are in the ALERT file : ORA-00600: internal error code, arguments: [koiofre970], [], [], [], [], [], [], [] ORA-04030: out of process memory when trying to allocate 16408 bytes (koh-kghu sessi,koicnfadt adt attr) ORA-07445: exception encountered: core dump [0000000101C464FC] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFE0] [] [] ORA-00600: internal error code, arguments: [koiofre970], [], [], [], [], [], [], [] ORA-04030: out of process memory when trying to allocate 16408 bytes (koh-kghu sessi,koicnfadt adt attr). 
CRITICAL 
May 31 2005 03:33:36 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 03:58:54 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 169545.0000000000000000000000000000000001 
CRITICAL 
May 31 2005 03:51:01 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 04:07:14 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 161069.0000000000000000000000000000000003 
CRITICAL 
May 31 2005 04:06:01 AM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Tue May 31 04:23:57 2005 -->
<pre>At 2005.05.29 11:12:28, one of the CDF online  
java programs that talks to cdfonprd gave an error: 
 
2005.05.29 11:12:28 java.sql.SQLException: Io exception: Connection refused(DESCRIPTION=(TMP=)(VSNNUM=153093632)(ERR=12500)(ERROR_S\ 
TACK=(ERROR=(CODE=12500)(EMFI=4))(ERROR=(CODE=12540)(EMFI=4))(ERROR=(CODE=12560)(EMFI=4))(ERROR=(CODE=510)(EMFI=4))(ERROR=(BUF='Sol\ 
aris Error: 12: Not enough space')))) 
 
Any ideas why this might have happened?   I notice above  
that at 2005.05.29 11:22 there was some sort of hardware  
error reported from bzora1 -- perhaps serious? 
</pre>
<!-- Author: W.Badgett -->
<!-- Comment: Tue May 31 04;25;32 comment by...WB -->
<pre>Note that the application error seems to have cleared 
up by itself within a minute or so after the error.
</pre>

<!-- Comment: Tue May 31 07;40;41 comment by...ns  -->
<pre>swap space had filled up   - no issue with hardware...as you state it was immediately fixed  -- we were preparing a tarball of the past 4 days of archive logs </pre>
<!-- Date: Tue May 31 04:27:38 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value:  
CLEARED 
May 31 2005 04:26:01 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 04:38:18 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 149236.9999999999999999999999999999999998 
CRITICAL 
May 31 2005 04:36:01 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 05:06:54 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 143328.9999999999999999999999999999999997 
CRITICAL 
May 31 2005 05:06:02 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 05:36:32 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 134769.9999999999999999999999999999999997 
CRITICAL 
May 31 2005 05:36:01 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 06:03:00 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 132717.9999999999999999999999999999999997 
CRITICAL 
May 31 2005 06:01:01 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 06:42:28 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 124973.9999999999999999999999999999999997 
CRITICAL 
May 31 2005 06:41:01 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 07:01:26 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 120160.9999999999999999999999999999999996 
CRITICAL 
May 31 2005 07:01:01 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 07:22:43 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 105734.9999999999999999999999999999999998 
CRITICAL 
May 31 2005 07:21:01 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 07:30:59 2005 -->
<pre>
Name : cdfofpr2_capture_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 829 
CRITICAL 
May 31 2005 07:29:11 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue May 31 07:30:59 2005 -->
<pre>
Name : cdfofpr2_apply_msg_spill_alert 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 1302 
CRITICAL 
May 31 2005 07:30:38 AM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Tue May 31 07:41:10 2005 -->
<pre>
Name : cdfofpr2_capture_latency_alert  
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

User Defined SQL Test 
Current Value: 100221.9999999999999999999999999999999998 
CRITICAL 
May 31 2005 07:41:01 AM 



</pre>
<!-- Author: oem-admin (email) -->