<!-- Title: 21:54:19  Thu Dec 28 2006 -->
<!-- Crewchief: New day -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>database activity</pre>


<!-- Date: Thu Dec 28 21:55:56 2006 -->
<pre>db bounced.   streaming had halted at 11:17   due to long transactions on cdfofpr2....some detail............ 
 
started and stopped the capture with no luck...so we did a bounce........... 
 
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 
replication catching up  should be up-to-date in 1hr 15 minutes or so 
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 
 
Kwangzoo, 
 
Sounds like you have several tasks going on and we haven't defined a  
reasonable plan for these.     The deletes have the same issue with the  
commits as do updates & inserts so they should be treated the same.  
Replication did result in a problem today (replication is now sync'g up  
since I bounced the db)  but  replication had an issue with your long  
transactions.  It was detecting long transactions txn but it was NOT seeing  
any commits, and there was not enough memory to hold the deletes. 
 
When I informed you about your long transactions this a.m. you did mention  
it was "a different type of thing" so I guess at that point I should of  
gotten more information.    Yes, the rollback creates similar activity  
altho' it still SEEMS you had no commits for the deletes? or am I wrong?  
and I/oracle just missed the commits?  The rollback will only affect those  
rows not yet committed, so if you were committing every 1000 rows only 1000  
rows would of been rolled back. 
 
I know you were working with Anil to define a plan but he is away until Jan  
18.   You need to do any single test and all tests on development or  
integration, let's discuss this further next week.  Also, Anil had mentioned  
that you would be doing some activity and that you discussed how often to  
commit and that if at any point replication would get behind you would  
simply hold off on any further activity until replication caught up.  We can  
work out a more concrete schedule where you run your scripts for 2 hours,  
then halt for 4 hours etc etc.  in an effort to not halt replication and get  
your tasks done. 
 
Thanks 
Nelly 
 
 
 
----- Original Message -----  
From: "K. Chung" &lt;kchung@cmu.edu&gt; 
To: "Nelly Stanfield" &lt;nelly@fnal.gov&gt; 
Cc: "Randolph J. Herber" &lt;herber@fnal.gov&gt;; "Jonathan D. Lewis"  
&lt;jdl@fnal.gov&gt;; &lt;cdfdb-support@fnal.gov&gt;; &lt;akumar@imapserver2.fnal.gov&gt;;  
"jtrumbo" &lt;jtrumbo@fnal.gov&gt; 
Sent: Thursday, December 28, 2006 6:09 PM 
Subject: Re: EM Alert: Critical: cdfofpr2.fnal.gov - UDM alert: key = %Key,  
value = 2 
 
 
&gt; Dear Nelly, 
&gt; 
&gt; Thank you for your support and help. 
&gt; - It was unfortunate that the rollback which you recommended made it even  
&gt; worse - 
&gt; 
&gt; This was planned as a small test but resulted in a big trouble. 
&gt; It was only for 1~2 inverse pb of data and I definitely overestimate the  
&gt; db performance, sorry. 
&gt; (FYI: the final goal is to deal with about thousand times bigger, i.e., 1  
&gt; fb^-1) 
&gt; 
&gt; There are things that I want to make clear: 
&gt; This deletion was a prep for the luminosity update. And I wasn't aware of  
&gt; a committing issue for this. 
&gt; The luminosity update process is surely committing quite frequently, which  
&gt; I couldn't get to run today. 
&gt; I think today's trouble was mostly due to replication rather than a  
&gt; committing issue. Two different things. 
&gt; 
&gt; I'll stop any single test until a solid and wise solution comes with  
&gt; agreements from DB group. 
&gt; 
&gt; Regards, 
&gt; Kwangzoo 
&gt; 
&gt; Nelly Stanfield wrote: 
&gt;&gt; Investigating the problem further I see the long transactions & your  
&gt;&gt; rollback. Some details below. I know you've worked with Anil in trying to  
&gt;&gt; keep the number of commits often but it's clear you need to commit more  
&gt;&gt; often.   Also, we should schedule this maintenance work, so that you run  
&gt;&gt; for a few hours and we ensure replication is caught up and then you can  
&gt;&gt; start again.   I thought this was the arrangement you had with Anil and  
&gt;&gt; it seems we are going to have to be even more conservative.   I don't see  
&gt;&gt; any commits below are you sure you are committing!!    I expect to see  
&gt;&gt; something like " C001: large txn committed, xid: 0x0001.00d.0041bb2c" 
&gt;&gt; 
&gt;&gt; PLEASE do not do any further deletes/maintenance work.  Replication is  
&gt;&gt; behind and it will not get restarted until we bounce the cdfonprd db.  I  
&gt;&gt; hope to get called/page  this evening allowing us a  5 minute  
&gt;&gt; interruption to cdfonline.      I will send additional information when  
&gt;&gt; the bounce is done & follow-up further once replication is complete. 
&gt;&gt; 
&gt;&gt; -Nelly 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; Details: 
&gt;&gt; 
&gt;&gt; cdfofprd2 
&gt;&gt; 
&gt;&gt; C001: large txn detected (465890 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 10:41:15 2006 
&gt;&gt; C001: large txn detected (1139885 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 10:51:14 2006 
&gt;&gt; C001: large txn detected (1476369 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 11:01:20 2006 
&gt;&gt; C001: large txn detected (1476369 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 11:11:33 2006 
&gt;&gt; C001: large txn detected (1690391 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 11:21:26 2006 
&gt;&gt; C001: large txn detected (1959563 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 11:31:36 2006 
&gt;&gt; C001: large txn detected (2228265 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 11:41:25 2006 
&gt;&gt; C001: large txn detected (2503718 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 11:51:31 2006 
&gt;&gt; C001: large txn detected (2699621 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 12:01:33 2006 
&gt;&gt; C001: large txn detected (2766358 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 12:11:38 2006 
&gt;&gt; C001: large txn detected (2766358 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 12:23:35 2006 
&gt;&gt; C001: large txn detected (2766358 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 12:31:41 2006 
&gt;&gt; C001: large txn detected (2766358 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 12:41:38 2006 
&gt;&gt; 
&gt;&gt; C001: large txn detected (2766358 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 12:51:35 2006 
&gt;&gt; C001: large txn detected (2766358 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 13:01:31 2006 
&gt;&gt; C001: large txn detected (2766358 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 13:11:31 2006 
&gt;&gt; 
&gt;&gt; C001: large txn detected (2766358 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 13:21:43 2006 
&gt;&gt; 
&gt;&gt; C001: large txn detected (2766358 LCRs), xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 13:31:32 2006 
&gt;&gt; 
&gt;&gt; C001: large txn rolled back, xid: 0x0006.01d.00139c61 
&gt;&gt; Thu Dec 28 13:37:42 2006 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; Long Running SQLs : 
&gt;&gt; 
&gt;&gt; OSUSER          DB User         SID SERIAL# PROGRAM            SYSTEM 
&gt;&gt; SQL Text                                           START_TIME 
&gt;&gt; --------------- ------------ ------ ------- ------------------  
&gt;&gt; ------------------ 
&gt;&gt; -------------------------------------------------- -------------------- 
&gt;&gt; Query Time 
&gt;&gt; ---------- 
&gt;&gt; kchung          KCHUNG          467   34205 sqlplus@ncmu03.fna 
&gt;&gt; ncmu03.fnal.gov    delete from filecatalog.cdf2_runsection_livetimes 
&gt;&gt; 12/28/06 10:34:31 
&gt;&gt;                                            l.gov (TNS V1-V3) 
&gt;&gt; where run_number&gt;139177 and run_number&lt;14015538.55 
&gt;&gt; 
&gt;&gt; kchung          KCHUNG          467   34205 sqlplus@ncmu03.fna 
&gt;&gt; ncmu03.fnal.gov    delete from filecatalog.cdf2_runsection_livetimes 
&gt;&gt; 12/28/06 10:34:31 
&gt;&gt;                                            l.gov (TNS V1-V3) 
&gt;&gt; where run_number&gt;140150 and run_number&lt;14103957.5166667 
&gt;&gt; 
&gt;&gt;&gt;&gt; Query Time 
&gt;&gt;&gt;&gt; ---------- 
&gt;&gt;&gt;&gt; kchung          KCHUNG          467   34205 sqlplus@ncmu03.fna  
&gt;&gt;&gt;&gt; ncmu03.fnal.gov    delete from filecatalog.cdf2_runsection_livetimes  
&gt;&gt;&gt;&gt; 12/28/06 10:34:31 
&gt;&gt;&gt;&gt;                                            l.gov (TNS V1-V3) where  
&gt;&gt;&gt;&gt; run_number&gt;140150 and run_number&lt;14103957.5166667 
&gt;&gt;&gt;&gt; 
&gt;&gt; 
&gt;&gt; ----- Original Message ----- From: "K. Chung" &lt;kchung@cmu.edu&gt; 
&gt;&gt; To: "Nelly Stanfield" &lt;nelly@fnal.gov&gt; 
&gt;&gt; Cc: "Randolph J. Herber" &lt;herber@fnal.gov&gt;; "Jonathan D. Lewis"  
&gt;&gt; &lt;jdl@fnal.gov&gt;; &lt;cdfdb-support@fnal.gov&gt; 
&gt;&gt; Sent: Thursday, December 28, 2006 2:01 PM 
&gt;&gt; Subject: Re: EM Alert: Critical: cdfofpr2.fnal.gov - UDM alert: key =  
&gt;&gt; %Key, value = 2 
&gt;&gt; 
&gt;&gt; 
&gt;&gt;&gt; I just issued a "rollback" and looks the deleted rows are back. 
&gt;&gt;&gt; Hope this help to solve the problem. 
&gt;&gt;&gt; 
&gt;&gt;&gt; 
&gt;&gt;&gt; Nelly Stanfield wrote: 
&gt;&gt;&gt;&gt; Replication is falling behind on cdfstrm1 & cdfofpr2.   Data is  
&gt;&gt;&gt;&gt; replicated up to 11:17 a.m.   There are no more long transactions that  
&gt;&gt;&gt;&gt; I can see but the activity is too 'high volume' with replication in  
&gt;&gt;&gt;&gt; place. Can you dial back your deletes.    You can see the status of  
&gt;&gt;&gt;&gt; replication at the following url: 
&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt; http://dbb.fnal.gov:8520/cdfr2/databases?type=ora-strms&fsrc=cdfofpr2&nsrc=cdfofpr2&gsrc=cdfofpr2&dcbk=FILECATALOG 
&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt; Normal activity support ~500 archives per day.... we are already past  
&gt;&gt;&gt;&gt; that. We are over 2 hours behind and replication needs to play catchup.  
&gt;&gt;&gt;&gt; Once there is a spillage it takes longer to catch up. 
&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt; fcdfora4&gt; ls -lrt | grep 'Dec 20' | wc -l 
&gt;&gt;&gt;&gt;     410 
&gt;&gt;&gt;&gt; fcdfora4&gt; ls -lrt | grep 'Dec 21' | wc -l 
&gt;&gt;&gt;&gt;     252 
&gt;&gt;&gt;&gt; fcdfora4&gt; ls -lrt | grep 'Dec 22' | wc -l 
&gt;&gt;&gt;&gt;     499 
&gt;&gt;&gt;&gt; fcdfora4&gt; ls -lrt | grep 'Dec 23' | wc -l 
&gt;&gt;&gt;&gt;     518 
&gt;&gt;&gt;&gt; fcdfora4&gt; ls -lrt | grep 'Dec 24' | wc -l 
&gt;&gt;&gt;&gt;     404 
&gt;&gt;&gt;&gt; fcdfora4&gt; ls -lrt | grep 'Dec 25' | wc -l 
&gt;&gt;&gt;&gt;     459 
&gt;&gt;&gt;&gt; fcdfora4&gt; ls -lrt | grep 'Dec 26' | wc -l 
&gt;&gt;&gt;&gt;     465 
&gt;&gt;&gt;&gt; fcdfora4&gt; ls -lrt | grep 'Dec 27' | wc -l 
&gt;&gt;&gt;&gt;     485 
&gt;&gt;&gt;&gt; fcdfora4&gt; ls -lrt | grep 'Dec 28' | wc -l 
&gt;&gt;&gt;&gt;     519 
&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt; Stream's Heartbeat 
&gt;&gt;&gt;&gt;      Description Updated 
&gt;&gt;&gt;&gt;      (text) (date&time) 
&gt;&gt;&gt;&gt;      STREAM STATUS 2006-12-28 11:17:01.0 
&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt; ----- Original Message ----- From: "K. Chung" &lt;kchung@cmu.edu&gt; 
&gt;&gt;&gt;&gt; To: "Nelly Stanfield" &lt;nelly@fnal.gov&gt; 
&gt;&gt;&gt;&gt; Cc: &lt;oem-admin@fnal.gov&gt;; "Randolph J. Herber" &lt;herber@fnal.gov&gt;;  
&gt;&gt;&gt;&gt; "Jonathan D. Lewis" &lt;jdl@fnal.gov&gt; 
&gt;&gt;&gt;&gt; Sent: Thursday, December 28, 2006 11:51 AM 
&gt;&gt;&gt;&gt; Subject: Re: EM Alert: Critical: cdfofpr2.fnal.gov - UDM alert: key =  
&gt;&gt;&gt;&gt; %Key, value = 2 
&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt; Thank you for the clarification. 
&gt;&gt;&gt;&gt;&gt; I'll try to avoid an expensive transaction but 
&gt;&gt;&gt;&gt;&gt; considering the limit, I'm afraid that more alerts would be caused.. 
&gt;&gt;&gt;&gt;&gt; - This is not my personal project but for the whole CDF collaboration  
&gt;&gt;&gt;&gt;&gt; and the size is unavoidably huge. - 
&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt; One question: So, there is a possibility that the alert could be  
&gt;&gt;&gt;&gt;&gt; fooled by slow connection 
&gt;&gt;&gt;&gt;&gt;                       even though there is not so many transactions,  
&gt;&gt;&gt;&gt;&gt; right? 
&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt; Best, 
&gt;&gt;&gt;&gt;&gt; Kwangzoo 
&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt; Nelly Stanfield wrote: 
&gt;&gt;&gt;&gt;&gt;&gt; Hi, 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; Yes,  any connection that is active longer then 5 mintues we get  
&gt;&gt;&gt;&gt;&gt;&gt; alerted on. 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; fcdfora4&gt; runscript cdfofpr2 show_long_transaction.sql 
&gt;&gt;&gt;&gt;&gt;&gt; Thu Dec 28 11:13:04 CST 2006 
&gt;&gt;&gt;&gt;&gt;&gt; Information for: cdfofpr2 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; Long Running SQLs : 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; OSUSER          DB User         SID SERIAL# PROGRAM            SYSTEM  
&gt;&gt;&gt;&gt;&gt;&gt; SQL Text                                           START_TIME 
&gt;&gt;&gt;&gt;&gt;&gt; --------------- ------------ ------ ------- ------------------  
&gt;&gt;&gt;&gt;&gt;&gt; ------------------  
&gt;&gt;&gt;&gt;&gt;&gt; --------------------------------------------------  
&gt;&gt;&gt;&gt;&gt;&gt; -------------------- 
&gt;&gt;&gt;&gt;&gt;&gt; Query Time 
&gt;&gt;&gt;&gt;&gt;&gt; ---------- 
&gt;&gt;&gt;&gt;&gt;&gt; kchung          KCHUNG          467   34205 sqlplus@ncmu03.fna  
&gt;&gt;&gt;&gt;&gt;&gt; ncmu03.fnal.gov    delete from filecatalog.cdf2_runsection_livetimes  
&gt;&gt;&gt;&gt;&gt;&gt; 12/28/06 10:34:31 
&gt;&gt;&gt;&gt;&gt;&gt;                                            l.gov (TNS V1-V3) where  
&gt;&gt;&gt;&gt;&gt;&gt; run_number&gt;139177 and run_number&lt;140155 
&gt;&gt;&gt;&gt;&gt;&gt;     38.55 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; OSUSER          DB User         SID SERIAL# PROGRAM            SYSTEM  
&gt;&gt;&gt;&gt;&gt;&gt; SQL Text                                           START_TIME 
&gt;&gt;&gt;&gt;&gt;&gt; --------------- ------------ ------ ------- ------------------  
&gt;&gt;&gt;&gt;&gt;&gt; ------------------  
&gt;&gt;&gt;&gt;&gt;&gt; --------------------------------------------------  
&gt;&gt;&gt;&gt;&gt;&gt; -------------------- 
&gt;&gt;&gt;&gt;&gt;&gt; Query Time 
&gt;&gt;&gt;&gt;&gt;&gt; ---------- 
&gt;&gt;&gt;&gt;&gt;&gt; kchung          KCHUNG          467   34205 sqlplus@ncmu03.fna  
&gt;&gt;&gt;&gt;&gt;&gt; ncmu03.fnal.gov    delete from filecatalog.cdf2_runsection_livetimes  
&gt;&gt;&gt;&gt;&gt;&gt; 12/28/06 10:34:31 
&gt;&gt;&gt;&gt;&gt;&gt;                                            l.gov (TNS V1-V3) where  
&gt;&gt;&gt;&gt;&gt;&gt; run_number&gt;140150 and run_number&lt;141039 
&gt;&gt;&gt;&gt;&gt;&gt; 57.5166667 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; ----- Original Message ----- From: "K. Chung" &lt;kchung@cmu.edu&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; To: "Nelly Stanfield" &lt;nelly@fnal.gov&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; Cc: &lt;oem-admin@fnal.gov&gt;; "Randolph J. Herber" &lt;herber@fnal.gov&gt;;  
&gt;&gt;&gt;&gt;&gt;&gt; "Jonathan D. Lewis" &lt;jdl@fnal.gov&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; Sent: Thursday, December 28, 2006 11:23 AM 
&gt;&gt;&gt;&gt;&gt;&gt; Subject: Re: EM Alert: Critical: cdfofpr2.fnal.gov - UDM alert: key =  
&gt;&gt;&gt;&gt;&gt;&gt; %Key, value = 2 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt; This was a different type of thing. 
&gt;&gt;&gt;&gt;&gt;&gt;&gt; If you can tell me about the criteria for "long transaction", 
&gt;&gt;&gt;&gt;&gt;&gt;&gt; it would be very helpful to plan ahead saving future alerts. 
&gt;&gt;&gt;&gt;&gt;&gt;&gt; Thank you, 
&gt;&gt;&gt;&gt;&gt;&gt;&gt; Kwangzoo 
&gt;&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt; Nelly Stanfield wrote: 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; Looks like your activity has resumed.........Pleaes ensure you are  
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; committing often! Long Running SQLs : 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; OSUSER          DB User         SID SERIAL# PROGRAM  
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; SYSTEM SQL Text  
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; START_TIME 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; --------------- ------------ ------ ------- ------------------  
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; ------------------  
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; --------------------------------------------------  
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; -------------------- 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; Query Time 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; ---------- 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; kchung          KCHUNG          467   34205 sqlplus@ncmu03.fna  
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; &lt;mailto:sqlplus@ncmu03.fna&gt; ncmu03.fnal.gov    delete from  
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; filecatalog.cdf2_runsection_livetimes  12/28/06 10:34:31 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;                                             l.gov (TNS V1-V3) where  
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; run_number&gt;139177 and run_number&lt;140155 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; 36.3333333 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     ----- Original Message ----- 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     *From:* Enterprise Manager &lt;mailto:oem-admin@fnal.gov&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     *To:* oem-admin@fnal.gov &lt;mailto:oem-admin@fnal.gov&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     *Sent:* Thursday, December 28, 2006 10:52 AM 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     *Subject:* EM Alert: Critical: cdfofpr2.fnal.gov - UDM alert:  
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; key 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     = %Key, value = 2 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     Target Name=*cdfofpr2.fnal.gov* 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; &lt;https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=cdfofpr2.fnal.gov&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     Target Type=*Database Instance* 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     Host=*fcdfora4.fnal.gov* 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     Metric=*User-Defined Numeric Metric* 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; &lt;https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=cdfofpr2.fnal.gov&metricColumn=NumValue&metric=SQLUDM&keyValue=long_transaction_count&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     Metric Value=*2* 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     Script=*long_transaction_count* 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     Timestamp=*Dec 28, 2006 10:50:17 AM CST* 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     Severity=*Critical* 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     Message=*UDM alert: key = %Key, value = 2* 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     Notification Rule Name=*fcdfora4 database alert* 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;     Notification Rule Owner=*SYSMAN* 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt; --  
&gt;&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt;  Kwangzoo Chung  -  CDF TR167K, (630)840-4543 
&gt;&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt; --  
&gt;&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt;&gt;  Kwangzoo Chung  -  CDF TR167K, (630)840-4543 
&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt; 
&gt;&gt;&gt;&gt; 
&gt;&gt;&gt; 
&gt;&gt;&gt; 
&gt;&gt;&gt; --  
&gt;&gt;&gt; 
&gt;&gt;&gt;  Kwangzoo Chung  -  CDF TR167K, (630)840-4543 
&gt;&gt;&gt; 
&gt;&gt;&gt; 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; 
&gt; 
&gt; 
&gt; --  
&gt; 
&gt;  Kwangzoo Chung  -  CDF TR167K, (630)840-4543  
</pre>
<!-- Author: nstanfield  -->

<!-- Date: Thu Dec 28 22:15:27 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFONPRD.FNAL.GOV"><b>CDFONPRD.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>bzora1.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFONPRD.FNAL.GOV&metricColumn=genericErrors&metric=alertLogStatus&keyValue="><b>Generic Alert Log Error Status</b></a><br>
Metric Value=<b>1</b><br>
Timestamp=<b>Dec 28, 2006 10:14:44 PM CST</b><br>
Severity=<b>Warning</b><br>
Message=<b>1 distinct types of ORA- errors have been found in the alert log.</b><br>
Notification Rule Name=<b>bzora1 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Dec 28 22:31:08 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFONPRD.FNAL.GOV"><b>CDFONPRD.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>bzora1.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFONPRD.FNAL.GOV&metricColumn=genericErrors&metric=alertLogStatus&keyValue="><b>Generic Alert Log Error Status</b></a><br>
Metric Value=<b>0</b><br>
Timestamp=<b>Dec 28, 2006 10:29:44 PM CST</b><br>
Severity=<b>Clear</b><br>
Message=<b>0 distinct types of ORA- errors have been found in the alert log.</b><br>
Notification Rule Name=<b>bzora1 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Dec 28 22:58:36 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=cdfofpr2.fnal.gov"><b>cdfofpr2.fnal.gov</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora4.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=cdfofpr2.fnal.gov&metricColumn=NumValue&metric=SQLUDM&keyValue=stream_heartbeat"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>12222</b><br>
Script=<b>stream_heartbeat</b><br>
Timestamp=<b>Dec 28, 2006 10:57:42 PM CST</b><br>
Severity=<b>Warning</b><br>
Message=<b>UDM alert: key = %Key, value = 12222</b><br>
Notification Rule Name=<b>fcdfora4 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Dec 28 23:00:07 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFSTRM1.FNAL.GOV"><b>CDFSTRM1.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora6.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFSTRM1.FNAL.GOV&metricColumn=NumValue&metric=SQLUDM&keyValue=stream_heartbeat"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>12144</b><br>
Script=<b>stream_heartbeat</b><br>
Timestamp=<b>Dec 28, 2006 10:58:24 PM CST</b><br>
Severity=<b>Warning</b><br>
Message=<b>UDM alert: key = %Key, value = 12144</b><br>
Notification Rule Name=<b>fcdfora6 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Dec 28 23:29:09 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=cdfofpr2.fnal.gov"><b>cdfofpr2.fnal.gov</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora4.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=cdfofpr2.fnal.gov&metricColumn=NumValue&metric=SQLUDM&keyValue=stream_heartbeat"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>6402</b><br>
Script=<b>stream_heartbeat</b><br>
Timestamp=<b>Dec 28, 2006 11:27:42 PM CST</b><br>
Severity=<b>Clear</b><br>
Message=<b>CLEARED - UDM alert: key = %Key, value = 6402</b><br>
Notification Rule Name=<b>fcdfora4 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Dec 28 23:29:09 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFSTRM1.FNAL.GOV"><b>CDFSTRM1.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora6.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFSTRM1.FNAL.GOV&metricColumn=NumValue&metric=SQLUDM&keyValue=stream_heartbeat"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>6444</b><br>
Script=<b>stream_heartbeat</b><br>
Timestamp=<b>Dec 28, 2006 11:28:24 PM CST</b><br>
Severity=<b>Clear</b><br>
Message=<b>CLEARED - UDM alert: key = %Key, value = 6444</b><br>
Notification Rule Name=<b>fcdfora6 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Dec 28 23:29:09 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFONPRD.FNAL.GOV"><b>CDFONPRD.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>bzora1.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFONPRD.FNAL.GOV&metricColumn=NumValue&metric=SQLUDM&keyValue=capture_msg_spill"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>1017</b><br>
Script=<b>capture_msg_spill</b><br>
Timestamp=<b>Dec 28, 2006 11:28:13 PM CST</b><br>
Severity=<b>Critical</b><br>
Message=<b>UDM alert: key = %Key, value = 1017</b><br>
Notification Rule Name=<b>bzora1 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Dec 28 23:37:25 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=cdfofpr2.fnal.gov"><b>cdfofpr2.fnal.gov</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora4.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=cdfofpr2.fnal.gov&metricColumn=Status&metric=Response&keyValue="><b>Status</b></a><br>
Metric Value=<b>0</b><br>
Timestamp=<b>Dec 28, 2006 11:34:42 PM CST</b><br>
Severity=<b>Critical</b><br>
Message=<b>Failed to connect to database instance: ORA-03135: connection lost contact (DBD ERROR: OCISessionBegin).</b><br>
Notification Rule Name=<b>fcdfora4 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Dec 28 23:41:20 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=cdfofpr2.fnal.gov"><b>cdfofpr2.fnal.gov</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora4.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=cdfofpr2.fnal.gov&metricColumn=Status&metric=Response&keyValue="><b>Status</b></a><br>
Metric Value=<b>1</b><br>
Timestamp=<b>Dec 28, 2006 11:39:42 PM CST</b><br>
Severity=<b>Clear</b><br>
Message=<b>Able to connect to database instance</b><br>
Notification Rule Name=<b>fcdfora4 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Dec 28 23:43:56 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFONPRD.FNAL.GOV"><b>CDFONPRD.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>bzora1.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFONPRD.FNAL.GOV&metricColumn=NumValue&metric=SQLUDM&keyValue=capture_latency"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>788</b><br>
Script=<b>capture_latency</b><br>
Timestamp=<b>Dec 28, 2006 11:43:13 PM CST</b><br>
Severity=<b>Warning</b><br>
Message=<b>UDM alert: key = %Key, value = 788</b><br>
Notification Rule Name=<b>bzora1 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Dec 28 23:43:56 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFONPRD.FNAL.GOV"><b>CDFONPRD.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>bzora1.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFONPRD.FNAL.GOV&metricColumn=NumValue&metric=SQLUDM&keyValue=capture_msg_spill"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>0</b><br>
Script=<b>capture_msg_spill</b><br>
Timestamp=<b>Dec 28, 2006 11:43:13 PM CST</b><br>
Severity=<b>Clear</b><br>
Message=<b>CLEARED - UDM alert: key = %Key, value = 0</b><br>
Notification Rule Name=<b>bzora1 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Dec 29 00:00:13 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFONPRD.FNAL.GOV"><b>CDFONPRD.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>bzora1.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFONPRD.FNAL.GOV&metricColumn=NumValue&metric=SQLUDM&keyValue=capture_latency"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>0</b><br>
Script=<b>capture_latency</b><br>
Timestamp=<b>Dec 28, 2006 11:58:13 PM CST</b><br>
Severity=<b>Clear</b><br>
Message=<b>CLEARED - UDM alert: key = %Key, value = 0</b><br>
Notification Rule Name=<b>bzora1 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Dec 29 00:00:13 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=cdfofpr2.fnal.gov"><b>cdfofpr2.fnal.gov</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora4.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=cdfofpr2.fnal.gov&metricColumn=NumValue&metric=SQLUDM&keyValue=apply_latency"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>1</b><br>
Script=<b>apply_latency</b><br>
Timestamp=<b>Dec 28, 2006 11:59:16 PM CST</b><br>
Severity=<b>Clear</b><br>
Message=<b>CLEARED - UDM alert: key = %Key, value = 1</b><br>
Notification Rule Name=<b>fcdfora4 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Dec 29 00:54:37 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=cdfofpr2.fnal.gov"><b>cdfofpr2.fnal.gov</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora4.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=cdfofpr2.fnal.gov&metricColumn=genericErrors&metric=alertLogStatus&keyValue="><b>Generic Alert Log Error Status</b></a><br>
Metric Value=<b>1</b><br>
Timestamp=<b>Dec 29, 2006 12:53:29 AM CST</b><br>
Severity=<b>Warning</b><br>
Message=<b>1 distinct types of ORA- errors have been found in the alert log.</b><br>
Notification Rule Name=<b>fcdfora4 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Dec 29 01:09:09 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=cdfofpr2.fnal.gov"><b>cdfofpr2.fnal.gov</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora4.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=cdfofpr2.fnal.gov&metricColumn=genericErrors&metric=alertLogStatus&keyValue="><b>Generic Alert Log Error Status</b></a><br>
Metric Value=<b>0</b><br>
Timestamp=<b>Dec 29, 2006 1:08:29 AM CST</b><br>
Severity=<b>Clear</b><br>
Message=<b>0 distinct types of ORA- errors have been found in the alert log.</b><br>
Notification Rule Name=<b>fcdfora4 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Dec 29 22:15:24 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFONPRD.FNAL.GOV"><b>CDFONPRD.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>bzora1.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFONPRD.FNAL.GOV&metricColumn=genericErrors&metric=alertLogStatus&keyValue="><b>Generic Alert Log Error Status</b></a><br>
Metric Value=<b>1</b><br>
Timestamp=<b>Dec 29, 2006 10:14:44 PM CST</b><br>
Severity=<b>Warning</b><br>
Message=<b>1 distinct types of ORA- errors have been found in the alert log.</b><br>
Notification Rule Name=<b>bzora1 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Dec 29 22:30:44 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFONPRD.FNAL.GOV"><b>CDFONPRD.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>bzora1.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFONPRD.FNAL.GOV&metricColumn=genericErrors&metric=alertLogStatus&keyValue="><b>Generic Alert Log Error Status</b></a><br>
Metric Value=<b>0</b><br>
Timestamp=<b>Dec 29, 2006 10:29:44 PM CST</b><br>
Severity=<b>Clear</b><br>
Message=<b>0 distinct types of ORA- errors have been found in the alert log.</b><br>
Notification Rule Name=<b>bzora1 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Dec 29 23:11:55 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=cdfofpr2.fnal.gov"><b>cdfofpr2.fnal.gov</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora4.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=cdfofpr2.fnal.gov&metricColumn=NumValue&metric=SQLUDM&keyValue=apply_msg_spill"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>5345</b><br>
Script=<b>apply_msg_spill</b><br>
Timestamp=<b>Dec 29, 2006 11:10:49 PM CST</b><br>
Severity=<b>Critical</b><br>
Message=<b>UDM alert: key = %Key, value = 5345</b><br>
Notification Rule Name=<b>fcdfora4 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Dec 29 23:13:59 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFONPRD.FNAL.GOV"><b>CDFONPRD.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>bzora1.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFONPRD.FNAL.GOV&metricColumn=NumValue&metric=SQLUDM&keyValue=capture_msg_spill"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>28756</b><br>
Script=<b>capture_msg_spill</b><br>
Timestamp=<b>Dec 29, 2006 11:13:13 PM CST</b><br>
Severity=<b>Critical</b><br>
Message=<b>UDM alert: key = %Key, value = 28756</b><br>
Notification Rule Name=<b>bzora1 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Dec 29 23:26:56 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=cdfofpr2.fnal.gov"><b>cdfofpr2.fnal.gov</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora4.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=cdfofpr2.fnal.gov&metricColumn=NumValue&metric=SQLUDM&keyValue=apply_msg_spill"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>0</b><br>
Script=<b>apply_msg_spill</b><br>
Timestamp=<b>Dec 29, 2006 11:25:49 PM CST</b><br>
Severity=<b>Clear</b><br>
Message=<b>CLEARED - UDM alert: key = %Key, value = 0</b><br>
Notification Rule Name=<b>fcdfora4 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Dec 30 00:58:24 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=cdfofpr2.fnal.gov"><b>cdfofpr2.fnal.gov</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora4.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=cdfofpr2.fnal.gov&metricColumn=NumValue&metric=SQLUDM&keyValue=stream_heartbeat"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>7240</b><br>
Script=<b>stream_heartbeat</b><br>
Timestamp=<b>Dec 30, 2006 12:57:41 AM CST</b><br>
Severity=<b>Warning</b><br>
Message=<b>UDM alert: key = %Key, value = 7240</b><br>
Notification Rule Name=<b>fcdfora4 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Dec 30 00:59:34 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFSTRM1.FNAL.GOV"><b>CDFSTRM1.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora6.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFSTRM1.FNAL.GOV&metricColumn=NumValue&metric=SQLUDM&keyValue=stream_heartbeat"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>7283</b><br>
Script=<b>stream_heartbeat</b><br>
Timestamp=<b>Dec 30, 2006 12:58:24 AM CST</b><br>
Severity=<b>Warning</b><br>
Message=<b>UDM alert: key = %Key, value = 7283</b><br>
Notification Rule Name=<b>fcdfora6 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Dec 30 02:59:01 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=cdfofpr2.fnal.gov"><b>cdfofpr2.fnal.gov</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora4.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=cdfofpr2.fnal.gov&metricColumn=NumValue&metric=SQLUDM&keyValue=stream_heartbeat"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>14440</b><br>
Script=<b>stream_heartbeat</b><br>
Timestamp=<b>Dec 30, 2006 2:57:41 AM CST</b><br>
Severity=<b>Critical</b><br>
Message=<b>UDM alert: key = %Key, value = 14440</b><br>
Notification Rule Name=<b>fcdfora4 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Dec 30 02:59:26 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFSTRM1.FNAL.GOV"><b>CDFSTRM1.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>fcdfora6.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFSTRM1.FNAL.GOV&metricColumn=NumValue&metric=SQLUDM&keyValue=stream_heartbeat"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>14484</b><br>
Script=<b>stream_heartbeat</b><br>
Timestamp=<b>Dec 30, 2006 2:58:25 AM CST</b><br>
Severity=<b>Critical</b><br>
Message=<b>UDM alert: key = %Key, value = 14484</b><br>
Notification Rule Name=<b>fcdfora6 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Dec 30 06:15:53 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFONPRD.FNAL.GOV"><b>CDFONPRD.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>bzora1.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFONPRD.FNAL.GOV&metricColumn=genericErrors&metric=alertLogStatus&keyValue="><b>Generic Alert Log Error Status</b></a><br>
Metric Value=<b>1</b><br>
Timestamp=<b>Dec 30, 2006 6:14:44 AM CST</b><br>
Severity=<b>Warning</b><br>
Message=<b>1 distinct types of ORA- errors have been found in the alert log.</b><br>
Notification Rule Name=<b>bzora1 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Dec 30 06:30:23 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFONPRD.FNAL.GOV"><b>CDFONPRD.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>bzora1.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFONPRD.FNAL.GOV&metricColumn=genericErrors&metric=alertLogStatus&keyValue="><b>Generic Alert Log Error Status</b></a><br>
Metric Value=<b>0</b><br>
Timestamp=<b>Dec 30, 2006 6:29:44 AM CST</b><br>
Severity=<b>Clear</b><br>
Message=<b>0 distinct types of ORA- errors have been found in the alert log.</b><br>
Notification Rule Name=<b>bzora1 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Dec 30 06:44:21 2006 -->
<pre>

Target Name=<a href="https://dba32.fnal.gov:1159/em/redirect?encoded=FALSE&targetType=oracle_database&pageType=TARGET_HOMEPAGE&targetName=CDFONPRD.FNAL.GOV"><b>CDFONPRD.FNAL.GOV</b></a><br>
Target Type=<b>Database Instance</b><br>
Host=<b>bzora1.fnal.gov</b><br>
Metric=<a href="https://dba32.fnal.gov:1159/em/redirect?targetType=oracle_database&pageType=METRIC_DETAILS&timePeriod=byDay&targetName=CDFONPRD.FNAL.GOV&metricColumn=NumValue&metric=SQLUDM&keyValue=capture_latency"><b>User-Defined Numeric Metric</b></a><br>
Metric Value=<b>26439</b><br>
Script=<b>capture_latency</b><br>
Timestamp=<b>Dec 30, 2006 6:43:13 AM CST</b><br>
Severity=<b>Critical</b><br>
Message=<b>UDM alert: key = %Key, value = 26439</b><br>
Notification Rule Name=<b>bzora1 database alert</b><br>
Notification Rule Owner=<b>SYSMAN</b><br>

</pre>
<!-- Author: oem-admin (email) -->