<!-- Title: 07:25:39  Tue Feb  4 2003 -->
<!-- Crewchief: New Day -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>database activity</pre>

<!-- Date: Tue Feb 4 10:31:19 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : STARTED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Tue Feb 4 10:31:19 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : COMPLETED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Feb 5 10:31:51 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : STARTED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Feb 5 10:31:51 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : COMPLETED 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Wed Feb 5 13:57:34 2003 -->
<pre>SQL&gt; alter system set undo_retention = 1500; 
 
System altered. 
 
SQL&gt;  
we should keep an eye on this..error in the alert log 
 
it's a query that did not have enough space for a CONSISTENT READ!&gt;&gt;&gt;&gt;&gt;&gt; 
 
 
01555, 00000, "snapshot too old: rollback segment number %s with name \"%s\" too small" 
 
// *Cause: rollback records needed by a reader for consistent read are 
 
// overwritten by other writers 
 
// *Action: If in Automatic Undo Management mode, increase undo_retention 
 
// setting. Otherwise, use larger rollback segments 
 
 
 
ORA-01555 caused by SQL statement below (Query Duration=1044246968 sec, SCN: 0x0000.450f5a01): 
 
Sun Feb 2 22:36:04 2003 
 
SELECT OBJID, NAME, OWNER, OWNERID, TABLESPACE, TSNO, FILENO, BLOCKNO, AUDIT$, COMMENT$, CLUSTERFLAG, PCTFREE$, PCTUSED$ 
 
, INITRANS, MAXTRANS, DEGREE, INSTANCES, CACHE, PROPERTY, DEFLOG, TSDEFLOG, ROID, ROWCNT, BLKCNT, AVGRLEN, T 
 
FLAGS, TRIGFLAG, OBJSTATUS FROM SYS.EXU9TAB T$ WHERE OWNERID = :1 AND NOT EXISTS ( SELECT N 
 
AME FROM SYS.EXU8NXP N$ WHERE N$.OWNERID = T$.OWNERID AND N$.NAME = T$.NAM 
 
E AND N$.TYPE = 2) ORDER BY T$.NAME 
 
ORA-01555 caused by SQL statement below (Query Duration=1044246968 sec, SCN: 0x0000.450f5a01): 
 
Sun Feb 2 22:36:04 2003 
 
SELECT sys_context('USERENV','DB_NAME') from dual 
 
Sun Feb 2 22:41:00 2003 
 
 
 
 
 
b0dau35&gt; 
 
----- Original Message ----- 
From: &lt;oem-admin@fnal.gov&gt; 
To: "CDFONLINEMGR" &lt;oem-admin@fnal.gov&gt;; "CDFONLINEMGR" &lt;cdfonoem@fnal.gov&gt; 
Sent: Sunday, February 02, 2003 10:37 PM 
Subject: EVENT cdfonprd_alert_log WARNING 
 
 
&gt; Name : cdfonprd_alert_log 
&gt; Target Name : cdfonprd 
&gt; Target Type : Database 
&gt; Node Name : b0dau35.fnal.gov 
&gt; Status : WARNING 
&gt; 
&gt; Alert 
&gt; The following errors are in the ALERT file : ORA-01555 caused by SQL statement below (Query Duration=1044246968 sec, SCN: 
0x0000.450f5a01): ORA-01555 caused by SQL statement below (Query Duration=1044246968 sec, SCN: 0x0000.450f5a01):. 
&gt; WARNING 
&gt; Feb 02 2003 10:37:44 PM 
&gt; 
</pre>
<!-- Author: nstanfield  -->


<!-- Date: Thu Feb 6 02:58:15 2003 -->
<pre>to quell Randy & SAM bureaucrats I added new column 
 
   	DATA_TYPE_ID to FILECATALOG.CDF2_DATASET_REGISTRIES 
 
Also I created table FILECATALOG.CDF2_DATA_TYPES 
 
	FILECATALOG.CDF2_DATASET_REGISTRIES has a FK to  
	FILECATALOG.CDF2_DATA_TYPES 
 
Current types are: 
 
	0 - RECONSTRUCTED 
	1 - RAW 
	2 - SIMULATION 
	3 - GENERATOR_LEVEL 
	4 - ROOT_NUPLE 
	5 - HBOOK_NTUPLE 
 
Registry page and necessary code were updated 
 
These changes were done on: 
 
	- cdfofdev 
	- cdfofint 
	- cdfofprd 
 
I need to go over existing entry and set DATA_TYPE on them</pre>
<!-- Author: Dmitry Litvintsev -->


<!-- Date: Thu Feb 6 04:51:00 2003 -->
<pre>added two columns to FILECATALOG.CDF2_RUNSECTION_LIVETIMES: 
 
alter table FILECATALOG.CDF2_RUNSECTION_LIVETIMES add ( 
	COUNT_BEFORE_PS NUMBER(38) NULL, 
	COUNT_AFTER_PS  NUMBER(38) NULL 
) 
/ 
 
this is per Jonathan Lewis' request 
 
did that on cdfofdev, cdfofprd, cdfofint</pre>
<!-- Author: Dmitry Litvintsev -->


<!-- Date: Thu Feb 6 08:48:35 2003 -->
<pre>Replication synced up for CDF2_RUNSECTIONS for DFC between  
cdfofprd and cdfrep01</pre>
<!-- Author: akumar -->

<!-- Date: Thu Feb 6 10:31:45 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : STARTED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Feb 6 10:31:45 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : COMPLETED 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Thu Feb 6 11:24:32 2003 -->
<pre>disk for cdfonint archived redo logs filled to 100%.  I deleted some of the oldest files to make some room. 
 diana</pre>
<!-- Author: diana -->


<!-- Date: Thu Feb 6 12:03:19 2003 -->
<pre>On cdfrep01,  CDF_READ role has been assigned to 
FILECATALOG_READUSER and Revoked the FILECATALOG_READ role from 
FILECATALOG_READUSER. 
 
And Application Co-ordinator verfied, everything is fine with DFC 
with this change. 
</pre>
<!-- Author: akumar -->


<!-- Date: Thu Feb 6 13:13:10 2003 -->
<pre>CALIBRATION/SCHEMA/V_5_4 
 
We need to add CEMPMTDeadChannels,CHAPMTDeadChannels, 
and WHAPMTDeadChannels to replication. We also need to  
resynch the USED_SETS table to get the US_VERSION column. 
 
Release Notes: 
 
1) Add data tables 
OffSCL (precut) 
CEMPMTDeadChannels 
CHAPMTDeadChannels 
WHAPMTDeadChannels 
 
2) Increase PASSCALIBS extents. (precut) 
 
3) Add markedjobset column to PROCESS_DEFINITIONS and  
trigger to fill it. (precut) 
 
4) Add US_VERSION column to USED_SETS to contain numeric version. 
Altered US_BEF_I trigger to fill it on insert. 
 
5) Added indices to USED_SETS 
  - US_VERSION 
  - PROCESS_NAME 
  - TO_NUMBER(PROC_CALIB_VERSION)</pre>
<!-- Author: jcranshaw -->
<!-- Comment: Thu Feb  6 15;18;15 comment by...akumar -->
<pre>Added the following tables to Replication on cdfofprd and cdfrep01 
CEMPMTDeadChannels,CHAPMTDeadChannels, 
and WHAPMTDeadChannels 

( Adding new tables to replication doesn't cause 
any interruption so decided to add those today ) Remaining 
Syncup for CALIB and DFC can be scheduled for Thurs 02/13/04

</pre>
<!-- Date: Thu Feb 6 14:44:57 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Feb 06 2003 02:43:24 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Feb 6 15:05:26 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Feb 06 2003 03:03:25 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Thu Feb 6 15:24:58 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Feb 06 2003 03:23:26 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Thu Feb 6 15:40:31 2003 -->
<pre>&gt; Alert 
&gt; The following errors are in the ALERT file : ORA-07445: exception encountered: core dump [__align_cpy_1()+48] [SIGSEGV] [Invalid permissions for mapped object] [0x1A47A74] [] []. 
&gt; WARNING 
&gt; Feb 06 2003 03:12:44 PM 
 
Information from Trace File  that  Session 18 and Serail# 31918 caused this execption : 
*** SESSION ID:(18.31918) 2003-02-06 15:12:04.414 
 
And Details from Login traces : 
 
       SID    SERIAL# TO_CHAR(LOGON_TIM OSUSER 
---------- ---------- ----------------- ------------------------------ 
PROGRAM 
------------------------------------------------ 
MACHINE 
---------------------------------------------------------------- 
        18      31918 02/06/03 13:26:58 torretta 
sqlplus@b0dap30.fnal.gov (TNS V1-V3) 
b0dap30.fnal.gov 
 
 
Need to follow-up with  Donatella T. What she was doing when "Invalid Permissions" caused the alert  in db. 
 
 
</pre>
<!-- Author: akumar -->


<!-- Date: Fri Feb 7 09:17:58 2003 -->
cdfonprd object calib_prd.pc_pk had hit maxextents.  I increased it from 100 to 120,  and the corresponding one in calib_prd_rep on cdfofprd. 
  diana
<!-- Author: diana -->

<!-- Date: Fri Feb 7 10:32:59 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : STARTED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Feb 7 10:32:59 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : COMPLETED 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Fri Feb 7 11:28:05 2003 -->
I've freed up disk space (again) for cdfonint.
<!-- Author: Diana -->

<!-- Date: Sat Feb 8 10:32:02 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : STARTED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Feb 8 10:32:02 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : COMPLETED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 9 10:42:59 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : STARTED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sun Feb 9 10:42:59 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : COMPLETED 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Sun Feb 9 13:52:16 2003 -->
<pre>Investigating apparent problem with CALIB_PRD replication 
from cdfonprd to cdfofprd.  Last successful replication of 
this application was on Thurs. Feb. 6.  See e-mails to 
cdfdb-support@fnal.gov on this topic via the listserv archives 
for more information.  In short form, CALIB_PRD replication 
(only) seems not to be working. </pre>
<!-- Author: Alan Sill -->

<!-- Date: Sun Feb 9 18:49:08 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Feb 09 2003 06:47:15 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Sun Feb 9 21:10:14 2003 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Tried to drop the CALIB_PRD refresh group  
for the cdfonprd-&gt;cdfofprd replication, but the  
command hangs forever.   Was not able to kill  
the session that was attempting to drop the  
refresh group (I don't have access to fcdfora1, so  
had to ALTER SYSTEM KILL..., but that did not  
work) 
 
Now I'm out of ideas;  let's wait until tomorrow  
to have experts handle it... 
</pre></td></tr></td></table>
<!-- Author: W.Badgett -->
<!-- Comment: Sun Feb  9 21;11;12 comment by...wfb -->
<pre>This was, of course, in an attempt to 
regenerate the refresh group and make 
sure the new table was included, and 
that replication was restarted...</pre>

<!-- Date: Mon Feb 10 10:32:23 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : STARTED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 10 10:32:23 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : COMPLETED 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 10 12:21:28 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CRITICAL 

Maximum Extents 
Segment CALIB_PRD_REP.PC_US_I in tablespace CALIB_PRD_REP_INDEX can only allocate 0 additional extents. Segment RLC.FILE_1_CREATE_TIME in tablespace USERINDEX_01 can only allocate 2 additional extents. Segment RLC.FILE_1_FS_FK_I in tablespace USERINDEX_01 can only allocate 2 additional extents. Segment RLC.FILE_DS_FK_I in tablespace USERINDEX_01 can only allocate 2 additional extents. 
CRITICAL 
Feb 10 2003 12:19:05 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 10 12:30:24 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CLEARED 

Unscheduled Jobs 
Event test "Unscheduled Jobs" is cleared 
CLEARED 
Feb 10 2003 12:28:10 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 10 13:03:56 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CRITICAL 

Alert 
The following errors are in the ALERT file : ORA-07445: exception encountered: core dump [colitn()+768] [SIGBUS] [Invalid address alignment] [1044903533] [] [] ORA-00604: error occurred at recursive SQL level 3 ORA-00028: your session has been killed ORA-02063: preceding line from CDFONPRD. 
CRITICAL 
Feb 10 2003 01:00:53 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Mon Feb 10 13:13:45 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : CRITICAL 

Alert 
The following errors are in the ALERT file : ORA-00600: internal error code, arguments: [15868], [26546656], [0], [37809], [], [], [], [] ORA-00604: error occurred at recursive SQL level 3 ORA-00028: your session has been killed ORA-02063: preceding line from CDFONPRD ORA-03106: fatal two-task communication protocol error ORA-02063: preceding line from CDFONPRD ORA-12008: error in snapshot refresh path ORA-00028: your session has been killed ORA-02063: preceding line from CDFONPRD ORA-12012: error on auto execute of job 3856 ORA-03106: fatal two-task communication protocol error ORA-02063: preceding line from CDFONPRD ORA-12008: error in snapshot refresh path ORA-00028: your session has been killed ORA-02063: preceding line from CDFONPRD ORA-06512: at "SYS.DBMS_SNAPSHOT", line 617 ORA-06512: at "SYS.DBMS_SNAPSHOT", line 674 ORA-06512: at "SYS.DBMS_IREFRESH", line 577 ORA-06512: at "SYS.DBMS_REFRESH", line 211 ORA-06512: at line 1 .... 
CRITICAL 
Feb 10 2003 01:10:55 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Mon Feb 10 15:44:27 2003 -->
cdfonprd -> cdfofprd replication is working again, with the exception of the calib table named TDCINFOS.  This table refuses to replicate and has been removed from the refresh group until we can solve the problem.
<!-- Author: Diana -->