<!-- Title: 06:48:27  Thu Oct 12 2000 -->
<!-- Crewchief: jtrumbo -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Database activities</pre>


<!-- Date: Thu Oct 12 06:49:23 2000 -->
<pre>Service=cdfonprd.fnal.gov 
Node=b0dau35.fnal.gov 
 
Result=The following errors are in the ALERT file : ORA-00600: internal error code, 
arguments: [12333], [0], [3], [81], [], [], [], [] ORA-00600: internal error code, 
arguments: [12333], [0], [3], [81], [], [], [], [] ORA-00600: internal error code, 
arguments: [12333], [0], [3], [81], [], [], [], [] ORA-00600: internal error code, 
arguments: [12333], [0], [3], [81], [], [], [], [] ORA-00600: internal error code, 
arguments: [12333], [0], [3], [81], [], [], [], []. 
Status=CRITICAL 
Timestamp=12-Oct-00 5:05:44 AM 
Test=Alert 
 
 
This has been the java inconsistent version problem.  
Can we verify this?????  I will look at this end.  jt</pre>
<!-- Author: jtrumbo -->


<!-- Date: Thu Oct 12 09:41:23 2000 -->
<pre> 
coffee@9 
 
 
o Replication:  
  Procedure: 
  o Refresh all of integration today in the online 
  o Refresh integration today in offline: file catalog, calibration, cdf views. 
  o Jack will try to do a cut of his new schema in both 
  o Can try jack's scripts tomorrow with john to test replication 
  o Replication test monday at 9, depending on approval and beam. 
  Problems: 
     Refresh takes long time to refresh replication. 
     Makes the integration testing hard: 1.5 hours at time. 
     Want to create integration as a subset of production? 
     Only make a full copy under dire circumstances where an investigation is needed. 
o lots of failures in development consumers: rick will follow up 
o dennis will follow up the bytes checking 
o DBANA cannot build because an include file moved for silicon 
o component test for byte to string and string to byte for oracle test was missed out because of a missed subdirs. 
o randy and rick looking at display of trigger for a given run 
o most calibrations in online not succeeding due to them not getting events and put in a procedure to only run calibrations in partition 1, and nothing else. 
o core dumps in oracle come when we run ctt and bill notes that since the time the core dumps came there is a funny date for the calibration (1969).  We see this in other consumers.  
</pre> 
  
<!-- Author: R. St. Denis -->


<!-- Date: Thu Oct 12 12:06:44 2000 -->
<pre>refresh of integration to production for both ON & OFF 
 
1)  export - setenv TWO_TASK & ORACLE_SID to  
cdfonprd on dau36 to do the export this saves having 
to SCP saves over an hour! 
2)  drop all applications plus drop replication HDW, CDFTRIG, 
RUNDB, & CALIB (kirsten, jack, billb) this includes 
dropping replication (john w) 
3) check clean all_objects, sequences & synonyms 
4) coalesce on ONLINE manually OFFLINE down by DBATOOLS 
COALESCE  BEFORE IMPORT 
Tablespace Tablespace                     Coalescable 
    Number Name                               Extents 
---------- ------------------------------ ----------- 
         6 CALIB_DATA                             165 
         7 CALIB_INDEX                            102 
         8 CDFTRIG_DATA                            32 
         9 CDFTRIG_IDX                             45 
        15 RUNDB_DATA                              20 
        16 RUNDB_INDEX                             42 
        18 MLOG_DATA                              162 
SQL> @writescript_coalesce.sql 
> sqlplus nelly@cdfonint 
 
ALTER TABLESPACE CALIB_DATA COALESCE 
ALTER TABLESPACE CALIB_INDEX COALESCE; 
ALTER TABLESPACE CDFTRIG_DATA COALESCE; 
ALTER TABLESPACE CDFTRIG_IDX COALESCE; 
ALTER TABLESPACE RUNDB_DATA COALESCE; 
ALTER TABLESPACE RUNDB_INDEX COALESCE; 
ALTER TABLESPACE MLOG_DATA COALESCE; 
7 rows 
 
5)  IMPORT in this order   HWD(6 min)  TRIG(6 min) RUN (20 min) then CALIB (started at 11:00 a.m.) expected finish time 
12:15 or so 
 
6)  IMPORT calib on the OFFLINE side also started at 
11:00 a.m. (10 minutes)  FILECATALOG done 5 minutes to import. 
warning: link not created also there were some sequences owned by 
litvinse that were cleaned up 
 
7) check for invalid objects - WHY ? 
RUNDB_INT                      PACKAGE 
RUNDB_INT                      PACKAGE BODY 
CDFTRIG_INT                    PROCEDURE 
 
8)  recut cdf_views_int on ONLINE only - call jack c at 2114 
 
9)  recreate  replication - expected time 1 hour 30 minutes  
 
</pre>
<!-- Author: nstanfield -->
<!-- Comment: Thu Oct 12 13;19;23 comment by...nstanfield -->
Synonyms need to be created after the imports or else
the cdf_views_int cutting scripts compile with errors
Jack C ran CALIB as well HDWDB & RUN in Bill's absence then all compiled
<!-- Comment: Thu Oct 12 13;43;06 comment by...nstanfield -->
CDFTRIG synonyms have also been created per Kirsten 1:30 Thursday

<!-- Comment: Thu Oct 12 16;15;59 comment by...John Weigand -->
The re-creation of the replication environment started at 13:06
and ended at 14:06.  There are 2 other individual postings that
follow this in the log with statistics and that process.
<!-- Date: Thu Oct 12 12:43:07 2000 -->
<pre> 
Date: Thu, 12 Oct 2000 12:36:24 -0500 (CDT) 
From: oem-admin@fnal.gov 
To: oem-admin@fnal.gov, cdfonoem@fnal.gov 
Subject: Event CDFONLINEMGR:cdfonprd_alert_log WARNING 
 
Service=cdfonprd.fnal.gov 
Node=b0dau35.fnal.gov 
 
Result=The following errors are in the ALERT file : ORA-000060: Deadlock 
detected. More info in file /ora1/admin/cdfonprd/udump/cdfonprd_ora_19235.trc. 
ORA-000060: Deadlock detected. More info in file 
/ora1/admin/cdfonprd/udump/cdfonprd_ora_20001.trc.. 
Status=WARNING 
Timestamp=12-Oct-00 12:33:09 PM 
Test=Alert 
</pre> 
I have checked the CDF Log and see that a run with the muon trigger started at 12:35.  So there was likely to have been a run startup in progress at the time of the deadlock.    
<!-- Author: R. St. Denis -->


<!-- Date: Thu Oct 12 12:48:27 2000 -->
<pre> 
Service=cdfonprd.fnal.gov 
Node=b0dau35.fnal.gov 
 
Result=The following errors are in the ALERT file : ORA-000060: Deadlock detected. 
More info in file /ora1/admin/cdfonprd/udump/cdfonprd_ora_19235.trc. ORA-000060: 
Deadlock detected. More info in file 
/ora1/admin/cdfonprd/udump/cdfonprd_ora_20001.trc.. 
Status=WARNING 
Timestamp=12-Oct-00 12:33:09 PM 
Test=Alert 
 
LOCK OUT, LOOKING INTO IT!</pre>
<!-- Author: jtrumbo -->
<!-- Comment: Thu Oct 12 13;06;55 comment by...jtrumbo -->
b0dau35> more cdfonprd_ora_20001.trc
Dump file /ora1/admin/cdfonprd/udump/cdfonprd_ora_20001.trc
Oracle8i Enterprise Edition Release 8.1.6.1.0 - Production
With the Partitioning option
JServer Release 8.1.6.1.0 - Production
ORACLE_HOME = /ora1/oracle/v8_1_6
System name:    SunOS
Node name:      b0dau35
Release:        5.7
Version:        Generic_106541-08
Machine:        sun4u
Instance name: cdfonprd
Redo thread mounted by this instance: 1
Oracle process number: 22
Unix process pid: 20001, image: oracle@b0dau35 (TNS V1-V3)

*** 2000-10-12 12:32:17.639
*** SESSION ID:(12.55116) 2000-10-12 12:32:17.615
DEADLOCK DETECTED
Current SQL statement for this session:
delete from datasets where dataset_name like '%120%'
The following deadlock is not an ORACLE error. It is a
deadlock due to user error in the design of an application
or from issuing incorrect ad-hoc SQL. The following
information may aid in determining the deadlock:
Deadlock graph:
                       ---------Blocker(s)--------  ---------Waiter(s)---------
Resource Name          process session holds waits  process session holds waits
TX-00040028-000016df        22      12     X             35      38           X
TX-000f004f-00000050        35      38     X             22      12           S
session 12: DID 0001-0016-00000002      session 38: DID 0001-0023-00000002
session 38: DID 0001-0023-00000002      session 12: DID 0001-0016-00000002
Rows waited on:
Session 38: obj - rowid = 0000106C - AAABBsAAKAAAAE+ABP
Session 12: no row
===================================================
PROCESS STATE
Process global information:
     process: 81396ff4, call: 8190c870, xact: 814c4134, curses: 813aed9c, usrses: 813aed9c
  ----------------------------------------
  SO: 81396ff4, type: 1, owner: 0, flag: INIT/-/-/0x00
  (process) Oracle pid=22, calls cur/top: 8190c870/8190c870, flag: (0) -
            int error: 0, call error: 0, sess error: 0, txn error 0
  (post info) last post received: 0 0 18
              last post received-location: ksqrcl
              last process to post me: 813978f4 48 0
              last post sent: 0 0 18
              last post sent-location: ksqrcl
              last process posted by me: 813996f4 67 0
    (latch info) wait_event=0 bits=10
      holding     8000351c Parent+children enqueue hash chains level=4 
        Location from where latch is held: ksqcmi: kslgpl: 
        Context saved from call: 0
        state=busy
        recovery area:
Dump of memory from 0x81392DD4 to 0x81392DDC
81392DD0          00000000 00000000               [........]    
    Process Group: DEFAULT, pseudo proc: 813a5bb0
    O/S info: user: oracle, term: UNKNOWN, ospid: 20001
    OSD pid info: 20001
    ----------------------------------------
    SO: 813aed9c, type: 3, owner: 81396ff4, flag: INIT/-/-/0x00
    (session) trans: 814c4134, creator: 81396ff4, flag: (41) USR/- BSY/-/-/-/-/-
              DID: 0001-0016-00000002, short-term DID: 0000-0000-00000000
              txn branch: 0
              oct: 7, prv: 0, user: 37/L1L2_PRD
    O/S info: user: cdfdaq, term: pts/9, ospid: 11610, machine: b0dap23.fnal.gov
              program: sqlplus@b0dap23.fnal.gov (TNS V1-V3)
    application name: SQL*Plus, hash value=3669949024
    last wait for 'enqueue' blocking sess=0x813bc41c seq=267 wait_time=308
                name|mode=54580004, id1=f004f, id2=50
      ----------------------------------------
etc
<!-- Date: Thu Oct 12 13:04:06 2000 -->
<pre>refresh integration  
 
CDF_VIEW_INT failing because public synonyms have not 
been recreated by the individual applications. 
Need to have each application owner create synonyms 
and then run CDF_VIEW_INT 
 
</pre>
<!-- Author: nstanfield -->




<!-- Date: Thu Oct 12 13:30:53 2000 -->
<pre>test for diana</pre>
<!-- Author: diana@fnal.gov -->
<!-- Comment: Thu Oct 12 13;31;38 comment by...diana@fnal.gov -->
ignore


<!-- Date: Thu Oct 12 13:33:20 2000 -->
<pre>sar statistics refresh replication 
 
on fcdfora1 two ways to view  sar interactively or  
look a the files at cd /var/admsa  the file 
here named sar digit digit these are the ascii interpretation 
files  
 
sa digit digit are binary and they are changed to 
sar digit digit in a cron job ran over night 
 
at the command line sar -f /var/admsa12   to see Oct12, 
because the sar is filling up while it collects but 
not converted to an ascii file till the cron job runs 
 
</pre>
<!-- Author: nstanfield -->


<!-- Date: Thu Oct 12 13:34:53 2000 -->
<pre>application refresh 
 
hwdb & rundb - bill badgett x6675 
cdftrig - kirsten 8271 or alexei 
calib - jack c x2114 
cdf_views_int - jack c x2114 
 
 
</pre>
<!-- Author: nstanfield -->



<!-- Date: Thu Oct 12 15:33:51 2000 -->
<pre>The re-creation of the replication environment for the 
calibration tables on the integration instance is complete. 
It took approx. 1 hour elapsed time.   
 
This was part of a full refresh of the online application from 
production and a partial refresh of selected offline  
applications. 
 
The statistics below are for the creation of the offline / 
snapshot tables which requires a complete copy of the 
corresponding master table from the online to the offline. 
 
The job processes parameter is set to 1 at this time which  
effectively single threads each snapshot copy.  After some 
initial testing of changes against the online tables, I will 
expand the number of job processes parameter in init.ora to  
try to determine some optimal setting. 
 
* *************************************************** 
* dbrep_crtSnapshots.sql 
* ------------------------------------------------- * 
* Log file: 
/cdf/home/weigand/dbrep/logs/int/dbrep_crtSnapshots.001012.130734 
* *************************************************** 
USER is "CALIB_INT_REP" 
* *************************** 
* Instance...cdfofint 
* Host name..fcdfora1 
* Version....8.1.6.1.0 
* *************************** 
                     Start	End    Elapsed	 Rows 
                    -------- -------- -------- --------- 
All CALIB_DEV	    13:06:44 14:06:59  1:00:15 
                   -   -   -   -   -  - 
CALDIGITOGEV	    13:06:51 13:06:53	   :07		1 
CALIBRUNLISTS	    13:06:53 13:06:54	   :01	    2,388 
CALIB_PERIODS	    13:06:54 13:06:54	   :00		8 
CALIB_PROPERTIES    13:06:54 13:06:55	   :01	       64 
CCRPED		    13:06:55 13:06:55	   :00		0 
CCRQIE		    13:06:55 13:06:56	   :01		0 
CEM1985TESTBEAM	    13:06:56 13:06:56	   :00	      960 
CEMLED		    13:06:56 13:07:14	   :18	  120,143 
CEMLINERESPONSE	    13:07:14 13:07:14	   :00	      960 
CEMPED		    13:07:14 13:07:35	   :21	  104,940 
CEMQIE		    13:07:35 13:24:22	 16:47  4,116,480 
CEMSRC		    13:24:22 13:24:24	   :02		0 
CEMXEF		    13:24:24 13:24:31	   :07	   51,650 
CESPED		    13:24:31 13:24:31	   :00		0 
CESQIE		    13:24:31 13:24:32	   :01	    2,048 
CHALASER	    13:24:32 13:24:32	   :00		0 
CHALINERESPONSE	    13:24:32 13:24:33	   :01	      768 
CHAPED		    13:24:33 13:24:50	   :17	   88,896 
CHAQIE		    13:24:50 13:38:07	 13:17  3,398,784 
CHASRC		    13:38:07 13:38:07	   :00		0 
CHATIMELASER	    13:38:08 13:38:08	   :00		0 
CLACQIE		    13:38:08 13:38:44	   :36	  199,168 
CLAPED		    13:38:44 13:38:44	   :00		0 
CLCAMPLCALIB	    13:38:44 13:38:49	   :05	    9,829 
CMPCH		    13:38:49 13:38:50	   :01		0 
CMPCHEL		    13:38:50 13:38:50	   :00		0 
CMPGLOBAL	    13:38:50 13:38:51	   :01		0 
CMUCH		    13:38:51 13:38:51	   :00		0 
CMUGLOBAL	    13:38:51 13:38:52	   :01		0 
CMXCHASP	    13:38:52 13:38:52	   :00		0 
CMXCHDRV	    13:38:52 13:38:53	   :01		0 
CMXGLOBAL	    13:38:53 13:38:53	   :00		0 
COTCTI		    13:38:53 13:38:54	   :01		0 
COTCTT		    13:38:54 13:38:55	   :01	   11,856 
COTSTROBEOFFSET	    13:38:55 13:38:56	   :01		6 
COTTIMEDISTANCE1    13:38:56 13:38:56	   :00		0 
COTWIREPOSITIONS    13:38:56 13:38:57	   :01		0 
CPRPED		    13:38:57 13:38:57	   :00		0 
CPRQIE		    13:38:57 13:38:58	   :01		0 
CSXCHAST	    13:38:58 13:38:58	   :00		0 
CSXGLOBAL	    13:38:58 13:38:59	   :01		0 
CTCPRO		    13:38:59 13:39:00	   :01		0 
CTCSPO		    13:39:00 13:39:00	   :00		0 
CTCTMS		    13:39:00 13:39:01	   :01		0 
CTCWPO		    13:39:01 13:39:01	   :00		0 
GEOMETRYTABLE	    13:39:01 13:39:02	   :01		0 
HARDWARE_DOWNLOADS  13:39:02 13:39:02	   :00		0 
_VIEW 
LSALASER	    13:39:02 13:39:03	   :01		0 
PEMLASER	    13:39:03 13:39:03	   :00		0 
PEMLINERESPONSE	    13:39:03 13:39:04	   :01	      960 
PEMPED		    13:39:04 13:39:25	   :21	   87,200 
PEMQIE		    13:39:25 13:50:45	 11:20  2,859,520 
PESCALIB	    13:50:45 13:50:46	   :01		0 
PESPED		    13:50:46 13:50:47	   :01		0 
PESQIE		    13:50:47 13:50:47	   :00		0 
PHALASER	    13:50:47 13:50:48	   :01		0 
PHALINERESPONSE	    13:50:48 13:50:48	   :00	      864 
PHAPED		    13:50:48 13:51:03	  1:15	   78,768 
PHAQIE		    13:51:03 14:00:58	  9:55  2,496,256 
SET_RUN_MAPS	    14:00:58 14:00:59	   :01		5 
SIXCHP		    14:00:59 14:01:12	   :13	  123,387 
SVXDOWNTEST	    14:01:12 14:01:17	   :05	    6,108 
SVXFIBPED	    14:01:17 14:01:17	   :00		0 
TESTTABLE	    14:01:17 14:01:18	   :01		0 
TIMESTAMP	    14:01:18 14:01:18	   :00	    2,412 
USED_SETS	    14:01:18 14:01:19	   :01		1 
VALID_SETS	    14:01:19 14:01:19	   :00		1 
WHALASER	    14:01:19 14:01:20	   :01		0 
WHALINERESPONSE	    14:01:20 14:01:20	   :00	      576 
WHAPED		    14:01:20 14:01:31	   :11	   53,568 
WHAQIE		    14:01:31 14:06:58	  5:27  1,542,144 
WHASRC		    14:06:58 14:06:59	   :01		0 
** ********************************************* ** 
** Total transactions.............    72 
**   - successful........    72 
**   - failed............     0 
** ********************************************* ** 
** All commands were processed successfully. 
** ********************************************* ** 
 
</pre>
<!-- Author: John Weigand -->


<!-- Date: Thu Oct 12 16:13:30 2000 -->
<pre>Keywords: replication integration snapshots offline refresh 
 
This is being posted as a new item although it relates to the  
re-creation of the offline integration replication environment 
posted early this same day.  It would have been added as comments 
but the comment page does not recognize new lines and this came 
out unreadable in the preview. 
 
The stats below is the 'sar' for fcdfora1 for the entire 
day up until 15:20.  This is the offline integration instance. 
 
Replication was running from 13:06 - 14:06. 
 
I am not sure of the time frame for the imports that preceeded  
it, but I believe it was a relatively short run since it was 
only the calibration tables. 
 
SunOS fcdfora1 5.8 Generic_108528-02 sun4u    10/12/00 
 
01:00:01    %usr    %sys    %wio   %idle 
02:00:00       7       4      72      18 
03:00:00       7       4      70      19 
04:00:00       9       6      60      25 
05:00:00      10       6      54      30 
06:00:01      11       7      45      37 
07:00:00      10       7      49      34 
08:00:00      10       7      47      36 
08:20:00      11       7      46      37 
08:40:00      11       7      41      42 
09:00:00      11       7      44      38 
09:20:01      11       6      49      33 
09:40:00      10       7      46      37 
10:00:00      10       7      40      43 
10:20:00      11       6      34      48 
10:40:00       9       6      40      45 
11:00:00       9       7      40      44 
11:20:00      11       7      46      37 
11:40:00       7       5      13      75 
12:00:00       6       3       2      90 
12:20:00       4       2       0      93 
12:40:01       4       2       1      93 
13:00:00       4       2       0      93 
13:20:00      10       7      26      57 
13:40:00      15      11      40      34 
14:00:00      14      10      39      37 
14:20:00       8       6      14      72 
14:40:00       5       3       2      90 
15:00:00       6       3       4      87 
15:20:00       5       3       4      88</pre>
<!-- Author: John Weigand -->