<!-- Title: 09:29:57  Wed Aug 10 2005 -->
<!-- Crewchief: A -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>New Activity </pre>


<!-- Date: Wed Aug 10 09:31:39 2005 -->
<pre>The store is effectively over now.  Please go ahead and reboot the tape 
drive this morning.  I have made a warning entry in the CDF e-log, 
including that the database itself may need to be rebooted. 
If you do need to do that, please call the CDF control room at 2080 
and ask them to note it in the e-log. (I will probably be in meetings). 
                                                               Ray 
 
 Ray Culbertson  CDF/FNAL  rlc@fnal.gov  630-840-6744  Trailers 170F (MS318) 
 
 
----- Original Message ----- 
From: "Steven W. Kovich" &lt;kovich@fnal.gov&gt; 
Date: Monday, August 8, 2005 11:34 am 
Subject: bzora1 - Tape Jukebox Problems 
 
Ray, 
 
The tape jukebox on bzora1 seems hung. We'd like to reboot and  
power cycle the tape jukebox only. Not the system. Can this be done live? If not, can we schedule a downtime for this? As we do not want to risk an interruption of bzora1. Currently we don't have control of the tape  jukebox and are unable to perform a tape backup at this time. 
 
Steven Kovich 
   
 
Steve Kovich is planning to do the above work at 10:00AM 08/10/05 Wed.  
 
</pre>
<!-- Author: akumar -->
<!-- Comment: Wed Aug 10 10;36;32 comment by...akumar -->
<pre>Power cycling the tape jukebox solved the problem.
We have access to the jukebox , have unloaded the older tapes, loaded 
today's tapes, and we are running today's Arch backup.
Anil has verified that the DB is ok.
So, the maintenance is completed.

</pre>


<!-- Date: Wed Aug 10 14:54:56 2005 -->
<pre>Hi, 
       I spoke with ops manager and we have tentative permission 
to take down the CDF online production database from 3-3:30PM 
for quarterly patches.  As the time gets close I will consult again to 
get final permission and send an annoucement.  I will assume DSG 
will be ready at 3 unless you tell me otherwise. 
                                                                 Ray 
 
 Ray Culbertson  CDF/FNAL  rlc@fnal.gov  630-840-6744  Trailers 170F (MS318) 
 
interruption will be approx. 30 minutes</pre>
<!-- Author: Diana -->
<!-- Comment: Wed Aug 10 15;19;24 comment by...Diana -->
<pre>The oracle security patch has been successfully installed on cdfonprd and the database is available to users.  The DB Timezone parameter has been set to US/Central vs -5:00.  This timezone parameter has already been set to US/Central for cdfondev and cdfonint.</pre>
<!-- Date: Wed Aug 10 15:05:15 2005 -->
<pre>
Name : cdfonprd_db_down 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : NODE UNREACHABLE 

Node UpDown 
 VNI-4009 : Cannot contact agent on the node. Agent may be down or network communication to the node has failed.  
NODE UNREACHABLE 
Aug 10 2005 03:03:38 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Aug 10 15:05:15 2005 -->
<pre>
Name : bzora1_node_down 
Target Name : bzora1.fnal.gov 
Target Type : Node 
Node Name : bzora1.fnal.gov 
Status : NODE UNREACHABLE 

Node UpDown 
 VNI-4009 : Cannot contact agent on the node. Agent may be down or network communication to the node has failed.  
NODE UNREACHABLE 
Aug 10 2005 03:03:38 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Aug 10 15:15:36 2005 -->
<pre>
Name : bzora1_node_down 
Target Name : bzora1.fnal.gov 
Target Type : Node 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Node UpDown 
Event test "Node UpDown" is cleared 
CLEARED 
Aug 10 2005 03:14:39 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Wed Aug 10 15:15:36 2005 -->
<pre>
Name : cdfonprd_db_down 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Node UpDown 
Event test "Node UpDown" is cleared 
CLEARED 
Aug 10 2005 03:14:39 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Wed Aug 10 19:14:01 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CRITICAL 

Alert 
The following errors are in the ALERT file : ORA-00600: internal error code, arguments: [kghfrempty:ds], [0xFFFFFFFF7BBCFBC0], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [0000000101DB168C] [SIGSEGV] [Address not mapped to object] [0x7600770078DA80] [] [] ORA-00600: internal error code, arguments: [kghfrempty:ds], [0xFFFFFFFF7BBCFBC0], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [0000000101DB168C] [SIGSEGV] [Address not mapped to object] [0x7600770078DA80] [] [] ORA-07445: exception encountered: core dump [0000000101DB168C] [SIGSEGV] [Address not mapped to object] [0x7600770078DA80] [] [] ORA-00600: internal error code, arguments: [kghfrempty:ds], [0xFFFFFFFF7BBCFBC0], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [0000000101DB168C] [SIGSEGV] [Address not mapped to object] [0x7600770078DA80] [] [] ORA-07445: exception encountered: core dump [0000000101DB168C] [SIGSEGV] [Address not mapped to object] [0x7600770078DA80]
CRITICAL 
Aug 10 2005 07:12:31 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Thu Aug 11 09;39;27 comment by...akumar -->
<pre>There is patch 4398475 for the error. Will be scheduled for dev first. </pre>
<!-- Date: Thu Aug 11 00:03:01 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Aug 11 2005 12:02:30 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Thu Aug 11 00:10:30 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Aug 11 2005 12:09:49 AM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Thu Aug 11 10:20:26 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CRITICAL 

Alert 
The following errors are in the ALERT file : ORA-04030: out of process memory when trying to allocate  bytes (,) ORA-02063: preceding line from CDFOFPR2 ORA-00600: internal error code, arguments: [kpotcpop1], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kpotcpop1], [], [], [], [], [], [], []. 
CRITICAL 
Aug 11 2005 10:19:49 AM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Thu Aug 11 10;31;11 comment by...akumar -->
<pre>Will revisit this error after applying patch patch 4398475. 
Patch 4398475 is pending and will be scheduled for dev machines 
after Aug 15. </pre>


<!-- Date: Thu Aug 11 12:08:25 2005 -->
<pre>Have executed HDWDB schema cut v1_2n -- add one column  
to CARDS (PULSARCONTROL) with a foreign key index & contraint,  
on cdfondev, cdfonint, cdfonprd. 
</pre>
<!-- Author: W.Badgett -->

<!-- Date: Thu Aug 11 17:33:33 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : WARNING 

Alert 
The following errors are in the ALERT file : ORA-07445: exception encountered: core dump [0000000101DAC5F4] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFF7CA83808] [] [] ORA-12805: parallel query server died unexpectedly. 
WARNING 
Aug 11 2005 05:32:30 PM 



</pre>
<!-- Author: oem-admin (email) -->
<!-- Comment: Fri Aug 12 11;01;16 comment by...akumar -->
<pre>No action. Workaround is already in place since cron job starts the streams broken job. 
No permanent resolution since case is not reproducible on demand.

</pre>
<!-- Date: Fri Aug 12 00:35:52 2005 -->
<pre>
Name : cdfonprd_alert_log 
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : bzora1.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Aug 12 2005 12:09:49 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Aug 12 00:35:52 2005 -->
<pre>
Name : cdfofpr2_alert_log 
Target Name : cdfofpr2.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Aug 12 2005 12:02:30 AM 



</pre>
<!-- Author: oem-admin (email) -->