<!-- Title: 08:39:28  Fri Mar  4 2005 -->
<!-- Crewchief: New day -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>database activity</pre>


<!-- Date: Fri Mar 4 08:41:05 2005 -->
<pre>ISSUE:    reporting 2 bad disks on b0dau35 
 
Last night we did a block level recovery & were able 
to stay in business yet today richard confirms that 
we have 2 bad disks.   Maurine is contacting Decision One. 
These disks need to be replaced. 
 
The goal would be to get these to fail over to a spare 
yet it's not clear how we would do this & are investigation. 
 
We need to swap out these disks  -- today!  
 
 
 
we have another bad disk: 
 
Mar  4 01:06:58 b0dau35 unix: WARNING:   
/sbus@a,0/SUNW,socal@d,10000/sf@0,0/ssd@w22000004cfe6ee9c,0 (ssd136): 
Mar  4 01:06:58 b0dau35         Error for Command: write(10)              
    Error Level: Retryable 
Mar  4 01:06:58 b0dau35 unix:   Requested Block: 6469                     
    Error Block: 20923040 
Mar  4 01:06:58 b0dau35 unix:   Vendor: SEAGATE                           
    Serial Number: 3HZ0L11B 
Mar  4 01:06:58 b0dau35 unix:   Sense Key: Hardware Error 
Mar  4 01:06:58 b0dau35 unix:   ASC: 0x32 (no defect spare location   
available), ASCQ: 0x0, FRU: 0x4 
 
</pre>
<!-- Author: nstanfield  -->
<!-- Comment: Fri Mar  4 09;23;57 comment by...nstanfield  -->
<pre>he spoke bob wagner & frank c  the situation is the luminosity is not high, the store ends this mid-afternoon with ashot this evening, so if we need to do any major work take the downtime in the early afternoon - they will terminate the run & waste luminisoity but this is better then wasting first luminosity of a store

-frank c states they are not ready -- online consumers are not tested & they are a large part -- prefer not to rush  & basically need to limp along with this machine  - try to get in touch with D1 and sysadmin & understanding mirroring & disk replacement

plan is not to insert new machine even though it's ok they want to be comfortable & need to test the consumers </pre>
<!-- Comment: Fri Mar  4 09;43;21 comment by...nstanfield  -->
<pre>frank c will check with maurine since she is over at cdf & so is he.    they will concur on whether the disks are mirrored 136 & 137 or NOT ...   if they are not she will work with d1 & replace the disks failing each over to a hot spare -- if they are mirrored she will take a downtime at ~noon & replace them

</pre>
<!-- Comment: Fri Mar  4 10;30;31 comment by...nstanfield  -->
<pre>I am trying to get this escalated with Decision One.   I got a hold of Sue Mcnamara & she was able to reach Adam Walters & they are escalating the ticket!  

So we hope to have D1 at CDF soon.  </pre>
<!-- Comment: Fri Mar  4 10;53;49 comment by...ns  -->
<pre>Maurine & Decision One together now.   Frank C will touch base & let us know. </pre>
<!-- Comment: Fri Mar  4 13;16;28 comment by...nstanfield  -->
<pre>the issue has been escalated to sun for diagnosis.  solaris 
hasn't reported the problem up the software stack to veritas, so no one  here knows what to do.   maurine is sending files to sun 
</pre>
<!-- Comment: Fri Mar  4 13;30;16 comment by...mmihalek -->
<pre>this morning i went over to cdf around 8:45 am.

on b0dau35, i saw no warning lights, no yellow lights, no monkey wrenches, 
nothing on the led panels of the raid units, and no yellow lights on any disks 
to indicate two disk problems, or even one disk problem.

i looked in /var/adm/messages and found the ssd136 and ssd137 warnings. i went 
into format and found which disks these went with.  one was c0t2d0 and the 
other was c3t2d0.  per sergei, there is no c3 on b0dau35. since veritas is 
involved, i was not sure which disks had the problems, and with no yellow 
lights, it made it that much harder to figure out which disks supposedly 
needed to be replaced.

eventually, i was able to get sergei to start the veritas command. within 
veritas, sergei said there was no indication of 2 failed disks. also, i had 
neglected to remove sergei from the alias file, so he is still in there as a 
root entry. he said he did not get any emails to indicate a failed disk.  per 
mike miletec at d1, we run the risk of bringing the system down if the disk 
swap is not handled properly within veritas.  mike cannot figure out which 
disks to replace either.  there are no error lights anywhere. we looked at the 
/var/adm/messages file and mike says these are warning, not errors.

the concensus between nelly, richard, mike, me, sergei, and frank was to send 
the pertinent parts of the /var/adm/messages file to sun. mike has already 
started the process with an earlier email that i sent mike with some of the 
warnings in it.

i have sent mike a larger part of the messages file. he will forward it to his 
contact at sun. as soon as sun can diagnose the problems and let mike know 
what to do, mike will let us know.

maurine</pre>
<!-- Comment: Fri Mar  4 15;42;33 comment by...ns -->
<pre>i spoke to mike & he believe it's worth trying to call sun and
escalate this ..   he is not sure of what kind of s/w support we have
yet it's worth a shot -- this is what i have been trying to do since 1:15
or so ...escalate it  -- similar to what we do with oracle.....so we will
keep trying -- maurine & mike are here for the duration &
will work on this till it gets resolved

----- Original Message ----- 
From: "Maurine_Mihalek" &lt;mmihalek@fnal.gov&gt;
To: &lt;mmiletic@fnal.gov&gt;
Cc: &lt;dsg-sysadmin@fnal.gov&gt;; &lt;mmihalek@fnal.gov&gt;; &lt;awalters@fnal.gov&gt;
Sent: Friday, March 04, 2005 3:29 PM
Subject: #58976@D1 (Traditional/PREMIUM)


&gt; hi mike,
&gt; i know you have ordered 2 disks, have sent the information on to sun for 
&gt; analysis, and are awaiting the tech support recommendations.
&gt; 
&gt; nelly has asked me to see if there is anything you can do to escalate this 
&gt; hardware ticket with sun.
&gt; 
&gt; maurine
&gt;</pre>
<!-- Date: Fri Mar 4 09:11:01 2005 -->
<pre>We are going to rerun  the recovery on bzora1 with March 3 rman & all current archives.   This test is necessary to confirm we can recover our archives........Is this goes fine we will then blow this away & do a restore only-- pending applying any archives till we make a decision on hardware for b0dau35. </pre>
<!-- Author: nstanfield  -->
<!-- Comment: Fri Mar  4 13;54;16 comment by...nstanfield  -->
<pre>Completed successfully a recovery with rman from March 3 and archives up to March 4  1:23 p.m.!  </pre>



<!-- Date: Fri Mar 4 16:50:47 2005 -->
<pre>We have lost our window of opportunity for today 
for hardware maintenance on B0dau35.   
 
Our current plan is to proceed this is evening,  & if possible 
schedule D1 for Saturday during low luminosity, there 
is no guarantee that this will be scheduled since we have 
not been able to contact Sun support directly.   
 
Altho' if any of the following two scenarios  occur 
please contact Maurine Mihalek at  630-255-6520  
 
-if we lose store  
 
-if we have a catastrophic  failure   
 
and she will try to pursue  the disk swap. 
 
 
Hardware & Operating System contact is Maurine Mihalek 
on all occurence or Julie Trumbo  630-218-9560 
 
 
DBA's to contact once Hardware is resolved:   
Nelly Stanfield  630-266-6580 
Richard Jetton 630-266-8446</pre>
<!-- Author: ns -->

<!-- Date: Fri Mar 4 17:35:03 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 685 
WARNING 
Mar 04 2005 05:34:34 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Mar 4 17:37:31 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : WARNING 

User Defined SQL Test 
Current Value: 648 
WARNING 
Mar 04 2005 05:14:34 PM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Fri Mar 4 17:40:14 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CLEARED 

User Defined SQL Test 
Current Value: .9999999999999999999999999999999999999996 
CLEARED 
Mar 04 2005 05:39:34 PM 



</pre>
<!-- Author: oem-admin (email) -->



<!-- Date: Fri Mar 4 21:06:13 2005 -->
<pre>There is only 1 bad disk vs 2.  We learned Veritas provides dual path support with load balancing, so we're seeing the errors each device is accessed. i.e.  the operating system was reporting one failed disk, that happens to have 2  
paths from the host system to the raid array. 
 
We know exactly which disk  array0, slot 2 of the front disks 
 
Sun has provided us with an email with instructions on how to do the necessary steps for replacing the disk before Decision One physically replaces it.  They also will allow us to call them at the time we replace it, and they can walk D1 & Maurine through the procedure over the phone. 
 
 
We did not get any information from Sun allowing this to be done minus an interruption. So we will bring the database down during this effort.  And it requires a reboot to make the new disk recognized.  Estimating 2 - 3 hours. 
 
 
Sun states "if we want to change the disks to go ahead" and we feel strongly enough regarding the hardware errors we are receiving we should proceed with the swap as soon as possible.  
 
So if there is an open window of  2 - 3 hours please page 
for Hardware Support. ( No's below)  
 
Altho' if any of the following two scenarios  occur 
please contact Maurine Mihalek at  630-255-6520  
 
-if we lose store  
 
-if we have a catastrophic failure of the database    
 
and she will try to pursue the disk swap. 
 
 
Hardware & Operating System contact is Maurine Mihalek 
on all occurences or Julie Trumbo  630-218-9560 
 
 
DBA's to contact once Hardware is resolved:   
Nelly Stanfield  630-266-6580 
Richard Jetton 630-266-8446 
</pre>
<!-- Author: nstanfield  -->

<!-- Date: Sat Mar 5 00:04:47 2005 -->
<pre>
Target Name : cdfonprd.fnal.gov 
Target Type : Database 
Node Name : b0dau35.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Mar 05 2005 12:04:00 AM 



</pre>
<!-- Author: oem-admin (email) -->


<!-- Date: Sat Mar 5 00:34:36 2005 -->
<pre>
Target Name : cdfofpr1.fnal.gov 
Target Type : Database 
Node Name : fcdfora4.fnal.gov 
Status : CLEARED 

Alert 
Event test "Alert" is cleared 
CLEARED 
Mar 05 2005 12:03:48 AM 



</pre>
<!-- Author: oem-admin (email) -->