<!-- Title: 03:52:41  Tue Sep 16 2008 -->
<!-- Crewchief: Rick St.Denis -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>New Shift</pre>


<!-- Date: Tue Sep 16 04:24:15 2008 -->
<pre>Sent mail to dcache-admin (CC: cdf_dh_help) 
 
r-fcdfcache01-1 to 4 offline  pings ok 
r-fcdfdata318-1 to 4 offline  pings ok 
 
Sent mail to helpdesk and run2-sys (CC: dcache-admin,cdf_dh_help) 
r-fcdfdata325-1      offline  does not ping 
 
 
Will wait 4 hours after 9am = 15 hence at 19 this evening to send email on these 
LTO3_01  bad mover 
LTO4_81  bad mover 
LTO4_83  bad mover 
</pre>
<!-- Author: r -->
<!-- Comment: Tue Sep 16 05;19;07 comment by...r -->
<pre>The first problem with 01 and 318 is JIRA ticket CDFDH-258
The second with 325 is CDFDH-257</pre>


<!-- Comment: Tue Sep 16 09;35;51 comment by...r -->
<pre>Ling Ho commented on CDFDH-258:
-------------------------------

The machine has been shut off because of hardware failure. We are still 
waiting for replacement part.
(this is in fact 325 that is being talked about. Confused about which JIRA ticket is which)
</pre>
<!-- Comment: Wed Sep 17 01;33;50 comment by...r -->
<pre>ngela Bellavance commented on CDFDH-259:
-----------------------------------------

The fcdfcache machines are in the process of being brought up, so you can ignore them for now.
fcdfdata318 just got moved and probably the dCache services need to be restarted. I'll check.Angela Bellavance updated CDFDH-258:
------------------------------------

    Status: Waiting  (was: Open)

Machine is awaiting hardware replacement.

- Angela


</pre>
<!-- Comment: Wed Sep 17 01;34;53 comment by...r -->
<pre>Your request has been put into a pending status by the expert working on the problem.  

Details are provided below.
___________________________________________________________________
Ticket #: 121711
___________________________________________________________________
Pending Reason: Waiting For Parts
___________________________________________________________________
This ticket is assigned to HO, LING of the CD-SF/FEF group.
</pre>
<!-- Comment: Wed Sep 17 01;44;24 comment by...r -->
<pre>Angela Bellavance commented on CDFDH-259:
-----------------------------------------

fcdfata318 is back online.
- Angela
</pre>
<!-- Comment: Wed Sep 17 02;20;25 comment by...r -->
<pre>Bad mover list reduced to LTO3_01  bad mover  Other bad movers from yesterday are not indicating as bad
                    Will discuss at 9am meeting and send mail if Angela agrees</pre>
<!-- Comment: Wed Sep 17 14;55;32 comment by...r -->
<pre>fcdfcache01 is not in production yet. We started it temporarily in Test pools, it get registered and is
reported on cells web page since then.
We will be starting nodes fcdfcache01-09 today.

pool 318 is alive now.

Thanks for reporting it.
Alex.
</pre>
<!-- Comment: Wed Sep 17 14;59;00 comment by...r -->
<pre>Sent mail to enstore admin (cc cdf dh help) about lt03_01.  Been red for a day now</pre>
<!-- Comment: Wed Sep 17 15;26;21 comment by...r -->
<pre>
Thanks for your note.  LTO3_01 has been down since Friday because of a hardware issue.  A service called has been placed. </pre>
<!-- Date: Tue Sep 16 06:43:15 2008 -->
<pre>Subject: SAM on CAF Error: CAF JID 3564839 summary 
 
CAF  : CAF 
JID  : 3564839 
User : sedov 
Group: medium 
DH   : SAM 
Nr. segments: 1 
Nr. errors  : 1 
Submitted   : Tue Sep 16 06:11:43 2008 
Ended       : Tue Sep 16 06:15:40 2008 
Job duration:   0:03:57 
 
Starter         Message 
    SAM:        Removed 
 
segment      node        exit      Wait     Real     CPU    WaitDH        Read      Written   
</pre>
<!-- Author: r -->
<!-- Comment: Tue Sep 16 06;43;54 comment by...r -->
<pre>Subject: SAM on CAF Error: CAF JID 3564837 summary

CAF  : CAF
JID  : 3564837
User : sedov
Group: medium
DH   : SAM
Nr. segments: 1
Nr. errors  : 1
Submitted   : Tue Sep 16 06:11:19 2008
Ended       : Tue Sep 16 06:15:45 2008
Job duration:   0:04:26

Starter         Message
    SAM:        Removed

segment      node        exit      Wait     Real     CPU    WaitDH        Read      Written  

    [ Note: This message contains email list management information ]
Subject: SAM on CAF Error: CAF JID 3564836 summary

CAF  : CAF
JID  : 3564836
User : sedov
Group: medium
DH   : SAM
Nr. segments: 1
Nr. errors  : 1
Submitted   : Tue Sep 16 06:10:59 2008
Ended       : Tue Sep 16 06:15:49 2008
Job duration:   0:04:50

Starter         Message
    SAM:        Removed

segment      node        exit      Wait     Real     CPU    WaitDH        Read      Written  

</pre>


<!-- Date: Tue Sep 16 06:44:28 2008 -->
<pre>Subject: check_file_size_and_CRC 
 
Filesize mismatches 
 
diff in MiB          # 
----------- ---------- 
          0    3798329 
          1          1 
          6          1 
 
CRC mismatches 
 
  COUNT(*) 
---------- 
         2 
</pre>
<!-- Author: r -->


<!-- Date: Tue Sep 16 08:31:37 2008 -->
<pre>&gt;&gt; Hello, 
&gt;&gt; 
&gt;&gt; I have a problem with some of the files on SAM and I believe this is the 
&gt;&gt; right list to ask. I am running the CDF luminosity script and have found 
&gt;&gt; that some files in my dataset have something wrong with the metadata. 
&gt;&gt; Specifically, it looks as though a certain block of files have no parent 
&gt;&gt; or children files, like the CDF luminosity script assumes. It is 
&gt;&gt; impossible to get an accurate count of the luminosity with the metadata 
&gt;&gt; as it is. 
&gt;&gt; 
&gt;&gt; At the bottom of this message I have printed out some of the error 
&gt;&gt; messages, basically the script crashes when `sam dump file 
&gt;&gt; --filename=ei031c2b.0189tlpa` does not have a 'parents' entry like, I 
&gt;&gt; believe, it should. 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; Any help would be appreciated. Thank you! 
</pre>
<!-- Author: r -->
<!-- Comment: Tue Sep 16 08;32;03 comment by...r -->
<pre>Raymond Culbertson wrote:
&gt;     This is an ntuple file, indicated by the "ai" instead of "0i".
&gt; The parents of ntuple files have not been carefully maintained.
&gt; I didn't realize anyone was running luminosity scripts on the
&gt; ntuple datasets.  I'm also not sure what script you are running, 
&gt; can you point me to the instructions for it?  I don't think
&gt; SamLumi.ksh, lumsum.pl or dpslum.sh work this way. Thanks.
&gt;                                       Ray
</pre>


<!-- Comment: Wed Sep 17 01;34;16 comment by...r -->
<pre> OK, I forgot about the top group method.  This is one of the
offical top group methods which we should support.  However my 
next point would be that the file you had trouble with is a Stntuple,
not a topntuple.  Stntuple has not normally supported this script.
In order to clean up the current problem, we would want to go through
the dataset (or as many as you need) and check the parents 
against the parent datasets and resolve any problems by modifying
the sam records.  It might not always be possible to resolve it
completely unambguously, depending on how widespread the
problems are.  
   Here is what most Stntuple users do - let me know if this 
OK for you.  For a dataset, for example, etlpai, you check the
total number of events in it (the number you ran over in your 
processing) to the total in the parent dataset 
sam list files --summaryOnly --dims="cdf.dataset etlp0i"
If those numbers are close (they should be within a few events),
then you know you have the whole dataset.  At this point,
you can sum the database luminosity for the runs in 
your goodrun list, using SamLumi.ksh, linked from here
http://www-cdf.fnal.gov/internal/dqm/goodrun/good.html
There are failure modes for this calculation, but no 
more scary than than other methods.  It should be accurate to 
0.1% or better.
                                      Ray
</pre>
<!-- Date: Tue Sep 16 08:32:25 2008 -->
<pre>i Ray, 
 
Thanks for the quick reply. Here are the script instructions: 
http://www-cdf.fnal.gov/internal/physics/top/RunIITopProp/gen6Sum06/instructions_scripts2.html 
And getLumiInfo.pl: 
http://www-cdf.fnal.gov/internal/physics/top/RunIITopProp/gen6Sum06/lumifiles/scripts/getLumiI 
nfo.pl 
I believe these are the scripts that the top group uses for their 
default numbers. 
 
 
I guess I defaulted to running lumi scripts on ntuple files and not 
production files because they are the actual files the analysis will use. 
 
 
Cheers, 
Rob 
</pre>
<!-- Author: r -->
<!-- Comment: Wed Sep 17 01;37;43 comment by...r -->
<pre>Hi Ray,

Yes, I'm using top tools for stntuples, but the Top group has such nice
tools, I use them as much as possible :) Ideally, the files would be
fixed, but if it takes some effort, I'll see if I can avoid that.

As for using the other datasets, here are the event counts for the same
data, different datasets:
etlp0i: 51610643
etlpmi: 76057863
etlpai: 73929558

So the top dataset 'mi' is off enough from my stntuple dataset 'ai' to
make me wonder. I don't know enough to say why the event numbers very so
much.

As for switching tools, I could do this, but the top tools are very nice
for a bunch of reasons. Using them would save a lot of time, and I think
get a more accurate answer (I can fix nonsensical dps fractions by hand).

Thanks again for your help.

Cheers,
Rob
</pre>


<!-- Comment: Wed Sep 17 01;44;41 comment by...r -->
<pre> The largest descrepancy is due to the fact that 0i was
originally P5-10, 203819-233111.  P10 had a bug
so it was reprocessed as 0j and the P10 0i files
were deleted.  Meanwhile the ntuples were run over 0i 
in P10, writing the data to ai and mi, and they happened 
to fix the bug.  So now we are in the situation where
the official production for P10 is in 0j but he ntuples are labeled i.
If you look in P5-P9 in 0i, you get better numbers
0i 51610643
ai 51609814
mi 51396263
and in P10:
0j 22325571
ai 22319744
mi 24661600
    Note that this is presented on the Stntuple 
datasets web page in a way that you don't need to know 
these details.
    I don't know why P5-9 mi is off by 0.4%.  Most likely 
there were a few files which were unreable, but this is 
an unusually large decrepancy.  I was not in charge of the 
top processing at that time, and since etlpmi is not one 
of the most important datasets to top, and this is long 
in the past, I am not immediatly moved to fix it.
Let me know if you object to that.
    The 10% descrepancy in P10 is more serious and I 
would ask Sasha Golossanov to look into it.  It may 
be that the sam listing is not the same as the file listing 
recommended (which I actually do not see on the top web page..)
                                   Ray
</pre>

<!-- Comment: Wed Sep 17 01;45;31 comment by...r -->
<pre>Hi Ray

This makes a lot more sense now. I agree with you that it is really not
a priority to fix the p5-9 mi dataset, but it is good to know that the
data has been moved like this. You probably saved me some frustration
trying to figure this out on my own. While you investigate, I will wait
for my Lumi scripts to run, and see if the output is good enough to use.

Thanks again.

Cheers,
Rob
</pre>


<!-- Date: Wed Sep 17 01:48:30 2008 -->
<pre>Subject: Re: Sept 18 Computing Downtime reminder 
 
We have suspended starts of jobs from medium queues earlier today. This affect 
most jobs except of special groups involved with offline tasks. Please refrain 
from submitting long and medium jobs unless your jobs are one of those special jobs categories 
and you know their run time as they either be waiting or will crash when the worker nodes go 
down for the power outage after 6am on Thursday. 
 
We had extended the time limit of short jobs to 12 hours till tomorrow when we will revert it 
back to 6 hours when we suspend the short queues. 
 
The medium job limit was set to 36 hours for all the three cafs. 
</pre>
<!-- Author: r -->


<!-- Date: Wed Sep 17 01:50:39 2008 -->
<pre>The diskpool node fcdfrdc3 has been repaired and the door configuration is back to normal. 
Be advised, however, that tomorrow two diskpool fileservers with ~64TB of disks will be moved 
from FCC go GCC, which will make some diskpool data unavailable during that transition. 
 
Alexei Varganov 
</pre>
<!-- Author: r -->



<!-- Date: Wed Sep 17 02:21:22 2008 -->
<pre>Wed Sep 17 02:15:43 2008 1221635743 Warning: 
b0csl23 -- Csl_fetch encountered problems 
Tue Sep 16 11:05:46 2008 1221581146 Action: Experts have been notified, please wait 
Tue Sep 16 15:10:44 2008 1221595844 Action: Experts have been notified, please wait 
Tue Sep 16 19:10:44 2008 1221610244 Action: Experts have been notified, please wait 
Tue Sep 16 23:10:45 2008 1221624645 Action: Experts have been notified, please wait 
</pre>
<!-- Author: r -->


<!-- Date: Wed Sep 17 02:46:15 2008 -->
<pre>restarted SAMTV and put log in log-save-rick-Oct172008</pre>
<!-- Author: r -->


<!-- Date: Wed Sep 17 10:07:55 2008 -->
<pre>We have set the fermigrid caf into draining mode yesterday evening. 
 
We have done this as our method of a selective shutdown had been observed not 
to work on some of the worker nodes of that caf although it did work on namcaf (and is the 
standard for groupcaf). 
 
We decided to revert to the method of draining fermigrid used for the previous downtimes. The 
method suspends the starts of all the jobs to avoid a large number of jobs crashing when the 
worker nodes go down for the power outage. (Which is scheduled for tomorrow morning.) 
 
We apologize for any inconvenience this may cause. 
 
Krzysztof 
</pre>
<!-- Author: r -->


<!-- Date: Wed Sep 17 14:52:15 2008 -->
<pre>We are experiencing a failure of the SL8500 robot at FCC.  The CDF and D0 LTO4 tape libraries have been 
paused and a service call has been placed with Sun.  Updates will be sent as necessary. 
</pre>
<!-- Author: r -->
<!-- Comment: Wed Sep 17 14;53;38 comment by...r -->
<pre>IAG211 is in the drive.</pre>





<!-- Comment: Thu Sep 18 01;29;59 comment by...r -->
<pre>
The CDF and D0 LTO4 libraries remain paused.  The problem has been traced to a faulty robot arm, and replacement of the arm is underway.  I will send another update when access to the libraries is restored.
 (sent at 12:00 am CDT)</pre>
<!-- Comment: Thu Sep 18 01;30;44 comment by...r -->
<pre>Hello,

The CDF and D0 LTO4 libraries were returned to service at approximately 6:30pm.  Thanks for your patience.
</pre>
<!-- Date: Thu Sep 18 01:28:15 2008 -->
<pre>All, 
 
fcdfora4 will be Oracle and OS patched tomorrow, 9/18/08. Downtime will be for 4 hour starting at 8:30am. 
 
Steven Kovich </pre>
<!-- Author: r -->


<!-- Date: Thu Sep 18 01:29:11 2008 -->
<pre>The short queue limit was reverted back to 6 hours for all three cafs. 
 
We will suspend job starts for all remaining job categories after 
10pm tonight. 
 
Krzysztof </pre>
<!-- Author: r -->




<!-- Date: Thu Sep 18 01:31:51 2008 -->
<pre>Job starts have been suspended for all three CAFs 
(and submissions have been blocked for fermigrid and namcaf). 
 
We will send announcements about CAFs status tomorrow afternoon. 
 
Krzysztof (for the CAF team) 
 
4:50 AM BST (10:50pm CST)</pre>
<!-- Author: r -->


<!-- Date: Thu Sep 18 15:10:21 2008 -->
<pre>Cell page reports pnfs manager down. 
I can not ping cdf pnfs server on virtual IP or any of pnfs servers directly 
at  cdfensrv[01234]n . 
Was it shut down on purpose in connection with power outage ? 
Alex. 
(sent at 9:30am CST)</pre>
<!-- Author: r -->


<!-- Date: Thu Sep 18 15:15:17 2008 -->
<pre> 
The work on the CDF web server has been completed. All web pages should now 
be accessible. 
 
Please report any problems to cdf_software_help@fnal.gov 
 
(sent at 10:11 CDT)</pre>
<!-- Author: r -->
<!-- Comment: Thu Sep 18 15;20;20 comment by...r -->
<pre>There appears to be a problem accessing the web server from at least some
off-site locations. We're investigating the problem and will send out
another announcement as soon as we know more.

I apologize for the additional delay,
(sent 11:42 CST)</pre>


<!-- Comment: Thu Sep 18 15;22;03 comment by...r -->
<pre>Problem accessing top internal web pages
----------------------------------------

Hi Rick,

  Were known changes made to the behavior of the internal physics webserver?
The shtml include on the top group page seems to no longer work?

Thanks,
  -Craig
</pre>
<!-- Comment: Thu Sep 18 15;23;09 comment by...r -->
<pre>Hello

I just tried posting a CDF Note update but got an error. It said one of
the scripts in cgi-bin was not available.
Is this related to the web-server work this morning?

Thank you,
Conor Henderson.
(sent 12:50 CDT)</pre>
<!-- Comment: Thu Sep 18 15;25;48 comment by...r -->
<pre>Off-site access to the web server has been restored. We apologize for the
delay.

It you still have a problem accessing the web pages, then the most probable
cause is that your nameserver has not been updated with the IP address of
the new web server. This situation should clear by itself. You should
contact your local system adminstrators if you have additional questions
about the update schedule for your DNS.

Rick
(sent 14:09 CDT)</pre>
<!-- Comment: Thu Sep 18 15;28;13 comment by...r -->
<pre>     Assignee: Rick Snider  (was: Simona Rolli)
    Resolution: Fixed

We found two problem preventing access. The first was that the border router
at Fermilab was blocking access to the web page. This has been fixed. 

The second was that, as of late morning, some nameservers had not yet been
updated with the IP address of the new web server. This should clear itself,
although the timescale for this is completely out of our control. The
translation for www-cdf.fnal.gov should be 
"fcdfweb.fnal.gov" with address 131.225.208.115". If this it not the case
for you, then the problem is your local nameserver. You should contact your
local sys admin if this is the case.
</pre>
<!-- Comment: Thu Sep 18 15;30;12 comment by...r -->
<pre>There was also a report of the twiki being down but that was fixed by Glenn rather quickly at about 12:00 CST</pre>
<!-- Comment: Thu Sep 18 16;06;27 comment by...r -->
<pre>www-cdf user database search problem
------------------------------------

Searches using the "Search any field" don't work.

</pre>
<!-- Comment: Thu Sep 18 17;15;35 comment by...r -->
<pre>
   This should be OK since there is a soft link:

ls -al /usr/local/bin/python
lrwxrwxrwx  1 root root 15 Aug  2  2006 /usr/local/bin/python -&gt; 
/usr/bin/python


Anyway, I changed it and I still get this error:

&lt;!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"&gt;
&lt;html&gt;&lt;head&gt;
&lt;title&gt;500 Internal Server Error&lt;/title&gt;
&lt;/head&gt;&lt;body&gt;
&lt;h1&gt;Internal Server Error&lt;/h1&gt;
&lt;p&gt;The server encountered an internal error or
misconfiguration and was unable to complete
your request.&lt;/p&gt;
&lt;p&gt;Please contact the server administrator,
  run2-sys@fnal.gov and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.&lt;/p&gt;
&lt;p&gt;More information about this error may be available
in the server error log.&lt;/p&gt;
&lt;/body&gt;&lt;/html&gt;


This has been working fine for more than a year..

Cheers,
   -Craig
</pre>
<!-- Comment: Fri Sep 19 02;20;21 comment by...r -->
<pre>Sasha made fcdfwww -&gt; www-cdf changes, but we still have a few problems.
First, the access to:
http://www-cdf.fnal.gov/physics/new/qcd/QCD_DetailedList.shtml
is not behaving right. This file use "server side includes" (SSI), but 
this doesn't seem
to be working. Secondly, the access to:
http://www-cdf.fnal.gov/internal/physics/qcd/minutes/
used to show all the text files in the /cdf/www/internal/physics/qcd/minutes
directory, but this doesn't work any longer. Is the httpd server 
configuration changed?
Can you enable SSI and indexing options? Or, at least allow us to 
control these via
.htaccess?

Best Regards,
Sasha and Ken
</pre>
<!-- Date: Thu Sep 18 15:20:56 2008 -->
<pre>All, 
 
fcdfora4 is back online after scheduled download. 
 
Steven Kovich 
 
Sent: 11:28 CDT</pre>
<!-- Author: r -->




<!-- Date: Thu Sep 18 15:23:51 2008 -->
<pre>Hello, 
 
Enstore was placed back in service at approximately 1:20pm, a bit later than 
anticipated.  I see that an LTO3 transfer has taken place. 
 
Please let us know if you encounter any lingering issues.  Thank you. 
(sent 13:36 CDT)</pre>
<!-- Author: r -->


<!-- Date: Thu Sep 18 15:24:40 2008 -->
<pre>Subject: COMPLETE_FILE_LISTING_cdf is not updating! 
 
Last update was: 
 
-- Listed at Thu Sep 18 06:37:17 2008 
-- 
-- STORAGE GROUP: cdf 
-- 
storage_group file_family volume location_cookie bfid size crc path 
(4031176 rows) 
 Thu, 18 Sep 2008 13:42:42  
 
</pre>
<!-- Author: r -->



<!-- Date: Thu Sep 18 15:26:30 2008 -->
<pre>Thu Sep 18 14:30:01 CDT 2008 
 
Thu Sep 18 14:30:01 CDT 2008 ERROR 
 
A /bin/ps axuwww did not contain 
    postgres: enstore dcache 131.225.211.78|postgres: enstore srm 
131.225.211.78|postgres: enstore replicas 131.225.211.78 
on node fcdfdcache4 
 
This means that dcap transactions are not being logged to the database. 
 
Things to check: 
 
  Check if postgres is running on fcdfdcache4 
     If not, then restart using  
        /etc/rc.d/init.d/postgres-boot stop 
        /etc/rc.d/init.d/postgres-boot start 
 
  Check if log indicates some kind of broken network pipe to postgres 
     If broken, then httpd domain needs to be restart, on head admin node: 
        /etc/rc.d/init.d/dcache-boot stop  fifo httpd 
        /etc/rc.d/init.d/dcache-boot start fifo httpd 
</pre>
<!-- Author: r -->
<!-- Comment: Thu Sep 18 15;27;10 comment by...r -->
<pre>     Assignee: Rick Snider  (was: Simona Rolli)
    Resolution: Fixed

We found two problem preventing access. The first was that the border router
at Fermilab was blocking access to the web page. This has been fixed. 

The second was that, as of late morning, some nameservers had not yet been
updated with the IP address of the new web server. This should clear itself,
although the timescale for this is completely out of our control. The
translation for www-cdf.fnal.gov should be 
"fcdfweb.fnal.gov" with address 131.225.208.115". If this it not the case
for you, then the problem is your local nameserver. You should contact your
local sys admin if this is the case.
</pre>


<!-- Comment: Thu Sep 18 15;28;53 comment by...r -->
<pre>Sorry: the last posting on the webserver issues was put to the wrong comment point. Ignore it here and read above.</pre>



<!-- Date: Thu Sep 18 15:48:50 2008 -->
<pre>My apologies for not sending mail earlier. 
 
dCache is available for tests. 
</pre>
<!-- Author: r -->


<!-- Date: Thu Sep 18 15:52:05 2008 -->
<pre>The DH at a glance shows no new problems. The hardware problems for 325 and the LT03 tape drive problem remain.  There is however no word on CAF yet so there is probably not much happening.</pre>
<!-- Author: r -->



<!-- Date: Thu Sep 18 17:14:42 2008 -->
<pre>It looks like some job from  host 131.225.234.75 (nkek22.fnal.gov) is 
constantly trying to access kerberized dcap doors. It has bad credential and 
fails. 
</pre>
<!-- Author: r -->
<!-- Comment: Fri Sep 19 02;17;27 comment by...r -->
<pre>
Thanks Alex!
I talked to the user and he is going to stop his jobs, kinit and
run them again. (They are running on the machine since over a day.)
Thanks for the information,
                               cheers,  Stephan
(posted at 17:50 yesterday)</pre>
<!-- Date: Fri Sep 19 02:13:10 2008 -->
<pre>IAG176            411.12GB   (NOACCESS   0918-2140 none              )   CDF-LTO4F1       cdf.bpel0m.cpio_odc                  
</pre>
<!-- Author: r -->


<!-- Date: Fri Sep 19 02:14:00 2008 -->
<pre>one tape no access but otherwise all looks fine</pre>
<!-- Author: r -->



<!-- Date: Fri Sep 19 02:19:13 2008 -->
<pre>Posted at 18:15 yesterday 
 
Dear colleagues, 
 
Group CAF and FermiGrid CAF are now up and running, their queues are open 
for 
job submission.  Not all nodes that were  available before the downtime  
are back due to hardware failures that occurred  during power cycling.  
As those nodes become available they will be added to the system. 
 
The NAMCAF is up and running with remote worker nodes, the onsite nodes are  
still experiencing network problems.  When these issues are resolved  
they will be added. 
 
We apologize for any confusion that occurred last night about  
queue availability. The development team took the opportunity  
to run load tests after the public queues were closed, knowing  
that many of the jobs they submitted would be lost during the  
instabilities caused by power cycling.   
 
The CAFs are now back in a stable configuration, please put them to 
good use.  
 
Regards, 
Dennis Box 
(for the  CDF CAF/Grid teams) 
</pre>
<!-- Author: r -->



<!-- Date: Fri Sep 19 02:20:56 2008 -->
<pre>Subject: ALERT from cdf-fncdfsrv0.SM 
 
09/18/08 18:54:29 cdf-fncdfsrv0.SM.FileUncachedInDBTimer 13958: Cannot 
uncache file in DB: 
CORBA::UserException caught in FileUncachedInDBTimer::doIt() 
 
</pre>
<!-- Author: r -->
<!-- Comment: Fri Sep 19 02;26;48 comment by...r -->
<pre>Got one of these messages every 10 minutes until 21:14. Then got
09/18/08 21:14:30 cdf-fncdfsrv0.SM.FileUncachedInDBTimer 13958: Cannot
uncache file in DB:
CORBA::UserException caught in FileUncachedInDBTimer::doIt()

as well.


Then one last message:

09/18/08 21:20:11 cdf-fncdfsrv0.SM.FileUncachedInDBTimer 12777: Warning,
will ignore CachedFileNotFound exception for CFID 42248056:
Cached file with id '42248056' not found.


It went quiet after that.
</pre>


<!-- Date: Fri Sep 19 09:19:21 2008 -->
<pre>Hi Angela, 
multiple dcache pool nodes trying to stage file from tape IAG176. It is marked NOACCESS. 
Are these requests coming from production jobs or is it from yesterdays tests ? 
Can you cancel these requests ? 
Alex. 
</pre>
<!-- Author: r -->


<!-- Date: Fri Sep 19 13:31:00 2008 -->
<pre>NOTALLOWED 0919-0940 none              )   CDF-LTO4F1       cdf.bpel0m.cpio_odc                  Tape in drop-off in robot - tech coming 9-19-08 
 
</pre>
<!-- Author: r -->
<!-- Comment: Fri Sep 19 14;45;16 comment by...r -->
<pre>
Due to a volume being stuck in an unobtainable (via console commands), we
need to pause the libraries for the SL8500 in FCC (D0-LTO4F1 and CDF-LTO4F1)
to retrieve this tape and re-insert it into production.

There will be a short - 15 minute or so - pause of those libraries
immediately. Running jobs should not be affected by this and submitted jobs
should be put in queue.

Thanks for your understanding and patience.

Stanley
</pre>
<!-- Comment: Fri Sep 19 15;15;36 comment by...r -->
<pre>
Sorry for the delays. Both libraries are un-paused and back in production.

Thanks for your patience.
Stanley
(sent 15:08)</pre>
<!-- Date: Fri Sep 19 13:33:44 2008 -->
<pre>PnfsManager 	pnfsDomain 	OFFLINE 
The machine itself is down.</pre>
<!-- Author: r -->
<!-- Comment: Fri Sep 19 13;41;29 comment by...r -->
<pre>PnfsManager 	pnfsDomain 	0 	21 	179 msec 	09/19 13:31:06 	production-1-7-0-1-7-0-48-20071205082024(1.44)

This is running again.</pre>


<!-- Date: Fri Sep 19 13:50:17 2008 -->
<pre>Pnfs manager down again 
 
 
PnfsManager 	pnfsDomain 	OFFLINE</pre>
<!-- Author: r -->
<!-- Comment: Fri Sep 19 13;59;57 comment by...r -->
<pre>This is chronic - Stefan and Dmitri are investigating. </pre>


<!-- Comment: Fri Sep 19 15;16;18 comment by...r -->
<pre>Yesterday evening our PNFS filesystem/server hung. Service was
restored quickly but an additional outage occurred during the
night. This caused SAM and dcache jobs to time out.
Today we experienced intermittent outages. The storage team has
traced the problem back to the server hosting the filesystem and
is in the process of replacing it.
We are holding all data handling jobs on groupCAF and FermiGrid
until the new machine is in place and functioning correctly.
We expect the problem to be resolved before the evening.
                                                cheers,  Stephan
</pre>
<!-- Comment: Fri Sep 19 16;01;46 comment by...r -->
<pre>
I believe the h/w problem was rectified. PnfsManager is up since 
3:44 PM. Things should go back to normal. 
Dmitri</pre>
<!-- Comment: Sun Sep 21 05;40;08 comment by...r -->
<pre>
There is some question as  to what caused this server problem:
===============================================================


Dmitry,
after unpausing library the encp error had changed.

Real-encp reports error in log on pool node :
   "Noticed the local file changed: [Errno 2] No such file or directory:
'/export/data2/read-pool-2/data/00170000000000000354DBE8'"
Well, it is reading from tape - why it shall be on disk.

More serious thing, access to /pnfs/fs on pool nodes (at least on fcdfdata138,
150) is hanging - but I can access pnfs IDs in pnfs on pnfs node itself.
Mounts hang ? I can not do ping to virtual IP cdfensrv1.
Can you have a look ?
Alex.


Alex Kulyavtsev wrote:
&gt; So what was the root cause problem causing reboots ? It was ~ four or so for
&gt; the last 24 hours. Alex.

</pre>
<!-- Date: Fri Sep 19 14:01:42 2008 -->
<pre>Angela restarted b0csl28 fss (in warning, then fss was gone). Noticed b0csl29 also was then yellow, so restarted that.</pre>
<!-- Author: r -->
<!-- Comment: Fri Sep 19 14;15;29 comment by...r -->
<pre>b0csl29 also was restarted.  Seems ok now.
Note: shifters cannot do this, must let Angela know.</pre>



<!-- Date: Fri Sep 19 14:49:21 2008 -->
<pre> 
 
dbs_cdf_user_srv1_prd dbs died at fcdfsam1: status 139 
 
New mail will not be send if the server crashes again  
within the same hour. 
&lt;HangMonitor 19-Sep-2008 12:45:17 (CDT)&gt; Pinging server 'SAMDbServer.dbs-cdf-user-srv1-prd:SAMDbServer' with timeout of 180 seconds ... 
&lt;HangMonitor 19-Sep-2008 12:45:17 (CDT)&gt; Ping was successful.  Waiting 600 seconds until the next scan. 
&lt;HangMonitor 19-Sep-2008 12:55:17 (CDT)&gt; Pinging server 'SAMDbServer.dbs-cdf-user-srv1-prd:SAMDbServer' with timeout of 180 seconds ... 
&lt;HangMonitor 19-Sep-2008 12:55:17 (CDT)&gt; Ping was successful.  Waiting 600 seconds until the next scan. 
&lt;HangMonitor 19-Sep-2008 13:05:17 (CDT)&gt; Pinging server 'SAMDbServer.dbs-cdf-user-srv1-prd:SAMDbServer' with timeout of 180 seconds ... 
&lt;HangMonitor 19-Sep-2008 13:05:17 (CDT)&gt; Ping was successful.  Waiting 600 seconds until the next scan. 
&lt;HangMonitor 19-Sep-2008 13:15:17 (CDT)&gt; Pinging server 'SAMDbServer.dbs-cdf-user-srv1-prd:SAMDbServer' with timeout of 180 seconds ... 
&lt;HangMonitor 19-Sep-2008 13:15:17 (CDT)&gt; Ping was successful.  Waiting 600 seconds until the next scan. 
&lt;HangMonitor 19-Sep-2008 13:25:17 (CDT)&gt; Pinging server 'SAMDbServer.dbs-cdf-user-srv1-prd:SAMDbServer' with timeout of 180 seconds ... 
&lt;HangMonitor 19-Sep-2008 13:25:17 (CDT)&gt; Ping was successful.  Waiting 600 seconds until the next scan. 
&lt;HangMonitor 19-Sep-2008 13:35:17 (CDT)&gt; Pinging server 'SAMDbServer.dbs-cdf-user-srv1-prd:SAMDbServer' with timeout of 180 seconds ... 
&lt;HangMonitor 19-Sep-2008 13:35:17 (CDT)&gt; Ping was successful.  Waiting 600 seconds until the next scan. 
&lt;HangMonitor 19-Sep-2008 13:45:17 (CDT)&gt; Pinging server 'SAMDbServer.dbs-cdf-user-srv1-prd:SAMDbServer' with timeout of 180 seconds ... 
&lt;HangMonitor 19-Sep-2008 13:45:17 (CDT)&gt; Ping was successful.  Waiting 600 seconds until the next scan. 
&lt;HangMonitor 19-Sep-2008 13:55:17 (CDT)&gt; Pinging server 'SAMDbServer.dbs-cdf-user-srv1-prd:SAMDbServer' with timeout of 180 seconds ... 
&lt;HangMonitor 19-Sep-2008 13:55:17 (CDT)&gt; Ping was successful.  Waiting 600 seconds until the next scan. 
&lt;HangMonitor 19-Sep-2008 14:05:17 (CDT)&gt; Pinging server 'SAMDbServer.dbs-cdf-user-srv1-prd:SAMDbServer' with timeout of 180 seconds ... 
&lt;HangMonitor 19-Sep-2008 14:05:17 (CDT)&gt; Ping was successful.  Waiting 600 seconds until the next scan. 
&lt;HangMonitor 19-Sep-2008 14:09:35 (CDT)&gt; I have received the signal number 15 and am dying now. 
&lt;HangMonitor 19-Sep-2008 14:09:35 (CDT)&gt; HangMonitor has been killed and is exiting now. 
</pre>
<!-- Author: r -->




<!-- Date: Fri Sep 19 16:00:54 2008 -->
<pre> 
Dear diskpool users, 
 
The diskpool fileservers fcdfdpool01 and fcdfdpool02 have been successfully 
transitioned to new GCC housing and commissioned back into the system. The 
diskpool is back to normal. 
 
Enjoy, 
Alexei Varganov 
</pre>
<!-- Author: r -->



<!-- Date: Fri Sep 19 16:04:25 2008 -->
<pre>LTO3_20 is now a bad mover.  But this will have to wait to Monday since we will not make 4hr in a business day. 
 
Also, files to restore is growing. But now the pfns server is fixed so perhaps this will work. Need to watch this. 
</pre>
<!-- Author: r -->


<!-- Date: Sun Sep 21 05:33:37 2008 -->
<pre>LTO4_80 now bad</pre>
<!-- Author: r -->
<!-- Comment: Mon Sep 22 08;43;33 comment by...rick -->
<pre>ok now.</pre>
<!-- Date: Sun Sep 21 05:45:49 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-79&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-79  width='300' align='top'></a></td><td>I was worried on Friday night when I saw the spike in files requested. I was watching this after the pnfs database server problems.  I mailed Angela suggesting some pools be added to the raw pool as the problem looked like it was running away. Not sure if that was done, but from the graph it can be seen that the requests dropped quickly. It seems odd that they practically went to zero and then went to a constant set.  I do wonder what was going on and hope this can be discussed at Monday's brief meeting.</td></tr></table>
<!-- Author: rick -->


<!-- Date: Sun Sep 21 17:00:21 2008 -->
<pre> 
There were two problems with cafs earlier this weekend: 
 
.) namcaf had a kerberos ticket renewal problems Fr/Sat which lead to many 
sections e.g. not being able to transfer output files to their destinations. 
 
.) groupcaf head node rebooted earlier today (no obvious reasons known so far) 
which lead to many sections crashing. 
 
The namcaf problems were corrected and groupcaf was restarted. All three cafs 
(groupcaf, fermigrid, namcaf) should be operational right now. 
 
Please report other problems seen using jira: 
http://cdfcaf.fnal.gov/jira/jira.html 
 
Krzysztof (for the CAF Team) 
</pre>
<!-- Author: r -->


<!-- Date: Sun Sep 21 17:07:25 2008 -->
<pre>Door 2 being hit a bit hard but plot not available.\ 
 
cdfopr_20080921_123042(1132411)  
seems to have 2 files requested (total) and not delivered. 
But this looks like  a test.  Will ask krzysztof. 
</pre>
<!-- Author: r -->


<!-- Date: Mon Sep 22 06:37:27 2008 -->
<pre>Doors traffic subsided.   
SAMTV seems stuck, but only at 2h. will wait a bit. 
Otherwise, no other new complaints.</pre>
<!-- Author: r -->
<!-- Comment: Mon Sep 22 08;42;21 comment by...r -->
<pre>samtv updating ok</pre>



<!-- Date: Mon Sep 22 08:44:29 2008 -->
<pre>LT03_20 is ok but now these are bad. 
 
LTO3_01 
LTO3_30 
 
Wait 4 hours.</pre>
<!-- Author: rick -->
<!-- Comment: Tue Sep 23 05;36;39 comment by...r -->
<pre>Sent mail on lt03_30</pre>
<!-- Comment: Tue Sep 23 06;21;21 comment by...r -->
<pre>This problem has already been reported to the vendor and should be resolved soon.

Thanks for bringing it to our attention, though.

Stanley

Richard St. Denis wrote:
&gt; Hi
&gt;  The mover LTO3_30 has been bad since Sunday. Can you please have a look at it today?
&gt; Thanks
&gt; Rick
&gt; (Sam Shifter)
</pre>
<!-- Comment: Tue Sep 23 06;23;10 comment by...r -->
<pre>This is CDFDH-270</pre>
<!-- Comment: Tue Sep 23 11;28;29 comment by...r -->
<pre>
Rick,
      We have found a problem with the fiber channel communications. We have
the vendor in this morning taking a look at the drive itself. What we have
is that there is a light path going to the drive but no light path returning
form it. We have swapped out the cable and it did not fix the problem. The
vendor was called in after we did that. I will update you as soon as we find
out something. Thank you for bringing this and have a good day.

Glenn Beagle
</pre>
<!-- Date: Mon Sep 22 10:04:54 2008 -->
<pre>Summary(thus far)1.  CDFDH-258/ Help desk ticket 121711 open 
Angela Bellavance updated CDFDH-258:  Ticket #: 121711 
------------------------------------ 
r-fcdfdata325-1  
    Status: Waiting  (was: Open) 
 
Machine is awaiting hardware replacement. 
 
2. CDFDH-257/259 
=========== 
r-fcdfcache01-1 to 4 offline  pings ok  
r-fcdfdata318-1 to 4 offline  pings ok  
The fcdfcache machines are in the process of being brought up, 
 so you can ignore them for now. 
fcdfdata318 just got moved and probably the dCache services need  
to be restarted. I'll check.Angela 
 
pool 318 is alive now. 
 
3. LT03_01 hardware problem: still a problem 
============================================ 
Thanks for your note.  LTO3_01 has been down 
 since Friday because of a hardware issue.  A service called has been placed.  
 
4. Lumi Scripts 
--------------- 
User running Lumi script asked about getting LUMI on top ntuple. 
There are a variety of these lumi scripts and the best is one that 
Randolph put together. We can mail it 'round but would be best if  
supported somewhere.  Also, these scripts can be hard on the offline db 
if not used properly. 
 
5. Downtime and recovery 
------------------------ 
fcdfora4 was back at 11:28 
Enstore was back at 13:20 
dCache was available at 15:48 
CAF was up at 18:00 
CDF Web pages had a lot of items to be fixed up. 
 
Noted lots of various other problems. We had to restart two csl fss's (not 
sure if related to downtime).   
 
Server of pnfs database failed and required attention.  
 
There was a rapidly growing demand for tapes but that dropped off quickly. 
 
7. Tape in  drop-off 
NOTALLOWED 0919-0940 none              )   CDF-LTO4F1    
    cdf.bpel0m.cpio_odc                 
 Tape in drop-off in robot - tech coming 9-19-08  
 
8. Misc. 
======== 
Some filesize mismatches 
 
Mail from "herber" instead of a common account 
 
Diskpool nodes repaired.  
The diskpool fileservers fcdfdpool01 and fcdfdpool02 have been successfully  
transitioned to new GCC housing 
 
Restarted SAM TV once 
 
One stuck job cdf_oper. What does this mean? 
 
A job from Japan was pounding on dcache - Stefan Lammel talked to the user 
and that was fixed. 
 
Had this error 09/18/08 18:54:29 to 09/18/08 21:14:30 every 10 min then it 
went away: 
09/18/08 21:14:30 cdf-fncdfsrv0.SM.FileUncachedInDBTimer 13958: Cannot 
uncache file in DB: 
CORBA::UserException caught in FileUncachedInDBTimer::doIt() 
 
 
 
</pre>
<!-- Author: r -->


<!-- Date: Tue Sep 23 05:36:12 2008 -->
<pre>The robot arm at GCC has a problem; we have paused the LTO3 libraries for cdfen and stken and the LTO4 
library for stken for the SL8500G1. We have contacted the help desk and they are contacting the vendor 
to have somebody come out and investigate/replace the arm. 
 
We will notify the respective experiments when the libraries are back in production. 
 
Thanks for your patience, 
 
Stanley and Glenn 
</pre>
<!-- Author: r -->
<!-- Comment: Tue Sep 23 08;59;24 comment by...rick -->
<pre>The problem with the robot (a tape caught in the gripper hand) has been resolved and the libraries are
back in production (with only a couple of minor drive problems being attended to currently).

Thanks for your patience and understanding.
Stanley
</pre>

<!-- Comment: Tue Sep 23 09;00;29 comment by...rick -->
<pre>The backlog of requests is already turning over and going down.</pre>