<!-- Title: 07:57:29  Tue Feb 19 2008 -->
<!-- Crewchief: ByeongRok Ko -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>New shift</pre>


<!-- Date: Tue Feb 19 08:10:12 2008 -->
<pre>fcdfdata320-1/2/3/4 still inactive 
 
MC Upload server still shows 
08:00:11 20080219 1203429611 
Unable to get the metadata structure for file /data/top3-131/submitted/pw02eae8.0001skt3 --- [Errno 2] No such file or directory: /data/top3-131/submitted/pw02eae8.0001skt3 
 
Two bad movers : 
9940B11 
9940B27 
 
Bad Tape : IAB313 
 
--&gt; All items are known  
</pre>
<!-- Author: ByeongRok -->


<!-- Date: Tue Feb 19 08:33:43 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-49&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-49  width='300' align='top'></a></td><td>It looks like all CAF file delivery stopped around 04:00. Station and nameservice appear to be running, but the optimizer is not responding to pings. Investigating.</td></tr></table>
<!-- Author: Angela Bellavance -->
<!-- Comment: Tue Feb 19 08;42;25 comment by...Angela Bellavance -->
<pre>I went to the optimizer log directory on cdfsam01 and noticed that there were a lot of very large old files. I zipped all daily logs from before 2008. Some plots (Number of Files to Restore, Requests to dCache Pool Groups) showed new activity as soon as I did the zipping. I went ahead and made a copy of the trace file (trace_20080219) and restarted the optimizer to see if it will restore full function.</pre>


<!-- Comment: Tue Feb 19 16;20;15 comment by...Angela Bellavance -->
<pre>It appears the optimizer was a red herring, or at least just a minor problem. 

Robert was able to attach to the station with gdb and figure out where it was getting stuck. From the variables in gdb he got the a file name. From the file name he got a project name. From the project name he got the project process ID. Executing "/usr/sbin/lsof -p &lt;PID&gt;" listed the number of open file handles which totaled to 1024. This is the maximum allowed. 

So, in the end, it's the same problem we've been warning users about for the last two weeks -- DO NOT run projects with more than 1000 sections! The station ended up stuck in a loop trying to process this project but failing to talk to it because it was out of handles. Robert killed the culprit project and that user was sent an email.

As an additional clean-up precaution, we also ran "samadmin purge zombie projects --startedbefore=today --station=cdf-caf" which ended up cleaning out 249 zombie projects going as far back as 2005. &lt;cough, cough, fanning away dust...&gt;</pre>
<!-- Date: Tue Feb 19 10:43:31 2008 -->
<pre>samTV was updated 7 hours ago</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Tue Feb 19 12;19;34 comment by...ByeongRok -->
<pre>email from Angela

I suspect this is tied to the cdf-caf station getting stuck. I tried to
restart samTV, but it was still getting stuck on re-trying to get project
dumps that returned "ProjectNotFound". I noticed that each project was doing
200 retries at 5 second intervals. If there are a lot of projects with
problems, then the script takes forever. I have modified the code to retry a
maximum of 5 times. If this seems to help, I'll update CVS.
</pre>

<!-- Comment: Tue Feb 19 16;11;13 comment by...Angela Bellavance -->
<pre>At Robert's suggestion I've changed the retries to zero and restarted.</pre>



<!-- Date: Wed Feb 20 08:00:46 2008 -->
<pre>CAF file delivery may have a problem 
</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Wed Feb 20 08;40;20 comment by...ByeongRok -->
<pre>email from Robert Illingworth

The current lack of activity is probably due to the code server problems.
</pre>


<!-- Date: Wed Feb 20 14:07:50 2008 -->
<pre>email from David Berg 
 
IAB313 - label replaced, tape is available 
 
 - David 
</pre>
<!-- Author: ByeongRok -->


<!-- Date: Thu Feb 21 08:04:16 2008 -->
<pre>Two bad movers 
 
9940B19 
9940B27 
</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Thu Feb 21 13;21;02 comment by...ByeongRok -->
<pre>9940B19 is back</pre>
<!-- Date: Thu Feb 21 10:35:42 2008 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-50&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-50' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-51&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=datahandle&action=view&page=-51' width='300'></a></center>
<!-- Author: Angela Bellavance -->
<!-- Comment: Thu Feb 21 10;38;12 comment by...Angela Bellavance -->
<pre>A discrepancy between requested files and the queue occurred last night around 17:00. Will investigate to try and find reason.

Also, a client is holding open a connection to the production database since around 18:00 last night. Will try to investigate.</pre>

<!-- Comment: Thu Feb 21 12;41;07 comment by...Angela Bellavance -->
<pre>Discrepancy has gone away just after noon. Tape IA5692 has also returned to service, so that was the culprit.</pre>



<!-- Comment: Fri Feb 22 11;13;05 comment by...Angela Bellavance -->
<pre>With Krzysztof's help, I contacted the user with two open database connections. Turns out he forgot processes running in background and has now killed them. Plot has returned to normal.

The most useful tool was a database query for connections:
http://cdfdbb.fnal.gov:8520/cdfr2/databases?type=ora-uaoff&do=r&fsrc=fp2&nsrc=np&gsrc=fp2&uaofforder=%2Buser</pre>
<!-- Date: Thu Feb 21 13:59:43 2008 -->
<pre>Red balls in cdf-caf 
 
!  linacre.1202342896:Project                       cdfsam02.fnal.gov:37619    unknown            unreachable 
!  oglez_20080206_204655:Project                    cdfsam02.fnal.gov:54604    unknown            unreachable 
!  oglez_20080206_220351:Project                    cdfsam02.fnal.gov:35461    unknown            unreachable 
!  ptl_cdf-caf_TestJob_24Jan07_b_1202382862_7923:Project cdfsam02.fnal.gov:55530    unknown            unreachable 
!  oglez_20080206_193829:Project                    cdfsam02.fnal.gov:46526    unknown            unreachable 
</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Thu Feb 21 14;37;39 comment by...ByeongRok -->
<pre>We usually are not concerned with red balls on individual jobs unless the user personally sends us a complaint. They will go away when they
time out.

- Angela
</pre>



<!-- Date: Mon Feb 25 15:58:24 2008 -->
<pre>LTO3_05 is offline 
LTO3_28 is dead 
</pre>
<!-- Author: ByeongRok -->


<!-- Date: Mon Feb 25 16:17:05 2008 -->
<pre>LTO3_10 is dead 
LTO3_31 is offline 
</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Mon Feb 25 17;34;43 comment by...ByeongRok -->
<pre>email from Stanley:

Three of the four drives seem to coincide with 
the tab flipping operation.
LTO3_05  has other problems.

I am restarting enstore on LTO3_10, LTO3_28 and LTO3_31.

</pre>

<!-- Comment: Tue Feb 26 07;56;37 comment by...ByeongRok -->
<pre>all of them are back now</pre>


<!-- Date: Tue Feb 26 08:25:36 2008 -->
<pre>shift report 
 
fcdfdata320-1/2/3/4 still inactive  
9940B27 is still being tested. 
 
Tue Feb 19 
all CAF file delivery stopped around 04:00 
Email from Angela Bellavance: 
Due to the projects with more than 1000 sections!  
The station ended up stuck in a loop trying to 
process this project but failing to talk to it because it was out of 
handles. Robert killed the culprit project. 
 
Wed Feb 20  
From David Berg : 
IAB313 - label replaced, tape is available. 
 
Mon Feb 25 
LTO3_05 was offline  
LTO3_10 was dead  
LTO3_28 was dead  
LTO3_31 was offline  
 
email from Stanley : 
Three of the four drives seem to coincide with the tab flipping 
operation. LTO3_05  has other problems. 
</pre>
<!-- Author: ByeongRok -->