<!-- Title: 02:39:37  Tue Apr  8 2008 -->
<!-- Crewchief: Thomas Kuhr -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>New SAM shift week</pre>


<!-- Date: Tue Apr 8 02:48:35 2008 -->
<pre>Known issues: 
- LTO3_05 down (powerup fails) 
- fcdfdata320 offline 
- samTV not working</pre>
<!-- Author: Thomas -->


<!-- Date: Tue Apr 8 02:56:18 2008 -->
<pre>There are many errors reported for encp processes by user sam on b0csl23, e.g.: 
 
encp copies to/from tape. It is not involved in copying unixfile to unixfile. : (SRC: /mnt/data1/logger/stager/Submitted/br03f924.0162phys) (DST: /pnfs/sam/crap)</pre>
<!-- Author: Thomas -->


<!-- Date: Thu Apr 10 03:04:29 2008 -->
<pre>Message from Federica: 
The DbServer for CAF died last night. We had notice of some jobs failure, and now should be fine.</pre>
<!-- Author: Thomas -->


<!-- Date: Thu Apr 10 03:11:00 2008 -->
<pre>dbs_cdf_rawdata_prd and opt_cdf_rawdata_prd are red at SAAG. Restarted them.</pre>
<!-- Author: Thomas -->
<!-- Comment: Thu Apr 10 06;56;56 comment by...Thomas -->
<pre>The restart didn't help. Sent email to oncall expert.</pre>
<!-- Comment: Thu Apr 10 09;28;36 comment by...Thomas -->
<pre>Angela said they are not used and can be ignored.</pre>
<!-- Date: Thu Apr 10 04:06:48 2008 -->
<pre>9940B19.mover is offline</pre>
<!-- Author: Thomnas -->
<!-- Comment: Fri Apr 11 02;44;12 comment by...Thomas -->
<pre>Fixed.</pre>
<!-- Date: Thu Apr 10 06:55:41 2008 -->
<pre>sam upload processes on fcdfdata321/322 fail since ~5:20am: 
05:20:04  [ERROR] Exit --- The name service is not available.  Contact your sam administrator.</pre>
<!-- Author: Thomas -->
<!-- Comment: Thu Apr 10 07;20;49 comment by...Thomas -->
<pre>This is caused by the wrong current version of sam_ns_ior (see next entry). I've added a temporary fix to the upload.sh script by explicitly setting the IOR.</pre>
<!-- Date: Thu Apr 10 07:17:07 2008 -->
<pre>The automatic cdfsoft update made the old version sam v7_4_2 and sam_ns_ior v7_0_1 current (instead of sam v8_2_2 and sam_ns_ior v7_1_0). Remote sites and sam upload servers are affected, fcdflnx machines are still ok.</pre>
<!-- Author: Thomas -->





<!-- Date: Fri Apr 11 13:35:03 2008 -->
<pre>Automatic alert email from cdf-caf: 
04/11/08 11:31:10 cdf-caf.SM.FileUncachedInDBTimer 29717: Cannot uncache file in DB: 
CORBA::SystemException caught in FileUncachedInDBTimer::doIt(): 
System exception `UNKNOWN' 
Completed: maybe 
Minor code: 1096024162</pre>
<!-- Author: Thomas -->


<!-- Date: Sat Apr 12 00:38:28 2008 -->
<pre>Some old DbServers got restarted on the cdfsam* machines by a cron job I forgot to turn off. The following "red balls" can be ignored in Sam-At-A-Glance: 
SAMDbServer.caf_srv2_prd:SAMDbServer 
SAMDbServer.caf_srv3_prd:SAMDbServer 
SAMDbServer.caf_srv4_prd:SAMDbServer 
SAMDbServer.farm1_srv2_prd:SAMDbServer 
SAMDbServer.farm1_srv3_prd:SAMDbServer 
SAMDbServer.farm1_srv4_prd:SAMDbServer 
SAMDbServer.interactive_srv2_prd:SAMDbServer 
SAMDbServer.interactive_srv3_prd:SAMDbServer 
SAMDbServer.interactive_srv4_prd:SAMDbServer 
SAMDbServer.web_srv2_prd:SAMDbServer 
SAMDbServer.web_srv3_prd:SAMDbServer 
SAMDbServer.web_srv4_prd:SAMDbServer 
</pre>
<!-- Author: Angela Bellavance -->


<!-- Date: Sat Apr 12 06:48:00 2008 -->
<pre>Giorgio reported an issue with jobs using a mixture of farm style and caf style access to files in sam. Anwered by Krzysztof.</pre>
<!-- Author: Thomas -->


<!-- Date: Sat Apr 12 06:51:15 2008 -->
<pre>Artur Apresyan reported a high crash rate of Stntuple production jobs. Ray is following up on it.</pre>
<!-- Author: Thomas -->


<!-- Date: Sat Apr 12 13:57:30 2008 -->
<pre>Diskpool door02 is not responding. Informed diskpool admins.</pre>
<!-- Author: Thomas -->
<!-- Comment: Sat Apr 12 15;23;03 comment by...Thomas -->
<pre>Alexei restarted it.</pre>


<!-- Date: Sun Apr 13 02:48:05 2008 -->
<pre>Mail from Alexei: 
My jobs are stuck in a very long queue waiting for the files. I am looking at cdfdca web page report and it looks that pool 
r-fcdfdata146-3 has 48 active movers with 183 jobs in a queue. 
 
Why files in demand are not replicated/cached on additional file servers? I am sure one could tweak dcache parameters such that p2p file replication starts when queue starts to build up on a fileservers. 
 
</pre>
<!-- Author: Thomas -->
<!-- Comment: Sun Apr 13 02;50;05 comment by...Thomas -->
<pre>The number of queued request seems to increase. Now it's at 247.</pre>


<!-- Date: Sun Apr 13 11:03:27 2008 -->
<pre>Automatic email: 
The serverName 'SAMDbServer.dbs-cdf-cafstation-srv1-prd:SAMDbServer' was killed because it appeared to be hung. 
It should restart automatically.</pre>
<!-- Author: Thomas -->


<!-- Date: Sun Apr 13 11:06:39 2008 -->
<pre>DB response time went up significantly at ~10:15am. SAAG is not updating since then. "sam locate foo" is very slow and then returns 
omniORB: From endpoint: giop:tcp:131.225.208.224:42336. Detected GIOP 1.2 protocol error in input message. giopImpl12.cc:863. Connection is closed. 
RetryHandler.getReplicaLocationList('foo')&gt; initial retriable exception COMM_FAILURE('CORBA.COMM_FAILURE(omniORB.COMM_FAILURE_UnMarshalResults, CORBA.COMPLETED_YES)') 
RetryHandler.getReplicaLocationList('foo')&gt; will retry in 0.59 seconds 
</pre>
<!-- Author: Thomas -->
<!-- Comment: Sun Apr 13 11;13;37 comment by...Thomas -->
<pre>Message from Nelly:

We received alerts at 10:32 that the machine fcdfora4  was not pingable.
 
Steve Kovich is investigating the fcdfora4~cdfofprd issue & as he learns
more we will give you a status report.</pre>

<!-- Comment: Sun Apr 13 11;32;56 comment by...Thomas -->
<pre>Mail from Nelly:
Steve Kovich has looked at the problem and has determined
he needs to do further diagnostics on-site.</pre>

<!-- Comment: Sun Apr 13 14;06;52 comment by...Thomas -->
<pre>Mail from Steve:
fcdfora4 crashed this morning and has now been brought back online.
The reason for the crash is being investigated.
</pre>

<!-- Comment: Sun Apr 13 15;14;01 comment by...Thomas -->
<pre>SAM seems fine again. File delivery is resumed.</pre>


<!-- Date: Sun Apr 13 17:38:28 2008 -->
<pre>DCache pool machine fcdfdata154 is offline, and pool r-fcdfdata146-3 has a backlog of 300+ requests. Moved r-fcdfdata146-3 to TestPools and am sending email to dCache admins.</pre>
<!-- Author: Angela Bellavance -->
<!-- Comment: Sun Apr 13 22;13;35 comment by...Angela Bellavance -->
<pre>Pool is drained and returned to nominal operation.</pre>


<!-- Date: Mon Apr 14 02:21:19 2008 -->
<pre>Automatic email: 
04/14/08 02:11:58 cdf-caf.SM.FileUncachedInDBTimer 29717: Cannot uncache file in DB: 
CORBA::UserException caught in FileUncachedInDBTimer::doIt()</pre>
<!-- Author: Thomas -->
<!-- Comment: Mon Apr 14 02;24;21 comment by...Thomas -->
<pre>Same message again 10 minutes later.</pre>

<!-- Comment: Mon Apr 14 02;48;35 comment by...Thomas -->
<pre>Message keeps reappearing every 10 minutes.
The file is bd02cd41.0069hmuk (CFID 37490106).</pre>


<!-- Date: Mon Apr 14 07:51:13 2008 -->
<pre>cdf-caf is not delivering files any more.</pre>
<!-- Author: Thomas -->
<!-- Comment: Mon Apr 14 07;59;54 comment by...Thomas -->
<pre>[sam@fcdfsam1 station__fcdfsam1__station_cdf_caf_prd__cdf-caf]$ grep -a "successfully opens" station_log__04_14_08 |tail
04/14/08 07:30:06 cdf-caf.benjamin_cdf-caf_bpel0i_258175_22453.GPM 28108: Proc { "UserName" : "benjamin" }(WorkGroup: test, Application: { "ApplicationFamily" : "unknown", "ApplicationName" : "demo", "ApplicationVersion" : "1" })@fcdfcaf997.fnal.gov[SAM Consumer Process](15445964) successfully opens file bi0358d6.07aapel0(7034151)
04/14/08 07:33:03 cdf-caf.benjamin_cdf-caf_bpel0i_258175_22453.GPM 28108: Proc { "UserName" : "benjamin" }(WorkGroup: test, Application: { "ApplicationFamily" : "unknown", "ApplicationName" : "demo", "ApplicationVersion" : "1" })@fcdfcaf1180.fnal.gov[SAM Consumer Process](15445957) successfully opens file bi0358f9.01bdpel0(7034172)
04/14/08 07:34:20 cdf-caf.benjamin_cdf-caf_bpel0i_258175_22453.GPM 28108: Proc { "UserName" : "benjamin" }(WorkGroup: test, Application: { "ApplicationFamily" : "unknown", "ApplicationName" : "demo", "ApplicationVersion" : "1" })@fcdfcaf997.fnal.gov[SAM Consumer Process](15445964) successfully opens file bi0358f8.0001pel0(7034157)
04/14/08 07:36:04 cdf-caf.benjamin_cdf-caf_bpel0i_258175_22453.GPM 28108: Proc { "UserName" : "benjamin" }(WorkGroup: test, Application: { "ApplicationFamily" : "unknown", "ApplicationName" : "demo", "ApplicationVersion" : "1" })@fcdfcaf1180.fnal.gov[SAM Consumer Process](15445957) successfully opens file bi035ae7.0175pel0(7056680)
04/14/08 07:38:26 cdf-caf.benjamin_cdf-caf_bpel0i_258175_22453.GPM 28108: Proc { "UserName" : "benjamin" }(WorkGroup: test, Application: { "ApplicationFamily" : "unknown", "ApplicationName" : "demo", "ApplicationVersion" : "1" })@fcdfcaf997.fnal.gov[SAM Consumer Process](15445964) successfully opens file bi035af9.011apel0(7060318)
04/14/08 07:39:06 cdf-caf.benjamin_cdf-caf_bpel0i_258175_22453.GPM 28108: Proc { "UserName" : "benjamin" }(WorkGroup: test, Application: { "ApplicationFamily" : "unknown", "ApplicationName" : "demo", "ApplicationVersion" : "1" })@fcdfcaf1180.fnal.gov[SAM Consumer Process](15445957) successfully opens file bi035af6.0001pel0(7060317)
04/14/08 07:42:10 cdf-caf.benjamin_cdf-caf_bpel0i_258175_22453.GPM 28108: Proc { "UserName" : "benjamin" }(WorkGroup: test, Application: { "ApplicationFamily" : "unknown", "ApplicationName" : "demo", "ApplicationVersion" : "1" })@fcdfcaf1180.fnal.gov[SAM Consumer Process](15445957) successfully opens file bi035b29.024dpel0(7060324)
04/14/08 07:42:32 cdf-caf.benjamin_cdf-caf_bpel0i_258175_22453.GPM 28108: Proc { "UserName" : "benjamin" }(WorkGroup: test, Application: { "ApplicationFamily" : "unknown", "ApplicationName" : "demo", "ApplicationVersion" : "1" })@fcdfcaf997.fnal.gov[SAM Consumer Process](15445964) successfully opens file bi037cb8.01a0pel0(8099460)
04/14/08 07:44:01 cdf-caf.seiya_cdf-caf_seiya_seiya_gjt30j_p12_v1_d1_258128_5692.GPM 15955: Proc { "UserName" : "seiya" }(WorkGroup: test, Application: { "ApplicationFamily" : "unknown", "ApplicationName" : "demo", "ApplicationVersion" : "1" })@fcdfcaf648[CafExe_CAF_CAF_JID_3491135_Section_Num_33](15443743) successfully opens file gj039bd7.0001jt30(10744500)
04/14/08 07:46:27 cdf-caf.benjamin_cdf-caf_bpel0i_258175_22453.GPM 28108: Proc { "UserName" : "benjamin" }(WorkGroup: test, Application: { "ApplicationFamily" : "unknown", "ApplicationName" : "demo", "ApplicationVersion" : "1" })@fcdfcaf997.fnal.gov[SAM Consumer Process](15445964) successfully opens file bi037cb8.020cpel0(8100582)
[sam@fcdfsam1 station__fcdfsam1__station_cdf_caf_prd__cdf-caf]$ date
Mon Apr 14 07:58:54 CDT 2008
</pre>

<!-- Comment: Mon Apr 14 08;08;20 comment by...Thomas -->
<pre>I restarted cdf-caf.</pre>

<!-- Comment: Mon Apr 14 09;20;58 comment by...Thomas -->
<pre>The station looks fine after the restart.</pre>


<!-- Comment: Mon Apr 14 11;11;28 comment by...Angela Bellavance -->
<pre>Since the file involved looks fine, I suspect this error was a result of the database outage on Sunday. The restart should have fixed it.</pre>
<!-- Date: Mon Apr 14 09:22:40 2008 -->
<pre>SAM jobs are crashing at CNAF: 
 
....... loading libdiskcache_i.so v2_07_05 ....... 
 
         Release version = 6.1.4 
 
04/14/08 14:55:52   sam_cpp_api.DBSProxy 10719: CorbaServerProxy::locate(): Establishing connection to server SAMDbServer.dbs-cdf-remotestation-prd:SAMDbServer... 
04/14/08 14:55:53   sam_cpp_api.DBSProxy 10719: CorbaServerProxy::locate(): Established connection to server SAMDbServer.dbs-cdf-remotestation-prd:SAMDbServer. 
%ERLOG-s Abnormal termination: caused by signal:  11 Framework abortJob() 
         14-Apr-2008 14:55:53 XYZ run = 0 event = 0 
Calling abortJob for DummyInput... </pre>
<!-- Author: Thomas -->
<!-- Comment: Mon Apr 14 10;22;36 comment by...Angela Bellavance -->
<pre>Since the file involved looks fine, I suspect this error was a result of the database outage on Sunday. The restart should have fixed it.</pre>

<!-- Comment: Mon Apr 14 11;11;06 comment by...Angela Bellavance -->
<pre>Oops, ignore previous comment - meant for a different issue.</pre>



<!-- Date: Mon Apr 14 13:37:56 2008 -->
<pre>As a test, I will be taking down the dbs-cdf-upload-srv2-prd DbServer temporarily. Service should not be affected.</pre>
<!-- Author: Angela Bellavance -->
<!-- Comment: Mon Apr 14 13;53;01 comment by...Angela Bellavance -->
<pre>Test completed. DbServer restarted.
Confirmed that the plot of the number of DbServers down is not updating properly. Eric investigating.</pre>


<!-- Comment: Mon Apr 14 15;58;08 comment by...Angela Bellavance -->
<pre>Trying test again.</pre>
<!-- Comment: Mon Apr 14 17;13;18 comment by...Angela Bellavance -->
<pre>Test failed again. Back to nominal.</pre>
<!-- Date: Mon Apr 14 14:23:32 2008 -->
<pre>dCache pool machine fcdfdata154 was down as of 18:30 Sunday night. Restarted 14:20 Monday. 
</pre>
<!-- Author: Angela Bellavance -->


<!-- Date: Mon Apr 14 15:16:23 2008 -->
<pre>LTO3_05 is back online.</pre>
<!-- Author: Thomas -->




<!-- Date: Mon Apr 14 17:31:45 2008 -->
<pre>A queue started to build on r-fcdfdata111-1. Moved to TestPools for draining.</pre>
<!-- Author: Angela Bellavance -->


<!-- Date: Tue Apr 15 04:48:02 2008 -->
<pre>9940B19.mover is offline</pre>
<!-- Author: Thomas -->


<!-- Date: Tue Apr 15 04:51:03 2008 -->
<pre>IA0342 is labeled NOACCESS.</pre>
<!-- Author: Thomas -->


<!-- Date: Tue Apr 15 05:56:03 2008 -->
<pre>Queued transfers on r-fcdfdata323-1 is 146, on r-fcdfdata327-1 its 79. Difference between restoreHandler and queueInfo is 32.</pre>
<!-- Author: Thomas -->
<!-- Comment: Tue Apr 15 07;18;05 comment by...Thomas -->
<pre>Still same number of queued transfers.</pre>