daitss/core — Issue #603: disk-fixity:  repeated crashes
Author: lydiam
Created: 2011-12-13 15:17:42+00:00
State: open

I'm not sure if this is a github issue or if this job was deliberately killed:

On tarchive, a disk-fixity.pid file exists that predates the latest run of disk-fixity and yet disk-fixity ran regardless.

[daitss@fclnx31 daemons]$ cd /var/run/fixity

[daitss@fclnx31 fixity]$ ls -l
total 8
-rw-r--r-- 1 silo daitss 6 Dec 5 11:00 disk-fixity.pid

[daitss@fclnx31 fixity]$ cat disk-fixity.pid
26696

The /var/log/daitss/daemons/fixity.log-20111211 ends with the job incomplete:

[daitss@fclnx31 daemons]$ tail fixity.log-20111211
Dec 9 11:45:53 fclnx31 DiskFixity[26696]: INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Beginning bulk fixity update for silo /daitssfs/038
Dec 9 11:45:53 fclnx31 DiskFixity[26696]: INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Starting silo /daitssfs/038 on Friday, December 09, 2011 at 11:45:53 AM
Dec 9 17:18:16 fclnx31 DiskFixity[26696]: INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Completed silo /daitssfs/038 on Friday, December 09, 2011 at 05:18:16 PM
Dec 9 17:18:16 fclnx31 DiskFixity[26696]: INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Total of 1921 packages: 0 skipped, 1921 succeeded, 0 failed
Dec 9 17:18:16 fclnx31 DiskFixity[26696]: INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Beginning bulk fixity update for silo /daitssfs/039
Dec 9 17:18:16 fclnx31 DiskFixity[26696]: INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Starting silo /daitssfs/039 on Friday, December 09, 2011 at 05:18:16 PM
Dec 9 23:16:13 fclnx31 DiskFixity[26696]: INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Completed silo /daitssfs/039 on Friday, December 09, 2011 at 11:16:13 PM
Dec 9 23:16:13 fclnx31 DiskFixity[26696]: INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Total of 2511 packages: 0 skipped, 2511 succeeded, 0 failed
Dec 9 23:16:13 fclnx31 DiskFixity[26696]: INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Beginning bulk fixity update for silo /daitssfs/021
Dec 9 23:16:13 fclnx31 DiskFixity[26696]: INFO silos.tarchive.fcla.edu: Disk Fixity Checks: Starting silo /daitssfs/021 on Friday, December 09, 2011 at 11:16:13 PM

There are no further entries for this job in any of the fixity logs.

Prior failed jobs issued XYMON messages and emails, but this one did not.

Tue Nov 29 15:47:11 EST 2011 -- Fixity Check --

green Collect-Fixities PID file does not exist and no running Collect-Fixities Process
red Disk-Fixity PID file exists, however no running Disk-Fixity Process
green Tape-Fixity PID file does not exist and no running Tape-Fixity Process

The message that triggered this error (email sent at 11/29/2011 3:46pm):

/opt/fda/etc/silo.weekly/run-disk-fixity:

/opt/web-services/sites/silos/current/lib/store/silo.rb:307:in `md5': Can't determine md5 for 'E20060816_AAACGI' (located at '/daitssfs/007/68f/ab940396a03793eb754264bc88a34'): No such file or directory - /daitssfs/007/68f/ab940396a03793eb754264bc88a34/md5 (Store::SiloError)
from /opt/web-services/sites/silos/current/lib/store/silo.rb:403:in `read_lock'
from /opt/web-services/sites/silos/current/lib/store/silo.rb:401:in `open'
from /opt/web-services/sites/silos/current/lib/store/silo.rb:401:in `read_lock'
from /opt/web-services/sites/silos/current/lib/store/silo.rb:303:in `md5'
from /opt/web-services/sites/silos/current/tools/disk-fixity:195
from /opt/web-services/sites/silos/current/lib/store/silodb.rb:141:in `each_package_record'
from /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/dm-core-1.2.0/lib/dm-core/collection.rb:507:in `each'
from /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/dm-core-1.2.0/lib/dm-core/support/lazy_array.rb:411:in `each'
from /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/dm-core-1.2.0/lib/dm-core/support/lazy_array.rb:411:in `each'
from /opt/web-services/sites/silos/shared/bundle/ruby/1.8/gems/dm-core-1.2.0/lib/dm-core/collection.rb:504:in `each'
from /opt/web-services/sites/silos/current/lib/store/silodb.rb:141:in `each_package_record'
from /opt/web-services/sites/silos/current/tools/disk-fixity:158
from /opt/web-services/sites/silos/current/tools/disk-fixity:152:in `each'
from /opt/web-services/sites/silos/current/tools/disk-fixity:152

Is it OK to delete the pid file at this point?