daitss/core — Issue #621: fclnx30: TapeFixity process "disappears" with no error message or log entry; user problem
Author: lydiam
Created: 2012-01-23 18:08:29+00:00
State: open

On January 21, 2011 at ~4:38am, a TapeFixity process ceased to run but left behind no error messages or log entries or emails, except for something that was attached to the end of the Collect-Fixities report email from Jan 21. (See the end of this comment.)

The log entry:

Jan 21 04:38:30 fclnx30 TapeFixity[9646]: INFO silos.darchive.fcla.edu: Configuration: fixity_disk => /daitssfs/fixity/0
Jan 21 04:38:30 fclnx30 TapeFixity[9646]: INFO silos.darchive.fcla.edu: Configuration: fixity_expired_days => 100
Jan 21 04:38:30 fclnx30 TapeFixity[9646]: INFO silos.darchive.fcla.edu: Configuration: fixity_stale_days => 25
Jan 21 04:38:30 fclnx30 TapeFixity[9646]: INFO silos.darchive.fcla.edu: Configuration: fresh_enough => 0
Jan 21 04:38:30 fclnx30 TapeFixity[9646]: INFO silos.darchive.fcla.edu: Configuration: hostname => silos.darchive.fcla.edu
Jan 21 04:38:30 fclnx30 TapeFixity[9646]: INFO silos.darchive.fcla.edu: Configuration: log_syslog_facility => LOG_LOCAL2
Jan 21 04:38:30 fclnx30 TapeFixity[9646]: INFO silos.darchive.fcla.edu: Configuration: pid_directory => /var/run/fixity
Jan 21 04:38:30 fclnx30 TapeFixity[9646]: INFO silos.darchive.fcla.edu: Configuration: silo_db => postgres://silo:********@localhost:6432/store_db
Jan 21 04:38:30 fclnx30 TapeFixity[9646]: INFO silos.darchive.fcla.edu: Configuration: silo_user => silo
Jan 21 04:38:30 fclnx30 TapeFixity[9646]: INFO silos.darchive.fcla.edu: Configuration: tivoli_server => BERNARD_SERVER

The next log entry begins Jan 22:
Jan 22 04:04:03 fclnx30 CollectFixities[28001]: INFO storage-master.fda.fcla.edu: Configuration: daitss_db => postgres://daitss2:********@localhost:5432/daitss_db

The pid file left behind:

-rw-r--r-- 1 daitss daitss 5 Jan 21 04:38 tape-fixity.pid

[lydiam@fclnx30 fixity]$ cat /var/run/fixity/tape-fixity.pid
9646

XYMON began sending alerts at:

Sat Jan 21 04:39:48 EST 2012

-- Fixity Checks --

green Collect-Fixities PID file does not exist and no running Collect-Fixities Process
green Disk-Fixity PID file does not exist and no running Disk-Fixity Process
red Tape-Fixity PID file exists, however no running Tape-Fixity Process

So obviously the process died shortly after the PID file was created.

Is there any process that could have killed TapeFixity at that hour?

From /etc/ctrontab I see:

04 4 * * * daitss run-parts /opt/fda/etc/daitss.daily

In that directory are the following jobs:
/opt/fda/etc/daitss.daily:
fiu-daily-ingest-report* reload-squid-cache* run-collect-fixities* run-tape-fixity* uf-daily-ingest-report* usf-daily-ingest-report*

Collect-fixities seems to have finished seconds before TapeFixity started. Could that interfere with the running of tape-fixity?

Jan 21 04:38:25 fclnx30 CollectFixities[32018]: WARN storage-master.fda.fcla.edu: Summary of DAITSS Package Fixity Checks: Additionally:
Jan 21 04:38:25 fclnx30 CollectFixities[32018]: WARN storage-master.fda.fcla.edu: Summary of DAITSS Package Fixity Checks: 612 unexpected packages (orphaned?) in silo pools
Jan 21 04:38:25 fclnx30 CollectFixities[32018]: WARN storage-master.fda.fcla.edu: Summary of DAITSS Package Fixity Checks: 0 packages had expired fixities
Jan 21 04:38:25 fclnx30 CollectFixities[32018]: INFO storage-master.fda.fcla.edu: CollectFixities finished at Sat Jan 21 04:38:25 -0500 2012
Jan 21 04:38:30 fclnx30 TapeFixity[9646]: INFO silos.darchive.fcla.edu: Configuration: fixity_disk => /daitssfs/fixity/0

My understanding of the fixity jobs is that they don't actually start working until after they report the configuration settings, so it appears that no processing was done on TapeFixity.

The Jan 21 CollectFixities email ends with:

/opt/fda/etc/daitss.daily/run-tape-fixity:

/opt/web-services/sites/silos/current/tools/tape-fixity:127:in `setup': Store::FatalFixityError: This program must be run as the silo user, but is being run as daitss. (Store::FatalFixityError)
from /opt/web-services/sites/silos/current/tools/tape-fixity:336

So it looks like the actual problem has to do with the change in users.