daitss/core — Issue #641: core services restarted itself March 4 4:05am
Author: lydiam
Created: 2012-03-07 18:20:20+00:00
State: open

This may be related to silo problems March 4, and most certainly is related to slowness.

E20090727_AAABFD took almost 4 hours to refresh on Sunday but it only contains 16 files and is 10MB in size:

2012 Mar 4 00:02:56 fclnx30 Pulse[8320]: INFO fda.fcla.edu: Beginning AIP download for E20090727_AAABFD.tar
2012 Mar 4 00:02:56 fclnx30 Pulse[22786]: INFO fda.fcla.edu: E20090727_AAABFD.d1refresh spawned
2012 Mar 4 00:02:58 fclnx30 Pulse[8320]: INFO fda.fcla.edu: AIP download for E20090727_AAABFD.tar complete
2012 Mar 4 00:02:58 fclnx30 Pulse[8320]: INFO fda.fcla.edu: Creating wip from tar for E20090727_AAABFD
2012 Mar 4 00:02:58 fclnx30 Pulse[8320]: INFO fda.fcla.edu: Finished creating wip from tar for E20090727_AAABFD
2012 Mar 4 00:06:34 fclnx30 Pulse[8320]: INFO fda.fcla.edu: Parsing AIP descriptor for E20090727_AAABFD
2012 Mar 4 00:06:35 fclnx30 Pulse[8320]: INFO fda.fcla.edu: Reserving storage for updated AIP for E20090727_AAABFD
2012 Mar 4 00:06:35 fclnx30 Pulse[8320]: INFO fda.fcla.edu: Writing updated AIP for E20090727_AAABFD to storage
2012 Mar 4 00:06:41 fclnx30 Pulse[8320]: INFO fda.fcla.edu: Updating AIP in database for E20090727_AAABFD
2012 Mar 4 00:06:43 fclnx30 Pulse[8320]: INFO fda.fcla.edu: Deleting old AIP for E20090727_AAABFD from storage

During that time core service was reported by XYMON as being down and core.log says:

2012 Mar 2 16:58:35 fclnx30 Core[23739]: INFO fda.fcla.edu: Deleting old AIP for E20090726_AAABAL from storage
2012 Mar 4 04:05:43 fclnx30 Core[19941]: INFO fda.fcla.edu: Starting up core service
2012 Mar 4 04:05:43 fclnx30 Core[19941]: INFO fda.fcla.edu: Using temp directory /var/daitss/tmp

No "daitss" command was issued at the time, and 4:05 is prior to the UF maintenance window. What would cause
core to start itself up like that? Pulse.log doesn't show any major activity.