sot/eng_archive — Issue #22: Corruption of archfiles.db3 for prop1eng
Author: taldcroft
Created: 2012-01-23 02:11:38+00:00
State: open
Labels: Bug

Around 2011:293 there was a corruption in the archfiles.db3 database index that is used to select row ranges based on time. This was in prop1eng on GRETA but not HEAD. The archfiles rowstart and rowstop got out of sync with the HDF5 files. This was associated with a large (-120000) jump in time in TIME.h5. Plotted data looked fine over the interval so the assumption is that there was a re-ingest but bad qualitity was set appropriately. Have not done full analysis though.

Bad files are in /proj/sot/ska/data/eng_archive/bad/data/prop1eng.

Need to check all other content types by using fetch.filetypes with a representative MSID. Spot check of two other MSIDs showed no problem.

An easy way to test for this corruption problem is to query on a short date interval, e.g one day. Then see if any data values are returned. I checked 'tephin' and 'aoattqt1' and did not see this issue.