bitly/nsq — pull request #203 by mreiferson (2013-05-17 17:16:29+00:00)
Title: nsqd: diskqueue corruption and depth accounting

When diskqueue encounters a corrupt datafile it skips to the next file and renames the corrupt file to `.bad`, logging all actions. I believe this is desirable and expected behavior.

The problem is that it cannot accurately determine the number of messages that were left unread in the corrupt file, so depth accounting becomes inconsistent (effectively never reaches 0 for the amount that was unread).

The proposed solution would be to automatically set `depth = 0` whenever diskqueue reaches the end of the last file on disk, effectively becoming "eventually consistent".

Question is... is this proposal inline with expected behavior? is keeping a `.bad` file around and logging sufficient feedback for administrators? (you could imagine suggesting in our documentation to monitor the `--data-path` for `.bad` files and alert when they are present).

cc @jehiah @jayridge @elubow