Corruption Detection and Containment
Channel: Andrea Ross
Date: 2013-08-18

This will not be the most exciting talk, nor is there (currently) a simple answer to make hardware corruption problems go away. But it's important -- without being careful, it's easy for corruption to spread to replicas and backups, leaving data hopelessly lost. Or, a strange crash due to corruption could take many engineering resources to analyze.

This talk is about kinds of hardware corruption that can and do happen, and the ways to detect and contain the corruption as quickly as possible. Additionally, we'll discuss a roadmap of improvements to postgresql to make this an easier process; as well as alternatives (such as detecting corruption in the filesystem).

Note: Some storage systems do provide strong protections against data corruption. This talk is primarily (though not exclusively) targeted at users of the local filesystem, particularly on Linux.

These are the topics that will be addressed:

    Why not deal with this in the filesystem?
    The different kinds of corruption.
    When to detect the corruption, and how to contain it.
    Data page checksums
    Backups and corruption
    Replication and corruption
    Background and offline detection
    More work to be done