wal-e/wal-e — pull request #141 by najalbert (2014-10-03 18:53:09+00:00)
Title: Tar partitioning is immune to directory clearing

Hi!

Thanks for this great project! Unfortunately, I've been getting intermittently corrupted base backups. The symptom is, after the base backup has been fetched and WAL replay is occurring, postgres will start spewing these errors:

```
postgres[28058]: [1191-1] 2014-10-03 00:16:45 UTC ERROR: could not access status of transaction 0
postgres[28058]: [1191-2] 2014-10-03 00:16:45 UTC DETAIL: Could not open file "pg_subtrans/08FD": No such file or directory.

```

and will eventually abort the WAL replay. In these cases, the ```pg_subtrans``` folder is missing from the ```$PGDATA``` directory, and I've been able to successfully complete WAL replay by manually creating the ```pg_subtrans``` folder. I've also seen this happen with the ```pg_multixact``` folder.

### Repro
I did some digging and I repro'd the behavior as follow:
* Create a directory in the postgres data dir (e.g. ```mkdir $PGDATA/disappearing_dir_test```).
* Create a single file in that directory (e.g. ```touch $PGDATA/disappearing_dir/foo```).
* Put a raw_input in tar partitioning [here](https://github.com/najalbert/wal-e/blob/master/wal_e/tar_partition.py#L522) to pause base backup after partitioning but before tarring and uploading.
* Run a base backup push.
* When the command hits the raw input, remove the file you created (e.g. ```rm $PGDATA/disappearing_dir/foo```).
* Press enter to continue the command.
* Fetch the backup. Note that you will neither have ```$PGDATA/disappearing_dir_test``` nor ```$PGDATA/disappearing_dir_test/foo``` in the restored backup.

The [postgres docs have a section](http://www.postgresql.org/docs/9.3/static/continuous-archiving.html#BACKUP-LOWLEVEL-BASE-BACKUP) about modified and disappearing files during base backup creation, so I believe the repro above is a reasonable simulation of postgres behavior.

### Fix

I took a swing at fixing this: the idea is to always include any directory you traverse in the tar partitioning. Because I'm not super familiar with the wal-e internals, me know if this approach makes sense. I'm happy to revise or hand-off as needed.

### Related

A few issues that might be related:
* [pg_subtrans directory not created during restore](https://github.com/wal-e/wal-e/issues/41)
* [Fetch on slave error](https://github.com/wal-e/wal-e/issues/71)
* [missing recovery.conf, postgresql.conf, and pg_subtrans dir on backup-fetch?](http://comments.gmane.org/gmane.comp.db.postgresql.wal-e/274)