Don't try to restore non-existent or empty directories during a git sync