Implement better folder checking

Fixes #2360, thanks @clinton_hall