Ensure our private dir is created

See fatbox/stackstrap#1