Ensure that local and remote keys have secure permissions - fixes 382

Merge after #403

Fixes #382 
No need for threads.