Keep wallet load/unload atomic

No partial data load/unload, nasty races may arise.

Basically reverts b0bfb4ec8c32dc9e4e0bbeb6f31d014425337b77.