More Updates to Configuration

Follow up to: I29e13d8990674641b2a4a633410c116249b057da

Change-Id: I2e2912aee0828b041dddd557b0db96c2572e0793
'Fix' the DiskFileDataStore

If the problem is that we remove things too early, just dont
remove them :p.

This will leave a reasonable number of keys directories in
the temp directory. However, they are really generic (they
have names like correlationId) so it's not such a bad thing.

Change-Id: I545ab8ca8b1ad84f179f3e85d5f2cf05821f1bec