Refectored so that different threads now get new database connections.

Still need to sort out getting locks with the heartbeat manager on new
file records as they are created. This has partly been done in this commit
but is unfinished. See TODOs