Settings cleanup

Also fixes some locking bugs @kahrl found while helping me debug this and makes an exception message more helpfull.