Settings cleanup

Also fixes some locking bugs @kahrl found while helping me debug this and makes an exception message more helpful.
Includes #1618.