Big keyholder cleanup.
Should hopefully also fix issues 1507, 1532 and 1533.