overhauled set loading to be faster and more robust in production