More general solution for refs and sets on chaperoned structs