Fix physics collision sets not being properly stored and found again