Bugfix: Rare crash-on-load fixed

Added proper concurrency handling so we no longer comodify the
controller set. Also changing it to a set to eliminate contains checks.