Use load balancer configuration, not load balancer id, in job-based LB removal logic

This is required to support RCv3.
Really love all these extra assertions about what has and hasn't been added :D