Fix slow-mo framerate drop

Technically only the updates of the rigidbodies slow down but anyways…