Well, it would break BC if you check that the controller is set when flushing the controller collection, indeed.
But if you rely on the Symfony validation done in the kernel, it could work (the validation would occur when the route is matched and the controller should be called)