5 is not a lot, but I understand it decreases the computation time by 2.
Is it related to some computational cost of M-estimators? If so, maybe we will have more discussion as I am reading the PR.