Remove early return in scaler_ctx_scale to avoid performance
regression