Remove early return in scaler_ctx_scale to avoid performance
regression
Rewrite scaler_ctx_scale to make it more efficient