Remove only worst user in calculate_residuals

From:

https://github.com/glinscott/fishtest/issues/26