handle overflow when retry count gets larger

When retry count gets to larger numbers, the returned value may overflow int. Added checks for this.