[polygamma] Fix integer overflow in branch check case, and bail out of transition code is the number of series evaluations is too high.