[RFC] fix strict-overflow cases

This fixes overflows caused by (X - 1) and (X - 3) that can result into negative numbers for values that are not intended to be like this.

Effectively fixes issues #464 and #513 .
Is this check still necessary? `keylen >= 3` should imply that `s[2] != NUL`.