OK, this will cast negative numbers to unsigned in some cases. That happens now, but it probably needs a check. This path is version dependent but we can use the `int(-1)` type to reach it in Python2.