I see, here is the relevant code:
```py
    with ctx.extraprec(extraprec):
        deg = len(coeffs) - 1
        # Must be monic
        lead = ctx.convert(coeffs[0])
        if lead == 1:
            coeffs = [ctx.convert(c) for c in coeffs]
        else:
            coeffs = [c/lead for c in coeffs]
        f = lambda x: ctx.polyval(coeffs, x)
```
So the `coeffs` get converted from ints to the working precision. That would explain why I need so high working precision (e.g. poly degree*10). I would still like to generate just integers, then there is no worry about this.