libdfp: Fix quantize decimal128 inputs

This patch fixes 2 decimal128 inputs for quantized that are too large
for decimal type.