fix: avoid int overflow in rlimit.c

(cherry picked from commit 0d44b3386987b041cec4321f5488994c04dff538)

Signed-off-by: Andreas Steenpass <steenpass@mathematik.uni-kl.de>