fix some integer conversions in the Table class

Sizes (x, y, z) are clamped to a non-negative int value.
Use NUM2INT instead of FIX2INT as the argument might not be a Fixnum.