VM: fixing some implicit casts from cell to fixnum

Small fix to make the fixnum/cell casts better. E.g  `unbox_array_size` is declared to return a cell so the variable that receives it should also be declared of type cell instead of fixnum.
Doesn't this change it from signed to unsigned?  Is that correct?