VM: fixing some implicit casts from cell to fixnum

Small fix to make the fixnum/cell casts better. E.g  `unbox_array_size` is declared to return a cell so the variable that receives it should also be declared of type cell instead of fixnum.
Same comment here, cell is ``uintptr``.  How can ``depth < 0`` be true?