VM: fixing some implicit casts from cell to fixnum

Small fix to make the fixnum/cell casts better. E.g  `unbox_array_size` is declared to return a cell so the variable that receives it should also be declared of type cell instead of fixnum.
Oops again. Otoh, I wonder if the depth < 0 branch is ever taken.. lets see.