Fix performance regression from c330b1d64efa566911cdd20921ca9419e58bacf9

StoredArray is more specific than AbstractVector{T}, so we were
performing unnecessary copies when calling `int` etc. This caused a
regression on the add1 benchmark, since we call `int` in `getindex`.