This looks like too much code. Seems to me there are two cases:
1. The baseDataType is fixed length, in which case, based on arrayIndex, we can just calculate what the offset should be:
    ptr.set(ptr.get(), ptr.getOffset() + arrayIndex * baseDataType.getByteSize(), baseDataType.getByteSize());
2. The baseDataType is variable length, in which case you should be able to position based on the index you built in the header of the array:
    * Calculate offset into your index where we know the offset of arrayIndex starts
    * Read the next offset in your index to know where the element ends (special case if we're at the last offset)
    * Set ptr as needed

Don't worry about any error checking - if your serialization format is valid, it'll work.

One thing that would be prudent would be to reserve the first byte of the array serialization as the "version", in case we want to tweak this or support other formats down the road.