Only optimize the byte array access if possible