Only optimize byte array access if its safe