Only optimize the byte array access if possible
Merge pull request #286 from normanmaurer/master

Only optimize byte array access if its safe