[Optimization] Fix pointer moving issue when given offset is negative