[#1136] Fix possible IllegalBufferAccessException when remove itself from the pipeline