Remove code block which handles duplicates, in fact duplicates can be allowed in chuncked data