remove offending chars
add option not to clean html; use set instead of list to store enetry id