non-duplicate entries filtering step added