complete duplicate filter and fix major bugs