finished filters and sorting for documents
added support to remove duplicates as resilience