ran some tests, added duplicate/sort function but need to optimize it