I appear to have fixed the duplicates issue by adding the pivot element to the smaller array, I am not sure why this fixes the issue.