Cleanup test code, remove the special case for size=1 since it just made things slower