Python | Removing duplicate dicts in list

02 Dec, 2018
Removal of duplicates is essential in many applications. List of dictionaries are quite common and sometimes we require to duplicate the duplicated. Lets discuss certain ways in which this can be achieved.
Method #1 : Naive methodThe basic method that comes to mind while performing this operation is the naive method of iterating the list of dictionaries and manually removing the duplicate dictionary and append in new list.
# Python3 code to demonstrate # remove duplicate dictionary# using naive method   # initializing listtest_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat" : 3}, {"Kil" : 2}, {"Akshat" : 3}]  # printing original list print ("Original list : " + str(test_list))  # using naive method to # remove duplicates res_list = []for i in range(len(test_list)):    if test_list[i] not in test_list[i + 1:]:        res_list.append(test_list[i])  # printing resultant listprint ("Resultant list is : " + str(res_list))
Output :
Original list : [{'Akash': 1}, {'Kil': 2}, {'Akshat': 3}, {'Kil': 2}, {'Akshat': 3}]
Resultant list is : [{'Akash': 1}, {'Kil': 2}, {'Akshat': 3}]

Method #2 : Using list comprehensionThe use of list comprehension and enumerate can possibly allow to achieve this particular task in a single line and hence is of a good utility.
# Python3 code to demonstrate # remove duplicate dictionary# using list comprehension  # initializing listtest_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat" : 3}, {"Kil" : 2}, {"Akshat" : 3}]  # printing original list print ("Original list : " + str(test_list))  # using list comprehension to # remove duplicates res_list = [i for n, i in enumerate(test_list) if i not in test_list[n + 1:]]  # printing resultant listprint ("Resultant list is : " + str(res_list))
Output :
Original list : [{'Akash': 1}, {'Kil': 2}, {'Akshat': 3}, {'Kil': 2}, {'Akshat': 3}]
Resultant list is : [{'Akash': 1}, {'Kil': 2}, {'Akshat': 3}]

Method #3 : Using frozensetfrozenset is used to assign a value to key in dictionary as a set. The repeated entries of dictionary are hence ignored and hence solving this particular task.
# Python3 code to demonstrate # remove duplicate dictionary# using frozenset  # initializing listtest_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat" : 3}, {"Kil" : 2}, {"Akshat" : 3}]  # printing original list print ("Original list : " + str(test_list))  # using frozenset to # remove duplicates res_list = {frozenset(item.items()) : item for item in test_list}.values()  # printing resultant listprint ("Resultant list is : " + str(res_list))
Output :
Original list : [{'Akash': 1}, {'Kil': 2}, {'Akshat': 3}, {'Kil': 2}, {'Akshat': 3}]
Resultant list is : [{'Kil': 2}, {'Akshat': 3}, {'Akash': 1}]

Python dictionary-programs
Python list-programs
python-dict
python-list
Python
python-dict
python-list
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Python Dictionary
Different ways to create Pandas Dataframe
Enumerate() in Python
Read a file line by line in Python
Python String | replace()
How to Install PIP on Windows ?
*args and **kwargs in Python
Python Classes and Objects
Python OOPs Concepts
Iterate over a list in Python