Normalized Discounted Cumulative Gain - Multilabel Ranking Metrics | ML - GeeksforGeeks

15 Jul, 2021
Discounted Cumulative Gain Discounted Cumulative Gain (DCG) is the metric of measuring ranking quality. It is mostly used in information retrieval problems such as measuring the effectiveness of the search engine algorithm by ranking the articles it displays according to their relevance in terms of the search keyword.
Let’s consider that a search engine that outputs 5 documents named ( D1, D2, D3, D4, D5) are output in that order. We need to define the relevance scale (0-3) where: 
0 : not relevant
1-2 : somewhat relevant
3 : completely relevant
Suppose these documents have relevance scores:  
D1 : 3
D2 : 2
D3 : 0
D4 : 0
D5 : 1
The Cumulative Gain is the sum of these relevance scores and can be calculated as: 
The discounted cumulative gain can be calculated by the formula: 
Therefore the discounted cumulative gain of above example is: 
Now we need to arrange these articles in descending order by rankings and calculate DCG to get the Ideal Discounted Cumulative Gain (IDCG) ranking. Now, we calculate our Normalized DCG using the following formula : 
Code : Python program for Normalized Discounted Cumulative Gain  
Python3
# import required packagefrom sklearn.metrics import ndcg_score, dcg_scoreimport numpy as np  # Relevance scores in Ideal ordertrue_relevance = np.asarray([[3, 2, 1, 0, 0]])  # Relevance scores in output orderrelevance_score = np.asarray([[3, 2, 0, 0, 1]])  # DCG scoredcg = dcg_score(true_relevance, relevance_score)print("DCG score : ", dcg)  # IDCG scoreidcg = dcg_score(true_relevance, true_relevance)print("IDCG score : ", idcg)  # Normalized DCG scorendcg = dcg / idcgprint("nDCG score : ", ndcg)  # or we can use the scikit-learn ndcg_score packageprint("nDCG score (from function) : ", ndcg_score(    true_relevance, relevance_score))
Output: 
DCG score :  4.670624189796882
IDCG score :  4.761859507142915
nDCG score :  0.980840401274087
nDCG score (from function) :  0.980840401274087
Limitations of Normalized Discounted Cumulative Gain (NDCG): 
NDCG metrics does not penalize the bad documents outputs. For Example: [3] and [3, 0, 0] has same NDCG but in second out there are two irrelevant documents. 
Because no specific standard defined for the number of output documents. DCG does not seem to deal with missing any relevant document in the output. For example, two outputs [3, 3, 3] and [3, 3, 3, 3] of similar input are considered equally good. For output-1 the DCG3 is calculated but for output-2 the DCG4 is calculated.
References: 
DCG Wikipedia article
Jarvelin, K., & Kekalainen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4), 422-446.
 
adnanirshad158
Machine Learning
Python
Machine Learning
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Comments
Old Comments
ML | Linear Regression
Decision Tree
Python | Decision tree implementation
Search Algorithms in AI
Difference between Informed and Uninformed Search in AI
Read JSON file using Python
Adding new column to existing DataFrame in Pandas
Python map() function
How to get column names in Pandas dataframe