How to get top 10 possible set of words to calculate Top-K accuracy and MRR?
Author: hmdgit
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help
Dear All,

I am a newcomer to build language model with GPT-2. 

I have [fine-tuned](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py) a  language model by using GPT-2. 

Now, I would like to calculate Top-k accuracy and Mean Reciprocal Rank (MRR) of my model. For this purpose, I am using following strategy to get top 10 next predicted words:

1. Get a sub-sequence of text of length such as 40, contained in a test.txt, from the start of the file.
2. Next sub-sequence is created by moving a window of 40 words to next step 1. This process will go on untill, we get a list of all sub-sequences of test.txt file.
3. Pass the sub-sequences one by one to the generated model, which should give next 10 possible set of words.

For this purpose, I am using following segment of code to get top 10 words by adapting a code mentioned at this [link](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py):

```
    prompt_text = 'hello world' #Its an example. In reality it will be a complete subsequence of 40 words
    encoded_prompt = tokenizer.encode(prompt_text, add_special_tokens=False, return_tensors="pt")
    encoded_prompt = encoded_prompt.to('cpu')
    output_sequences = model.generate(
        input_ids=encoded_prompt,
        max_length=1+len(encoded_prompt[0]),
        top_p=0.9,
        do_sample=True,
        num_return_sequences=10
    )
```

Is it a right way to get top 10 next words  on the basis of input string, which helps me in calculating Top-k accuracy and MRR of my model? Kindly let me know about your concerns.



Comments:
I have written a function to calculate Top-k accuracy and MRR of a model, which is trained by using GPT-2. However, the function gives me very low values of Top-k accuracy and MRR. 

Kindly let me know anything wrong in this function?

```
def calculateModelPerformance(model,tokenizer):
    testData=readFileData('dataset/test.txt') # file contain words separated by space.
    step=1
    block_size=128
    top1 = 0.0
    top3 = 0.0
    top5 = 0.0
    top10 = 0.0
    mrr=0.0
	totalIterations=0.0
    for i in range(0, len(testData)-block_size, step):
        print("Iteration " + str(i+1))
        sequence=testData[i: i + block_size]
        next_word=testData[i + block_size]
        input_ids = torch.tensor(tokenizer.encode(sequence)).unsqueeze(0)
        # get logits of last predicted token
        next_word_logits = model(input_ids)[0][0, -1].detach()
        probabilities, indices = next_word_logits.topk(10)
        words = [tokenizer.decode(tir.item()) for tir in indices]
        rank = 1.0

        for word in words:
            if word == next_word:
                 mrr += 1.0/rank
                 if rank<=1.0:
                    top1+=1.0
                 if rank<=3.0:
                    top3+=1.0
                 if rank<=5.0:
                    top5+=1.0
                 if rank<=10.0:
                    top10+=1.0
                 print("MRR ", str(mrr))
                 print("Top 1 ",str(top1))
                 print("Top 3 ", str(top3))
                 print("Top 5 ",str(top5))
                 print("Top 10 ",str(top10))
                 break
            rank = rank + 1.0
		totalIterations +=1.0
		
    print("Total MRR ",str(mrr/totalIterations))
    print("Total Top-1 Accuracy ", str(top1 / totalIterations))
    print("Total Top-3 Accuracy ",str(top3/totalIterations))
    print("Total Top-5 Accuracy ", str(top5 / totalIterations))
    print("Total Top-10 Accuracy ", str(top10 / totalIterations))
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.