DistilBERT MCQ Correctness Classifier

Author: Anurag Singh
Framework: Hugging Face Transformers + PyTorch
Base Model: distilbert-base-uncased
Task: Binary classification for multiple-choice question answering

Model Description

This model is a fine-tuned DistilBERT model developed for the Smart MCQ Solver Challenge.

The original task contains a question with five answer options (A-E). The problem was converted into a binary classification task by creating one training example for each (question, option) pair.

Label 1: The option is correct.
Label 0: The option is incorrect.

During inference, the model predicts a probability for each option, and the option with the highest probability is selected as the final answer.

Problem Formulation

Original MCQ

Question: What is the capital of France?

A. Berlin
B. Madrid
C. Paris
D. Rome
E. Lisbon

Converted Binary Samples

Input	Label
Question: What is the capital of France? Option: Berlin	0
Question: What is the capital of France? Option: Madrid	0
Question: What is the capital of France? Option: Paris	1
Question: What is the capital of France? Option: Rome	0
Question: What is the capital of France? Option: Lisbon	0

Model Architecture

This model is based on DistilBERT, a compressed version of BERT that retains most of BERT's language understanding capability while being significantly smaller and faster.

Architecture

                      text Question + Option Text
                                  ↓
                          DistilBERT Tokenizer
                                  ↓
                              Input IDs
                                  ↓
                      DistilBERT Encoder (6 Transformer Layers)
                                  ↓
                      Contextual Representation
                                  ↓
                      Classification Head (Linear Layer)
                                  ↓
                      Binary Correctness Score

Model Details

Component	Value
Base Model	`distilbert-base-uncased`
Transformer Layers	6
Hidden Size	768
Attention Heads	12
Vocabulary Size	30,522
Max Position Embeddings	512
Classification Head	Linear layer (`768 → 2`)
Output Labels	2 (`incorrect`, `correct`)

Input Representation

Each training sample is constructed as:

Question: <question> Option: <option>

The text is tokenized using the DistilBERT WordPiece tokenizer and converted into:

input_ids
attention_mask

Classification

The contextual representation produced by DistilBERT is passed to a linear classification head that outputs logits for two classes:

Class 0: Incorrect option
Class 1: Correct option

During inference, the probability of class 1 is used as the correctness score for each option. The option with the highest score among A-E is selected as the final answer.

distilbert_output

This model is a fine-tuned version of bert-base-uncased on the private dataset. It achieves the following results on the evaluation set:

Loss: 0.0985
Accuracy: 0.986
Precision: 0.9769
Recall: 0.9525
F1: 0.9646

Intended uses & limitations

Intended Uses

Multiple-choice question answering
Educational NLP applications
Research on transformer-based text classification
Binary text classification
Demonstration of parameter-efficient NLP models

Limitations

The model is trained only on the project dataset and may not generalize well to unrelated domains.
Performance depends on the quality and wording of the question and options.
The model evaluates each question-option pair independently and does not explicitly model relationships between all options simultaneously.
It is intended for educational and research purposes and should not be used in high-stakes decision-making systems without further validation.

Training and evaluation data

The model was trained on a multiple-choice question dataset where each question contained five candidate answers (A-E).

The original dataset was converted into a binary classification dataset by generating one training sample for every (question, option) pair.

Training labels:

0 → Incorrect option
1 → Correct option

Each input was tokenized using the DistilBERT tokenizer with appropriate padding and truncation before training.

Model performance was evaluated using:

Accuracy
Precision
Recall
F1-score
Validation Loss

Training procedure

Preprocessing

Combined each question with one candidate option to create the input text.
Tokenized the input using the DistilBERT WordPiece tokenizer.
Applied padding and truncation to a fixed maximum sequence length.
Converted labels into binary classes.

Fine-tuning

The model was fine-tuned using Hugging Face Transformers with the following configuration:

Parameter	Value
Base Model	distilbert-base-uncased
Task	Binary Sequence Classification
Loss Function	CrossEntropyLoss
Optimizer	AdamW
Learning Rate	2e-5
Number of Labels	2
Framework	PyTorch + Hugging Face Transformers

The trained model predicts the probability that a candidate option is correct. For each MCQ, all options are scored independently, and the option with the highest probability is selected as the final answer.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 32
eval_batch_size: 32
seed: 42
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 5

Training results

The DistilBERT model achieved the best overall performance among the evaluated models in this project, outperforming both the custom BiLSTM and BERT models in terms of validation performance and Kaggle leaderboard score. The model was selected as the final submission because it provided the best balance between prediction accuracy, computational efficiency, and inference speed.

Training Loss	Epoch	Step	Validation Loss	Accuracy	Precision	Recall	F1
0.2479	1.0	250	0.3284	0.9475	0.9743	0.7575	0.8523
0.2102	2.0	500	0.2452	0.952	0.8781	0.8825	0.8803
0.1279	3.0	750	0.1781	0.975	0.9605	0.9125	0.9359
0.1124	4.0	1000	0.1519	0.9775	0.9449	0.9425	0.9437
0.0485	5.0	1250	0.0985	0.986	0.9769	0.9525	0.9646

Framework versions

Transformers 5.0.0
Pytorch 2.10.0+cu128
Datasets 4.8.5
Tokenizers 0.22.2

Downloads last month: -

Safetensors

Model size

67M params

Tensor type

F32

Model tree for anuragsinghiitm/distilbert-nppe

Base model

distilbert/distilbert-base-uncased

Finetuned

(12210)

this model

anuragsinghiitm
/

distilbert-nppe