|
--- |
|
language: |
|
- Python |
|
tags: |
|
- NLP |
|
- Fake News Detection |
|
- XLM RoBERTa |
|
datasets: |
|
- https://www.kaggle.com/datasets/clmentbisaillon/fake-and-real-news-dataset |
|
metrics: |
|
- Accuracy |
|
- F1-score |
|
--- |
|
|
|
# Write up: |
|
|
|
## Link to hugging face model: |
|
https://huggingface.co/Sajib-006/fake_news_detection_xlmRoberta |
|
|
|
## Model Description: |
|
* Used pretrained XLM-Roberta base model. |
|
* Added classifier layer after bert model |
|
* For tokenization, i used max length of text as 512(which is max bert can handle) |
|
|
|
## Result: |
|
* Using bert base uncased english model, the accuracy was near 85% (For all samples) |
|
* Using XLM Roberta base model, the accuracy was almost 100% ( For only 2k samples) |
|
|
|
## Limitations: |
|
* Pretrained XLM Roberta is a heavy model. Training it with the full dataset(44k+ samples) was not possible using google colab free version. So i had to take small sample of 2k size for my experiment. |
|
* As we can see, there is almost 100% accuracy and F1-score for 2000 dataset, so i haven't tried to find misclassified data. |
|
* I couldn't run the model for the whole dataset as i used google colab free version, there was RAM and disk restrictions. XLMRoberta is a heavy model, so training it for the full dataset tends to take huge time. Colab doesn't provide GPU for long time. |
|
* As one run for one epoch took huge time, i had to save checkpoint after 1 epoch and retrain the model loading weights for 2nd time. After 2 epoch it showed almost 100% accuracy, so i didn't continue to train again. |
|
* A more clear picture could have been seen if it could be run for the full dataset. I thought of some ideas about better model but couldn't implement for hardware restriction as mentioned and time constraint. My ideas are given below. |
|
|
|
## Ideas to imrove on full dataset: |
|
* Using XLM Roberta large instead of base can improve |
|
* Adding dense layer and dropout layer to reduce overfitting(Though in my result there is 100% accuracy on hold-out test set, so no overfitting seems to be there) |
|
* Adding convolutional layer after the bert encoder work even better. |
|
* Combination of different complex convolution layers can be added to check if accuracy increases further more. |
|
* Hyperparameter tuning of the layers to ensure best result. |
|
|
|
|
