|
--- |
|
license: mit |
|
language: en |
|
tags: |
|
- bert |
|
- cloze |
|
- distractor |
|
- generation |
|
datasets: |
|
- cloth |
|
widget: |
|
- text: "I feel [MASK] now. [SEP] happy" |
|
- text: "The old man was waiting for a ride across the [MASK]. [SEP] river" |
|
--- |
|
|
|
# cdgp-csg-scibert-cloth |
|
|
|
## Model description |
|
|
|
This model is a Candidate Set Generator in **"CDGP: Automatic Cloze Distractor Generation based on Pre-trained Language Model", Findings of EMNLP 2022**. |
|
|
|
Its input are stem and answer, and output is candidate set of distractors. It is fine-tuned by [**CLOTH**](https://www.cs.cmu.edu/~glai1/data/cloth/) dataset based on [**allenai/scibert_scivocab_uncased**](https://huggingface.co/allenai/scibert_scivocab_uncased) model. |
|
|
|
For more details, you can see our **paper** or [**GitHub**](https://github.com/AndyChiangSH/CDGP). |
|
|
|
## How to use? |
|
|
|
1. Download the model by hugging face transformers. |
|
```python |
|
from transformers import BertTokenizer, BertForMaskedLM, pipeline |
|
|
|
tokenizer = BertTokenizer.from_pretrained("AndyChiang/cdgp-csg-scibert-cloth") |
|
csg_model = BertForMaskedLM.from_pretrained("AndyChiang/cdgp-csg-scibert-cloth") |
|
``` |
|
|
|
2. Create a unmasker. |
|
```python |
|
unmasker = pipeline("fill-mask", tokenizer=tokenizer, model=csg_model, top_k=10) |
|
``` |
|
|
|
3. Use the unmasker to generate the candidate set of distractors. |
|
```python |
|
sent = "I feel [MASK] now. [SEP] happy" |
|
cs = unmasker(sent) |
|
print(cs) |
|
``` |
|
|
|
## Dataset |
|
|
|
This model is fine-tuned by [CLOTH](https://www.cs.cmu.edu/~glai1/data/cloth/) dataset, which is a collection of nearly 100,000 cloze questions from middle school and high school English exams. The detail of CLOTH dataset is shown below. |
|
|
|
| Number of questions | Train | Valid | Test | |
|
| ------------------- | ----- | ----- | ----- | |
|
| Middle school | 22056 | 3273 | 3198 | |
|
| High school | 54794 | 7794 | 8318 | |
|
| Total | 76850 | 11067 | 11516 | |
|
|
|
You can also use the [dataset](https://huggingface.co/datasets/AndyChiang/cloth) we have already cleaned. |
|
|
|
## Training |
|
|
|
We use a special way to fine-tune model, which is called **"Answer-Relating Fine-Tune"**. More detail is in our paper. |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
|
|
- Pre-train language model: [allenai/scibert_scivocab_uncased](https://huggingface.co/allenai/scibert_scivocab_uncased) |
|
- Optimizer: adam |
|
- Learning rate: 0.0001 |
|
- Max length of input: 64 |
|
- Batch size: 64 |
|
- Epoch: 1 |
|
- Device: NVIDIA® Tesla T4 in Google Colab |
|
|
|
## Testing |
|
|
|
The evaluations of this model as a Candidate Set Generator in CDGP is as follows: |
|
|
|
| P@1 | F1@3 | F1@10 | MRR | NDCG@10 | |
|
| ---- | ---- | ----- | ----- | ------- | |
|
| 8.10 | 9.13 | 12.22 | 19.53 | 28.76 | |
|
|
|
## Other models |
|
|
|
### Candidate Set Generator |
|
|
|
| Models | CLOTH | DGen | |
|
| ----------- | ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | |
|
| **BERT** | [cdgp-csg-bert-cloth](https://huggingface.co/AndyChiang/cdgp-csg-bert-cloth) | [cdgp-csg-bert-dgen](https://huggingface.co/AndyChiang/cdgp-csg-bert-dgen) | |
|
| **SciBERT** | [*cdgp-csg-scibert-cloth*](https://huggingface.co/AndyChiang/cdgp-csg-scibert-cloth) | [cdgp-csg-scibert-dgen](https://huggingface.co/AndyChiang/cdgp-csg-scibert-dgen) | |
|
| **RoBERTa** | [cdgp-csg-roberta-cloth](https://huggingface.co/AndyChiang/cdgp-csg-roberta-cloth) | [cdgp-csg-roberta-dgen](https://huggingface.co/AndyChiang/cdgp-csg-roberta-dgen) | |
|
| **BART** | [cdgp-csg-bart-cloth](https://huggingface.co/AndyChiang/cdgp-csg-bart-cloth) | [cdgp-csg-bart-dgen](https://huggingface.co/AndyChiang/cdgp-csg-bart-dgen) | |
|
|
|
### Distractor Selector |
|
|
|
**fastText**: [cdgp-ds-fasttext](https://huggingface.co/AndyChiang/cdgp-ds-fasttext) |
|
|
|
|
|
## Citation |
|
|
|
None |
