Update README.md
Browse files
README.md
CHANGED
@@ -3,4 +3,85 @@ license: gpl-3.0
|
|
3 |
datasets:
|
4 |
- medalpaca/medical_meadow_medical_flashcards
|
5 |
pipeline_tag: question-answering
|
6 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3 |
datasets:
|
4 |
- medalpaca/medical_meadow_medical_flashcards
|
5 |
pipeline_tag: question-answering
|
6 |
+
---
|
7 |
+
# Model Description
|
8 |
+
This is a fine-tuned version of the Minerva model, trained on the [Medical Meadow Flashcard Dataset](https://huggingface.co/datasets/medalpaca/medical_meadow_medical_flashcards) for question answering. The model was developed by the Sapienza NLP Team in collaboration with Future Artificial Intelligence Research (FAIR) and CINECA; specifically, I used the version with 350 million parameters due to computational limits, though versions with 1 billion and 3 billion parameters also exist. For more details, please refer to their repositories: [Sapienza NLP on Hugging Face](https://huggingface.co/sapienzanlp) and [Minerva LLMs](https://nlp.uniroma1.it/minerva/).
|
9 |
+
|
10 |
+
# Issues and possible Solutions
|
11 |
+
- In the original fine-tuned version, my model tended to generate answers that continued unnecessarily, leading to repeated sentences and a degradation in quality over time. Parameters like '*max_length*' or '*max_new_tokens*' were ineffective as they merely stopped the generation at a specified point without properly concluding the sentence. To address this issue, I redefined the stopping criteria to terminate the generation at the first period ('.'), as demonstrated in the code below:
|
12 |
+
- ```python
|
13 |
+
class newStoppingCriteria(StoppingCriteria):
|
14 |
+
|
15 |
+
def __init__(self, stop_word):
|
16 |
+
self.stop_word = stop_word
|
17 |
+
|
18 |
+
def __call__(self, input_ids, scores, **kwargs):
|
19 |
+
|
20 |
+
decoded_text = tokenizer.decode(input_ids[0], skip_special_tokens=True)
|
21 |
+
return self.stop_word in decoded_text
|
22 |
+
|
23 |
+
|
24 |
+
criteria = newStoppingCriteria(stop_word = ".")
|
25 |
+
stoppingCriteriaList = StoppingCriteriaList([criteria])
|
26 |
+
```
|
27 |
+
|
28 |
+
- Since the preprocessed text was formatted as "BoS token - Question - EoS token - BoS token - Answer - EoS token," the model generated answers that included the question as well. To resolve this, I implemented a method to remove the question from the generated text, leaving only the answer:
|
29 |
+
|
30 |
+
- ```python
|
31 |
+
outputText = tokenizer.decode(output_ids[0], skip_special_tokens = True)
|
32 |
+
inputText = tokenizer.decode(inputEncoding.input_ids[0], skip_special_tokens = True)
|
33 |
+
answer = outputText[len(inputText):].strip()
|
34 |
+
```
|
35 |
+
|
36 |
+
# Use Example
|
37 |
+
|
38 |
+
```python
|
39 |
+
question = 'What causes Wernicke encephalopathy?'
|
40 |
+
|
41 |
+
inputEncoding = tokenizer(question, return_tensors = 'pt').to('cuda')
|
42 |
+
output_ids = model.generate(
|
43 |
+
|
44 |
+
inputEncoding.input_ids,
|
45 |
+
max_length = 128,
|
46 |
+
do_sample = True,
|
47 |
+
temperature = 0.7,
|
48 |
+
top_p = 0.97,
|
49 |
+
top_k = 2,
|
50 |
+
pad_token_id = tokenizer.eos_token_id,
|
51 |
+
repetition_penalty = 1.2,
|
52 |
+
stopping_criteria = stoppingCriteriaList
|
53 |
+
)
|
54 |
+
|
55 |
+
outputText = tokenizer.decode(output_ids[0], skip_special_tokens = True)
|
56 |
+
inputText = tokenizer.decode(inputEncoding.input_ids[0], skip_special_tokens = True)
|
57 |
+
answer = outputText[len(inputText):].strip()
|
58 |
+
|
59 |
+
# Generated Answer: Wernicke encephalopathy is caused by a defect in the Wern-Herxheimer reaction, which leads to an accumulation of acid and alkaline phosphatase activity.
|
60 |
+
# Effective Answer: The underlying pathophysiologic cause of Wernicke encephalopathy is thiamine (B1) deficiency.
|
61 |
+
```
|
62 |
+
|
63 |
+
# Training Information
|
64 |
+
The model was fine-tuned for 3 epochs using the parameters specified in its original repository:
|
65 |
+
|
66 |
+
```python
|
67 |
+
trainingArgs = TrainingArguments(
|
68 |
+
|
69 |
+
output_dir = "MedicalFlashcardsMinerva",
|
70 |
+
evaluation_strategy = "steps",
|
71 |
+
save_strategy = "steps",
|
72 |
+
learning_rate = 2e-4,
|
73 |
+
per_device_train_batch_size = 6,
|
74 |
+
per_device_eval_batch_size = 6,
|
75 |
+
gradient_accumulation_steps = 8,
|
76 |
+
num_train_epochs = 3,
|
77 |
+
lr_scheduler_type = "cosine",
|
78 |
+
warmup_ratio = 0.1,
|
79 |
+
adam_beta1 = 0.9,
|
80 |
+
adam_beta2 = 0.95,
|
81 |
+
adam_epsilon = 1e-8,
|
82 |
+
weight_decay = 0.01,
|
83 |
+
logging_steps = 100,
|
84 |
+
report_to = "none",
|
85 |
+
|
86 |
+
)
|
87 |
+
```
|