YAML Metadata Warning:The pipeline tag "text2text-generation" is not in the official list: text-classification, token-classification, table-question-answering, question-answering, zero-shot-classification, translation, summarization, feature-extraction, text-generation, fill-mask, sentence-similarity, text-to-speech, text-to-audio, automatic-speech-recognition, audio-to-audio, audio-classification, audio-text-to-text, voice-activity-detection, depth-estimation, image-classification, object-detection, image-segmentation, text-to-image, image-to-text, image-to-image, image-to-video, unconditional-image-generation, video-classification, reinforcement-learning, robotics, tabular-classification, tabular-regression, tabular-to-text, table-to-text, multiple-choice, text-ranking, text-retrieval, time-series-forecasting, text-to-video, image-text-to-text, image-text-to-image, image-text-to-video, visual-question-answering, document-question-answering, zero-shot-image-classification, graph-ml, mask-generation, zero-shot-object-detection, text-to-3d, image-to-3d, image-feature-extraction, video-text-to-text, keypoint-detection, visual-document-retrieval, any-to-any, video-to-video, other
Model Card for Model ID
The model is a LoRA trained Flan-T5-Large. The shift in performace came from the LoRA Adapter Layer that was introduced to the model. It helped in training the model using cheaper hardware while not losing any of the accuracy, on the contrary it increased the accuracy. The takes as input 3 elements: Previous_Question, Previous_Answer and Current_Question. It will rephrase the Current_Question based on all the previouses.
Model Details
Model Description
- Model type: Language model
- Language(s) (NLP): English
- Finetuned from model: Flan-T5-Large
Uses
Direct Use
The primary use of the model is to generate a rephrased version of a follow-up question. The rephrased sentence will be a standalone sentence, that can be understood independantaly from the previous context. The output is supposed to be a sentence that has the same meaning/context as the combination of (Previous_Question, Previous_Answer, Current_Question)
Out-of-Scope Use
The model is not intended to be used in fixing grammar or rephrasing using a format not similar to the one the model was trained on.
Bias, Risks, and Limitations
The data used for rephrasing was mainly data that roams around the context that can be seen when dealing with the HR Chatbot for Azadea or the News Chatbot for AL Qabas.
Recommendations
It is always recommended to follow the input format that the model was trained on. The format is the following:
text = """
Question:
What about Mondays?
History:
ASSISTANT: No it is not allowed to work remotely on Fridays per company policy.
HUMAN: Can we work remotely on Fridays?
"""
How to Get Started with the Model
Use the code below to get started with the model.
from transformers import pipeline
pipe = pipeline("text2text-generation", model="EuriskoMobility/Flan-T5-Large-LoRA-Rephraser-Merged", max_new_tokens=64)
Training Details
Training Data
The data used will be provided shortly.
Training Procedure
Using a LoRA adapter while still using 32-bit precision, the model managed to generalize better. With a mix of dropout and weight decay, overfitting was reduced. Using both approaches the model reached performances never reached previously.
Training Hyperparameters
- Training regime:
- max_input_length: 512
- max_target_length: 64
- learning_rate: 1e-4
- per_device_train_batch_size: 8
- per_device_eval_batch_size: 8
- num_train_epochs: 10
- fp16: False
- seed: 42
- metric_for_best_model: "rougel"
Speeds, Sizes, Times [optional]
The model needs to be tested using pipeline for single input and batched input as well as deployment on Triton and TGI.
Evaluation
Testing Data, Factors & Metrics
The model was tested on both a test set and a list of key examples to tackle that were gathered around by the team. While "rougel" was taken into consideration, the aim of course was also to decrease training and validation loss as much as possible. Then the model was tested on the 63 examples gathered aside where he only rephrased incorrerctly 6 sentences. Some of them also being in a condition acceptable for the retriever system. as in the retriever would return the desired chunk.
Testing Data
The data used for testing was augmented data generated mainly from GPT-4 based on a few examples gathered by the team. It is a mix of HR related inputs as well as News related input. It has some conversational examples as well with the goal of generalizing more. Then there is a 63 input list that were manually tested by the team to judge the performance of the model.
Metrics
During training the following metrics were mainly taken into consideration:
- rouge1
- rougel
- training_loss
- evaluation_loss
Results
The best performing checkpoint that we got during training had the followng results:
- rouge1: 0.642358
- rougel: 0.604891
- training_loss: 0.657600
- evaluation_loss: 0.840903
Summary
The model still has room for improvement, but it would require an extensive amount of data that is much more diversified.
Technical Specifications [optional]
Model Architecture and Objective
The model uses Flan-T5-Large architecture as a base but adds to it a LoRA Adapter to help in generalization.
Hardware
1 x A100 GPU
- Downloads last month
- 3