|
--- |
|
license: apache-2.0 |
|
inference: false |
|
pipeline_tag: text-generation |
|
tags: |
|
- axolotl |
|
- generated_from_trainer |
|
- text-generation-inference |
|
model-index: |
|
- name: Mistral-7B-instruct-v0.2 |
|
results: [] |
|
model_type: mistral |
|
widget: |
|
- messages: |
|
- role: user |
|
content: I want to cancel an order |
|
--- |
|
|
|
# Mistral-7B-Customer-Support-v1 |
|
|
|
## Model Description |
|
|
|
This model is a fine-tuned version of the [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2), specifically tailored for customer service. It is designed to handle question answering tasks, providing responses based on a specialized customer support dataset. |
|
|
|
## Intended Use |
|
|
|
- **Recommended applications**: This model is intended for automated customer service. You can use it in chatbots, virtual assistants and copilots to answer user questions. |
|
- **Out-of-scope**: The model is not intended for general conversational purposes and should not be used for medical, legal, or safety-critical advice. |
|
|
|
## Usage Example |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
model = AutoModelForCausalLM.from_pretrained("bitext-llm/Mistral-7B-Customer-Support-v1") |
|
tokenizer = AutoTokenizer.from_pretrained("bitext-llm/Mistral-7B-Customer-Support-v1") |
|
|
|
inputs = tokenizer("<s>[INST] I want to change to the standard account [/INST] ", return_tensors="pt") |
|
outputs = model.generate(inputs['input_ids'], max_length=50) |
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
``` |
|
|
|
## Model Architecture |
|
|
|
The model employs the `MistralForCausalLM` architecture with a `LlamaTokenizer`. It maintains the configuration of the base Mistral model but has been fine-tuned to better understand and generate responses related to customer service. |
|
|
|
## Training Data |
|
|
|
The model was fine-tuned using the [bitext/Bitext-customer-support-llm-chatbot-training-dataset](https://huggingface.co/datasets/bitext/Bitext-customer-support-llm-chatbot-training-dataset), which is designed for question and answer interactions in the customer service sector. This dataset includes instructions and responses across a variety of customer service topics, ensuring that the model can handle a wide range of inquiries related to this field. The dataset covers 27 intents assigned to 10 categories such as `cancel_order`, `place_order`, `change_order`, and `check_invoice`. Each intent has around 1000 examples, illustrating a training process aimed at understanding and generating accurate responses for customer service interactions. |
|
|
|
## Training Procedure |
|
|
|
### Hyperparameters |
|
|
|
- **Optimizer**: AdamW with betas=(0.9, 0.999) and epsilon=1e-08 |
|
- **Learning Rate**: 0.0002 with a cosine learning rate scheduler |
|
- **Epochs**: 1 |
|
- **Batch Size**: 8 |
|
- **Gradient Accumulation Steps**: 4 |
|
- **Maximum Sequence Length**: 1024 tokens |
|
|
|
### Environment |
|
|
|
- **Transformers Version**: 4.40.0.dev0 |
|
- **Framework**: PyTorch 2.2.1+cu121 |
|
- **Tokenizers**: Tokenizers 0.15.0 |
|
|
|
## Training Results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | |
|
| :-----------: | :---: | :--: | :-------------: | |
|
| 1.6865 | 0.01 | 1 | 2.0557 | |
|
| 0.6351 | 0.25 | 32 | 0.8355 | |
|
| 0.5724 | 0.5 | 64 | 0.7859 | |
|
| 0.5249 | 0.75 | 96 | 0.7711 | |
|
| 0.516 | 1.0 | 128 | 0.7667 | |
|
|
|
## Dataset Description |
|
|
|
### Overview |
|
|
|
The dataset used for fine-tuning can train Large Language Models for both Fine Tuning and Domain Adaptation. It includes: |
|
|
|
- **Use Case**: Intent Detection |
|
- **Vertical**: Customer Service |
|
- **27 intents** assigned to 10 categories |
|
- **26,872 question/answer pairs**, around 1000 per intent |
|
- **30 entity/slot types** |
|
- **12 different types of language generation tags** |
|
|
|
### Categories and Intents |
|
|
|
The dataset covers the following categories and intents: |
|
|
|
- **ACCOUNT**: create_account, delete_account, edit_account, switch_account |
|
- **CANCELLATION_FEE**: check_cancellation_fee |
|
- **DELIVERY**: delivery_options |
|
- **FEEDBACK**: complaint, review |
|
- **INVOICE**: check_invoice, get_invoice |
|
- **NEWSLETTER**: newsletter_subscription |
|
- **ORDER**: cancel_order, change_order, place_order |
|
- **PAYMENT**: check_payment_methods, payment_issue |
|
- **REFUND**: check_refund_policy, track_refund |
|
- **SHIPPING_ADDRESS**: change_shipping_address, set_up_shipping_address |
|
|
|
### Entities |
|
|
|
The dataset includes various entities such as: |
|
|
|
- {{Order Number}}, {{Invoice Number}}, {{Online Order Interaction}}, {{Online Payment Interaction}}, {{Online Navigation Step}}, {{Online Customer Support Channel}}, {{Profile}}, {{Profile Type}}, {{Settings}}, {{Online Company Portal Info}}, {{Date}}, {{Date Range}}, {{Shipping Cut-off Time}}, {{Delivery City}}, {{Delivery Country}}, {{Salutation}}, {{Client First Name}}, {{Client Last Name}}, {{Customer Support Phone Number}}, {{Customer Support Email}}, {{Live Chat Support}}, {{Website URL}}, {{Upgrade Account}}, {{Account Type}}, {{Account Category}}, {{Account Change}}, {{Program}}, {{Refund Amount}}, {{Money Amount}}, {{Store Location}} |
|
|
|
### Language Generation Tags |
|
|
|
The dataset contains tags for various linguistic phenomena: |
|
|
|
- **Lexical Variation**: Morphological (M), Semantic (L) |
|
- **Syntactic Structure Variation**: Basic (B), Interrogative (I), Coordinated (C), Negation (N) |
|
- **Language Register Variations**: Politeness (P), Colloquial (Q), Offensive (W) |
|
- **Stylistic Variations**: Keyword (K), Abbreviations (E), Errors and Typos (Z) |
|
- **Other Tags**: Indirect Speech (D), Regional Variations (G), Respect Structures (R), Code Switching (Y) |
|
|
|
## Limitations and Bias |
|
|
|
- The model is fine-tuned on a domain-specific dataset and may not perform well outside the scope of customer service. |
|
- Users should be aware of potential biases in the training data, as the model's responses may inadvertently reflect these biases. The dataset aims to cover general customer service inquiries, but biases may exist for specific use cases. |
|
|
|
## Ethical Considerations |
|
|
|
This model should be used responsibly, considering ethical implications of automated customer service. It is important to ensure that the model's advice complements human expertise and adheres to relevant customer service guidelines. |
|
|
|
## Acknowledgments |
|
|
|
This model was developed by Bitext and trained on infrastructure provided by Bitext. |
|
|
|
## License |
|
|
|
This model, "Mistral-7B-Customer-Support-v1," is licensed under the Apache License 2.0 by Bitext Innovations International, Inc. This open-source license allows for free use, modification, and distribution of the model but requires that proper credit be given to Bitext. |
|
|
|
### Key Points of the Apache 2.0 License |
|
|
|
- **Permissibility**: Users are allowed to use, modify, and distribute this software freely. |
|
- **Attribution**: You must provide proper credit to Bitext Innovations International, Inc. when using this model, in accordance with the original copyright notices and the license. |
|
- **Patent Grant**: The license includes a grant of patent rights from the contributors of the model. |
|
- **No Warranty**: The model is provided "as is" without warranties of any kind. |
|
|
|
You may view the full license text at [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0). |
|
|
|
This licensing ensures the model can be used widely and freely while respecting the intellectual contributions of Bitext. For more detailed information or specific legal questions about using this license, please refer to the official license documentation linked above. |
|
|
|
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl) |
|
|
|
<details><summary>See axolotl config</summary> |
|
|
|
axolotl version: `0.4.0` |
|
|
|
```yaml |
|
base_model: mistralai/Mistral-7B-Instruct-v0.2 |
|
model_type: MistralForCausalLM |
|
tokenizer_type: LlamaTokenizer |
|
is_mistral_derived_model: true |
|
|
|
hub_model_id: malmarjeh/Mistral-7B-instruct-v0.2 |
|
|
|
load_in_8bit: false |
|
load_in_4bit: true |
|
strict: false |
|
|
|
datasets: |
|
- path: bitext/Bitext-customer-support-llm-chatbot-training-dataset |
|
type: |
|
system_prompt: 'You are an expert in customer support.' |
|
field_instruction: instruction |
|
field_output: response |
|
format: '[INST] {instruction} [/INST]' |
|
no_input_format: '[INST] {instruction} [/INST]' |
|
|
|
#datasets: |
|
# - path: json |
|
# type: alpaca_w_system.load_open_orca |
|
#data_files: file.zip |
|
|
|
dataset_prepared_path: |
|
|
|
val_set_size: 0.05 |
|
output_dir: ./qlora-out |
|
|
|
adapter: qlora |
|
lora_model_dir: |
|
|
|
sequence_len: 1024 |
|
sample_packing: true |
|
pad_to_sequence_len: true |
|
|
|
eval_sample_packing: False |
|
|
|
lora_r: 32 |
|
lora_alpha: 16 |
|
lora_dropout: 0.05 |
|
lora_target_linear: true |
|
lora_fan_in_fan_out: |
|
lora_target_modules: |
|
- gate_proj |
|
- down_proj |
|
- up_proj |
|
- q_proj |
|
- v_proj |
|
- k_proj |
|
- o_proj |
|
|
|
wandb_project: axolotl |
|
wandb_entity: |
|
wandb_watch: |
|
wandb_name: |
|
wandb_log_model: |
|
|
|
gradient_accumulation_steps: 4 |
|
micro_batch_size: 8 |
|
num_epochs: 1 |
|
optimizer: adamw_bnb_8bit |
|
lr_scheduler: cosine |
|
learning_rate: 0.0002 |
|
|
|
train_on_inputs: false |
|
group_by_length: false |
|
bf16: auto |
|
fp16: |
|
tf32: false |
|
|
|
gradient_checkpointing: true |
|
early_stopping_patience: |
|
resume_from_checkpoint: |
|
local_rank: |
|
logging_steps: 1 |
|
xformers_attention: |
|
flash_attention: true |
|
|
|
loss_watchdog_threshold: 5.0 |
|
loss_watchdog_patience: 3 |
|
|
|
warmup_steps: 10 |
|
evals_per_epoch: 4 |
|
eval_table_size: |
|
eval_max_new_tokens: 128 |
|
saves_per_epoch: 1 |
|
debug: |
|
deepspeed: |
|
weight_decay: 0.0 |
|
fsdp: |
|
fsdp_config: |
|
special_tokens: |
|
bos_token: '<s>' |
|
eos_token: '</s>' |
|
unk_token: '<unk>' |
|
``` |
|
|
|
</details><br> |
|
|
