QueryCategorizer

This model is a fine-tuned version of distilbert-base-uncased on the bitext/Bitext-customer-support-llm-chatbot-training-dataset. It achieves the following results on the evaluation set:

• Loss: 0.0038
• Accuracy: 0.9996
• Precision: 0.9996
• Recall: 0.9996
• F1: 0.9996

Model description

A fine-tuned DistilBERT model that classifies customer support queries into one of 9 departments, designed to auto-route incoming support tickets to the correct team.

Model Details

• Base model: distilbert-base-uncased
• Task: Multi-class text classification (9 classes)
• Language: English
• Fine-tuned on: bitext/Bitext-customer-support-llm-chatbot-training-dataset

The model predicts one of the following 9 departments:

ACCOUNT, CONTACT, DELIVERY, FEEDBACK, INVOICE, ORDER, PAYMENT, REFUND, SHIPPING

Intended uses & limitations

• Routing incoming customer support emails/messages to the appropriate department
• Prototyping multi-class text classification pipelines
• Educational demonstrations of fine-tuning transformer models for classification
• It is limited to just classify the message but not forward them to the appropriate department
• Out of 11 labels mentioned in dataset, we only took 9 and removed "Cancel" and "Subscription" because of limited no of data samples for that class.

Training and evaluation data

The bitext/Bitext-customer-support-llm-chatbot-training-dataset was used for training and evaluation of the model. 80% was reserved for training, 10% for validation and 10% for testing.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Precision	Recall	F1
0.0079	1.0	1125	0.0053	0.9988	0.9988	0.9988	0.9988
0.0036	2.0	2250	0.0061	0.9984	0.9984	0.9984	0.9984
0.0009	3.0	3375	0.0038	0.9996	0.9996	0.9996	0.9996
0.0005	4.0	4500	0.0038	0.9992	0.9992	0.9992	0.9992
0.0001	5.0	5625	0.0029	0.9996	0.9996	0.9996	0.9996
0.0000	6.0	6750	0.0028	0.9996	0.9996	0.9996	0.9996

Framework versions

• Transformers 5.16.1
• Pytorch 2.11.0+cu128
• Datasets 4.8.5
• Tokenizers 0.23.1