TalkTix/roberta-base-priority-type-generator-55k

Model Details

This model is designed to classify customer service inquiries into four priorites: Low, Medium, High and Very High.

Training Data

The model was trained on a balanced dataset composed of anonymized customer service inquiries. Each category contained a similar number of examples to prevent class imbalance. https://github.com/amosproj/amos2023ws01-ticket-chat-ai/tree/main/Backend/app/model/test_data/test_data_with_gpt

Training Procedure

The model was fine-tuned over four epochs for a sequence classification task. We utilized a batch size of 4 and an Adam optimizer with a learning rate of 2e-5.

Model Performance

The model's performance was evaluated using a confusion matrix and a learning curve, as detailed below:

• Confusion Matrix Analysis

  

  • High: The model has performed well in classifying high-priority items, with 1800 correct predictions. However, there are 28 instances where high priority is confused with low, 460 with medium, and 84 with very high.
  • Low: There is some confusion in the low-priority classification, with 140 instances classified correctly, but 360 instances confused with medium priority and 200 with very high. The model rarely misclassifies low as high priority.
  • Medium: The model has classified medium priority with moderate accuracy, with 700 correct predictions. However, there's notable confusion with high priority (150 instances) and very high priority (36 instances).
  • Very High: This category shows significant confusion. While the model correctly identifies 410 very high priority instances, it also confuses 5 with low, 160 with high, and 200 with medium.

• Learning Curve Analysis

  

  • Training Loss: This line starts at approximately 0.94 and steadily decreases to around 0.82, indicating that the model is effectively learning from the training data.
  • Validation Loss: The validation loss begins just below 0.90 and decreases slightly after the first epoch, then levels off around 0.86. This behavior suggests that the model is not overfitting since the validation loss is not increasing as the model trains. However, the plateauing of the validation loss also suggests that the model may not be improving significantly after the first epoch.

• Interprating Model's Output:

  • LABEL_0 stands for High
  • LABEL_1 stands for Low
  • LABEL_2 stands for Medium
  • LAVEL_3 stands for Very High