|
--- |
|
license: mit |
|
--- |
|
### Synthetic IT Call Center Data Sentence Quality Predictor |
|
A RoBERTa-base model fine-tuned on a synthetic dataset of good and bad sentences that would be found in IT call center tickets. This model aims to predict the quality of sentences in the context of IT support communications, providing a numerical score from 0 to 10, where 0 represents a poor quality sentence, and 10 represents an ideal quality sentence. |
|
|
|
#### Model Background |
|
This model was created out of the necessity to objectively measure the quality of IT call center journaling and improve overall customer service. By leveraging OpenAI's GPT-4 to simulate both effective and ineffective call center agent responses, and then using GPT-4-turbo to rank these responses, we've synthesized a unique dataset that reflects a wide range of possible interactions in an IT support context. The dataset comprises 1,464 items, each scored and annotated with insights into what constitutes quality journaling vs. poor journaling. |
|
|
|
#### Approach |
|
The foundation of this model is the RoBERTa-base transformer, chosen for its robust performance in natural language understanding tasks. I extended and fine-tuned the last four layers of RoBERTa to specialize in our sentence quality prediction task. This fine-tuning process involved manual adjustments and iterative training sessions to refine the model's accuracy and reduce the Mean Squared Error (MSE) on the validation set. |
|
|
|
#### Performance |
|
After several rounds of training and manual tweaks, the model achieved a validation MSE of approximately 1.66. This metric indicates the model's ability to closely predict the quality scores assigned by the simulated call center manager, with a lower MSE reflecting higher accuracy in those predictions. |
|
|
|
#### Future Work |
|
The journey to perfecting this model is ongoing. Plans to improve its performance include: |
|
|
|
- Expanding the training dataset with more synthesized examples to cover a broader spectrum of potential customer interactions. |
|
- Experimenting with adjusting and fine-tuning additional layers of the RoBERTa model to see if that yields better predictive accuracy. |
|
- Exploring other evaluation metrics beyond MSE to ensure the model's predictions are as useful and actionable as possible in a real-world IT call center environment. |
|
|
|
#### How to Use This Model |
|
This model is designed for integration into IT call center software systems, where it can automatically score incoming and outgoing ticket responses for quality. To use this model: |
|
|
|
1. Ensure you have the Hugging Face Transformers library installed in your Python environment. |
|
2. Load the model using the following code snippet: |
|
|
|
```python |
|
from transformers import AutoModelForSequenceClassification, AutoTokenizer |
|
|
|
model_name = "KameronB/sitcc-roberta" |
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
model = AutoModelForSequenceClassification.from_pretrained(model_name) |
|
|
|
def predict(sentence): |
|
inputs = tokenizer(sentence, return_tensors="pt") |
|
outputs = model(**inputs) |
|
return outputs.logits |
|
|
|
sentence = "Your example sentence here." |
|
score = predict(sentence) |
|
print(f"Quality Score: {score}") |
|
|
|
|
