SEMANTIC APPROACH TO AGENT ROUTING USING A HYBRID ATTRIBUTE-BASED RECOMMENDER SYSTEM
Authors: Anwitha Paruchuri
Venue:  (2021)
Keywords: Hybrid Recommendation
Customer Relationship Management
Semantic Embedding
Approximate Nearest Neighbor

Abstract: Traditionally contact centers route an issue to an agent based on ticket load or skill of the agent. When a ticket comes into the system, it is either manually analyzed and pushed to an agent or automatically routed to an agent based on some business rules. A Customer Relationship Management (CRM) system often has predefined categories that an issue could belong to. The agents are generally proficient in handling multiple categories, the categories in the CRM system are often related to each other, and a ticket typically contains content across multiple categories. This makes the traditional approach sub-optimal. We propose a Hybrid Recommendation based approach that recommends top N agents for a ticket by jointly modelling on the interactions between the agents and categories as well as on the semantic features of the categories and the agents.

SEMANTIC APPROACH TO AGENT ROUTING USING A HYBRID ATTRIBUTE-BASEE RECOMMENDER SYSTEM
Traditionally contact centers route an issue to an agent based on ticket load or skill of the agent. When a ticket comes into the system, it is either manually analyzed and pushed to agent automatically routed to an agent based on some business rules. A Customer Relationship Management (CRM) system often has predefined categories that an issue could belong to. The agents are generally proficient in handling multiple categories, the categories in the CRM system are often related to each other, and ticket typically contains content across multiple categories. This makes the traditional approach sub-optimal. We propose a Hybrid Recommendation based approach that recommends top N agents for a ticket by jointly modelling on the interactions between the agents and categories as well as on the semantic features of the categories and the agents.
Motivation:
Agents
Model Architecture: We have available a dataset of tickets categorized into 41 categories, which are in principle mutually exclusive. However, some categories are more related to each other and therefore we must account for sematic information in the model. We also have a dataset of interactions between agents and categories of tickets. To describe the model formally, let A be the set of Agents, C be the set of categories. Each Agent interacts with number of categories, either favorably (a positive interaction), or unfavorably (a negative interaction). Each Agent and Category is represented by its meta data FA: (Total Case Completion Time, Average Sentiment Score, Average Customer Rating) and Fe: [N === E where N Number of Tickets in a Category, E=Word2Vec semantic embedding [1] vector of dimension [d: 300]. The interaction dataset between the categories and the agents is represented by the count of tickets the agent has resolved for a particular category: = fa1ici faz .. lam een (a E Cn E C) and counts by R = (Fai,cı, aaz,c2. "M Famm E A, Cn E C) where m is equal to the total number of the agents and the is equal to the number of the categories and ram,en the count of tickets solved by Agent am for Category Cm- We formulate the recommendation problem using a hybrid attribute based model called LightFM[2] which is based on jointly factorizing the agent-category interaction matrix, category-feature, and agent- feattre matrices. When new ticket (qn) comes into the system we encode 9n into a Word2Vee vector of dimension d: 300. We implemented Approximate Nearest Neighbor search (ANN) using Locality Sensitive Hashing [3] on the issue encodings to predict the category of the ticket (qn)) Given different
can't find
consice. Couls help plaxms?
Category
Reccmrandation Engine
ML Model
Figurel: Proposed Recommender System
categories such as c1,c2,....cn Rd, where d: 300 is the of dimension of the category encoding and query point q Rd. top category is given by Ar Argmin [D(ci, where D is distance metric, Cosine Similarity. The hybrid recommender (Figure 1) calculates the similarity scores between the ticket and the categories to predict the category with the highest similarity score as the suggested category(Cp). Category (cp) is used to retrieve the top N agents from the recommender model.
Model Evaluation: We measure model accuracy using the mean receiver operating characteristics area under the curve (ROC AUC) metric. For a category, AUC corresponds to the probability that a randomly chosen positive agent will be ranked higher than randomly chosen negative agent. We compute this metric for all categories in the test set and average it for the final score as shown in Table 1.
Metric
logistic
BRP**
WARPss
Train AUC (no features)
0.81
0.738
0.761
Test AUC (no features)
0.84
0.785
0.782
Train AUC (with fcatures)
0,8235
0.75
0.773
Test AUC (with features)
0.8562
0.812
0.79
Table 1: AUC metric computed for different loss functions. We see that the
The proposed recommender system, which considers meta data features along with the historic interactions between agents and categories, is better when compared to the traditional approach of agent routing. This is of high importance when a ticket is of a new category or if the top recommended agent is busy. Our results also show that the model augmented by the meta data features outperforms the standard collaborative filtering model. This recommender system recommends top N agents who can best solve the problem despite their experience and background.
Conclusion:
* Refer to [2] [10] WARP and BPR loss functions

---

References:
[1] Tomáš Mikolov, Ilya Sutskever, Kai Chen, Greg S. Cor- rado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Adv. NIPS.
[2] Kula, M.: Metadata embeddings for user and item cold-start recommendations. In: CEUR Workshop Proceedings (2015)
[3] Cayton, Lawerence (2008). "Fast nearest neighbor retrieval for bregman divergences". Proceedings of the 25th International Conference on Machine Learning: 112 119. doi: 15/11111///11115..33900 ISBN 978 1781655522
[5] Pennington, Jeffrey, Richard Socher, and Christopher Manning. (2014) "Glove: Global vectors for word representation." In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) 1532-1543.
[6] Z. Shou, D. Wang, and S.-F. Chang. Intent Discovery Through Unsupervised Semantic Text Clustering. In Interspeech 2018.
[7] Shao, Y. Chen, S. Tao, X. Yan, and N. Anerousis. Efficient ticket routing by resolution sequence mining. In SIGKDD'OS, pages 605 613, 2008.
[8] K. Balog, L. Azzopardi, and M. de Rijke. Formal models for expert finding enterprise corpora. In SIGIR '06, pages 43-50, 2006.
[9] H. Fang and C. Zhai. Probabilistic models for expert finding. In ECIR 07, pages 418 4300, 2007.
[10] S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty Artificial Intelligence, pages 452-461, AUAI Press, 2009.
* Refer to [10] WARP loss functions


Reviews:
[{"review_id": "sdzlugc2grb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submissions deals with the problem of assigning tickets to agents in a Customer Relationship Management (CRM) system. The motivation is that the traditional way of assigning tickets does not take into account the interactions between agents and the categories of the tickets. Three algorithms (Table 1) are compared on the AUC measure, however the details of the experiment are not clear. It is also not clear how the experiments were implemented. Figure 1 is hard to read and the caption on Table 1 is incomplete. Several features are used for agents and tickets but their details are not provided, e.g., how are the word2vec features described in Section \"Model Architecture\" obtained? The results are hard to understand too. Which model are being compared exactly, and what are their parameters? The \"Model Architecture\" section talks about a LightFM based model, while the table describes a model named \"logistic\", another named \"BRP\" (is it supposed to be the well known BPR-MF algorithm?) and the third one named \"WARP.\" None of these are described. The conclusion refers to a \"proposed recommender system\", but no such system is proposed in the paper. This submission cannot be accepted to the conference.", "extra_scores": {}, "extra_text": {}}, {"review_id": "gppyPgU9-vg", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use lightFM to solve the matching of the agent and the ticket belonging to some categories.\n\nUsing lightFM to solve this application is interesting. However, there is no modification on lightFM, except testing three loss fuction. The theoretical contribution and novelty are limited.\n\nThe experiments are also not enough. They are lightFMs of three loss fuctions, without other compared algorithms. So the statements “is better when compared to the traditional approach of agent routing” and  “outperforms the standard collaborative filtering model” in the Conclusion section are not convincing.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UoYuz-hpxxt", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "* First of all, the submission is not following the style.\n\n* The paper only has one page.\n\n* The proposed method is not novel, and it is not described clearly. \n\n* Lack of experiments and comparisons. Similarly, a one-page paper will have a very hard time to deliver enough story. There is only one table in the whole paper.\n\n* BRP or BPR?\n\n* I also found that it is on a YouTube video (https://www.youtube.com/watch?v=Rj5E724O4nA&ab_channel=anwithaparuchuri) mentioning it is a submission to another venue (WeCNLP). It looks the results (the only table) are totally identical, so it probably should be considered as dual submission. Also from the video, probably it also violates the double-blind policy.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Does not seem to be a complete submission (only one page), all reviewers agree on rejecting.
Decision: Reject