# Building a Knowledge-Intensive, Intent-Lean, Question Answering Chatbot in the Telecom Industry - Challenges and Solutions

Antonios Misargopoulos, Filippos Nikolopoulos-Gkamatsis, Konstantinos Nestorakis, Alexandros Tzoumas, Georgios Giannakopoulos,

Christos-Antonios Gizelis, Michalis Kefalogiannis

<figure><img src="image_1.png" /></figure>

## To cite this version:

Antonios Misargopoulos, Filippos Nikolopoulos-Gkamatsis, Konstantinos Nestorakis, Alexandros Tzoumas, Georgios Giannakopoulos, et al. Building a Knowledge-Intensive, Intent-Lean, Question Answering Chatbot in the Telecom Industry - Challenges and Solutions. 18th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2022, Hersonissos, Greece. pp.87-97, 10.1007/978-3-031-08341-9_8. hal-05335640

HAL Id: hal-05335640

https://inria.hal.science/hal-05335640v1

Submitted on 28 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Building a Knowledge-Intensive, Intent-Lean, Question Answering Chatbot in the Telecom Industry - Challenges and Solutions

Antonios Misargopoulos<fnref n="1" /> ID, Filippos Nikolopoulos-Gkamatsis<fnref n="1" /> ID, Konstantinos Nestorakis<fnref n="1" /> ID, Alexandros Tzoumas<fnref n="2" /> ID, Georgios Giannakopoulos<fnref n="3" /> ID, Christos-Antonios Gizelis<fnref n="1" />(✉) ID, and Michalis Kefalogiannis<fnref n="1" /> ID

**Abstract.** Artificial Intelligence (AI) is one of the most emerging tech-
nologies of the past decade, leading the way towards human and machine
interactions in terms of efficiency, accuracy and the overall value gained.
This paper describes an intent-lean AI chatbot solution that handles
user queries posed in natural language upon business related documents
of a single-domain of Hellenic Telecommunications Organization S.A.
(HTO). Unlike other traditional chatbot solutions that strictly rely on
intent identification, our approach infers the implicit user need in order
to provide the most relative documents and text snippets within, as the
proper answer. To do this, we proceeded with a custom implementa-
tion based on Elasticsearch engine and most common NLP techniques
tailored to our needs; i.e., tokenization, lowercase filtering, stop words
removal, stemming, fuzzy searching, synonyms, etc. The main challenges
as well as the architectural models that thrive to overcome them are
being described in detail. Finally, the effectiveness of the proposed solu-
tion is being measured and the identified features for improvement are
being presented.

**Keywords:** Artificial Intelligence (AI) · NLP · Chatbot · Intent-lean · Lemmatization · Stemming · Synonyms · Elasticsearch

# 1 Introduction

Artificial Intelligence (AI) technologies have been evolving in a tremendous pace
during the past few years expanding research opportunities and multi-domain

<footnote n="1">¹ Hellenic Telecommunications Organization S.A. (HTO), Marousi Attiki, Greece `{amisargopo,fnikolop,knestorak,cgkizelis,mkefalogiannis}@ote.gr`</footnote>

<footnote n="2">² SciFY PNPC, Ag. Paraskevi Attiki, Greece `a.tzoumas@scify.org`</footnote>

<footnote n="3">³ National Centre of Scientific Research “Demokritos”, Ag. Paraskevi Attiki, Greece `ggianna@iit.demokritos.gr`</footnote>
capabilities in large enterprises. AI chatbots offer profound contribution acting as interactive agents to handle costly user requests; i.e., place a new order, open a trouble ticket or navigate in a knowledge base.

The Hellenic Telecommunications Organization S.A. (HTO), as the largest telecommunications provider in Southeast Europe, serves its customers with a wide variety of ICT services addressing a large customer base. HTO owns a vast amount of official corporate documents governed by BPMN Specification [1] in Greek & English language; i.e., policies, procedure, processes, work instructions etc., that support the core operation on a daily basis. At the moment, this knowledge base is published to a custom intranet portal and utilized via a simple searching mechanism based on keywords filtering; but lacks to answer user questions. As a result, users fail to retrieve the required information; and so, seniors and subject-matter experts SMEs get involved inflicting an undesirable effort and labor expenses overhead.

This work proposes a knowledge-intensive, intent-lean AI chatbot solution to tackle the need to handle openly-posed queries *inferring* the implicit user need so that to provide correct and adequate answers; i.e., pairs of a relative document and text snippet within. In Sect. 2 we discuss the limitation of other chatbot implementations that strictly rely on intent identification; while in Sects. 3 and 4 we highlight some challenges and present our methodology and technical architecture in detail based on a custom implementation over *Elasticsearch* [2] functionality and other well-known AI Natural Language Processing (NLP) techniques. Later, in Sect. 5 we present our experimental results and in Sect 6 we conclude with some potential future work to enhance the algorithms' effectiveness and improve the overall user experience.

## 2 Related Work

AI has drawn the immense attention of both the academic community and the industry during last decade worldwide. One of the AI-based applications expected to have the most emerging development in near future, is conversational agents [3]. Conversational agents, also known as chatbots, or chatterbots, are machine conversation systems that interact with human users through natural conversational language. The Bank of America announced in December 2019 that its AI-driven virtual financial assistant Erica has surpassed 10mil users since its nationwide roll-out in 2018 [4]. Back in 2011 Apple Inc. introduced AI-powered virtual assistant *Siri* as part of all Apple devices operating systems, while Google came up with *Google Assistant* with advanced two-way conversations capabilities [5]. Moreover, chatbot applications leave influential imprint on multiple other domains such as education [6] or healthcare [7].

During the past years numerous research and academic publications aimed to designate the challenges around chatbot implementations and propose sustainable solutions. As already mentioned, they are based mainly on intent identification [8] to manage specific flow-based business cases like opening a fault ticket or placing a new order [9]. In this paper we present an end-to-end solution
that is not based in strict intent detection and can be deployed within industries
with diversity in knowledge domain almost transparently.

## 3 Challenges and Problem Definition

The existing corporate portal in place supports keyword-based search function-
ality upon specific document attributes, like title or description; but is totally
incapable of handling free-text questions. In 2020, more than 300 incidents have
been recorded, where users experienced inadequate search response. To handle
those cases, SMEs have to be involved in an expensive and time-consuming effort
overhead. Thus, a sophisticated, custom chatbot solution has been promoted to
eliminate this operational cost. Some of the most critical challenges, that need
to be addressed, are presented below.

**Challenge 1. Data Diversity Makes Data Modelling Difficult:** The knowledge base consists of hundreds of documents mapped to approximately 10 different domains like Human Resource (HR), IT, Technology, Legal etc. The documents in each domain embed different vocabulary, notation or even formatting type, i.e., plain text, tabular, or images.

**Challenge 2. Complexity on the Type of Questions Asked by the Users:** Based on the questions that Procurement & Finance domain SMEs handle in daily basis, we concluded the following challenges:

(a) variety in the type of questions that can be made by the users; i.e., (i) simple questions that can be answered from a single and inseparable portion of the text inside one document, (ii) complex questions where the desired answer can be found in different locations inside one or more documents.

(b) diverse vocabulary and wording used by users of different expertise or role in the company

(c) vast number of semantically different questions. As already stated in the Challenge 1 above, this is a multi-domain knowledge base; consequently, the queries can vary in linguistic meaning dramatically.

**Challenge 3. Variety of User Intents:** Traditional chatbot engines focus on serving answers based on user intents. The user query is analyzed and fragmented into one or more intents to describe distinct scenarios and different business logic each; e.g., in case of a vacation leave request, the chatbot would prompt a specific absence form to be filled in by the requestor, while if a user asks how much a new service costs, the chatbot would need more information about specific service features. The large number of documents and the diversity in the domains made it very challenging to a-priori define the full list of intents expected to be faced throughout the system use. Thus, a more generic solution should be introduced.
## 3.1 Problem Definition

The aforementioned challenges form an information retrieval problem, where given a user query the chatbot engine should identify and return the related set of documents and text paragraphs (snippets) of the knowledge domain(s) in scope where the answers reside. We introduce a method that calculates the similarity relevance of two text arguments and concludes a matching performance result. In our case as Eq. (1) depicts, given an initial user query $q$, consider $f_{match}$ as the function that returns a set of {$d_i, sp_j$} pairs, where $sp_j$ is a snippet located in document $d_i$ of the knowledge base $\Omega$ in scope and $sp_j$ matches user query $q$ with a matching performance $score_j$.

$$f_{match}(q, \Omega) = \bigcup_{i,j=1}^{k} (d_i, sp_j, score_j), sp_j \in d_i \cap d_i \in \Omega \quad (1)$$

To assure the highest level of performance, we filter out the result set to remove the snippets with $score_j$ lower than a threshold, defined with respect to business needs. Eq. (2) depicts that for a given user query $q$ and a threshold $thres$, in $res(q, \Omega, thres)$ the chatbot would use $f_{match}$ and retrieve all {$d_i, sp_j$} pairs where the matching performance $score_j$ satisfies the performance threshold.

$$res(q, \Omega, thres) = f_{match}(q, \Omega), thres \leq score_j \quad (2)$$

In the following section, the entire solution and all implemented architecture modules are presented in detail.

## 4 Methodology and Proposed Solution

The methodology consists of the following main components:

1. Definition of a Generic Data model (related to challenge 1 in Sect. 3), see Sect. 4.1.2.

2. Definition of an evaluation model to measure performance each time changes are made, see Sect. 4.3.1.

3. Focus on term-based question answering first (related to challenges 2, 3 in Sect. 3), see Sects. 4.1.3, 4.2.1 and 4.2.2.

4. Innovation in User Interface design (related to challenge 3 in Sect. 3), see Sect. 4.2.3.

5. Support for feedback from the experts (related to challenge 3 in Sect. 3), see Sect. 4.3.2.

6. Ontology engineering (related to challenge 3 in Sect. 3), see Sect. 4.2.1.

7. Query classification (related to challenges 2, 3 in Sect. 3), see Sect. 4.2.1.

8. Definition of frequently asked questions (FAQs) (related to challenge 2 in Sect. 3). For questions that cannot be answered directly by the content of the document.

In the sections below, the aforementioned components are described in detail.
## 4.1 Data Management

The first component of our architecture is the *data management* that consists of three main modules as Fig. 1 depicts.

<figure><img src="image_4.png" /><figcaption>Fig. 1. High level overview</figcaption></figure>

### 4.1.1 File Parsers

HTO has documents from multiple domains. The approach was to first incorporate one of the most complex domains; i.e., Finance & Procurement, and provided that the results are satisfying, incorporate more and more domains. The input dataset consists of 32 word documents (with information related to activities and processes), 6 excel spreadsheets (that contain FAQs and manuals), and 3 powerpoint presentations (that contain manuals). The Word documents which contain the core process information are being revised over time, iteratively. Thus, not all documents follow the same template. The same applies for Excel spreadsheets and the PowerPoints presentations.

In order to incorporate and ingest all the available data into the database, five distinct parsers have been implemented to accommodate the different file formats and templates. The parsers split the document into smaller chunks of information (e.g., parsing a document as an *html* page) and discard portion of text that is out of scope, like footers, headers etc. Moreover, all documents have been classified into categories based on a set of regular expressions rules; i.e. a) FAQ, b) Processes, c) Manuals, d) Definitions and Abbreviations.

### 4.1.2 Snippet Extraction

The string chunks generated from the previous step are transformed into a custom common data model that consists of the following fields: Filename, Document Title, Document Summary, Document Text, Snippet Title, Snippet Body, Snippet Type, Document Summary, Snippet Additional Info, Snippet Hash.

### 4.1.3 NLP Analyzers

Finally the snippets are processed via a NLP Pipeline, before their storage into the database, as demonstrated in Fig. 2. The NLP Pipeline follows the common and standard procedures of tokenization (standard tokenizer), lowercase filter,
<figure><img src="image_5.png" /><figcaption>Fig. 2. NLP pipeline during data ingestion</figcaption></figure>

stop words removal and stemmer. The outcome of this pipeline is the creation of an inverted index [11]. Not all fields of the snippet data model are being processed. Those that undergo processing are those that carry the important information ( snippet title, snippet body and additional info) that will be queried to answer the users’ questions.

## 4.2 Chatbot Engine/Question Answering System

The high-level overview of the question answering system is depicted in Fig. 3. The user submits a query via the custom User Interface (UI), then the query gets processed and executed; and finally the result set is displayed back to the user.

### 4.2.1 Query Processing

Query processing module follows a similar pipeline with Sect. 4.1.3, with the addition of *query expansion* and *query classification*. Query expansion is based on a flat ontology and associates the query with synonyms, translations, abbreviations and business-related terms. The ontology was created by business matter experts and information extracted by web scrappers. Query classification was built with the goal to classify questions to a predefined list of categories (similar to intents) [12]. It is based on simple regex rules that identify terms in the query and assign a category.

### 4.2.2 Query Execution

During this step the chatbot engine utilizes information from the previous step and creates multiple requests to the document database implemented in the ELK<fnref n="1" /> stack [13]. In general, snippets are filtered first by their generic type (Sect. 4.1.2) and a set of term-based queries are executed towards the related

<footnote n="1">¹ ELK stands for three open source projects: Elasticsearch, Logstash, and Kibana.</footnote>
inverted index. Then these results are ranked, filtered and grouped in categories by their data type. A set of custom rules define the filters and the type of requests that will be made depending on the classification type of the query. Furthermore, the search type, defines the different weights at corresponding snippet fields. The specification of those weights has been derived via a machine learning process that used the initial test queries in order to determine the proper weights depending on the search type. Our model was based on the gradient descent algorithm. In addition, the searches include an approximate string-matching feature; i.e., *fuzzy searching* [14], which allows the user to make spelling errors without negatively impacting the results. In essence, the user query returns documents that contain terms similar to the search terms. Similarity is defined by a maximum threshold of the *Levenshtein edit distance* [14] between the query term and the document terms. Furthermore, Query execution utilizes the query expansion information from Sect. 4.2.1 in order to perform phrase matching for business related terms (e.g. 'sap expense order') that is important to be searched as a whole phrase, instead of each term individually. As a last step the TF-IDF score of each retrieved snippet is normalized into the [0,1] range. Snippets with score less than 60%<fnref n="2" />, are filtered out to reduce the size of the result set. The results are categorized based on their type as discussed in Sect. 4.1.2 and handled in an intelligent manner by the UI, as explained in the following section.

<figure><img src="image_6.png" /><figcaption>Fig. 3. Chatbot core engine</figcaption></figure>

### 4.2.3 Innovative User Interface

At challenge 3 in Sect. 3 we argued about the difficulty of a-priori defining the user intents. Having no insights on which kind of intents will be more valuable for the business led to the implementation of more generic, intent-agnostic user-interface that is different from traditional chatbot based systems. Let's consider

<footnote n="2">² 60% threshold has been chosen after experimentation with the test questions at hand.</footnote>
a sample question "*What does 'R' mean in the second digit of the payment code*".
Even though the intent of the user is agnostic to the chatbot i.e., not recognized by the Query classification component, the chatbot engine performs a search in every snippet available and returns a ranked list of potential answers. A single concrete answer is intentionally avoided. Instead, the response is generic, displaying a set of suggested documents found, and inviting the users to explore and identify the information requested themselves. Selecting one the available answers, opens an embedded pdf document viewer that scrolls automatically to the page where the answer is identified.

## 4.3 Evaluation System

The evaluation system described below allowed us to immediately compare different versions and understand if the changes have a positive or negative effect to the desired results of the test set.

### 4.3.1 Evaluating the Chatbot Engine Automatically

The automatic evaluation framework was exploited with the usage of 100 test questions along with their “gold standard” answers. For every new chatbot version, the evaluation framework was executed to quantify the impact of the change. The evaluation framework measured the following Key Performance Indicators (KPIs): Snippet Recall@1, Snippet Recall@3 and Snippet Recall@5, the percentage of the test queries that the desired snippet returned in the first place, within the first three, or within the first five responses respectively. Other valuable metrics are the Execution time of experiment, Snippet std, Document Recall@1, Document Recall@3 and Document Recall@5. The execution time of the experiment is important to be as low as possible. The Snippet_std is the standard deviation of the correct snippet within the returned responses. Finally, the last three mentioned metrics are the equivalents to the Snippet Recall@1, Snippet Recall@3 and Snippet Recall@5 but instead of snippets we assess the document correctness.

### 4.3.2 Evaluating the Achievement of the Business Need, Manually, via the User Interface

During an extensive period of User Acceptance Tests (UATs), SMEs evaluated the performance of the chatbot, as well as the overall level of satisfaction with respect to the initial business need; i.e., “How satisfied are you from the results? - The chatbot: a) Helped you find the quickly the correct information b) Gave you the correct information, but not in the first order c) Helped you slightly d) Did not help you at all”. The results are presented extensively in Sect. 5 below.

## 5 Experiments and Discussion

For the automatic evaluation of the chatbot engine (see Sect. 4.3.1) 100 test questions have been collected and 41 documents from the domain in scope have been
processed, resulting into about 1500 snippets in our database. We measured the performance for two distinct iterations (chatbot versions) as presented at Table 1. “Baseline” iteration included the first implementation of the NLP pipeline (see Sect. 4.1.3) and term-based matching. “Enhanced” iteration included improvements in the NLP pipeline (utilization of Skroutz Greek stemmer [10]), query classification, fuzziness search, expansion of term-based matching to take into account phrases, query expansion and weights of the search fields; as anticipated, a significant performance improvement has been measured.

<table><caption>**Table 1.** Comparing iteration results</caption><thead><tr><th>Iteration</th><th>Doc. Recall@1</th><th>Doc. Recall@3</th><th>Doc. Recall@5</th><th>Snippet Recall@1</th><th>Snippet Recall@3</th><th>Snippet Recall@5</th><th>Execution time</th></tr></thead><tbody><tr><td>Baseline</td><td>35%</td><td>55%</td><td>62%</td><td>27%</td><td>45%</td><td>51%</td><td>0.05 s</td></tr><tr><td>Enhanced</td><td>70%</td><td>87%</td><td>91%</td><td>65%</td><td>84%</td><td>88%</td><td>0.14 s</td></tr></tbody></table>

It is important to note that during both iterations a set of *irrelevant documents*<fnref n="3" /> have also been included and evaluated in order to monitor the KPIs values variation. The hypothesis was denied by the results, indicating that the solution is quite stable and unbiased.

The technology stack consisted of Python and related libraries (Natural Language Toolkit NLTK, Scikit-Learn, Numpy, Pandas etc.) for the chatbot NLP engine, Elasticsearch for storing the snippets and performing term-based matching and MySQL for storing the feedback from the users.

Moreover, a user study has been conducted via a series of User Acceptance Test (UAT) sessions (see Sect. 4.3.2). User perception is subjective comprising performance, responsiveness and reliability. In the question “How satisfied are you from the results?-, the SMEs gave positive feedback for 75% of the questions they submitted, neutral: 11% and negative: 14%, as demonstrated in Table 2.

<table><caption>**Table 2.** UAT evaluation results</caption><thead><tr><th>Test queries</th><th>Helped to find quickly the correct info</th><th>Gave the correct info but not in the first place</th><th>Helped slightly</th><th>Did not help at all</th></tr></thead><tbody><tr><td>180</td><td>41%</td><td>34%</td><td>11%</td><td>14%</td></tr></tbody></table>

## 6 Next Steps

Following the UAT results, the future work about the chatbot would include new documents from other domains like HR. In addition the NLP engine will

<footnote n="3">³ Documents not related to the Procurement & Finance domain in scope.</footnote>
be enhanced with *entity recognition* capabilities, trained to identify products, references to internal systems, monetary values and arithmetic operators in the snippets and the user query. This will allow us to build a dialog system asking for disambiguation of terms and to further expand the mechanism in Sect. 4.2.2 with business rules like "if an product entity is recognized in the user query, give preferences to snippets that contain information related with this product family". Finally, given the initial positive response from the users, UI enhancements will be evaluated to replace the 'Process Web' portal completely.

# 7 Conclusion

In this paper we have analyzed the methodology to build a question answering chatbot that does not strictly rely on intent identification. We described the NLP components, the user interface approach and the evaluation mechanisms to measure performance. The challenges presented at Sect. 3 along with the iterative approach of building the chatbot engine led us to the following lessons learned, that we would like to share with the reader as a conclusion: a) when building a system that encapsulates knowledge that is hard to model, start with the simplest approach possible and increment it gradually, b) user interface design should be an integral part of any AI application acting as an ally towards facilitating the user c) the system performance should be measured independently from any technical KPIs and should be based on business-related human-centric goals. In principle, it is clear that AI chatbots is a promising technology constantly evolving in the next years with significant academic interest and potential applications in various enterprises.

# References

1. Business Process Model and Notation (BPMN) Homepage. https://www.bpmn.org/

2. Elasticsearch Homepage. https://www.elastic.co/what-is/elasticsearch

3. Carter, D.: How real is the impact of artificial intelligence. Bus. Inf. Rev. (Bus. Inf. Surv.) **35**(3), 99–115 (2018)

4. Bank of America: Bank of America's Erica® Surpasses 10 Million Users (2019). https://newsroom.bankofamerica.com/press-releases/consumer-banking/bank-americas-ericar-surpasses-10-million-users-introduces-new

5. López, G., Quesada, L., et al.: Alexa vs. Siri vs. Cortana vs. Google Assistant: a comparison of speech-based natural user interfaces. In: Nunes, I. (ed.) AHFE 2017. AISC, vol. 592, pp. 241–250. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-60366-7_23

6. Kerly, A., Hall, P.D., Bull, S.: Bringing chatbots into education: towards natural language negotiation of open learner models. In: International Conference on Innovative Techniques and Applications of Artificial Intelligence, pp. 179–192 (2006). https://doi.org/10.1007/978-1-84628-666-7_14

7. Oh, K.-J., Lee, D., Ko, B.: A chatbot for psychiatric counseling in mental healthcare service based on emotional dialogue analysis and sentence generation. In: 18th IEEE International Conference on Mobile Data Management MDM (2017). https://doi.org/10.1109/MDM.2017.64
8. Ngai, E.W.T., Lee, M.C.M., et al.: An intelligent knowledge-based chatbot for customer service. Electron. Commerce Res. Appl. (2021). https://doi.org/10.1016/j.elerap.2021.101098

9. Hussain, S., Sianaki, O.A., Ababneh, N.: A survey on conversational agents/chatbots classification and design techniques. Artif. Intell. Netw. Appli. Proc. 946–956 (2019). https://doi.org/10.1007/978-3-030-15035-8_93

10. Skroutz Greek Stemmer Homepage. https://github.com/skroutz/greek_stemmer/blob/master/README.md

11. Cutting, D., Pedersen, J.: Optimization for dynamic inverted index maintenance. In: Proceedings, pp. 405–411 (1980). https://doi.org/10.1145/96749.98245

12. Kang, Y., Cai, Z., et al.: Natural language processing (NLP) in management research: a literature review. J. Manag. Anal. 139–172 (2020). https://doi.org/10.1080/23270012.2020.1756939

13. ELK Stack Homepage. https://www.elastic.co/what-is/elk-stack

14. Siahaan, A.P.U., Aryza, A., et al.: Combination of Levenshtein distance and Rabin-Karp to improve the accuracy of document equivalence level. Int. J. Eng. Technol. 7, 17–21 (2018). http://dx.doi.org/10.14419/ijet.v7i2.27.12084