Introduction
QnAMaker aims to simplify the process of bot creation by extracting Question-Answer (QA) pairs from data given by users into a Knowledge Base (KB) and providing a conversational layer over it. KB here refers to one instance of azure search index, where the extracted QA are stored. Whenever a developer creates a KB using QnAMaker, they automatically get all NLP capabilities required to answer user's queries. There are other systems such as Google's Dialogflow, IBM's Watson Discovery which tries to solve this problem. QnAMaker provides unique features for the ease of development such as the ability to add a persona-based chit-chat layer on top of the bot. Additionally, bot developers get automatic feedback from the system based on end-user traffic and interaction which helps them in enriching the KB; we call this feature active-learning. Our system also allows user to add Multi-Turn structure to KB using hierarchical extraction and contextual ranking. QnAMaker today supports over 35 languages, and is the only system among its competitors to follow a Server-Client architecture; all the KB data rests only in the client's subscription, giving users total control over their data. QnAMaker is part of Microsoft Cognitive Service and currently runs using the Microsoft Azure Stack.
System description ::: Architecture
As shown in Figure FIGREF4, humans can have two different kinds of roles in the system: Bot-Developers who want to create a bot using the data they have, and End-Users who will chat with the bot(s) created by bot-developers. The components involved in the process are:
QnAMaker Portal: This is the Graphical User Interface (GUI) for using QnAMaker. This website is designed to ease the use of management APIs. It also provides a test pane.
QnaMaker Management APIs: This is used for the extraction of Question-Answer (QA) pairs from semi-structured content. It then passes these QA pairs to the web app to create the Knowledge Base Index.
Azure Search Index: Stores the KB with questions and answers as indexable columns, thus acting as a retrieval layer.
QnaMaker WebApp: Acts as a layer between the Bot, Management APIs, and Azure Search Index. WebApp does ranking on top of retrieved results. WebApp also handles feedback management for active learning.
Bot: Calls the WebApp with the User's query to get results.
System description ::: Bot Development Process
Creating a bot is a 3-step process for a bot developer:
Create a QnaMaker Resource in Azure: This creates a WebApp with binaries required to run QnAMaker. It also creates an Azure Search Service for populating the index with any given knowledge base, extracted from user data
Use Management APIs to Create/Update/Delete your KB: The Create API automatically extracts the QA pairs and sends the Content to WebApp, which indexes it in Azure Search Index. Developers can also add persona-based chat content and synonyms while creating and updating their KBs.
Bot Creation: Create a bot using any framework and call the WebApp hosted in Azure to get your queries answered. There are Bot-Framework templates provided for the same.
System description ::: Extraction
The Extraction component is responsible for understanding a given document and extracting potential QA pairs. These QA pairs are in turn used to create a KB to be consumed later on by the QnAMaker WebApp to answer user queries. First, the basic blocks from given documents such as text, lines are extracted. Then the layout of the document such as columns, tables, lists, paragraphs, etc is extracted. This is done using Recursive X-Y cut BIBREF0. Following Layout Understanding, each element is tagged as headers, footers, table of content, index, watermark, table, image, table caption, image caption, heading, heading level, and answers. Agglomerative clustering BIBREF1 is used to identify heading and hierarchy to form an intent tree. Leaf nodes from the hierarchy are considered as QA pairs. In the end, the intent tree is further augmented with entities using CRF-based sequence labeling. Intents that are repeated in and across documents are further augmented with their parent intent, adding more context to resolve potential ambiguity.
System description ::: Retrieval And Ranking
QnAMaker uses Azure Search Index as it's retrieval layer, followed by re-ranking on top of retrieved results (Figure FIGREF21). Azure Search is based on inverted indexing and TF-IDF scores. Azure Search provides fuzzy matching based on edit-distance, thus making retrieval robust to spelling mistakes. It also incorporates lemmatization and normalization. These indexes can scale up to millions of documents, lowering the burden on QnAMaker WebApp which gets less than 100 results to re-rank.
Different customers may use QnAMaker for different scenarios such as banking task completion, answering FAQs on company policies, or fun and engagement. The number of QAs, length of questions and answers, number of alternate questions per QA can vary significantly across different types of content. Thus, the ranker model needs to use features that are generic enough to be relevant across all use cases.
System description ::: Retrieval And Ranking ::: Pre-Processing
The pre-processing layer uses components such as Language Detection, Lemmatization, Speller, and Word Breaker to normalize user queries. It also removes junk characters and stop-words from the user's query.
System description ::: Retrieval And Ranking ::: Features
Going into granular features and the exact empirical formulas used is out of the scope of this paper. The broad level features used while ranking are:
WordNet: There are various features generated using WordNet BIBREF2 matching with questions and answers. This takes care of word-level semantics. For instance, if there is information about “price of furniture" in a KB and the end-user asks about “price of table", the user will likely get a relevant answer. The scores of these WordNet features are calculated as a function of:
Distance of 2 words in the WordNet graph
Distance of Lowest Common Hypernym from the root
Knowledge-Base word importance (Local IDFs)
Global word importance (Global IDFs)
This is the most important feature in our model as it has the highest relative feature gain.
CDSSM: Convolutional Deep Structured Semantic Models BIBREF3 are used for sentence-level semantic matching. This is a dual encoder model that converts text strings (sentences, queries, predicates, entity mentions, etc) into their vector representations. These models are trained using millions of Bing Query Title Click-Through data. Using the source-model for vectorizing user query and target-model for vectorizing answer, we compute the cosine similarity between these two vectors, giving the relevance of answer corresponding to the query.
TF-IDF: Though sentence-to-vector models are trained on huge datasets, they fail to effectively disambiguate KB specific data. This is where a standard TF-IDF BIBREF4 featurizer with local and global IDFs helps.
System description ::: Retrieval And Ranking ::: Contextual Features
We extend the features for contextual ranking by modifying the candidate QAs and user query in these ways:
$Query_{modified}$ = Query + Previous Answer; For instance, if user query is “yes" and the previous answer is “do you want to know about XYZ", the current query becomes “do you want to know about XYZ yes".
Candidate QnA pairs are appended with its parent Questions and Answers; no contextual information is used from the user's query. For instance, if a candidate QnA has a question “benefits" and its parent question was “know about XYZ", the candidate QA's question is changed to “know about XYZ benefits".
The features mentioned in Section SECREF20 are calculated for the above combinations also. These features carry contextual information.
System description ::: Retrieval And Ranking ::: Modeling and Training
We use gradient-boosted decision trees as our ranking model to combine all the features. Early stopping BIBREF5 based on Generality-to-Progress ratio is used to decide the number of step trees and Tolerant Pruning BIBREF6 helps prevent overfitting. We follow incremental training if there is small changes in features or training data so that the score distribution is not changed drastically.
System description ::: Persona Based Chit-Chat
We add support for bot-developers to directly enable handling chit-chat queries like “hi", “thank you", “what's up" in their QnAMaker bots. In addition to chit-chat, we also give bot developers the flexibility to ground responses for such queries in a specific personality: professional, witty, friendly, caring, or enthusiastic. For example, the “Humorous" personality can be used for a casual bot, whereas a “Professional" personality is more suited in case of banking FAQs or task-completion bots. There is a list of 100+ predefined intents BIBREF7. There is a curated list of queries for each of these intents, along with a separate query understanding layer for ranking these intents. The arbitration between chit-chat answers and user's knowledge base answers is handled by using a chat-domain classifier BIBREF8.
System description ::: Active Learning
The majority of the KBs are created using existing FAQ pages or manuals but to improve the quality it requires effort from the developers. Active learning generates suggestions based on end-user feedback as well as ranker's implicit signals. For instance, if for a query, CDSSM feature was confident that one QnA should be ranked higher whereas wordnet feature thought other QnA should be ranked higher, active learning system will try to disambiguate it by showing this as a suggestion to the bot developer. To avoid showing similar suggestions to developers, DB-Scan clustering is done which optimizes the number of suggestions shown.
Evaluation and Insights
QnAMaker is not domain-specific and can be used for any type of data. To support this claim, we measure our system's performance for datasets across various domains. The evaluations are done by managed judges who understands the knowledge base and then judge user queries relevance to the QA pairs (binary labels). Each query-QA pair is judged by two judges. We filter out data for which judges do not agree on the label. Chit-chat in itself can be considered as a domain. Thus, we evaluate performance on given KB both with and without chit-chat data (last two rows in Table TABREF19), as well as performance on just chit-chat data (2nd row in Table TABREF19). Hybrid of deep learning(CDSSM) and machine learning features give our ranking model low computation cost, high explainability and significant F1/AUC score. Based on QnAMaker usage, we observed these trends:
Around 27% of the knowledge bases created use pre-built persona-based chitchat, out of which, $\sim $4% of the knowledge bases are created for chit-chat alone. The highest used personality is Professional which is used in 9% knowledge bases.
Around $\sim $25% developers have enabled active learning suggestions. The acceptance to reject ratio for active learning suggestions is 0.31.
25.5% of the knowledge bases use one URL as a source while creation. $\sim $41% of the knowledge bases created use different sources like multiple URLs. 15.19% of the knowledge bases use both URL and editorial content as sources. Rest use just editorial content.
Demonstration
We demonstrate QnAMaker: a service to add a conversational layer over semi-structured user data. In addition to query-answering, we support novel features like personality-grounded chit-chat, active learning based on user-interaction feedback (Figure FIGREF40), and hierarchical extraction for multi-turn conversations (Figure FIGREF41). The goal of the demonstration will be to show how easy it is to create an intelligent bot using QnAMaker. All the demonstrations will be done on the production website Demo Video can be seen here.
Future Work
The system currently doesn't highlight the answer span and does not generate answers taking the KB as grounding. We will be soon supporting Answer Span BIBREF9 and KB-grounded response generation BIBREF10 in QnAMaker. We are also working on user-defined personas for chit-chat (automatically learned from user-documents). We aim to enhance our extraction to be able to work for any unstructured document as well as images. We are also experimenting on improving our ranking system by using semantic vector-based search as our retrieval and transformer-based models for re-ranking.

Question: What experiments do the authors present to validate their system?
Answer:  we measure our system's performance for datasets across various domains, evaluations are done by managed judges who understands the knowledge base and then judge user queries relevance to the QA pairs

Question: How does the conversation layer work?
Answer: Unanswerable

Question: What components is the QnAMaker composed of?
Answer: QnAMaker Portal, QnaMaker Management APIs, Azure Search Index, QnaMaker WebApp, Bot

QnAMaker Portal, QnaMaker Management APIs, Azure Search Index, QnaMaker WebApp, Bot