title: Definition and evaluation of a conversational agent system utilising a rule-based knowledge-base system that maintains conversational context
Creators: name: Herbert, DP
Publisher: name: University of Tasmania
Year: 2020

description: Conversational Agent (CA) systems are artificially intelligent software programs using natural language interfaces to simulate real human conversation. They are important as they can provide a human-centric interaction between humans and digital services. This importance is increasing due to the recent, exponential increase of connected devices that provide many disparate services available in homes, businesses and industry. The nature of these services is also changing. Contemporary services that fulfil requests such as web-based information retrieval and setting timer-based reminders are being complemented by new types of services that are emerging due to the enormous expansion of network-connected devices (the Internet of Things). The ability to interact with these services through conversation and modify device behaviour by imparting human knowledge conversationally is a challenge that needs to be addressed for these devices to be truly integrated within society. The objective of this thesis is to start to address this challenge by defining and evaluating a Rule based CA system that naturally retains conversational context and that can be easily maintained by conversational authors who are experts in their own domains. This initial approach is focused to consider domains that do not contain large repositories of existing conversational data for training, and CA systems that do not have a reliance on complex scripting and programming skills or knowledge of formal grammatical syntax that would be needed by the conversational author. This research is undertaken over two phases. The first phase determines a suitable rule-based knowledge-base system (KBS) methodology subject to conversational and domain constraints that addresses brittleness (lacking knowledge or being unclear as to what knowledge a system contains) which is a common criticism levelled against knowledge-base systems. The Multiple Classification Ripple Down Rules (MCRDR) KBS methodology partially satisfies these requirements and it was adopted as the foundation KBS. This research phase concludes by defining and evaluating Contextual MCRDR (C-MCRDR) through augmentation of MCRDR to facilitate the creation of CA systems for suitable domains. The CA system implemented (C-MCRDR CA) had speech capabilities via specific web browser support, meaning speech input and output were tied to a client computer's speaker and microphone as well as the automatic speech recognition (ASR) client supported by the browser. For improved versatility and speech-to-text performance, integration of an Intelligent Personal Assistant (IPA) was required. The second phase of the research evaluated properties of two market-leading IPA systems to determine which is the best-performing device to integrate as part of the CA system's user interface. The Google Home and the Amazon Echo were evaluated, with the Google Home demonstrating superior performance. This research phase concluded by defining and evaluating two methods for isolated word transcription error-correction using the Google Home directly coupled to the C-MCRDR CA system. Significant findings of this research are: 1. C-MCRDR can facilitate an effective, efficient way of maintaining knowledge base systems by acquiring and classifying knowledge incrementally and unlike MCRDR, it maintains conversation context, a factor that assists with allowing utterance classification to change over time as a conversation progresses; 2. Substantial rule-count reduction can be achieved by C-MCRDR (in comparison to standard MCRDR) due to post-inference deferred classifications that include database querying expressions; 3. A natural language interfaces to databases (NLIDB) framework can be engendered by C-MCRDR by way of post-inference query binding and a pattern-matched NL interface with the advantages of knowledge acquisition and maintenance through the ripple-down methodology; 4. Brittleness can be mitigated by C-MCRDR due to the incremental approach to knowledge acquisition, the use of paraphrasal terms in pattern matching, and paraphrasal look-ahead prompting; 5. The Google Home values for the isolated word and phrasal recognition word error rate (WER) measurements were significantly lower than the Amazon Echo; 6. Two error correction schemes have been defined that can be applied to IPAs when coupled to a C-MCRDR CA system to significantly improve the average WER across all datasets. This doctoral study has made a significant contribution to the body of knowledge through the definition and evaluation of a C-MCRDR CA system. This research demonstrates a contextual rule-based conversational agent system can achieve proven excellent performance results in domains lacking existing corpora of conversational knowledge, while at the same time it does not encumber conversational authors to require complex scripting or programming skills, or knowledge of formal grammatical syntax. No other human-authored, rule-based contextual conversational agent system based on a derivative of RDR, as opposed to machine learning or statistical approaches, has been detailed and evaluated in the literature. In addition, no other identified studies evaluate and measure the isolated word or sentence-level WER of contemporary IPA devices comprehensively nor do they examine whether attributes of word ranking, and word and sentence lengths, affect the IPA's WER. The key implication of this study is that C-MCRDR CA systems can perform well without significant technical overheads in their conversational knowledge construction. In domains that do not contain sufficient domain-specific conversational training data, a C-MCRDR CA system may be suitable for the conversational author to build a conversation system that is relevant to their domain. Future work could investigate how such systems can be extended to create a conversational platform that could accommodate many different types of services by extending the conversational knowledge to include behavioural programming, for example, in the form of rule-based actions that are associated with dialog responses.