An Anatomy of an Answer: Open NLP & Discourse Analysis - based Indexing
Channel: The ASF
Date: 17/10/2020

An Anatomy of an Answer: Open NLP & Discourse Analysis - based Indexing
Boris Galitsky

A presentation from ApacheCon @Home 2020
https://apachecon.com/acah2020/

Indexers usually index all text in documents. However, once we learn to "understand" the logic of a plain text, we will see how bad for a search it is to index the whole thing. Discourse analysis helps to select text fragment which should be matched with a potential query, and throw away the rest In this talk we will apply discourse linguistic to practical text search and discover that the majority of indexers which index all text perform very poorly for complex queries. Relying on standard relevance means such as TF*IDF does not alleviate this problem. We will explore how discourse analysis helps search by identifying text fragments which should be indexed and matched with potential queries, and those text fragments which would mislead the search and make its precision low. We will demonstrate how a discourse analysis - based indexer can be employed relying on Apache Open NLP project. The audience will learn how discourse analysis formalizes a logic of text to be searched and represents it as a discourse tree, a structure to represent a domain-independent logical organization of text essential for finding relevant fragments. We will also discuss how to proceed from search engines like SOLR to chatbots, where discourse analysis helps with dialogue management.

Boris Galitsky has been presenting talks on AI over last two decades and at Apache conferences over last few years. He contributed linguistic and machine learning technologies to Silicon Valley startups for last 25 years, as well as eBay and Oracle, where he is currently an architect of the Digital Assistant project. An author of three computer science books, 150+ publications and 20+ patents related to search, he is now working on a book "AI for CRM" to be published by Springer in 2021. Boris is Apache committer to OpenNLP where he created OpenNLP.Similarity component which is a basis for search engine and chatbot development.