Army ANT : A Workbench for Innovation in Entity-Oriented Search External Option : Scientific Activities – TREC

This report has been submitted in partial fulfillment of the requirements for the approval of the External Option curricular unit, of the MAP-i Doctoral Programme in Computer Science, 2016/2017 edition. In this document, we present the scientific output regarding the work developed in preparation for the Open Search track of the Text REtrieval Conference (TREC), in the context of the Scientific Activities curricular option. We being by presenting an overview of how TREC works. Search engine evaluation in TREC has been historically based on a set of fixed topics, accompanied by relevance judgments provided by human assessors. The Open Search track changed this approach by introducing the concept of a Living Labs, where results can be evaluated by real users of real web search engines. The calendar for the 2017’s edition of this track hasn’t been released as of May 2017. Accordingly, we present the system we prepared to enable the participation in the TREC Open Search track, when it becomes available. Army ANT is a workbench we developed to support innovation in entity-oriented search, which also makes it an essential tool for this doctoral work. While it provides an adequate level of freedom for information retrieval researchers to build their own search engine implementations, it also provides reusable code, such as a basic analyzer for tokenization and stopword removal, as well as some structure to enable the integration of different engines. Each engine should provide documentation on its ranking functions and indexed datasets, as well as the values for the score components of individual documents. Army ANT then provides a parallel coordinates visualization to support the understanding of the ranking function through its individual components, as an aid to explain the final score of each document. We implemented two graph-based document representation and retrieval models that we integrated with Army ANT. First, we studied and explored the graph-of-word model. This was originally implemented using an inverted file, replacing the term frequency by a term weight that was computed based on the graph-of-word for each individual document. Our implementation, however, was done directly on a graph database, and the ranking function was implemented using the Gremlin domain-specific language (DSL). While this severely impacted efficiency, both for indexing and retrieval, it provided a purely graph-based approach that enabled a focused and clear method to explore the relations among terms, entities, and terms and entities. As one of the goals of this doctoral thesis is to prove that several retrieval tasks can be unified in a single model, in particular through a graph data structure, we felt this was a fundamental step, in spite of expected performance issues. While the graph-of-word only contained a single dimension of nodes representing terms, we also proposed an improvement and extension of this model, which we called graph-of-entity. The graph-of-entity slightly redefines the edges between term nodes (it does not use sliding windows) and it also includes entity nodes. Nodes are connected through edges that model all the available structured and unstructured information as a unified document representation. We then integrate two essential tasks of entity-oriented search through this model — query entity linking and entity ranking — and show that query entity linking can directly contribute to the ranking process instead of being considered a separate step. The intuition was that the uncertainty of query entity linking should seamlessly contribute to the weighting scheme as a regular feature. Given the unexpected delay of the TREC Open Search track, we were required to implement our own evaluation benchmark, based on available datasets. In particular, we used the Wikipedia Relation Extraction Data v1.0, which provides textual content about an entity, along with labeled relations to other entities. This dataset enabled us to build the first implementations of the graph-of-word and graph-of-entity models. However, since it did not provide any relevance judgments, we also indexed a subset of the INEX 2009 Wikipedia Collection, which provided a two-fold contribution to this work. On one hand, it enabled us to test the indexing process of a dataset with a different format and, on the other hand, it provided a list of topics, assessed by real users, that could be used to evaluate the models, calculating metrics like the mean average precision (MAP) or the normalized discounted cumulative gain for the top-p results (NDCG@p). We close with some final remarks on Army ANT and the two graph-based document representation and retrieval models that we implemented. Given the lack of enough evidence, we could not conclude whether there is an improved effectiveness for the graph-of-entity over the graph-of-word model. With a little tuning, however, our platform should be able to successfully index the complete INEX 2009 Collection, enabling us to assess effectiveness and further extend the range of unified entity-oriented search subtasks supported by the graph-of-entity.