Enterprise Alexandria: Online High-Precision Enterprise Knowledge Base Construction with Typed Entities

1 . Introduction . As the volume of textual information produced within organizations continues to grow dramatically [ Radicati and Levenstein , 2019 ] , it is critical for Automated Knowledge Base Construction ( AKBC ) systems applied to enterprise scenarios to process data automatically and at large scale . This is becoming a key research area in enterprise knowledge discovery [ Loshin , 2001 ] where a key task is to turn unstructured text present in emails , documents and intranet pages into a single , continuously available , consistent and up-to-date knowledge base . In doing so , due to the restricted access of enterprise documents that are continuously edited by the users , the entire knowledge base must be constructed incrementally and eyes-off ( i.e. , documents are not accessible for labeling ) for each organization [ Voigt and Von dem Bussche , 2017 ] . Furthermore , since any automated system will inevitably have gaps in the extracted knowledge , the management of such systems also involves the role of knowledge curators acting as a trusted source that can editorially curate or expand the mined knowledge [ Maedche et al. , 2003 ] . Therefore it is important for an AKBC system to be designed to accommodate and learn from curated content . The aim is to achieve a 1. www.microsoft.com/en-us/microsoft-viva/topics virtuous cycle where people and machines continually work together to keep the knowledge accurate and complete . A key characteristic of enterprise documents is their structured nature , including both structured text and associated metadata . For instance , emails have a subject , sender , list of recipients and attached documents while documents have a creator and a list of editors , viewers and commentators as well as structure within the document , such as sections and titles . We argue that both kinds of information are critical to build an accurate knowledge base . In this respect , there is a wealth of knowledge discovery methods that process text without explicitly modeling metadata [ Yamada et al. , 2020 , Devlin et al. , 2019 , Zhang et al. , 2019 ] . Other systems can leverage the full data structure if some amount of training labels are available to infer the relationship between each structured field ( e.g . email sender ) and the entity properties ( e.g . project lead ) [ Zhang , 2015 ] . Unfortunately , the privacy restrictions of enterprise data do not allow for their labeling by human annotators , making these methods unfeasible . We address this gap by developing an automated system , Enterprise Alexandria ( EA ) , as an extension of Alexandria [ Winn et al. , 2019 ] , an existing system designed for Web-based information extraction . EA is able to jointly discover entities and their associated types with very minimal supervision in compliance with the privacy requirements of enterprise data , which makes it the first entity discovery method applicable to enterprise scenarios to the best of our knowledge . In EA , the probabilistic program is extended to generate both the document and its metadata from a set of latent entities . Inference is partly performed automatically using an efficient message-passing algorithm [ Minka et al. , 2018 ] , and partly with a handwritten scalable algorithm that can process a large set of documents in batches and build the knowledge base incrementally , making use of any manual curations that have happened in the interim . Furthermore , EA adds an important ability to learn the set of types associated with each entity ( e.g . product , event , team , project ) from a pre-defined type hierarchy . In order to ensure that the type hierarchy is sufficient , EA is also able to semi-automatically discover type names from enterprise data . The importance of determining the types of extracted entities receives support from a study of workers in two organizations , which showed that a wide variety of knowledge types are important for work , and that workers seek to engage with these different types in different ways . We argue that types are needed not just generically , but to support work in particular industries and organizations . Using real-world datasets , we show that the quality of the entities and their associated types extracted by EA outperforms the baselines by significantly increasing the precision of the extracted knowledge . In summary , this paper provides the following contributions to the state of the art : ( i ) Findings from a qualitative study of members of two multinational organizations , which highlights how entity types are important for work . ( ii ) An extended probabilistic program that improves the existing Alexandria model and can perform entity discovery jointly from structured text and the associated metadata . ( iii ) A new mechanism for updating the discovered knowledge incrementally with the ability to support curated content contributed by human users . ( iv ) A real-world evaluation showing that EA performs efficiently at scale while significantly improving the precision of the extracted knowledge compared to four baselines on several real-world datasets . 2 . User Study on Knowledge Categories . To best inform the design of our model , and to gather data on real entity types as a baseline for our evaluation , we undertook a qualitative study with prospective end users who were employed in two different industries . In the study we asked : How do organization members think about knowledge types , and what types of entities need to be discovered to support everyday work ? The study was conducted in two workshops with 12 ( 5 = men , 7 = women ) knowledge workers who were employed by one of two multinational organizations ( 6 participants in each ) . The first organization was a large professional services firm ; the second was part of the travel and tourism industry . Participants came from a variety of backgrounds in terms of role ( e.g . designer , program manager , analyst , information architect ) and ranged in seniority from junior to national leadership . In order to provide context , participants were first shown a short video that described how an automatically-generated knowledge base might support work . Participants were then asked to think about the different components that make up their work , which they might want represented in the AKBC system . Participants were given four examples of such components ( project , budget , product launch , designer ) in order to stimulate ideas . The study received ethical approval from the host Institutional Review Board ( IRB ) . Each participant provided informed consent for data collection and use prior to participation . Table 1 shows the 48 unique knowledge types generated by participants in total , which we manually grouped bottom-up ( via an affinity diagram ) into eight superordinate classes to help summarize results . The knowledge categories were grouped based on the explicit meanings of the category names , descriptions and discussions provided by participants in the workshop . While three of the knowledge categories reported by participants were confirmations of the examples we gave to participants ( project , budget , designers ) , the remaining 45 categories were generated by participants independently . Rather than suggesting that all of these categories are necessary for every knowledge base , we suggest these categories capture this set of prospective users ’ aspirations for a knowledge base . Overall , this qualitative study demonstrates why types matter : a wide variety of entity types are important for work , and workers seek to engage with these different types in different ways . In the following discussion , we move on to describe a model to learn the entity types automatically as part of the AKBC task . 3 . The Enterprise Alexandria Model . The Enterprise Alexandria model is an extended form of the probabilistic program used by Alexandria [ Winn et al. , 2019 ] . The main extensions are described below . 3.1 Structured Templates for joint processing of Text and Metadata . The Alexandria probabilistic program uses a set of templates to generate text . EA extends this program to generate entire data items , such as emails or calendar events , using manuallyspecified structured templates . Figure 1 illustrates how a structured template can be used to generate an observed data item from a latent knowledge base entity . In this example , the structured template is applied to a Project entity named “ Tokyo ” to create a recurring calendar event ‘ Tokyo project sync ’ with suitable organizers and attendees . During inference this process is inverted , so a compatible calendar event will be matched against this structured template to produce the knowledge base entity . A structured template consists of the type or types of data items which can be generated ( such as emails , calendar events or documents ) along with a set of properties for the generated data items each with a suitable value generator . Value generators can be one of three kinds : ( i ) A template-based generator which generates a string given a template . If more than one template is provided , one is selected at random . The Name property in Figure 1 uses this kind of value generator . ( When ‘ Types ’ is used in a template , one type is selected at random . ) ( ii ) A subset generator which generates a collection value as a random subset of a specified collection in the entity . The Organizers and Attendees property in Figure 1 use this kind of generator to generate subsets of the Members property . ( iii ) A constant value generator which always gives a particular value . The IsRecurring property in Figure 1 uses a constant value generator which always returns ‘ True ’ . 3.2 Entity Names and Variants . In the original Alexandria model , the prior probability of an entity name was uniform over the set of valid names . EA uses a more sophisticated model ( tuned on Wikipedia titles ) which takes into account the number of words , the length of each word , and more . This means that two mentions of the same long name are less likely to be a random collision and more likely to be referring to the same entity than two mentions of the same short name . This improves the ability of the system to disambiguate mentions of different entities sharing the same name . In enterprise data , it is common for an entity to be referred to by different variations of a name , such as abbreviated forms , capitalizations , etc . EA allows an entity to have alternative names , as long as the alternatives are compatible variants of each other under a set of variants models . The supported variants models are : Case and diacritic variants where names differ only in case or in the presence/absence of accents or diacritics ; Separator variants where the names differ only in separators ( such as ‘ & ’ or ‘ + ’ ) or the separators are removed entirely ; Name phrase variants where a name “ Tokyo ” and a name phrase “ Project Tokyo ” can be identified ; Acronym variants where one name is an expansion and the other an acronym or partial acronym . When computing the probability that two mentions refer to the same entity , if their names do not match but are compatible alternatives , then the probability is the same as if the names matched ( on the more probable name ) , times a constant penalty . 3.3 Type Discovery . As in Alexandria , facts about an entity are represented by named , typed properties . EA includes a property named ‘ Types ’ which holds the types for the entity , selected from a predefined set . EA semi-automatically learns this set of allowed types through a process of type discovery . This process starts with a small manually-provided seed set of types , such as { Project , Team } . New types are then added iteratively as follows : ( i ) Entity discovery is run to extract entities from a set of enterprise data items , given these seed types . ( ii ) Fact retrieval is run using the names of these discovered entities , as described in Winn et al . [ 2019 ] . However , the schema is modified to allow values of the Types property to be any string of 1–3 words , rather than one of the fixed set of known types . The resulting type names are ‘ type candidates ’ . ( iii ) The type candidates are aggregated across entities and automatically filtered based on their frequency and other checks , such as whether there is uncertainty in the posterior distribution over the type name . ( iv ) The remaining type names are manually filtered to exclude people types and file types . ( v ) The manually approved types are added to the type set and the process is repeated using this updated set . Iteration continues until the number of newly added types becomes sufficiently small to ensure that good overall coverage has been achieved . This type discovery process works because entities usually have multiple types . For example , ‘ Tokyo ’ might be referred to as a project but also as a framework , system and toolkit . Each iteration of type discovery first finds entities whose types are any known type and then uses fact retrieval to add in the other types for these entities , which may be unknown . Iteration of this process will discover any type , so long as there exists a chain of entities which connect that type to a seed type . In practice , given a large set of enterprise data , the process appears to give good coverage of types across a wide variety of domains . Specifically , on our largest dataset , it was able to discover a universe of 171 types across different domain that include “ university ” , “ case ” , “ campaign ” , etc . Note that the EA ’ s entity discovery process will surface only the subset of types that appear in the input dataset . For instance , types like “ university ” or “ case ” will be prevalent in the data belonging to an academic institution or a law firm , respectively .

Summary: The authors have presented a dedicated new system for automatically constructing a knowledge base from private enterprise data. This system is an extension of an existing system designed for Web-based information extraction.  It employs a probabilistic program to generate both the document and its metadata from a set of latent entities and adds the ability to learn the set of types associated with each entity from a pre-defined type hierarchy.  . This work also presents a real-world evaluation showing that the developed system scales well and significantly improves the precision of the extracted knowledge compared to four baselines on three real-world datasets.