Package: mapintel
MapIntelCategoryToolsDevelopmentPackageDocumentationCommunicationIntroductionMapIntel is a system for acquiring intelligence from vast collections of text data by representing each document as a
multidimensional vector that captures its semantics. The system is designed to handle complex Natural Language queries while it
provides Question-Answering functionality. Additionally, it allows for a visual exploration of the corpus. The MapIntel uses a
retriever engine that first finds the closest neighbors to the query embedding and identifies the most relevant documents. It
also leverages the embeddings by projecting them onto two dimensions while preserving the multidimensional landscape, resulting in
a map where semantically related documents form topical clusters which we capture using topic modeling. This map aims to promote a
fast overview of the corpus while allowing a more detailed exploration and interactive information encountering process. MapIntel
can be used to explore many types of corpora.InstallationFor user installation,mapintelis currently available on the PyPi's repository, and you can install it viapip:pipinstallmapintelDevelopment installation requires cloning the repository and then usingPDMto install the
project as well as the main and development dependencies:gitclonehttps://github.com/NOVA-IMS-Innovation-and-Analytics-Lab/MapIntel.gitcdmapintel
pdminstallConfigurationMapIntel aims to be a flexible system that can run with any user provided corpus. In order to achieve this goal, it standardizes
the data and models, while the deployment of all services is expected to be on AWS. An example of how to fully set up a MapIntel
instance can be found atMapIntel-News. After deploying
the required services, a file.envshould be created at the root of the project with environmental variables that are described
below.AWS credentialsThe following environmental variable should be included in the.envfile:AWS_PROFILE_NAMEThe user should have permissions to interact with the services described below.DataAn OpenSearch database instance should be deployed in AWS with documents contained in an index calleddocument. Each document is
expected to have thecontent,date,embedding,embedding2dandtopicfields with the following types:content: text type that contains the main text of the document.date:longtype that represents the ordinal format of a date.embedding:knn_vectortype that represents the embedding vector of the document.embedding2d:floattype that represents the 2D embedding vector of the document.topic:keywordtype that assigns a topic label to each document.The relevant environmental variables are the following:OPENSEARCH_ENDPOINT: The AWS endpoint of the OpenSearch deployed instance.OPENSEARCH_PORT: The port of the instance.OPENSEARCH_USERNAME: The username.OPENSEARCH_PASSWORD: The password.ModelsMapIntel uses three models trained on the user provided data. The first is a Haystack retriever model, the second is a model that
reduces the dimensions of the embeddings to 2D, while the third is a generator model used for question-answering. The
corresponding environmental variables are the following:HAYSTACK_RETRIEVER_MODEL: The value of the parameterembedding_modelof the Haystack classEmbeddingRetriever.SAGEMAKER_DIMENSIONALITY_REDUCTIONER_ENDPOINT: The SageMaker endpoint of the deployed dimensionality reductioner.SAGEMAKER_GENERATOR_MODEL_ENDPOINT: The SageMaker endpoint of the deployed generator.UsageTo run the application use the following command:mapintelThen the server starts and listens to connections athttp://localhost:8080. You may open the browser and use this URL to
interact with the MapIntel UI.