Package: principledinvestigator
principledinvestigatorA papers recomendation toolprincipledinvestigator compares papers in your library against a database of scientific papers to find new papers that you might be interested in.
While there's a few services out there that try to do the same, principledinvestigator is unique in several ways:principledinvestigator is completely open source, you can get the code and tweak it to improve the recomendation engineprincipledinvestigator doesn't just use a single paper or a subset of (overly generic) keywords to find new papers, instead it comparesallof your papers' abstracts against a database of papers metadata, producing much more relevant resultsdisclaimerThe dataset used here is a subset of alarger dataset of scientific papers. The dataset if focused on neuroscience papers published in the latest 30 years. If you want to include older papers or are interested in another field, then follow the instructions to create your custom database.(possible) future improvementsusescibertinstead of tf-idf for creating the embedding. This should also make it possible to embed the database's papers before use (unlike tf-idf which needs to run on the entire corpus every time).OverviewThe core feature making principledinvestigator unique among papers recomendation systems is that it analyzesyour entire libraryof papers and matches it against avast databaseof scientific papers to find new relevant papers. This is obviously an improvement compared e.g. to finding papers similar toone paper you like.
In addition, principledinvestigator doesn't just use things like "title", "authors", "keywords"... to find new matches, instead it finds similar papers usingTerm Frequency-Inverse Document Frequencyto asses the similarity acrosspapers abstracts, thus using much more information about the papers' content.UsageFirst, you need to get data about your papers you want to use for the search. The best way is to export your library (or a subset of it) directly to a.bibfile using your references menager of choice.Then, you can use...Making your own databaseprincipledinvestigator uses a subset of the vast and eccelent corpus of scientific publications' metadata fromSemanthic Scholar.
The dataset used by principledinvestigator is focused on neuroscience papers written in english and published in the last 30 years. If you wish to include a different set of papers in your database, you can make your custom database and use it with principledinvestigator by executing the following steps.1. Download whole corpusYou'll first need to download the whole corpus from Semantic Scholar. You can find the data and download instructionshere. Once the data are downloaded, save them in a folder where you want to base your dataset-creation process2. Uncompressing dataThe downloaded corpus is compressed. To uncompress the files useprincipledinvestigator.database_preprocessing.upack_databasepasing to it the path to the folder where you've downloaded the data.3. Specifying your parametersThe selection of a subset of papers from the corpus is based on a set of parameters (e.g. year of publication) matched against criteria specified (and described) inprincipledinvestigator.settings. Edit the criteria to adapt the dataset selection to your needs4. Creating the datasetSimply runprincipledinvestigator.database_preprocessing.make_database5. Training doc2vec modelPapers semanthic similarity is estimated using a doc2vec model trained on the entire dataset.
After modifying the dataset to your needs, you'll have to re-train the model by runningprincipledinvestigator.doc2vec.train_doc2vec_modelsummary:An example code for creating your dataset (after having downloaded the corpus and edited the settings)fromprincipledinvestigator.database_preprocessingimportupack_database,make_databasefromprincipledinvestigator.doc2vecimporttrain_doc2vec_modelfrompathlibimportPathfolder=Path('path to your data')# unpack and createunpack_database(folder)make_database(folder)# train new d2v modeltrain_doc2vec_model()