Package: sciterra
sciterra: a python library for similarity-based scientometricsSciterra is a software libary to support data-driven analyses of scientific literature, with a focus on unifying different bibliographic database APIs and document-embedding methods for systematic scientometrics research.OverviewThe main purpose of sciterra is to perform similarity-based retrieval of scientific publications for metascience/scientometrics research. While there are many services/existing software that makes this simple, this software library exists toUnify the different APIs and vector-based retrieval methodsSupport scientometrics analyses of citation dynamics, especially with respect to a vectorized 'landscape' of literature.Installing sciterraFirst, set up a virtual environment (e.g. viaminiconda,conda create -n sciterra, andconda activate sciterra).Install sciterra via git:python -m pip install 'sciterra @ git+https://github.com/nathimel/sciterra.git'Alternatively, download or clone this repository and navigate to the root folder, and install locally:pip install -e .UsageAtlasThe central object in sciterra is theAtlas. This is a basic data structure for containing scientific publications that are returned from calls to various bibliographic database APIs.An Atlas minimally requires a list ofPublications.PublicationA publication object is a minimal wrapper of the publication data, and minimally should have a string identifier. It is designed to encompass the basic metadata contained in the results from some bibliographic database API.fromsciterraimportAtlas,Publicationatl=Atlas([Publication({"identifier":"id"})])Alternatively, you can construct an Atlas by passing in a .bib file. The entries in this bibtex file will be parsed for unique identifiers (e.g., DOIs), and sent in an API call, and returned as Publications, which then populate an Atlas.atl=crt.bibtex_to_atlas(bibtex_filepath)In the line of code above, the variablecrtis an instance of aCartographerobject, which encapsulates the bookkeeping involved in querying a bibliographic database for publications.CartographerThe Cartographer class is named because interfaces with an Atlas to build out a library of publications. Since it does so via similarity-based retrieval, the resulting Atlas can be considered a 'region' of publications.To do this, a Cartographer needs two things: an API with which to interface, and a way of getting document embeddings. Both are encapsulated, respectively, by theLibrarianand theVectorizerclasses.fromsciterraimportCartographerfromsciterra.librariansimportSemanticScholarLibrarian# or ADSLibrarianfromsciterra.vectorizationimportSciBERTVectorizer# or Word2VecVectorizercrt=Cartographer(librarian=SemanticScholarLibrarian(),vectorizer=SciBERTVectorizer(),)LibrarianEach Librarian subclass is designed to be a wrapper for an existing python API service, such as theadspackage or thesemanticscholarclient library.A Librarian subclass also overrides two methods. The first isget_publications, which takes a list of identifiers, should query the specific API for that Librarian, and returns a list of Publications. Keywords arguments can be passed to specify the metadata that is kept for each publication (e.g. date, title, journal, authors, etc.) The second method isconvert_publication, which defines how the result of an API call is converted to a sciterra Publication object.Contributions to sciterra in the form of new Librarian subclasses are encouraged and appreciated.VectorizerVectorizer subclasses override one function,embed_documents, which takes a list of strings, representing the text of a publication (currently, just its abstract), and returns annp.ndarrayof embeddings.Under the hood, theprojectmethod of Cartographer, which is used during similarity-based retrieval, uses the vectorizer roughly as follows# Get abstractsdocs=[atlas[identifier].abstractforidentifierinidentifiers]# Embed abstractsresult=vectorizer.embed_documents(docs)embeddings=result["embeddings"]# depending on the vectorizer, sometimes not all embeddings can be obtained due to out-of-vocab issuessuccess_indices=result["success_indices"]# shape `(len(embeddings),)`fail_indices=result["fail_indices"]# shape `(len(docs) - len(embeddings))``Currently, sciterra has vectorizers usingSciBERT,SBERT, andWord2Vec. Contributions to sciterra in the form of new Vectorizer subclasses are also encouraged and appreciated.Putting it all togetherThe main use case for all of these ingredients is to iteratively build out a region of publications. This is done usingiterate_expand:fromsciterra.mapping.cartographyimportiterate_expand# Assuming the initial atlas contains just one publication(atl.center,)=atl.publications.values()# build out an atlas to contain 10,000 publications, with increasing dissimilarity to the initial publication, saving progress in binary files to the directory named "atlas".iterate_expand(atl=atl,crt=crt,atlas_dir="atlas",target_size=10000,center=atl.center,)This method has a number of useful keyword arguments that enable tracking the Atlas expansion, limiting the number of publications per expansion, how many times to try to get a response if there are connection issues, etc.Additional featuresThetopographysubmodule contains similarity-based metrics for publications, to support scientometrics analyses.AcknowledgmentsThis software is an reimplimentation of Zachary Hafen-Saavedra's library,cc.To cite sciterra, please use the following workshop paper,@inproceedings{Imel2023,
 author = {Imel, Nathaniel, and Hafen, Zachary},
 title = {Citation-similarity relationships in astrophysics},
 booktitle = {AI for Scientific Discovery: From Theory to Practice Workshop (AI4Science @ NeurIPS)},
 year = {2023},
 url = {https://openreview.net/pdf?id=mISayy7DPI},
}