Package: openalex-raw
OpenAlex-RAWThis is a python module to process the OpenAlex dataset from the snapshot raw files available from theOpenAlex website.InstallationTo use the package you need to have a python (>=3.7) environment installed in your system. The package can be installed viapipor by downloading the source code from this repository.Downloading the OpenAlex snapshotIf you did not already download the snapshot, you can follow the instalation instructions from the OpenAlex website inDownload OpenAlex Snapshot to your machine. Here we provide a summary of the steps to download the dataset. Please, check the OpenAlex website for the most up to date instructions.First, install theaws clitool by following the instructions on theAWS-cli website.Next, use the following command to download the snapshot:awss3sync's3://openalex''openalex-snapshot'--no-sign-requestA new folder namedopenalex-snapshotwill be created in your current working directory containing the dataset. Note that this process can take a long time as the dataset is over 300GB.Installing the OpenAlex-RAW packageThe package can be installed via pip by running the following command:pipinstallopenalex-rawAll the required packages are installed automatically.You can also download the source code from this repository and install it manually. This can be done usinggit:gitclonehttps://github.com/filipinascimento/openalex-rawNext, you need to install the package usingpiporsetup.py:pipinstall-e./openalex-raworcdopenalex-raw
pythonsetup.pyinstallUsage RAW accessTo go over all the entries of a certain type in the dataset, you can use the following code:frompathlibimportPath# tqdm is used to print a nice progress bar# install it using `pip install tqdm`fromtqdm.autoimporttqdmimportopenalexrawasoaraw# Path to the OpenAlex snapshotopenAlexPath=Path("<Location of the OpenAlex Snapshot>")# Initializing the OpenAlex object with the OpenAlex snapshot pathoa=oaraw.OpenAlex(openAlexPath=openAlexPath)# Which entity to process# "works" | "authors" | "institutions" | "venues" | "concepts"entityType="works"# Getting the number of entriesentitiesCount=oa.getRawEntityCount(entityType)# Iterating over all the entities of a certain typeforentityintqdm(oa.rawEntities(entityType),total=entitiesCount):openAlexID=entity["id"]# do something with the entityOn fast storage, it may take a couple of hours to iterate over all the entities forworksorauthorstypes. Forinstitutionsandvenues, andconceptstypes, it may take just a few minutes.Generating Schema and ReportSchemas and reports for each entity type can be found respectively in the foldersSchemaandReportsof this repository. Schema files are in machine-readable JSON format and contain all the fields and non-null counts, nested structures and lists are included. The reports show the number and percentage of the coverage of the fields in the dataset. Both Schema and Report files are named according to the OpenAlex entity type. Schema files also include the most common values (samples) for each field. Two schema files are provided: one with samples (e.g.,Schema/schema_works_samples.json) and another without (e.g.,Schema/schema_works.json).To generate/update all the reports and schema, check the fileExamples/create_report.py. Building the report can take a long time. You can use the provided schema files when generatingdbgzarchives.Coming soonRandom access based on the OpenAlex ID viadbgz.Better documentation for Schema/Report generators.Full API documentationThe following is the documentation of the package's API.classOpenAlexOpenAlex(openAlexPath,verbose=False):Class to access the OpenAlex data snapshots.openAlexPath:strorpathlib.PathThe path to the OpenAlex directory. (default: current working directory)verbose:boolIf True, print out more information. (default: False)ReturnsOpenAlexobject
The OpenAlex instance that can be used to access the dataset.methodgetRawEntityCountOpenAlex.getRawEntityCount(entityType):Get the number of raw entities of the given entity type.entityType:strEntity type can be"authors","concepts","institutions","venues"or"works".ReturnsintThe number of entities for the providedentityType.methodrawEntitiesOpenAlex.rawEntities(entityType):Iterate over the entities of the selected type directly from the raw snapshot.entityType:strEntity type can be"authors","concepts","institutions","venues"or"works".ReturnsiterableAn iterable collection of entities of the providedentityType.