Package: exaparser
Exabyte Parser (ExaParser)Exabyte parser is a python package to extract and convert materials modeling data (eg. Density Functional Theory, Molecular Dynamics) on disk toESSE/EDCformat.FunctionalityAs below:Extract structural information and material properties from simulation dataSerialize extracted information according toESSE/EDCStore serialized data on disk or remote databasesSupport for multiple simulation engines, including:VASPQuantum ESPRESSOothers, to be addedThe package is written in a modular way easy to extend for additional applications and properties of interest. Contributions can be in the form of additionalfunctionalityandbug/issue reports.InstallationExaParser can be installed as below.Installgit-lfsin order to pull the files stored on Git LFS.Clone repository:gitclonegit@github.com:Exabyte-io/exaparser.gitInstallvirtualenvusingpipif not already present:pipinstallvirtualenvCreate virtual environment and install required packages:cdexaparser
virtualenvvenvsourcevenv/bin/activateexportGIT_LFS_SKIP_SMUDGE=1pipinstall-rrequirements.txtUsageExaparser will look in the following locations for theconfigfile, and use the first one it finds:The existing file in the root of this repository,if installed as editable source.  This won't work for production installs, and is just for testing scenarios.Your user's home directory at~/.exabyte/exaparser/configA global system configuration at/etc/exabyte/exaparser/configCopy theconfigfile from the root of this repo to one of the above locations and edit it.Edit the config file and adjust parameters as necessary. The most important ones are listed below.AddExabyteRESTfulAPItodata_handlersparameters list (comma-separated), if not already present. This will enable upload the data into Exabyte.io account.New users can registerhereto obtain an Exabyte.io account.Setowner_slug,project_slug,api_account_id, andapi_auth_tokenifExabyteRESTfulAPIis enabled.SeeRESTful API Documentationto learn how to obtain authentication parameters.Adjustworkflow_template_nameparameter in case a different template should be used.By default aShell Workflowis constructed. SeeTemplatessection for more details.Adjustpropertiesparameter to extract desired properties; all listed properties will be attempted for extraction.Run the below commands to extract the data.sourcevenv/bin/activate
exaparser-wPATH_TO_JOB_WORKING_DIRECTORYor just call exaparser with the explicit path to the virtualenv binary:venv/bin/activate/exaparser-wPATH_TO_JOB_WORKING_DIRECTORYTestsRun the following command to run the tests../run-tests.sh-p=PYTHON_BIN-v=VENV_NAME-t=TEST_TYPEAll the passed parameters are optional, with the defaults beingpython3,venv, andunit, respectively.The script will create a virtual environment and populate it, so there's no need to create one manually for testing.Note that the testing virtualenv uses therequirements-dev.txtfile, where a production usage should use therequirements.txtfile.  This avoids installing test dependencies when not needed.ContributionThis repository is anopen-sourcework-in-progress and we welcome contributions. We suggest forking this repository and introducing the adjustments there, the changes in the fork can further be considered for merging into this repository as explained inGitHub Standard Fork and Pull Request Workflow.ArchitectureThe following diagram presents the package architecture.Here's an example flow of data/events:User invokes the parser with a path to a job working directory.The parser initializes aJobclass to extract and serialize the job.Job class usesWorkflowparser to extract and serialize the workflow.The Workflow is initialized with aTemplateto help the parser to construct the workflow.Users can add new templates or adjust the current ones to support complex workflows.Workflow parser iterates over theUnitsto extractapplication-related datainput and output filesmaterials (initial/final structures) and propertiesThe job utilizesComputeclasses to extract compute configuration from the resource management system.Once the job is formed it is passed toData Handlerclasses to handle data, e.g. storing data in Exabyte platform.TemplatesWorkflow templates are used to help the parser extracting the data as users follow different approaches to name their input/output files and organize their job directories. Readers are referred toExabyte.io Documentationfor more information about the structure of workflows. As explain above aShell Workflow Templateis used by default to construct the workflow. For each unit of the workflow one should specifystdoutFile, the relative path to the file containing the standard output of the job,workDir, the relative path to directory containing data for the unit and the name ofinputfiles.TODO ListDesirable features for implementation:Implement PBS/Torque and SLURM compute parsersImplement VASP and Espresso execution unit parsersAdd other data handlersAdd complex workflow templatesLinksExabyte Source of Schemas and Examples (ESSE), Github RepositoryVienna Ab-initio Simulation Package (VASP), official websiteQuantum ESPRESSO, Official Website