Package: matbench-genmetrics
This is not an official repository of Matbench, but eventually, it may beincorporated into Matbenchmatbench-genmetricsGenerative materials benchmarking metrics, inspired byguacamolandCDVAE.This repository provides standardized benchmarks for benchmarking generative models for
crystal structure. Each benchmark has a fixed dataset, a predefined split, and a notion
of best (i.e. metric) associated with it.Getting StartedInstallation, a dummy example, output metrics for the example, and descriptions of the benchmark metrics.InstallationCreate a conda environment with thematbench-genmetricspackage installed from theconda-forgechannel. Then activate the environment.**NOTE: not available on conda-forge as of 2022-07-30, recipe under review by
conda-forge team. So usepip install matbench-genmetricsfor nowcondacreate--namematbench-genmetrics--channelconda-forgepython==3.9.*matbench-genmetrics
condaactivatematbench-genmetricsNOTE: It doesn't have to be Python 3.9; you can removepython==3.9.*altogether or
change this to e.g.python==3.8.*. SeeAdvanced InstallationExampleNOTE: be sure to setdummy=Falsefor the real/full benchmark run. MPTSMetrics10 is
intended for fast prototyping and debugging, as it assumes only 10 generated structures.>>>fromtqdmimporttqdm>>>frommp_time_split.utils.genimportDummyGenerator>>>frommatbench_genmetrics.coreimportMPTSMetrics10,MPTSMetrics100,MPTSMetrics1000,MPTSMetrics10000>>>mptm=MPTSMetrics10(dummy=True)>>>forfoldinmptm.folds:>>>train_val_inputs=mptm.get_train_and_val_data(fold)>>>dg=DummyGenerator()>>>dg.fit(train_val_inputs)>>>gen_structures=dg.gen(n=mptm.num_gen)>>>mptm.evaluate_and_record(fold,gen_structures)Outputprint(mptm.recorded_metrics){0:{"validity":0.4375,"coverage":0.0,"novelty":1.0,"uniqueness":0.9777777777777777,},1:{"validity":0.4390681003584229,"coverage":0.0,"novelty":1.0,"uniqueness":0.9333333333333333,},2:{"validity":0.4401197604790419,"coverage":0.0,"novelty":1.0,"uniqueness":0.8222222222222222,},3:{"validity":0.4408740359897172,"coverage":0.0,"novelty":1.0,"uniqueness":0.8444444444444444,},4:{"validity":0.4414414414414415,"coverage":0.0,"novelty":1.0,"uniqueness":0.9111111111111111,},}MetricsMetricDescriptionValidityOne minus (Wasserstein distance between distribution of space group numbers for train and generated structures divided by distance of dummy case between train andspace_group_number == 1). See alsohttps://github.com/sparks-baird/matbench-genmetrics/issues/44CoverageMatch counts between held-out test structures and generated structures divided by number of test structures ("predict the future").NoveltyOne minus (match counts between train structures and generated structures divided by number of generated structures).UniquenessOne minus (non-self-comparing match counts within generated structures divided by total possible non-self-comparing matches).A match is whenStructureMatcher(stol=0.5, ltol=0.3, angle_tol=10.0).fit(s1, s2)evaluates toTrue.Detailed descriptions of the metrics are given inhttps://self-driving-lab-demo.readthedocs.io/en/latest/metrics.html.Advanced InstallationAnaconda (conda) installation (recommended)(2022-07-30, conda-forge installation pending, fallback topip install matbench-genmetricsas separate command)Create and activate a newcondaenvironment namedmatbench-genmetrics(-n) that will search for and install thematbench-genmetricspackage from theconda-forgeAnaconda channel (-c).condaenvcreate-nmatbench-genmetrics-cconda-forgematbench-genmetrics
condaactivatematbench-genmetricsAlternatively, in an already activated environment:condainstall-cconda-forgematbench-genmetricsIf you run into conflicts with packages you are integrating withmatbench-genmetrics, please try installing all packages in a single line of code (or two if mixingcondaandpippackages in the same environment) and installing withmamba(source).PyPI (pip) installationCreate and activate a newcondaenvironment namedmatbench-genmetrics(-n) withpython==3.9.*or your preferred Python version, then installmatbench-genmetricsviapip.condacreate-nmatbench-genmetricspython==3.9.*
condaactivatematbench-genmetrics
pipinstallmatbench-genmetricsEditable installationIn order to set up the necessary environment:clone and enter the repository via:gitclonehttps://github.com/sparks-baird/matbench-genmetrics.gitcdmatbench-genmetricscreate and activate a new conda environment (optional, but recommended)condaenvcreate--namematbench-genmetricspython==3.9.*
condaactivatematbench-genmetricsperform an editable (-e) installation in the current directory (.):pipinstall-e.NOTE:Some changes, e.g. insetup.cfg, might require you to runpip install -e .again.Optional and needed only once aftergit clone:install severalpre-commitgit hooks with:pre-commitinstall# You might also want to run `pre-commit autoupdate`and checkout the configuration under.pre-commit-config.yaml.
The-n, --no-verifyflag ofgit commitcan be used to deactivate pre-commit hooks temporarily.installnbstripoutgit hooks to remove the output cells of committed notebooks with:nbstripout--install--attributesnotebooks/.gitattributesThis is useful to avoid large diffs due to plots in your notebooks.
A simplenbstripout --uninstallwill revert these changes.Then take a look into thescriptsandnotebooksfolders.Dependency Management & ReproducibilityAlways keep your abstract (unpinned) dependencies updated inenvironment.ymland eventually
insetup.cfgif you want to ship and install your package viapiplater on.Create concrete dependencies asenvironment.lock.ymlfor the exact reproduction of your
environment with:condaenvexport-nmatbench-genmetrics-fenvironment.lock.ymlFor multi-OS development, consider using--no-buildsduring the export.Update your current environment with respect to a newenvironment.lock.ymlusing:condaenvupdate-fenvironment.lock.yml--pruneProject Organization├── AUTHORS.md              <- List of developers and maintainers.
├── CHANGELOG.md            <- Changelog to keep track of new features and fixes.
├── CONTRIBUTING.md         <- Guidelines for contributing to this project.
├── Dockerfile              <- Build a docker container with `docker build .`.
├── LICENSE.txt             <- License as chosen on the command-line.
├── README.md               <- The top-level README for developers.
├── configs                 <- Directory for configurations of model & application.
├── data
│   ├── external            <- Data from third party sources.
│   ├── interim             <- Intermediate data that has been transformed.
│   ├── processed           <- The final, canonical data sets for modeling.
│   └── raw                 <- The original, immutable data dump.
├── docs                    <- Directory for Sphinx documentation in rst or md.
├── environment.yml         <- The conda environment file for reproducibility.
├── models                  <- Trained and serialized models, model predictions,
│                              or model summaries.
├── notebooks               <- Jupyter notebooks. Naming convention is a number (for
│                              ordering), the creator's initials and a description,
│                              e.g. `1.0-fw-initial-data-exploration`.
├── pyproject.toml          <- Build configuration. Don't change! Use `pip install -e .`
│                              to install for development or to build `tox -e build`.
├── references              <- Data dictionaries, manuals, and all other materials.
├── reports                 <- Generated analysis as HTML, PDF, LaTeX, etc.
│   └── figures             <- Generated plots and figures for reports.
├── scripts                 <- Analysis and production scripts which import the
│                              actual PYTHON_PKG, e.g. train_model.
├── setup.cfg               <- Declarative configuration of your project.
├── setup.py                <- [DEPRECATED] Use `python setup.py develop` to install for
│                              development or `python setup.py bdist_wheel` to build.
├── src
│   └── matbench_genmetrics <- Actual Python package where the main functionality goes.
├── tests                   <- Unit tests which can be run with `pytest`.
├── .coveragerc             <- Configuration for coverage reports of unit tests.
├── .isort.cfg              <- Configuration for git hook that sorts imports.
└── .pre-commit-config.yaml <- Configuration of pre-commit git hooks.NoteThis project has been set up usingPyScaffold4.2.2.post1.dev2+ge50b5e1 and thedsproject extension0.7.2.post1.dev2+geb5d6b6.