Package: astartes
astartes(as-tar-tees)Train:Validation:Test Algorithmic Sampling for Molecules and Arbitrary ArraysStatus BadgesUsageContinuous IntegrationReleaseOnline DocumentationFollowthis linkfor a nicely-rendered version of this README along with additional tutorials formoving from train_test_split in sklearn to astartes.
Keep reading for a installation guide and links to tutorials!InstallingastartesWe recommend installingastarteswithin a virtual environment, using eithervenvorconda(or other tools) to simplify dependency management. Python versions 3.8, 3.9, 3.10, 3.11, and 3.12 are supported on all platforms.WarningWindows (PowerShell) and MacOS Catalina or newer (zsh) require double quotes around text using the'[]'characters (i.e.pip install "astartes[molecules]").pipastartesis available onPyPIand can be installed usingpip:To include the featurization options for chemical data, usepip install astartes[molecules].To install only the sampling algorithms, usepip install astartes(this install will have fewer dependencies and may be more readily compatible in environments with existing workflows).condaastartespackage is also available onconda-forgewith this command:conda install -c conda-forge astartes.
To installastarteswith support for featurizing molecules, use:conda install -c conda-forge astartes aimsim.
This will download the baseastartespackage as well asaimsim, which is the backend used for molecular featurization.SourceTo installastartesfrom source for development, see theContributing & Developer Notessection.Statement of NeedMachine learning has sparked an explosion of progress in chemical kinetics, materials science, and many other fields as researchers use data-driven methods to accelerate steps in traditional workflows within some acceptable error tolerance.
To facilitate adoption of these models, there are two important tasks to consider:use a validation set when selecting the optimal hyperparameter for the model and separately use a held-out test set to measure performance on unseen data.evaluate model performance on both interpolative and extrapolative tasks so future users are informed of any potential limitations.astartesaddresses both of these points by implementing ansklearn-compatibletrain_val_test_splitfunction.
Additional technical detail is provided below as well as in our companion paper in the Journal of Open Source Software:Machine Learning Validation via Rational Dataset Sampling with astartes.
For a demo-based explainer using machine learning on a fast food menu, see theastartesReproducible Notebook published at the United States Research Software Engineers Conference atthis page.Target Audienceastartesis generally applicable to machine learning involving both discovery and inferenceandmodel validation.
There are specific functions inastartesfor applications in cheminformatics (astartes.molecules) but the methods implemented are general to all numerical data.Quick Startastartesis designed as a drop-in replacement forsklearn'strain_test_splitfunction (see thesklearn documentation). To switch toastartes, changefrom sklearn.model_selection import train_test_splittofrom astartes import train_test_split.Likesklearn,astartesaccepts any iterable object asX,y, andlabels.
Each will be converted to anumpyarray for internal operations, and returned as anumpyarray with limited exceptions: ifXis apandasDataFrame,yis aSeries, orlabelsis aSeries,astarteswill cast it back to its original type including its index and column names.NoteThe developers recommend passingX,y, andlabelsasnumpyarrays and handling the conversion to and from other types explicitly on your own. Behind-the-scenes type casting can lead to unexpected behavior!By default,astarteswill split data randomly. Additionally, a variety of algorithmic sampling approaches can be used by specifying thesamplerargument to the function (see theTable of Implemented Samplersfor a complete list of options and their corresponding references):fromsklearn.datasetsimportload_diabetesX,y=load_diabetes(return_X_y=True)X_train,X_test,y_train,y_test=train_test_split(X,# preferably numpy arrays, but astartes will cast it for youy,sampler='kennard_stone',# any of the supported samplers)NoteExtrapolation sampling algorithms will return an additional set of arrays (the cluster labels) which will result in aValueError: too many values to unpackif not called properly. See thesplit_comparisonsGoogle colab demofor a full explanation.That's all you need to get started withastartes!
The next sections include more examples and some demo notebooks you can try in your browser.Example NotebooksClick the badges in the table below to be taken to a live, interactive demo ofastartes:DemoTopicLinkComparing Sampling Algorithms with Fast FoodVisual representations of how different samplers affect data partitioningUsingtrain_val_test_splitwith thesklearnexample datasetsDemonstrating how witholding a test set withtrain_val_test_splitcan impact performanceCheminformatics sample set partitioning withastartesExtrapolation vs. Interpolation impact on cheminformatics model accuracyComparing partitioning approaches for alkanesVisualizing how sampler impact model performance with simple chemicalsTo execute these notebooks locally, clone this repository (i.e.git clone https://github.com/JacksonBurns/astartes.git), navigate to theastartesdirectory, runpip install .[demos], then open and run the notebooks in your preferred editor.
You donotneed to execute the cells prefixed with%%capture- they are only present for compatibility with Google Colab.Withhold Testing Data withtrain_val_test_splitFor rigorous ML research, it is critical to withhold some data during training to use atestset.
The model shouldneversee this data during training (unlike the validation set) so that we can get an accurate measurement of its performance.Withastartesperforming this three-way data split is readily available withtrain_val_test_split:fromastartesimporttrain_val_test_splitX_train,X_val,X_test=train_val_test_split(X,sampler='sphere_exclusion')You can now train your model withX_train, optimize your model withX_val, and measure its performance withX_test.Evaluate the Impact of Splitting Algorithms on Regression ModelsFor data with many features it can be difficult to visualize how different sampling algorithms change the distribution of data into training, validation, and testing like we do in some of the demo notebooks.
To aid in analyzing the impact of the algorithms,astartesprovidesgenerate_regression_results_dict.
This function allows users to quickly evaluate the impact of different splitting techniques on anysklearn-compatible model's performance.
All results are stored in a nested dictionary ({sampler:{metric:{split:score}}}) format and can be displayed in a neatly formatted table using the optionalprint_resultsargument.fromsklearn.svmimportLinearSVRfromastartes.utilsimportgenerate_regression_results_dictasgrrdsklearn_model=LinearSVR()results_dict=grrd(sklearn_model,X,y,print_results=True,)TrainValTest----------------------------MAE1.415223.134352.17091RMSE2.030623.737212.40041R20.907450.807870.78412Additional metrics can be passed togenerate_regression_results_dictvia theadditional_metricsargument, which should be a dictionary mapping the name of the metric (as astring) to the function itself, like this:fromsklearn.metricsimportmean_absolute_percentage_erroradd_met={"mape":mean_absolute_percentage_error}grrd(sklearn_model,X,y,additional_metric=add_met)See the docstring forgenerate_regression_results_dict(withhelp(generate_regression_results_dict)) for more information.Usingastarteswith Categorical DataAny of the implemented sampling algorithms whose hyperparameters allow specifying themetricordistance_metric(effectively1-metric) can be co-opted to work with categorical data.
Simply encode the data in a format compatible with thesklearnmetric of choice and then callastarteswith that metric specified:fromsklearn.metricsimportjaccard_scoreX_train,X_test,y_train,y_test=train_test_split(X,y,sampler='kennard_stone',hopts={"metric":jaccard_score},)Other samplers which do not allow specifying a categorical distance metric did not provide a method for doing so in their original inception, though it is possible that they can be adapted for this application.
If you are interested in adding support for categorical metrics to an existing sampler, consider opening aFeature Request!Access Sampling Algorithms DirectlyThe sampling algorithms implemented inastartescan also be directly accessed and run if it is more useful for your applications.
In the below example, we import the Kennard Stone sampler, use it to partition a simple array, and then retrieve a sample.fromastartes.samplers.interpolationimportKennardStonekennard_stone=KennardStone([[1,2],[3,4],[5,6]])first_2_samples=kennard_stone.get_sample_idxs(2)All samplers inastartesimplement a_sample()method that is called by the constructor (i.e. greedily) and either aget_sampler_idxsorget_cluster_idxsfor interpolative and extrapolative samplers, respectively.
For more detail on the implementaiton and design of samplers inastartes, see theDeveloper Notessection.Theory and Application ofastartesThis section of the README details some of the theory behind why the algorithms implemented inastartesare important and some motivating examples.
For a comprehensive walkthrough of the theory and implementation ofastartes, followthis linkto read the companion paper (freely available and hosted here on GitHub).NoteWe reference open-access publications wherever possible. For articles locked behind a paywall (denoted with :small_blue_diamond:), we instead suggest readingthis Wikipedia pageand absolutelynotattempting to bypass the paywall.Rational Splitting AlgorithmsWhile much machine learning is done with a random choice between training/validation/test data, an alternative is the use of so-called "rational" splitting algorithms.
These approaches use some similarity-based algorithm to divide data into sets.
Some of these algorithms include Kennard-Stone (Kennard & Stone:small_blue_diamond:), Sphere Exclusion (Tropsha et. al:small_blue_diamond:),as well as the OptiSim as discussed inApplied Chemoinformatics: Achievements and Future Opportunities:small_blue_diamond:.
Some clustering-based splitting techniques have also been incorporated, such asDBSCAN.There are two broad categories of sampling algorithms implemented inastartes: extrapolative and interpolative.
The former will force your model to predict on out-of-sample data, which creates a more challenging task than interpolative sampling.
See the table below for all of the sampling approaches currently implemented inastartes, as well as the hyperparameters that each algorithm accepts (which are passed in withhopts) and a helpful reference for understanding how the hyperparameters work.
Note thatrandom_stateis defined as a keyword argument intrain_test_splititself, even though these algorithms will use therandom_statein their own work.
Do not provide arandom_statein thehoptsdictionary - it will be overwritten by therandom_stateyou provide fortrain_test_split(or the default if none is provided).Implemented Sampling AlgorithmsSampler NameUsage StringTypeHyperparametersReferenceNotesRandom'random'Interpolativeshufflesklearn train_test_splitDocumentationThis sampler is a direct passthrough tosklearn'strain_test_split.Kennard-Stone'kennard_stone'InterpolativemetricOriginal Paper byKennard & Stone:small_blue_diamond:Euclidian distance is used by default, as described in the original paper.Sample set Partitioning based on joint X-Y distances (SPXY)'spxy'Interpolativedistance_metricSaldhana et. aloriginal paper:small_blue_diamond:Extension of Kennard Stone that also includes the response when sampling distances.Mahalanobis Distance Kennard Stone (MDKS)'spxy'(MDKS is derived from SPXY)Interpolativenone, see NotesSaptoro et. aloriginal paperMDKS is SPXY using Mahalanobis distance and can be called by using SPXY withdistance_metric="mahalanobis"Scaffold'scaffold'Extrapolativeinclude_chiralityBemis-Murcko Scaffold:small_blue_diamond: as implemented in RDKitThis sampler requires SMILES strings as input (use themoleculessubpackage)Molecular Weight'molecular_weight'Extrapolativenone~Sorts molecules by molecular weight as calculated by RDKitSphere Exclusion'sphere_exclusion'Extrapolativemetric,distance_cutoffcustom implementationVariation on Sphere Exclusion for arbitrary-valued vectors.Time Based'time_based'ExtrapolativenonePapers using Time based splitting:Chen et al.:small_blue_diamond:,Sheridan, R. P:small_blue_diamond:,Feinberg et al.:small_blue_diamond:,Struble et al.This sampler requireslabelsto be an iterable of either date or datetime objects.Target Property'target_property'Extrapolativedescending~Sorts data by regression target yOptimizable K-Dissimilarity Selection (OptiSim)'optisim'Extrapolativen_clusters,max_subsample_size,distance_cutoffcustom implementationVariation onOptiSimfor arbitrary-valued vectors.K-Means'kmeans'Extrapolativen_clusters,n_initsklearn KMeansPassthrough tosklearn'sKMeans.Density-Based Spatial Clustering of Applications with Noise (DBSCAN)'dbscan'Extrapolativeeps,min_samples,algorithm,metric,leaf_sizesklearn DBSCANDocumentationPassthrough tosklearn'sDBSCAN.Minimum Test Set Dissimilarity (MTSD)~~upcoming inastartesv1.x~~Restricted Boltzmann Machine (RBM)~~upcoming inastartesv1.x~~Kohonen Self-Organizing Map (SOM)~~upcoming inastartesv1.x~~SPlit Method~~upcoming inastartesv1.x~~Domain-Specific ApplicationsBelow are some field specific applications ofastartes. Interested in adding a new sampling algorithm or featurization approach? SeeCONTRIBUTING.md.Chemical Data and theastartes.moleculesSubpackageMachine Learning is enormously useful in chemistry-related fields due to the high-dimensional feature space of chemical data.
To properly apply ML to chemical data for inferenceordiscovery, it is important to know a model's accuracy under the two domains.
To simplify the process of partitioning chemical data,astartesimplements a pre-built featurizer for common chemistry data formats.
After installing withpip install astartes[molecules]one can import the new train/test splitting function like this:from astartes.molecules import train_test_split_moleculesThe usage of this function is identical totrain_test_splitbut with the addition of new arguments to control how the molecules are featurized:train_test_split_molecules(molecules=smiles,y=y,test_size=0.2,train_size=0.8,fingerprint="daylight_fingerprint",fprints_hopts={"minPath":2,"maxPath":5,"fpSize":200,"bitsPerHash":4,"useHs":1,"tgtDensity":0.4,"minSize":64,},sampler="random",random_state=42,hopts={"shuffle":True,},)To see a complete example of usingtrain_test_split_moleculeswith actual chemical data, take a look in theexamplesdirectory and the briefcompanion paper.Configuration options for the featurization scheme can be found in the documentation forAIMSimthough most of the critical configuration options are shown above.Reproducibilityastartesaims to be completely reproducible across different platforms, Python versions, and dependency configurations - any version ofastartesv1.x should result in theexactsame splits, always.
To that end, the default behavior ofastartesis to use42as the random seed andalwaysset it.
Runningastarteswith the default settings will always produce the exact same results.
We have verified this behavior on Debian Ubuntu, Windows, and Intel Macs from Python versions 3.7 through 3.11 (with appropriate dependencies for each version).Known Reproducibility LimitationsInevitably external dependencies ofastarteswill introduce backwards-incompatible changes.
We continually run regression tests to catch these, and will list allknownlimitations here:sklearnv1.3.0 introduced backwards-incompatible changes in theKMeanssampler that changed how the random initialization affects the results, even given the same random seed. Different version ofsklearnwill affect the performance ofastartesand we recommend including the exact version ofscikit-learnandastartesused, when applicable.NoteWe are limited in our ability to test on M1 Macs, but from our limited manual testing we achieve perfect reproducbility in all casesexcept occasionallywithKMeanson Apple silicon.astartesis still consistent between runs on the same platform in all cases, and other samplers are not impacted by this apparent bug.How to CiteIf you useastartesin your work please follow the link below to our (Open Access!) paper in the Journal of Open Source Software or use the "Cite this repository" button on GitHub.Machine Learning Validation via Rational Dataset Sampling with astartesContributing & Developer NotesSeeCONTRIBUTING.mdfor instructions on installingastartesfor development, making a contribution, and general guidance on the design ofastartes.