Package: astrape
ASTRAPE : A STrategic, Reproducible & Accessible Project and ExperimentCreator : Woosog Benjamin Chaybenchay1@gmail.com (preferred) / benchay@kaist.ac.krpip install astrapedocstutorial for 'Experiment'tutorial for 'Project'ASTRAPE : A STrategic, Reproducible & Accessible Project and Experiment1. Overview2. Project2-1. Visulaizing Data2-2. CreatingExperiments With Different Random Seeds2-3. Set Models in Each Experiment2-4. Fit Models in Each Experiment2-5. Save Fitted Models in Each Experiment2-6. Plot Results2-6-1. Plot Performances With an Identical Model Type2-6-2. Plot Performances With an Identical Model Structure2-6-3. Plot Performances of All Model Structures3. Experiment3-1. Specifying Models3-2. (Optional) Specifying Trainers3-3. Fitting the Model3-4. Stacking Fitted Models3-5. Saving Models3-6. Checking the Best Model Thus Far3-7. (Stratified) K-Fold Cross-ValidationAstrape :https://en.wikipedia.org/wiki/Astrape_and_BronteAstrape is a package that would help you organize machine learning projects. It is written mostly in PyTorch Lightning(https://pytorchlightning.ai).This project is motivated by the need to provide packages, of which only "human-language-level codings" are necessary, to whom that is not familiar with machine learning or with programming. Even though there are high-level machine learning frameworks such as PyTorch(and PyTorch-Lightning) or Tensorflow, it is difficult for beginners even to run a simple Perceptron due to the presence of magic commands e.g., codes regarding saving results, hyperparameter tuning, etc. Even if one has a strong background in learning theory, it would be a long journey to conduct an experiment when he/she hasn't acquired basic programming skills.Astrape eliminated most of the low-level codings of the entire machine learning process so that every process in machine learning experiment can be done by typing nearly-human-level languages.I am currently writing lecture notes/slides about machine learning with practice sessions using astrape. Most of the materials are originated from machine learning courses held inSK Hynix(May, 2021 ~ Oct, 2021 : 16 days) andHyundai NGV(Feb, 2022: 10 days), where I joined as a teaching assistant developing such lecture materials and practice sessions.UpdatesAdd generative models inastrape.models.models_lightning(~ Aug, 2022)Enable real-time tracking of training process for sci-kit learn models(~ Oct, 2022)Add features regarding pre/post-processing of data(~ Oct, 2022)Release publicly available machine learning lectures with practice sessions integrated w/ astrape(~ Dec, 2022)1. Overview:zap: Astrape :zap:"Project" and "Experiment" conspire to the soul of astrape. The term "Project" here refers to "all set of possible machine learningexperimentsfor analyzing the given data". Wait, what is anexperimentanyway? Anexperimenthere means "a process of train/validation/test phase with certain random state acquired for all random operations such as splitting scheme, initialization scheme, etc.". "Experiment" is a collection ofexperimentswith the same random state.For stability's sake, you are tempted to (and should) conduct several "Experiments" with different random states to verify that your data analysis is indeed accurate. Astrape organizes such "Experiments" in a way that makes this sanity-checking process succinct and reproducible.2. ProjectProjectis defined as a set ofexperiments, with different random seed allocated to eachexperiment. It performs the A to Z of the process of a machine learning experiment.Projectcan visualize the data according to its type of domain e.g., image data, points.Check details in theproject tutorial.2-1. Visulaizing DataYou can visualize data using.plot_data()method. Depending on the domain of the data, such as image data or simple points in Euclidean space,.plot_data()automatically visulizes and saves the visualized figure. If the data is image data, you should specify the argumentdomain_typeas"image". Else if the data is points in Euclidean space, you should specify the argumentdomain_typeas"points". When the dimensionality of the points data is higher than 3,.plot_data()plots a 2D figure with 2 principal axes.2-2. CreatingExperiments With Different Random SeedsYou can createexperiments with different random states using.create_experiments()method. Random seeds will be generated as per theamountof experiments you want to create.2-3. Set Models in Each ExperimentYou can set identical models among the createdexperiments using.set_models()method. You should pass the type (class) of the model and its hyperparameters.2-4. Fit Models in Each ExperimentYou can train the model sequentially usingfit_experiments()method.
Event file for the real-time tracking of the experiment via TensorBoard is saved in {path}/{project_name}/FIT.Astrape uses Rich ProcessBar2-5. Save Fitted Models in Each ExperimentYou can save the fitted models using.save_stacks(), which is basically performing.save_stack()for the createdexperiments. Read3-5. Saving Modelsfor details.You can also save the fitted models using.save_project()method.2-6. Plot ResultsAstrape supports plotting results of the following:2-6-1. Plot Performances With an Identical Model Type.plot_identical_model_type()method plots & saves a figure showing performance of a specified model type (e.g., MLP, UNet) generated in each experiment.AUC among different random states2-6-2. Plot Performances With an Identical Model Structure.plot_identical_model_structure()method plots & saves a figure showing performance of a specified model structure generated in each experiment.AUC among different random states2-6-3. Plot Performances of All Model Structures.plot_all_model_structures()method plots & saves two figures showing performances of all model structrues created in theproject. One is a line plot of performances among different random states and the second is a box plot of performances of all model structures.Upper image shows performances of all model structures and the lower image shows a boxplot of all model structures trained in the project3. ExperimentWhen using Astrape, we expect you to conduct all experiments inside theexperiment.Experimentclass. This class takes a number of parameters, and you can check the details in thetutorial.Once you declare an experiment, all random operations are governed by the same random seed you defined as a parameter for the experiment. When initialized (with a given random state) and the train/validation/test data are specified, you should now declare models for the task.3-1. Specifying ModelsDeclare a model using.set_model()method. There are pre-defined models inastrape.models.models_lightningthat are easy to use, but you can also set anypl.LightningModules or descendants ofsklearn.base.BaseEstimatoras well.You can also declare sci-kit learn models and their variants(e.g., xgboost) as well using.set_model(). Astrape is compatible with sci-kit learn and PyTorch-lightning modules.3-2. (Optional) Specifying TrainersPyTorch Lightning usesTrainerfor training, validating, and testing models. You can specify it using the.set_trainer()method with trainer configurations as parameters. If you don't, default values will be set for theTrainer. Check thetutorialfor details.3-3. Fitting the ModelYou can fit the model using.fit()method. When you didn't specify aTrainerin previous step, default settings would be used in the fitting. Else, you can specifyTrainerimplicitly by passing the trainer configurations as parameters for.fit().Training and valiation process of a LightningModule-Based models are visualized in real-time using TensorBoard.3-4. Stacking Fitted ModelsExperimentclass has.stackas an attribute. If.stack_modelsis set toTrue, fitted models will automatically be saved to.stack. If.stack_modelsis set toFalse, it would stop stacking fitted models to the stack. However, it would still save the model that is just fitted i.e., it will have memory of 1 fit. You can toggle.stack_modelsusing.toggle_stack_models()method.Plus, you can check which model in the stack has the best performance using.best_ckpt_in_stack().3-5. Saving ModelsYou can save the current model using.save_ckpt()method, or you can save the models in the stack using.save_stack()method. After.save_stack(),.stackwill be flushed.An example of a.stack.stackis flushed after.save_stack().Example of a file tree3-6. Checking the Best Model Thus FarWith.best_ckpt_thus_far()method, you can check the best model saved (in local) thus far. It searches for the checkpoint with the best performance as per the validation metric you specify inval_metricargument.When definingpl.LightningModule, you can make your model to log metrics during each step/epoch, like the way in the following figure.you can log metrics using.log()method inpl.LightningModuleclassThe value that should be passed intoval_metricin.best_ckpt_thus_far()is the name of the metric you logged using.log().Models inastrape.models.models_lightninglogs metrics in the following way:training phase:"train/loss","train/acc","train/auc"validation phase:"val/loss","val/acc","val/auc"prediction phase:"pred/loss","pred/acc","pred/auc"test phase:"test/loss","test/acc","test/auc"In regression, only"{phase}/loss"is logged.For sci-kit learn models and its variants, The metrics are logged after the entire training.3-7. (Stratified) K-Fold Cross-ValidationYou can perform (stratified) k-fold cross-validation using.cross_validation()method. See details in thetutorial.One of the arguments that is passed tocross_validation()isparameters.parametersshould be a dictionary containing the name of each hyperparameter to tune as keys and its configuration as values.The values corresponding to the hyperparameter is also a dictionary and should follow the below format:Suppose you want to tune the hyperparameters with the following scheme.hyperparametervaluesdistributionbatch_sizein {32,64,128,256}gridlrin [1e-4,1e-2]sampled from uniform distribution(in log scale)For the batch size, the value corresponding to the key"batch_size"should be:>>> {"values": [32,64,128,256]}and for the learning rate, the value corresponding to the key"lr"should be:>>> {"distribution" : "log_scale_uniform", "min" : -4, "max" : -2}To sum up, this should be passed as a value of the argumentparameters.>>> parameters = {
        "batch_size" : {
              "values" : [32,64,128,256]
        },
        "lr" : {
              "distribution" : "log_scale_uniform",
              "min" : -4,
              "max" : -2
        }
  }