Package: miles-guess
MILES-GuessGeneralized Uncertainty for Earth System Science (GUESS)Developed by the Machine Ingetration and Learning for Earth Systems (MILES) group at the National Center for Atmospheric Research (NCAR), Boulder CO, USAContributersJohn SchreckDavid John GagneCharlie BeckerGabrielle GantosDhamma KimparaThomas MartinSetup from ScratchInstall the Miniconda Python installer availablehere.First clone the miles-guess repo from github.gitclonehttps://github.com/ai2es/miles-guess.git`cdmiles-guessCreate a conda environment for non-Casper/Derecho users:mambaenvcreate-fenvironment.yml`condaactivateguess`Create a conda environment for Casper/Derecho users including Tensorflow 2.15 with GPU support.mambaenvcreate-fenvironment_casper.yml`condaactivateguessUsing miles-guessThe package contains three scripts for training three regression models, and one for training categorical models.
The regression examples are trained on our surface layer ("SL") dataset for predicting latent heat and other quantities,
and the categorical example is trained on a precipitation dataset ("p-type").The law of total variance for each model prediction target may be computed as$$LoTV = E[\sigma^2] + Var[\mu]$$which is the sum of aleatoric and epistemic contributions, respectively.1a. Train/evaluate a deterministic multi-layer perceptrion (MLP) on the SL dataset:python3applications/train_mlp_SL.py-cconfig/model_mlp_SL.yml1b. Train/evaluate a parametric "Gaussian" MLP on the SL dataset:pythonapplications/train_gaussian_SL.py-cconfig/model_gaussian_SL.yml1c. Train/evaluate a parametric "normal-inverse gamma" (evidential) MLP on the SL dataset:pythonapplications/train_evidential_SL.py-cconfig/model_evidential_SL.yml2a. Train a categorical MLP classifier on the p-type dataset:pythonapplications/train_classifier_ptype.py-cconfig/model_classifier_ptype.yml2b. Train an evidential MLP classifier on the p-type dataset:pythonapplications/train_classifier_ptype.py-cconfig/model_evidential_ptype.yml2c. Evaluate a categorical/evidential classifier on the p-type dataset:pythonapplications/evaluate_ptype.py-cconfig/model_classifier_ptype.yml3. Ensembling modes for the deterministic model (1a)There are four "modes" for training the deterministic MLP (1a) that are controlled using the "ensemble" field in a model configuration.ensemble:n_models:100n_splits:20monte_carlo_passes:0where n_models means the number of models to train using a fixed data split with variable initial weight initializations, n_splits means the number of models to train using variable training and validation splits (random initializations), and mc_carlo_passes means the number of MC-dropout evaluations performed on a given input to the model.3a. Single Modeensemble:n_models:1n_splits:1monte_carlo_passes:0Train a single deterministic model (no uncertainty evaluation). If MC passes > 0, an ensemble is created after the model finishes training.3b. Data Modeensemble:n_models:1n_splits:10monte_carlo_passes:100Create an ensemble of models (random initialization) using cross-validation splits. If MC passes > 0, an ensemble is created after each model finishes training on the test holdout. The LOTV may then be applied to the ensemble created from cross-validation. Otherwise a single ensemble is created but the LOTV is not applied.3c. Model Modeensemble:n_models:10n_splits:1monte_carlo_passes:100Create an ensemble of models using a fixed train/validation/test data split and variable model layer weight initializations. If MC passes > 0, an ensemble is created after each model finishes training. The LOTV may then be applied to the ensemble created from variable weight initializations to obtain uncertainty estimations for each prediction target. Otherwise a single ensemble is created but the LOTV is not applied.3d. Ensemble Modeensemble:n_models:1n_splits:1monte_carlo_passes:0Create an ensemble of ensembles. The first ensemble is created using cross validation and a fixed weight initialization, from which a mean and variance may be obtained for each prediction target. The second ensemble is created by varying the weight initalization that can then be used with the LOTV to obtain uncertainty estimations for each prediction target. The MC steps field is ignored in ensemble mode.4. Ensembling modes for the Gaussian parametric model (1b)There are three "modes" for training the Gaussian MLP (1b).4a. Single Modeensemble:n_models:1n_splits:1monte_carlo_passes:0Train a single deterministic model (no LOTV evaluation). If MC passes > 0, an ensemble is created after the model finishes training (LOTV evaluation).4b. Data Modeensemble:n_models:1n_splits:10monte_carlo_passes:0Create an ensemble of models using cross-validation splits, and then LOTV evaluation.4c. Model Modeensemble:n_models:10n_splits:1monte_carlo_passes:0Create an ensemble of models using different random initializations and a fixed cross-validation split, and then LOTV evaluation.Configuration filesIn addition to the ensemble field, the other fields in the configuration file areseed:1000save_loc:"/path/to/save/directory"training_metric:"val_mae"direction:"min"where seed allows for reproducability, save_loc is where data will be saved, and training metric and direction are used as the validation metric (and direction).For regression tasks, other fields in a configuration file are model and callbacks:model:activation:relubatch_size:193dropout_alpha:0.2epochs:200evidential_coef:0.6654439861214466hidden_layers:3hidden_neurons:6088kernel_reg:l2l1_weight:0.0l2_weight:7.908676527243475e-10lr:3.5779279071474884e-05metrics:maeoptimizer:adamuncertainties:trueuse_dropout:trueuse_noise:falseverbose:2callbacks:EarlyStopping:monitor:"val_loss"patience:5mode:"min"verbose:0ReduceLROnPlateau:monitor:"val_loss"factor:0.1patience:2min_lr:1.0e-12min_delta:1.0e-08mode:"min"verbose:0CSVLogger:filename:"training_log.csv"separator:","append:FalseModelCheckpoint:filepath:"model.h5"monitor:"val_loss"save_weights:Truesave_best_only:Truemode:"min"verbose:0For categorical tasks, the model field changes slightly:model:activation:leakybalanced_classes:1batch_size:3097dropout_alpha:0.31256692323263807epochs:200hidden_layers:4hidden_neurons:6024loss:categorical_crossentropyloss_weights:-21.465788717561477-83.31367732936326-136.50944842077058-152.62042204485107lr:0.0004035503144482269optimizer:adamoutput_activation:softmaxuse_dropout:1verbose:0where the user has two options:(1) A standard deterministic classifier is trained whenloss:cateogical_crossentropyoutput_activation:softmax(2) An evidential classifier is trained whenloss:dirichletoutput_activation:linearCallbacks are not required in regression training, however a custom callback which tracks the current epoch is requried for the categorical model and is added automatically (the user does not need to specify it in the callbacks field). The user may add any other supported keras callback by adding the relevant fields to the callbacks field.Depending on the problem, a data field is customized and also present in the configuration files. See the examples for surface layer and p-type data sets for more details.ECHO hyperparameter optimizationConfiguration files are also supplied for use with the Earth Computing Hyperparameter Optimization (ECHO) package. See the echo packagehttps://github.com/NCAR/echo-opt/tree/main/echofor more details on the configuration fields.