Repository: lanl/Yoke
Pull request #8: Adding a mini test harness.
Author: login: galegozi
Created: 2025-03-10T22:19:24Z | Merged: 2025-03-26T17:49:09Z

This test harness is designed to be a lightweight test that functions like an integration test. It runs one batch/epoch just to make sure that the code runs.
NOTE: this is not designed to be used for production, this is something we would use (internally) just to test code. This would be useful when making a new piece of code (such as a loss function).


Diff:
diff --git a/applications/harnesses/mini-run-test/README.md b/applications/harnesses/mini-run-test/README.md
new file mode 100644
index 00000000..34ec8946
--- /dev/null
+++ b/applications/harnesses/mini-run-test/README.md
@@ -0,0 +1,15 @@
+# Mini Run Test Harness
+
+This folder contains the **Mini Run Test Harness**, a lightweight framework designed for an end-to-end test. This ensures a full-weight study is ran, just significantly faster.
+
+## Purpose
+
+This serves as an extra validation step when the user has already tested their code. This should be used after all automated testing has passed (all CI testing), but before using HPC to run a large-scale model.
+
+## Usage
+
+After making the desired modifications to the ML script, (currently `train_density_LodeRunner.py`), run the harness in the typical way. To do this, load the python environment then run:
+
+```bash
+python START_study.py
+```
\ No newline at end of file
diff --git a/applications/harnesses/mini-run-test/START_study.py b/applications/harnesses/mini-run-test/START_study.py
new file mode 120000
index 00000000..e1155eab
--- /dev/null
+++ b/applications/harnesses/mini-run-test/START_study.py
@@ -0,0 +1 @@
+../START_study.py
\ No newline at end of file
diff --git a/applications/harnesses/mini-run-test/cp_files.txt b/applications/harnesses/mini-run-test/cp_files.txt
new file mode 100644
index 00000000..dcd3857e
--- /dev/null
+++ b/applications/harnesses/mini-run-test/cp_files.txt
@@ -0,0 +1 @@
+train_density_LodeRunner.py
diff --git a/applications/harnesses/mini-run-test/hyperparameters.csv b/applications/harnesses/mini-run-test/hyperparameters.csv
new file mode 100644
index 00000000..bd32c259
--- /dev/null
+++ b/applications/harnesses/mini-run-test/hyperparameters.csv
@@ -0,0 +1,2 @@
+studyIDX,YOKE_TORCH_ENV,channel_map_size,EMBED_DIM,B0,B1,B2,B3,NUM_WORKERS,PREFETCH_FACTOR,BATCH_SIZE,NTRN_BATCH,NVAL_BATCH,LEARN_RATE,LR_EPOCH_PER_STEP,LR_DECAY_RATE,train_script
+1,torch_ch_gpu_241112,8,128,1,1,3,1,8,2,4,2,1,1.00E-04,110,0.5,train_density_LodeRunner.py
diff --git a/applications/harnesses/mini-run-test/local-yoke-runner.sh b/applications/harnesses/mini-run-test/local-yoke-runner.sh
new file mode 100755
index 00000000..53676cc7
--- /dev/null
+++ b/applications/harnesses/mini-run-test/local-yoke-runner.sh
@@ -0,0 +1,2 @@
+export PYTHONPATH=../../../src:$PYTHONPATH
+python START_study.py
\ No newline at end of file
diff --git a/applications/harnesses/mini-run-test/train_density_LodeRunner.py b/applications/harnesses/mini-run-test/train_density_LodeRunner.py
new file mode 100644
index 00000000..3a43f6bc
--- /dev/null
+++ b/applications/harnesses/mini-run-test/train_density_LodeRunner.py
@@ -0,0 +1,566 @@
+"""Initial training setup for LodeRunner on LSC material densities.
+
+This version of training uses only the lsc240420 data with only per-material
+density along with the velocity field. A single timestep is input, a single
+timestep is predicted. The number of input variables is fixed throughout
+training.
+
+"""
+
+#############################################
+# Packages
+#############################################
+import os
+import time
+import argparse
+import torch
+import torch.nn as nn
+import random
+import logging
+import numpy as np
+
+from yoke.models.vit.swin.bomberman import LodeRunner
+from yoke.datasets.lsc_dataset import LSC_rho2rho_temporal_DataSet
+import yoke.torch_training_utils as tr
+from yoke.parallel_utils import LodeRunner_DataParallel
+import yoke.utils.logger as yl
+
+#############################################
+# Inputs
+#############################################
+descr_str = (
+    "Trains LodeRunner architecture on single-timstep input and output of the "
+    "lsc240420 per-material density fields."
+)
+parser = argparse.ArgumentParser(
+    prog="Initial LodeRunner Training", description=descr_str, fromfile_prefix_chars="@"
+)
+
+#############################################
+# Channel Subset Study Param
+#############################################
+parser.add_argument(
+    "--channel_map_size",
+    action="store",
+    type=int,
+    default=0,
+    help="Index into the list of tuple of input and output channel subsets",
+)
+
+#############################################
+# Data Parallelism
+#############################################
+parser.add_argument(
+    '--multigpu',
+    action='store_true',
+    help='Supports multiple GPUs on a single node.'
+)
+
+#############################################
+# Learning Problem
+#############################################
+parser.add_argument(
+    "--studyIDX",
+    action="store",
+    type=int,
+    default=1,
+    help="Study ID number to match hyperparameters",
+)
+
+#############################################
+# File Paths
+#############################################
+parser.add_argument(
+    "--FILELIST_DIR",
+    action="store",
+    type=str,
+    default=os.path.join(os.path.dirname(__file__), "../../filelists/"),
+    help="Directory where filelists are located.",
+)
+
+parser.add_argument(
+    "--LSC_NPZ_DIR",
+    action="store",
+    type=str,
+    default=os.path.join(os.path.dirname(__file__), "../../../data_examples/lsc240420/"),
+    help="Directory in which LSC *.npz files live.",
+)
+
+parser.add_argument(
+    "--train_filelist",
+    action="store",
+    type=str,
+    default="lsc240420_prefixes_train_80pct.txt",
+    help="Path to list of files to train on.",
+)
+
+parser.add_argument(
+    "--validation_filelist",
+    action="store",
+    type=str,
+    default="lsc240420_prefixes_validation_10pct.txt",
+    help="Path to list of files to validate on.",
+)
+
+parser.add_argument(
+    "--test_filelist",
+    action="store",
+    type=str,
+    default="lsc240420_prefixes_test_10pct.txt",
+    help="Path to list of files to test on.",
+)
+
+#############################################
+# Model Parameters
+#############################################
+parser.add_argument(
+    "--block_structure",
+    action="store",
+    type=int,
+    nargs="+",
+    default=[1, 1, 3, 1],
+    help="List of number of SW-MSA layers in each SWIN block.",
+)
+
+parser.add_argument(
+    "--embed_dim",
+    action="store",
+    type=int,
+    default=128,
+    help="Initial embedding dimension for SWIN-Unet.",
+)
+
+#############################################
+# Training Parameters
+#############################################
+parser.add_argument(
+    "--init_learnrate",
+    action="store",
+    type=float,
+    default=1e-3,
+    help="Initial learning rate",
+)
+
+parser.add_argument(
+    "--LRepoch_per_step",
+    action="store",
+    type=float,
+    default=10,
+    help="Number of epochs per LR reduction.",
+)
+
+parser.add_argument(
+    "--LRdecay", action="store", type=float, default=0.5, help="LR decay factor."
+)
+
+parser.add_argument(
+    "--batch_size", action="store", type=int, default=64, help="Batch size"
+)
+
+parser.add_argument(
+    "--num_workers",
+    action="store",
+    type=int,
+    default=4,
+    help=("Number of processes simultaneously loading batches of data. "
+          "NOTE: If set too big workers will swamp memory!!")
+)
+
+parser.add_argument(
+    "--prefetch_factor",
+    action="store",
+    type=int,
+    default=2,
+    help=("Number of batches each worker preloads ahead of time. "
+          "NOTE: If set too big preload will swamp memory!!")
+)
+
+#############################################
+# Epoch Parameters
+#############################################
+parser.add_argument(
+    "--total_epochs", action="store", type=int, default=10, help="Total training epochs"
+)
+
+parser.add_argument(
+    "--cycle_epochs",
+    action="store",
+    type=int,
+    default=5,
+    help=(
+        "Number of epochs between saving the model and re-queueing "
+        "training process; must be able to be completed in the "
+        "set wall time"
+    ),
+)
+
+parser.add_argument(
+    "--train_batches",
+    action="store",
+    type=int,
+    default=250,
+    help="Number of batches to train on in a given epoch",
+)
+
+parser.add_argument(
+    "--val_batches",
+    action="store",
+    type=int,
+    default=25,
+    help="Number of batches to validate on in a given epoch",
+)
+
+parser.add_argument(
+    "--TRAIN_PER_VAL",
+    action="store",
+    type=int,
+    default=10,
+    help="Number of training epochs between each validation epoch",
+)
+
+parser.add_argument(
+    "--trn_rcrd_filename",
+    action="store",
+    type=str,
+    default="./default_training.csv",
+    help="Filename for text file of training loss and metrics on each batch",
+)
+
+parser.add_argument(
+    "--val_rcrd_filename",
+    action="store",
+    type=str,
+    default="./default_validation.csv",
+    help="Filename for text file of validation loss and metrics on each batch",
+)
+
+parser.add_argument(
+    "--continuation",
+    action="store_true",
+    help="Indicates if training is being continued or restarted",
+)
+
+parser.add_argument(
+    "--checkpoint",
+    action="store",
+    type=str,
+    default="None",
+    help="Path to checkpoint to continue training from",
+)
+
+
+############################################
+# Select n channles randomly for an epoch
+############################################
+def rand_channel_map(
+        max_number_channels: int,
+        num_subchannels: int,
+        seed: int = None,
+        _seed_set: list[bool] = [False]
+) -> list:
+    """Choose list of subsampled channels."""
+    if num_subchannels > max_number_channels:
+        raise ValueError("Subsampled channels cannot be greater than maximum channels.")
+
+    if seed is not None and not _seed_set[0]:
+        random.seed(seed)
+        _seed_set[0] = True  # Mark the seed as set
+
+    return sorted(random.sample(range(0, max_number_channels), num_subchannels))
+
+
+if __name__ == "__main__":
+    #############################################
+    # Process Inputs
+    #############################################
+    yl.configure_logger("yoke_logger", level=logging.INFO)
+
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+    args = parser.parse_args()
+
+    # Study ID
+    studyIDX = args.studyIDX
+
+    # Data Paths
+    train_filelist = args.FILELIST_DIR + args.train_filelist
+    validation_filelist = args.FILELIST_DIR + args.validation_filelist
+    test_filelist = args.FILELIST_DIR + args.test_filelist
+
+    # Model Parameters
+    embed_dim = args.embed_dim
+    block_structure = tuple(args.block_structure)
+
+    # Training Parameters
+    initial_learningrate = args.init_learnrate
+    LRepoch_per_step = args.LRepoch_per_step
+    LRdecay = args.LRdecay
+    batch_size = args.batch_size
+
+    # Number of workers controls how batches of data are prefetched and,
+    # possibly, pre-loaded onto GPUs. If the number of workers is large they
+    # will swamp memory and jobs will fail.
+    num_workers = args.num_workers
+    prefetch_factor = args.prefetch_factor
+
+    # Epoch Parameters
+    total_epochs = args.total_epochs
+    cycle_epochs = args.cycle_epochs
+    train_batches = args.train_batches
+    val_batches = args.val_batches
+    train_per_val = args.TRAIN_PER_VAL
+    trn_rcrd_filename = args.trn_rcrd_filename
+    val_rcrd_filename = args.val_rcrd_filename
+    CONTINUATION = args.continuation
+    START = not CONTINUATION
+    checkpoint = args.checkpoint
+
+    #############################################
+    # Check Devices
+    #############################################
+    print("\n")
+    print("Slurm & Device Information")
+    print("=========================================")
+    print("Slurm Job ID:", os.environ["SLURM_JOB_ID"])
+    print("Pytorch Cuda Available:", torch.cuda.is_available())
+    print("GPU ID:", os.environ["SLURM_JOB_GPUS"])
+    print("Number of System CPUs:", os.cpu_count())
+    print("Number of CPUs per GPU:", os.environ["SLURM_JOB_CPUS_PER_NODE"])
+
+    print("\n")
+    print("Model Training Information")
+    print("=========================================")
+
+    #############################################
+    # Initialize Model
+    #############################################
+    hydro_fields =[
+        'density_case',
+        'density_cushion',
+        'density_maincharge',
+        'density_outside_air',
+        'density_striker',
+        'density_throw',
+        'Uvelocity',
+        'Wvelocity',
+    ]
+
+    model = LodeRunner(
+        default_vars=hydro_fields,
+        image_size=(1120, 800),
+        patch_size=(10, 10),
+        embed_dim=embed_dim,
+        emb_factor=2,
+        num_heads=8,
+        block_structure=block_structure,
+        window_sizes=[
+            (8, 8),
+            (8, 8),
+            (4, 4),
+            (2, 2),
+        ],
+        patch_merge_scales=[
+            (2, 2),
+            (2, 2),
+            (2, 2),
+        ],
+    )
+
+    print("Lode Runner parameters:", tr.count_torch_params(model, trainable=True))
+    # Wait to move model to GPU until after the checkpoint load. Then
+    # explicitly move model and optimizer state to GPU.
+
+    #############################################
+    # Initialize Optimizer
+    #############################################
+    optimizer = torch.optim.AdamW(
+        model.parameters(),
+        lr=initial_learningrate,
+        betas=(0.9, 0.999),
+        eps=1e-08,
+        weight_decay=0.01,
+    )
+
+    #############################################
+    # Initialize Loss
+    #############################################
+    # Use `reduction='none'` so loss on each sample in batch can be recorded.
+    loss_fn = nn.MSELoss(reduction="none")
+
+    print("Model initialized.")
+
+    #############################################
+    # Load Model for Continuation
+    #############################################
+    if CONTINUATION:
+        starting_epoch = tr.load_model_and_optimizer_hdf5(model, optimizer, checkpoint)
+        print("Model state loaded for continuation.")
+    else:
+        starting_epoch = 0
+
+    #############################################
+    # Move model and optimizer state to GPU
+    #############################################
+    if args.multigpu:
+        model = LodeRunner_DataParallel(model)
+
+    model.to(device)
+
+    for state in optimizer.state.values():
+        for k, v in state.items():
+            if isinstance(v, torch.Tensor):
+                state[k] = v.to(device)
+
+    #############################################
+    # Setup LR scheduler
+    #############################################
+    stepLRsched = torch.optim.lr_scheduler.StepLR(
+        optimizer,
+        step_size=LRepoch_per_step,
+        gamma=LRdecay,
+        last_epoch=starting_epoch - 1,
+    )
+
+    #############################################
+    # Training Loop
+    #############################################
+    # Train Model
+    print("Training Model . . .")
+    starting_epoch += 1
+    ending_epoch = min(starting_epoch + cycle_epochs, total_epochs + 1)
+
+    # For reproducibility of channel map per epoch
+    SEED = 42
+
+    max_channels = len(hydro_fields)
+    yl.logger.info(f"Max Channel  : {max_channels}")
+
+    channel_map_size = args.channel_map_size
+    yl.logger.info(f"channel_map_size = {channel_map_size}")
+
+    # Change hydrofields to array to enable slicing with channel map
+    hydro_fields = np.array(hydro_fields)
+    for epochIDX in range(starting_epoch, ending_epoch):
+        # Randomly select 'channel_map_size' number of channels from for the epoch
+        channel_map = rand_channel_map(max_channels, channel_map_size, SEED)
+
+        log_str = (
+            f"Epoch {epochIDX:04d}, "
+            f"Nchannels:{channel_map_size:03d}, "
+            f"Channel Map:{channel_map}"
+            )
+        yl.logger.info(log_str)
+
+        #############################################
+        # Initialize Data
+        # For varying channels subset per epoch, the
+        # data must be initialized for each epoch.
+        #############################################
+        train_dataset = LSC_rho2rho_temporal_DataSet(
+            args.LSC_NPZ_DIR,
+            file_prefix_list=train_filelist,
+            max_timeIDX_offset=2,  # This could be a variable.
+            max_file_checks=10,
+            half_image=False,
+            hydro_fields=hydro_fields[channel_map],
+        )
+        val_dataset = LSC_rho2rho_temporal_DataSet(
+            args.LSC_NPZ_DIR,
+            file_prefix_list=validation_filelist,
+            max_timeIDX_offset=2,  # This could be a variable.
+            max_file_checks=10,
+            half_image=False,
+            hydro_fields=hydro_fields[channel_map],
+        )
+        test_dataset = LSC_rho2rho_temporal_DataSet(
+            args.LSC_NPZ_DIR,
+            file_prefix_list=test_filelist,
+            max_timeIDX_offset=2,  # This could be a variable.
+            max_file_checks=10,
+            half_image=False,
+            hydro_fields=hydro_fields[channel_map],
+        )
+
+        print("Datasets initialized...")
+
+        # Setup Dataloaders
+        train_dataloader = tr.make_dataloader(
+            train_dataset,
+            batch_size,
+            train_batches,
+            num_workers=num_workers,
+            prefetch_factor=prefetch_factor
+        )
+        val_dataloader = tr.make_dataloader(
+            val_dataset,
+            batch_size,
+            val_batches,
+            num_workers=num_workers,
+            prefetch_factor=prefetch_factor
+        )
+        print("DataLoaders initialized...")
+
+        # Time each epoch and print to stdout
+        startTime = time.time()
+
+        # Train an Epoch
+        tr.train_simple_loderunner_epoch(
+            channel_map,
+            training_data=train_dataloader,
+            validation_data=val_dataloader,
+            model=model,
+            optimizer=optimizer,
+            loss_fn=loss_fn,
+            epochIDX=epochIDX,
+            train_per_val=train_per_val,
+            train_rcrd_filename=trn_rcrd_filename,
+            val_rcrd_filename=val_rcrd_filename,
+            device=device,
+            verbose=False
+        )
+
+        # Increment LR scheduler
+        stepLRsched.step()
+
+        endTime = time.time()
+        epoch_time = (endTime - startTime) / 60
+
+        # Print Summary Results
+        print("Completed epoch " + str(epochIDX) + "...", flush=True)
+        print("Epoch time (minutes):", epoch_time, flush=True)
+
+        # Clear GPU memory after each epoch
+        torch.cuda.empty_cache()
+
+    # Save Model Checkpoint
+    print("Saving model checkpoint at end of epoch " + str(epochIDX) + ". . .")
+
+    # Move the model back to CPU prior to saving to increase portability
+    model.to("cpu")
+    # Move optimizer state back to CPU
+    for state in optimizer.state.values():
+        for k, v in state.items():
+            if isinstance(v, torch.Tensor):
+                state[k] = v.to("cpu")
+
+    # Save model and optimizer state in hdf5
+    h5_name_str = "study{0:03d}_modelState_epoch{1:04d}.hdf5"
+    new_h5_path = os.path.join("./", h5_name_str.format(studyIDX, epochIDX))
+    tr.save_model_and_optimizer_hdf5(
+        model, optimizer, epochIDX, new_h5_path, compiled=False
+    )
+
+    #############################################
+    # Continue if Necessary
+    #############################################
+    FINISHED_TRAINING = epochIDX + 1 > total_epochs
+    if not FINISHED_TRAINING:
+        new_slurm_file = tr.continuation_setup(
+            new_h5_path, studyIDX, last_epoch=epochIDX
+        )
+        os.system(f"sbatch {new_slurm_file}")
+
+    ###########################################################################
+    # For array prediction, especially large array prediction, the network is
+    # not evaluated on the test set after training. This is performed using
+    # the *evaluation* module as a separate post-analysis step.
+    ###########################################################################
diff --git a/applications/harnesses/mini-run-test/training_START.input b/applications/harnesses/mini-run-test/training_START.input
new file mode 100644
index 00000000..2187de63
--- /dev/null
+++ b/applications/harnesses/mini-run-test/training_START.input
@@ -0,0 +1,41 @@
+--studyIDX
+<studyIDX>
+--FILELIST_DIR
+/usr/projects/artimis/mpmm/hickmank/chicoma/ch_yoke/applications/filelists/
+--LSC_NPZ_DIR
+/lustre/scratch4/exempt/artimis/lsc240420/
+--train_filelist
+lsc240420_prefixes_train_80pct.txt
+--validation_filelist
+lsc240420_prefixes_validation_10pct.txt
+--test_filelist
+lsc240420_prefixes_test_10pct.txt
+--init_learnrate
+<LEARN_RATE>
+--LRepoch_per_step
+<LR_EPOCH_PER_STEP>
+--LRdecay
+<LR_DECAY_RATE>
+--trn_rcrd_filename
+./training_study<studyIDX>_epoch<epochIDX>.csv
+--val_rcrd_filename
+./validation_study<studyIDX>_epoch<epochIDX>.csv
+--batch_size
+<BATCH_SIZE>
+--num_workers
+<NUM_WORKERS>
+--prefetch_factor
+<PREFETCH_FACTOR>
+--total_epochs
+4
+--cycle_epochs
+2
+--train_batches
+<NTRN_BATCH>
+--val_batches
+<NVAL_BATCH>
+--TRAIN_PER_VAL
+2
+--multigpu
+--channel_map_size
+<channel_map_size>
diff --git a/applications/harnesses/mini-run-test/training_START.slurm b/applications/harnesses/mini-run-test/training_START.slurm
new file mode 100644
index 00000000..84d8750d
--- /dev/null
+++ b/applications/harnesses/mini-run-test/training_START.slurm
@@ -0,0 +1,62 @@
+#!/bin/bash
+
+# This is a setup for GPU training on Chicoma. Find out how much
+# memory per node, number of CPUs/node.
+
+# NOTE: Number of CPUs per GPU must be an even number since there are
+# 2 threads per core. If an odd number is requested the next higher
+# even number gets used.
+
+# The following are one set of SBATCH options for the Chicoma GPU
+# partition. There are optional other constraints. Namely, `gpu80` and
+# `gpu40only`.
+
+#SBATCH --job-name=loderunner_chfe_study<studyIDX>
+#SBATCH --account=w25_artimis_g
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=64
+#SBATCH --partition=gpu_debug
+#SBATCH --reservation=gpu_debug
+#SBATCH --constraint=gpu40
+#SBATCH --time=00:20:00
+#SBATCH --output=study<studyIDX>_epoch0001.out
+#SBATCH --error=study<studyIDX>_epoch0001.err
+#SBATCH -vvv
+
+
+# Check available GPUs
+sinfo  -o "%P %.24G %N"
+srun /usr/bin/echo $CUDA_AVAILABLE_DEVICES
+
+# Load correct conda environment
+module load python/3.11-anaconda-2023.07
+source activate
+#conda activate <YOKE_TORCH_ENV>
+conda activate /usr/projects/artimis/conda/envs/torch_ch_gpu_241112/
+
+# Currently need to set an environment variable for MKL
+# threading. Believed to be related to Numpy.
+export MKL_SERVICE_FORCE_INTEL=TRUE
+
+# Get start time
+export date00=`date`
+
+# # Start GPU monitoring
+# nvidia-smi --loop 5 --query-gpu=timestamp,gpu_bus_id,memory.used,memory.free,memory.total --format=csv > nvidia_smi_epoch0001.out &
+# nvpid=$!
+
+# Start the Code
+python <train_script> @study<studyIDX>_START.input
+#pypid=$!
+
+# Kill GPU monitoring after python process finishes
+#wait $pypid
+#kill $nvpid
+
+# Get end time and print to stdout
+export date01=`date`
+
+echo "===================TIME STARTED==================="
+echo $date00
+echo "===================TIME FINISHED==================="
+echo $date01
diff --git a/applications/harnesses/mini-run-test/training_input.tmpl b/applications/harnesses/mini-run-test/training_input.tmpl
new file mode 100644
index 00000000..aed18bf3
--- /dev/null
+++ b/applications/harnesses/mini-run-test/training_input.tmpl
@@ -0,0 +1,44 @@
+--studyIDX
+<studyIDX>
+--FILELIST_DIR
+/usr/projects/artimis/mpmm/hickmank/chicoma/ch_yoke/applications/filelists/
+--LSC_NPZ_DIR
+/lustre/scratch4/exempt/artimis/lsc240420/
+--train_filelist
+lsc240420_prefixes_train_80pct.txt
+--validation_filelist
+lsc240420_prefixes_validation_10pct.txt
+--test_filelist
+lsc240420_prefixes_test_10pct.txt
+--init_learnrate
+<LEARN_RATE>
+--LRepoch_per_step
+<LR_EPOCH_PER_STEP>
+--LRdecay
+<LR_DECAY_RATE>
+--trn_rcrd_filename
+./training_study<studyIDX>_epoch<epochIDX>.csv
+--val_rcrd_filename
+./validation_study<studyIDX>_epoch<epochIDX>.csv
+--batch_size
+<BATCH_SIZE>
+--num_workers
+<NUM_WORKERS>
+--prefetch_factor
+<PREFETCH_FACTOR>
+--total_epochs
+4
+--cycle_epochs
+2
+--train_batches
+<NTRN_BATCH>
+--val_batches
+<NVAL_BATCH>
+--TRAIN_PER_VAL
+2
+--multigpu
+--continuation
+--checkpoint
+<CHECKPOINT>
+--channel_map_size
+<channel_map_size>
diff --git a/applications/harnesses/mini-run-test/training_slurm.tmpl b/applications/harnesses/mini-run-test/training_slurm.tmpl
new file mode 100644
index 00000000..f0921f2d
--- /dev/null
+++ b/applications/harnesses/mini-run-test/training_slurm.tmpl
@@ -0,0 +1,62 @@
+#!/bin/bash
+
+# This is a setup for GPU training on Chicoma. Find out how much
+# memory per node, number of CPUs/node.
+
+# NOTE: Number of CPUs per GPU must be an even number since there are
+# 2 threads per core. If an odd number is requested the next higher
+# even number gets used.
+
+# The following are one set of SBATCH options for the Chicoma GPU
+# partition. There are optional other constraints. Namely, `gpu80` and
+# `gpu40only`.
+
+#SBATCH --job-name=loderunner_chfe_study<studyIDX>
+#SBATCH --account=w25_artimis_g
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=64
+#SBATCH --partition=gpu_debug
+#SBATCH --reservation=gpu_debug
+#SBATCH --constraint=gpu40
+#SBATCH --time=00:20:00
+#SBATCH --output=study<studyIDX>_epoch<epochIDX>.out
+#SBATCH --error=study<studyIDX>_epoch<epochIDX>.err
+#SBATCH -vvv
+
+
+# Check available GPUs
+sinfo  -o "%P %.24G %N"
+srun /usr/bin/echo $CUDA_AVAILABLE_DEVICES
+
+# Load correct conda environment
+module load python/3.11-anaconda-2023.07
+source activate
+conda activate <YOKE_TORCH_ENV>
+
+# Currently need to set an environment variable for MKL
+# threading. Believed to be related to Numpy.
+export MKL_SERVICE_FORCE_INTEL=TRUE
+
+# Get start time
+export date00=`date`
+
+# # Start GPU monitoring
+# nvidia-smi --loop 5 --query-gpu=timestamp,gpu_bus_id,memory.used,memory.free,memory.total --format=csv > nvidia_smi_epoch<epochIDX>.out &
+# nvpid=$!
+
+# Assumes running from inside harnesses directory
+# Start the Code
+python <train_script> @<INPUTFILE>
+#pypid=$!
+
+# Kill GPU monitoring after python process finishes
+#wait $pypid
+#kill $nvpid
+
+# Get end time and print to stdout
+export date01=`date`
+
+echo "===================TIME STARTED==================="
+echo $date00
+echo "===================TIME FINISHED==================="
+echo $date01


Reviews:
user: login: hickmank
body: @galegozi please see my comment in the conversation. I like the idea of this work but I want to get it to be able to run in CI from a 'flit' install.
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-03-11T15:47:03Z
user: login: hickmank
body: A few minor changes. I still think a CI job that just calls `python START_study.py` in this directory with random data would be more useful but I'll leave it up to you.
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-03-19T15:40:10Z
user: login: galegozi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T15:41:20Z
user: login: galegozi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T15:42:29Z
user: login: galegozi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T20:30:44Z
user: login: galegozi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T20:31:22Z
user: login: galegozi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T20:31:35Z
user: login: galegozi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T20:36:59Z
user: login: galegozi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T20:40:14Z
user: login: hickmank
body: The slurm file and template code be cleaned a little more. Other than that, looks good!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-26T16:10:13Z
user: login: galegozi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T17:48:00Z

Review comments:
diff_hunk: @@ -0,0 +1,2 @@
+export PYTHONPATH=../../../src:$PYTHONPATH
path: applications/harnesses/mini-run-test/local-yoke-runner.sh
user: login: hickmank
body: I don't want to include shell scripts with python path manipulation in the repo. If `yoke` is installed properly then a user can execute `python START_study.py` from the command line.
created_at: 2025-03-19T15:36:20Z
author_association: COLLABORATOR
path: applications/harnesses/mini-run-test/hyperparameters.csv
user: login: hickmank
body: Your description of this PR is that a single batch will be run as a test. However, this `hyperparameters.csv` file has many more studies included. Also, you have one study that is labelled 926. Let's call it 1 and only have one study if this is just for testing.
created_at: 2025-03-19T15:38:36Z
author_association: COLLABORATOR
path: applications/harnesses/mini-run-test/hyperparameters.csv
user: login: galegozi
body: Sounds good, I'll make that change, thanks!
created_at: 2025-03-19T15:41:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,2 @@
+export PYTHONPATH=../../../src:$PYTHONPATH
path: applications/harnesses/mini-run-test/local-yoke-runner.sh
user: login: galegozi
body: The point of this is to have the ability to have Yoke running from a different version (not the flit installed version). However, if you think that's a bad idea, I'll remove it, thanks!
created_at: 2025-03-19T15:42:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,62 @@
+#!/bin/bash
+
+# This is a setup for GPU training on Chicoma. Find out how much
+# memory per node, number of CPUs/node.
+
+# NOTE: Number of CPUs per GPU must be an even number since there are
+# 2 threads per core. If an odd number is requested the next higher
+# even number gets used.
+
+# The following are one set of SBATCH options for the Chicoma GPU
+# partition. There are optional other constraints. Namely, `gpu80` and
+# `gpu40only`.
+
+#SBATCH --job-name=loderunner_chfe_study<studyIDX>
+#SBATCH --account=w25_artimis_g
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=64
+#SBATCH --partition=gpu_debug
+#SBATCH --reservation=gpu_debug
+#SBATCH --constraint=gpu40
+#SBATCH --time=02:00:00
path: applications/harnesses/mini-run-test/training_START.slurm
user: login: galegozi
body: shorten this
created_at: 2025-03-25T20:30:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,44 @@
+--studyIDX
+<studyIDX>
+--FILELIST_DIR
+/usr/projects/artimis/mpmm/hickmank/chicoma/ch_yoke/applications/filelists/
+--LSC_NPZ_DIR
+/lustre/scratch4/exempt/artimis/lsc240420/
+--train_filelist
+lsc240420_prefixes_train_80pct.txt
+--validation_filelist
+lsc240420_prefixes_validation_10pct.txt
+--test_filelist
+lsc240420_prefixes_test_10pct.txt
+--init_learnrate
+<LEARN_RATE>
+--LRepoch_per_step
+<LR_EPOCH_PER_STEP>
+--LRdecay
+<LR_DECAY_RATE>
+--trn_rcrd_filename
+./training_study<studyIDX>_epoch<epochIDX>.csv
+--val_rcrd_filename
+./validation_study<studyIDX>_epoch<epochIDX>.csv
+--batch_size
+<BATCH_SIZE>
+--num_workers
+<NUM_WORKERS>
+--prefetch_factor
+<PREFETCH_FACTOR>
+--total_epochs
+1
+--cycle_epochs
+10
path: applications/harnesses/mini-run-test/training_input.tmpl
user: login: galegozi
body: run cycle epochs = 2
created_at: 2025-03-25T20:31:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,44 @@
+--studyIDX
+<studyIDX>
+--FILELIST_DIR
+/usr/projects/artimis/mpmm/hickmank/chicoma/ch_yoke/applications/filelists/
+--LSC_NPZ_DIR
+/lustre/scratch4/exempt/artimis/lsc240420/
+--train_filelist
+lsc240420_prefixes_train_80pct.txt
+--validation_filelist
+lsc240420_prefixes_validation_10pct.txt
+--test_filelist
+lsc240420_prefixes_test_10pct.txt
+--init_learnrate
+<LEARN_RATE>
+--LRepoch_per_step
+<LR_EPOCH_PER_STEP>
+--LRdecay
+<LR_DECAY_RATE>
+--trn_rcrd_filename
+./training_study<studyIDX>_epoch<epochIDX>.csv
+--val_rcrd_filename
+./validation_study<studyIDX>_epoch<epochIDX>.csv
+--batch_size
+<BATCH_SIZE>
+--num_workers
+<NUM_WORKERS>
+--prefetch_factor
+<PREFETCH_FACTOR>
+--total_epochs
+1
path: applications/harnesses/mini-run-test/training_input.tmpl
user: login: galegozi
body: total epochs = 4
created_at: 2025-03-25T20:31:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,44 @@
+--studyIDX
+<studyIDX>
+--FILELIST_DIR
+/usr/projects/artimis/mpmm/hickmank/chicoma/ch_yoke/applications/filelists/
+--LSC_NPZ_DIR
+/lustre/scratch4/exempt/artimis/lsc240420/
+--train_filelist
+lsc240420_prefixes_train_80pct.txt
+--validation_filelist
+lsc240420_prefixes_validation_10pct.txt
+--test_filelist
+lsc240420_prefixes_test_10pct.txt
+--init_learnrate
+<LEARN_RATE>
+--LRepoch_per_step
+<LR_EPOCH_PER_STEP>
+--LRdecay
+<LR_DECAY_RATE>
+--trn_rcrd_filename
+./training_study<studyIDX>_epoch<epochIDX>.csv
+--val_rcrd_filename
+./validation_study<studyIDX>_epoch<epochIDX>.csv
+--batch_size
+<BATCH_SIZE>
+--num_workers
+<NUM_WORKERS>
+--prefetch_factor
+<PREFETCH_FACTOR>
+--total_epochs
+1
+--cycle_epochs
+10
+--train_batches
+<NTRN_BATCH>
+--val_batches
+<NVAL_BATCH>
+--TRAIN_PER_VAL
+5
path: applications/harnesses/mini-run-test/training_input.tmpl
user: login: galegozi
body: set it to 2
created_at: 2025-03-25T20:36:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,30 @@
+studyIDX,YOKE_TORCH_ENV,channel_map_size,EMBED_DIM,B0,B1,B2,B3,NUM_WORKERS,PREFETCH_FACTOR,BATCH_SIZE,NTRN_BATCH,NVAL_BATCH,LEARN_RATE,LR_EPOCH_PER_STEP,LR_DECAY_RATE,train_script
+#1,torch_ch_gpu_241112,8,128,1,1,3,1,8,2,128,1000,500,1.00E-03,110,0.5,train_density_LodeRunner.py
+#2,torch_ch_gpu_241112,8,128,1,1,3,1,8,2,128,1000,500,1.00E-04,110,0.5,train_density_LodeRunner.py
+#3,torch_ch_gpu_241112,8,128,1,1,3,1,8,2,128,1000,500,1.00E-05,110,0.5,train_density_LodeRunner.py
+#4,torch_ch_gpu_241112,8,128,1,1,3,1,8,2,64,1000,500,1.00E-04,110,0.5,train_density_LodeRunner.py
+#5,torch_ch_gpu_241112,8,128,1,1,3,1,8,2,64,1000,500,1.00E-04,110,0.5,train_density_LodeRunner.py
+#5,torch_ch_gpu_241112,8,96,1,1,3,1,8,2,128,1000,500,1.00E-06,110,0.5,train_density_LodeRunner.py
+#6,torch_ch_gpu_241112,8,96,1,1,9,1,8,2,128,1000,500,1.00E-06,110,0.5,train_density_LodeRunner.py
+#7,torch_ch_gpu_241112,8,128,1,1,9,1,8,2,32,1000,500,1.00E-06,110,0.5,train_density_LodeRunner.py
+#8,torch_ch_gpu_241112,8,192,1,1,9,1,8,2,16,1000,500,1.00E-06,110,0.5,train_density_LodeRunner.py
+#9,torch_ch_gpu_241112,8,352,1,1,9,1,8,2,16,1000,500,1.00E-06,110,0.5,train_density_LodeRunner.py
+#10,torch_ch_gpu_241112,8,512,1,1,11,2,8,2,8,1000,500,1.00E-06,110,0.5,train_density_LodeRunner.py
+#11,torch_ch_gpu_241112,8,512,1,1,11,2,8,2,16,1000,500,1.00E-06,110,0.5,train_density_LodeRunner.py
+#12,torch_ch_gpu_241112,8,512,1,1,11,2,8,2,32,1000,500,1.00E-06,110,0.5,train_density_LodeRunner.py
+#13,torch_ch_gpu_241112,8,512,1,1,11,2,8,2,64,1000,500,1.00E-06,110,0.5,train_density_LodeRunner.py
+#14,torch_ch_gpu_241112,8,512,1,1,11,2,8,2,128,1000,500,1.00E-06,110,0.5,train_density_LodeRunner.py
+#604,torch_ch_gpu_241112,4,128,1,1,3,1,8,2,64,1000,500,1.00E-04,110,0.5,train_density_LodeRunner.py
+#605,torch_ch_gpu_241112,5,128,1,1,3,1,8,2,64,1000,500,1.00E-04,110,0.5,train_density_LodeRunner.py
+#610,torch_ch_gpu_241112,6,128,1,1,3,1,8,2,64,1000,500,1.00E-04,110,0.5,train_density_LodeRunner.py
+#611,torch_ch_gpu_241112,4,128,1,1,3,1,8,2,64,1000,500,1.00E-04,110,0.5,train_density_LodeRunner.py
+#620,torch_ch_gpu_241112,8,128,1,1,3,1,8,2,64,1000,500,1.00E-04,110,0.5,train_density_LodeRunner.py
+#621,torch_ch_gpu_241112,1,128,1,1,3,1,8,2,64,1000,500,1.00E-04,110,0.5,train_density_LodeRunner.py
+#922,torch_ch_gpu_241112,2,128,1,1,3,1,8,2,64,1000,500,1.00E-04,110,0.5,train_density_LodeRunner.py
+#923,torch_ch_gpu_241112,3,128,1,1,3,1,8,2,64,1000,500,1.00E-04,110,0.5,train_density_LodeRunner.py
+#924,torch_ch_gpu_241112,4,128,1,1,3,1,8,2,64,1000,500,1.00E-04,110,0.5,train_density_LodeRunner.py
+#925,torch_ch_gpu_241112,5,128,1,1,3,1,8,2,64,1000,500,1.00E-04,110,0.5,train_density_LodeRunner.py
+926,torch_ch_gpu_241112,6,128,1,1,3,1,8,2,64,1,1,1.00E-04,110,0.5,train_density_LodeRunner.py
path: applications/harnesses/mini-run-test/hyperparameters.csv
user: login: galegozi
body: study ID 1, batch size = 4
created_at: 2025-03-25T20:40:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,62 @@
+#!/bin/bash
+
+# This is a setup for GPU training on Chicoma. Find out how much
+# memory per node, number of CPUs/node.
+
+# NOTE: Number of CPUs per GPU must be an even number since there are
+# 2 threads per core. If an odd number is requested the next higher
+# even number gets used.
+
+# The following are one set of SBATCH options for the Chicoma GPU
+# partition. There are optional other constraints. Namely, `gpu80` and
+# `gpu40only`.
+
+#SBATCH --job-name=loderunner_chfe_study<studyIDX>
+#SBATCH --account=w25_artimis_g
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=64
+#SBATCH --partition=gpu_debug
+#SBATCH --reservation=gpu_debug
+#SBATCH --constraint=gpu40
+#SBATCH --time=00:20:00
+#SBATCH --output=study<studyIDX>_epoch0001.out
+#SBATCH --error=study<studyIDX>_epoch0001.err
+#SBATCH -vvv
+
+
+# Check available GPUs
+sinfo  -o "%P %.24G %N"
+srun /usr/bin/echo $CUDA_AVAILABLE_DEVICES
+
+# Load correct conda environment
+module load python/3.11-anaconda-2023.07
+source activate
+#conda activate <YOKE_TORCH_ENV>
+conda activate /usr/projects/artimis/conda/envs/torch_ch_gpu_241112/
+
+# Currently need to set an environment variable for MKL
+# threading. Believed to be related to Numpy.
+export MKL_SERVICE_FORCE_INTEL=TRUE
+
+# Get start time
+export date00=`date`
+
+# # Start GPU monitoring
path: applications/harnesses/mini-run-test/training_START.slurm
user: login: hickmank
body: It is not important but I'd get rid of this old commented code. It is not used anymore and I can't foresee a time when we'd want it.
created_at: 2025-03-26T16:09:14Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,62 @@
+#!/bin/bash
+
+# This is a setup for GPU training on Chicoma. Find out how much
+# memory per node, number of CPUs/node.
+
+# NOTE: Number of CPUs per GPU must be an even number since there are
+# 2 threads per core. If an odd number is requested the next higher
+# even number gets used.
+
+# The following are one set of SBATCH options for the Chicoma GPU
+# partition. There are optional other constraints. Namely, `gpu80` and
+# `gpu40only`.
+
+#SBATCH --job-name=loderunner_chfe_study<studyIDX>
+#SBATCH --account=w25_artimis_g
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=64
+#SBATCH --partition=gpu_debug
+#SBATCH --reservation=gpu_debug
+#SBATCH --constraint=gpu40
+#SBATCH --time=00:20:00
+#SBATCH --output=study<studyIDX>_epoch0001.out
+#SBATCH --error=study<studyIDX>_epoch0001.err
+#SBATCH -vvv
+
+
+# Check available GPUs
+sinfo  -o "%P %.24G %N"
+srun /usr/bin/echo $CUDA_AVAILABLE_DEVICES
+
+# Load correct conda environment
+module load python/3.11-anaconda-2023.07
+source activate
+#conda activate <YOKE_TORCH_ENV>
+conda activate /usr/projects/artimis/conda/envs/torch_ch_gpu_241112/
+
+# Currently need to set an environment variable for MKL
+# threading. Believed to be related to Numpy.
+export MKL_SERVICE_FORCE_INTEL=TRUE
+
+# Get start time
+export date00=`date`
+
+# # Start GPU monitoring
path: applications/harnesses/mini-run-test/training_START.slurm
user: login: galegozi
body: I will resolve this comment for now, given that the plan is to "de-slurm" the repo :)
created_at: 2025-03-26T17:47:59Z
author_association: CONTRIBUTOR

PR comments:
user: login: hickmank
created_at: 2025-03-11T15:27:55Z
body: > This test harness is designed to be a lightweight test that functions like an integration test. It runs one batch/epoch just to make sure that the code runs. NOTE: this is not designed to be used for production, this is something we would use (internally) just to test code. This would be useful when making a new piece of code (such as a loss function).

@galegozi I get your idea here but I think, if we're going to have a "test-harness" it should be able to be run through the CI (even if it only uses CPUs and randomly generated data) and use the `flit` install there (no modification of PYTHONPATH). That would mean it wouldn't have a SLURM submission script, just a shell script. 

If you can modify this PR into that I'll approve. It'd be great to have!!
author_association: COLLABORATOR
user: login: galegozi
created_at: 2025-03-12T15:33:13Z
body: Hi @hickmank ,
I understand what you want; however, this is not designed for that.
The way I think about it, there are four levels of testing:

1. Full model test: an end-to-end test where a model is trained. These runs typically take 100+ hours. These should never, NEVER be in a CICD, unless it's for a very, very good reason (such as training a baseline once per month).
2. Quick manual test: This is a full model test except it is designed to run for a shorter period of time. This is designed to resemble the full training process as closely as possible while requiring as little time as possible. _**This (and only this) is what this PR was designed for.**_
3. CICD integration test: A lighter version of the model that can be run on CPU in 1-2 minutes or less. This kind of test can be ran on CICD periodically (such as once or twice per day). Depending on the complexity of the integration tests, it may be possible and reasonable to run it every hour; however, running it once per day is likely sufficient.
    1. Depending on the complexity of these tests, it may be reasonable to configure them to run after the PR gets all approvals and the user presses the "merge" button. Please note: we should be very, very careful what we run here, so that we are not spending 1-2 hours trying to get a PR merged; therefore, only a set of a few critical integration tests should be run after the PR merge.
4. Unit tests: The quickest (and shortest) tests that are designed to test a small portion of code (such as a method). These tests should be extremely quick, and should run in about 5-10 minutes total.

I have seen what happens when this testing fails, and it's really, really bad... Given that this project is built to be a software project, it is vital to do this right, and I will not stand to have us fail at this!
Please feel free to respond to this comment (or contact me any other way) if you have questions about this.
author_association: CONTRIBUTOR
user: login: galegozi
created_at: 2025-03-12T22:30:21Z
body: @hickmank : Do you understand my previous comment? I don't mind writing for CLI integration tests, but I recommend refactoring the harnesses to config files first (because then we can test everything, and just have a config for integration tests) :)
My point is that the harness structure makes testing a lot more complicated... What I am aiming for is eventually to have one config file that gets loaded in with all parameters... eventually I am aiming to have a config file with defaults and where you can use regex (regular expressions), or at a bare minimum an asterisk (*) to describe the experiments.

Consider this example for the channel subsampling experiments:
```json
{
    "defaults" : {
        "param": "value"
    },
    "**1": {
        "numChannels": 1
    },
    "*1*": {
        "numEpochs": 100000,
        "numBatches": 1
    }
}
```
If that doesn't make sense, the defaults are where defaults for all experiments are stored, then the second example would match 011, 021, 031, etc, the third example would match 011, 012, 013, etc. Basically you put most of your config in defaults, then the rest of the config in the experiment specific. With this infrastructure, we could have everything (loss functions, learning rate schedulers, etc) configurable from a file. It would not support hot swap, but it could support multi-component scheduled swapping (such as swapping loss functions on a schedule).

I hope that makes sense because once this behavior is achieved, then the harnesses just become a config file, and when you run start study, it will generate the slurm script and run everything automatically. I do not expect this to be ready by tomorrow, but once all this is ready, then we can just have a config.json with all relevant parameters that generates the scripts. Eventually I'd like to have a better "run on debug" option that will try to start a study on debug (if possible), otherwise add it to normal queue. That option would not bypass debug rules, but it would fully utilize the debug queue. With the "harness as JSON", that would just be one option (set useDebug=true).
author_association: CONTRIBUTOR
user: login: galegozi
created_at: 2025-03-14T17:39:45Z
body: @hickmank Do you insist on the CI test bed for this PR to make it in? Or can we merge this in separately (this change is a completely separate test bed)?

If you insist on CI, then I will close this PR (because doing CI integration testing will take time)
author_association: CONTRIBUTOR
user: login: hickmank
created_at: 2025-03-19T15:42:16Z
body: > @hickmank : Do you understand my previous comment? I don't mind writing for CLI integration tests, but I recommend refactoring the harnesses to config files first (because then we can test everything, and just have a config for integration tests) :) My point is that the harness structure makes testing a lot more complicated... What I am aiming for is eventually to have one config file that gets loaded in with all parameters... eventually I am aiming to have a config file with defaults and where you can use regex (regular expressions), or at a bare minimum an asterisk (*) to describe the experiments.
> 
> Consider this example for the channel subsampling experiments:
> 
> ```json
> {
>     "defaults" : {
>         "param": "value"
>     },
>     "**1": {
>         "numChannels": 1
>     },
>     "*1*": {
>         "numEpochs": 100000,
>         "numBatches": 1
>     }
> }
> ```
> 
> If that doesn't make sense, the defaults are where defaults for all experiments are stored, then the second example would match 011, 021, 031, etc, the third example would match 011, 012, 013, etc. Basically you put most of your config in defaults, then the rest of the config in the experiment specific. With this infrastructure, we could have everything (loss functions, learning rate schedulers, etc) configurable from a file. It would not support hot swap, but it could support multi-component scheduled swapping (such as swapping loss functions on a schedule).
> 
> I hope that makes sense because once this behavior is achieved, then the harnesses just become a config file, and when you run start study, it will generate the slurm script and run everything automatically. I do not expect this to be ready by tomorrow, but once all this is ready, then we can just have a config.json with all relevant parameters that generates the scripts. Eventually I'd like to have a better "run on debug" option that will try to start a study on debug (if possible), otherwise add it to normal queue. That option would not bypass debug rules, but it would fully utilize the debug queue. With the "harness as JSON", that would just be one option (set useDebug=true).

This seems like a much bigger task than this PR. Sounds good but maybe it should be outlined in an issue first.
author_association: COLLABORATOR
user: login: hickmank
created_at: 2025-03-19T15:44:28Z
body: > @hickmank Do you insist on the CI test bed for this PR to make it in? Or can we merge this in separately (this change is a completely separate test bed)?
> 
> If you insist on CI, then I will close this PR (because doing CI integration testing will take time)

I made a few comments in the review with minor changes requested. Notably, there does not seem to be a need for the extra shell script. 

It seems like it should be simple to have a CI job that just calls `python START_study.py` in this directory after a `flit install` and passes if one cycle of training runs. What am I missing?
author_association: COLLABORATOR
user: login: galegozi
created_at: 2025-03-19T15:44:35Z
body: > A few minor changes. I still think a CI job that just calls `python START_study.py` in this directory with random data would be more useful but I'll leave it up to you.

That's a different kind of change. This is just a quick test using the real data to ensure that everything runs correctly.

A CPU test for CI would be much smaller!

I'll work on that and get a separate PR for a CPU CICD test, thanks!
author_association: CONTRIBUTOR
user: login: galegozi
created_at: 2025-03-19T15:46:43Z
body: > > @hickmank : Do you understand my previous comment? I don't mind writing for CLI integration tests, but I recommend refactoring the harnesses to config files first (because then we can test everything, and just have a config for integration tests) :) My point is that the harness structure makes testing a lot more complicated... What I am aiming for is eventually to have one config file that gets loaded in with all parameters... eventually I am aiming to have a config file with defaults and where you can use regex (regular expressions), or at a bare minimum an asterisk (*) to describe the experiments.
> > Consider this example for the channel subsampling experiments:
> > ```json
> > {
> >     "defaults" : {
> >         "param": "value"
> >     },
> >     "**1": {
> >         "numChannels": 1
> >     },
> >     "*1*": {
> >         "numEpochs": 100000,
> >         "numBatches": 1
> >     }
> > }
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > If that doesn't make sense, the defaults are where defaults for all experiments are stored, then the second example would match 011, 021, 031, etc, the third example would match 011, 012, 013, etc. Basically you put most of your config in defaults, then the rest of the config in the experiment specific. With this infrastructure, we could have everything (loss functions, learning rate schedulers, etc) configurable from a file. It would not support hot swap, but it could support multi-component scheduled swapping (such as swapping loss functions on a schedule).
> > I hope that makes sense because once this behavior is achieved, then the harnesses just become a config file, and when you run start study, it will generate the slurm script and run everything automatically. I do not expect this to be ready by tomorrow, but once all this is ready, then we can just have a config.json with all relevant parameters that generates the scripts. Eventually I'd like to have a better "run on debug" option that will try to start a study on debug (if possible), otherwise add it to normal queue. That option would not bypass debug rules, but it would fully utilize the debug queue. With the "harness as JSON", that would just be one option (set useDebug=true).
> 
> This seems like a much bigger task than this PR. Sounds good but maybe it should be outlined in an issue first.

Of course this is not an issue for one PR! but once that's done (as in "harness as JSON"), then testing would be much easier and more comprehensive...
Imagine there being unit tests for the harnesses!
author_association: CONTRIBUTOR
user: login: galegozi
created_at: 2025-03-19T15:52:30Z
body: > > @hickmank Do you insist on the CI test bed for this PR to make it in? Or can we merge this in separately (this change is a completely separate test bed)?
> > If you insist on CI, then I will close this PR (because doing CI integration testing will take time)
> 
> I made a few comments in the review with minor changes requested. Notably, there does not seem to be a need for the extra shell script.
> 
> It seems like it should be simple to have a CI job that just calls `python START_study.py` in this directory after a `flit install` and passes if one cycle of training runs. What am I missing?

The script handles multiple versions of Yoke. If this behavior does not need to be supported, then there is no need for the script.
author_association: CONTRIBUTOR
user: login: galegozi
created_at: 2025-03-25T20:40:43Z
body: ADD A README
author_association: CONTRIBUTOR
user: login: galegozi
created_at: 2025-03-26T04:21:30Z
body: @hickmank : I made the requested changes.
Could you please re-review this PR? thanks!
author_association: CONTRIBUTOR