Package: weaver-core
WeaverWeaveraims at providing a streamlined yet flexible machine learning R&D framework for high energy physics (HEP) applications. Seeweaver-examplesfor a number of use cases.Weaver puts particular emphases on:handling common HEP dataset formats (ROOT, HDF5,awkward array) efficiently, up to terabyte-scaleproviding a simple way to perform input processingon-the-fly(e.g., sample selections, new variable definition, inputs transformation/standardization, sample reweighting, etc.)bridging the gap between development and production: neural networks are trained withPyTorchand exported to theONNXformat for fast inference (e.g., usingONNXRuntime)Compared to its predecessorNNTools,Weaversimplifies the data processing pipeline by running all the pre-processing on-the-fly, without the necessity of creating an intermediate transformed dataset (though it still supports that). The neural network training now uses the more widely adoptedPyTorchinstead ofApache MXNet.WeaverSet up your environmentInstall Miniconda (if you don't already have it)Set up a conda environment and install the packagesPrepare your configuration filesData configuration fileModel configuration fileStart running!TrainingPrediction/InferenceModel exportationMore about data loading and processingTraining modePrediction/Inference modePerformance considerations on data loadingSet up your environmentTheWeaverpackage requires Python 3.7+ and a number of packages likenumpy,scikit-learn,PyTorch, etc.
To run the neural network training, an Nvidia GPU withCUDAsupport is needed.To manage the Python environment, we recommend usingMiniconda.
Below are the instructions to set up Miniconda and install the required packages. This needs to be done only once.Install Miniconda (if you don't already have it)wgethttps://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bashMiniconda3-latest-Linux-x86_64.sh# Follow the instructions to finish the installation# Make sure to choose `yes` for the following one to let the installer initialize Miniconda3# > Do you wish the installer to initialize Miniconda3# > by running conda init? [yes|no]# [Optional] Disable auto activation of the base environmentcondaconfig--setauto_activate_basefalseVerify the installation is successful by runningconda infoand checking if the paths are pointing to your Miniconda installation.If you cannot run thecondacommand, check if the conda path has been correctly set up in your.bashrc/.zshrcfile. You may need to log out and log in again for the changes to take effect.Set up a conda environment and install the packages# create a new conda environmentcondacreate-nweaverpython=3.10# activate the environmentcondaactivateweaver# install pytorch, follow instructions for your OS/CUDA version at:# https://pytorch.org/get-started# pip install torch# install weaver, this will install also all the dependencies except for PyTorchpipinstallweaver-coreAlternatively, if you want to installweaver-corein editable/development mode:git clone git@github.com:hqucms/weaver-core.git
cd weaver-core
pip install -e .Prepare your configuration filesTo train a neural network usingWeaver, you need to prepare:A YAMLdata configuration filedescribing how to process the input data.A pythonmodel configuration fileproviding the neural network module and the loss function.Data configuration fileThe data configuration file is aYAMLformat file describing how to process the input data. It needs the following sections:selection(optional): event selection for trainingtest_time_selection(optional): event selection for testing; if not specified, use the same asselectionnew_variables(optional): new variable definitioninputs(required): input groups, variables for each group, variable transformation (mean/scale/min/max for standardization, length/pad values for padding/clipping, etc.)labels(required): label definitionobservers(optional): additional variables that are not used in the training, but will be added to the output files when running predictionweights(optional): instance weight information for sampling the instances during the trainingAn example of the data configuration file isJetClass_full.yaml.
For more details, checkutils/data/config.pyandutils/dataset.py.Model configuration fileThe model configuration file specifies the neural network model and the loss function. It needs to implement theget_modelfunction (required) and theget_lossfunction (optional, default istorch.nn.CrossEntropyLoss()) in the following signatures:defget_model(data_config,**kwargs):model=...# instance of PyTorch Modulemodel_info={'input_names':...,'input_shapes':...,'output_names':...,'dynamic_axes':...,}returnmodel,model_info# `get_loss` is optional# if not provided, fallback to use `torch.nn.CrossEntropyLoss()`defget_loss(data_config,**kwargs):loss_func=...returnloss_funcAn example of the model configuration file isexample_ParticleTransformer.py.Start running!Theweavercommand is the top-level entry to run for training a neural net, getting prediction from trained models, and exporting trained models to ONNX for production. The corresponding script file isweaver/train.py.
To check all the command-line options forweaver, runweaver -h. Examples for training, inference and model exportation are shown below:Trainingweaver--data-train'/path/to/train_files/*/*/*/*/output_*.root'\--data-test'/path/to/train_files/*/*/*/*/output_*.root'\--data-configdata/ak15_points_pf_sv.yaml\--network-confignetworks/particle_net_pf_sv.py\--model-prefix/path/to/models/prefix\--gpus0,1,2,3--batch-size512--start-lr5e-3--num-epochs20--optimizerranger\--loglogs/train.logNote:--data-trainand--data-testsupports providing multiple entries, e.g.,--data-train /path/to/A /path/to/B /path/to/C, and each entry also supports wildcards (*,?, etc. -- The pythonglobpackage is used to parse the paths).--data-testis optional: if specified, the performance on the testing dataset will be automatically evaluated after the training, using the epoch giving the best performance on the validation set. The prediction output can be saved if--predict-outputis also set.for training,--model-prefixsets theprefixpart of the paths to save model snapshots.
At the end of each epoch, the model parameters will be saved to/path/to/models/prefix_epoch-%d_state.pt,
and the optimizer states will be saved to/path/to/models/prefix_epoch-%d_optimizer.ptin case the training is interrupted and needed to be resumed from a certain epoch.One can also use an auto-generated path by including{auto}as part of the--model-prefix, then{auto}will be replaced by a string based on the timestamp and the hash of the network configuration.for small datasets, it's more efficient to use--in-memory --fetch-step 1to load the whole dataset (and perform the preprocessing) only once and keep it in memory for the entire run.when training on remote files (e.g., from EOS filesystem), one could consider adding--copy-inputsso the files are copied to the local workdir to speed up data loading.training can be resumed by adding--load-epoch [last_epoch]: with this option, the training will continue atlast_epoch + 1, and the optimizer states and the learning rate will be properly restored.Prediction/InferenceOnce you have a trained model, you can load it to run prediction and test its performance, e.g.,weaver--predict--data-test'/path/to/test_files/*/*/*/*/output_*.root'\--data-configdata/ak15_points_pf_sv.yaml\--network-confignetworks/particle_net_pf_sv.py\--model-prefix/path/to/models/prefix_best_epoch_state.pt\--gpus0,1,2,3--batch-size512\--predict-output/path/to/output.rootNote:--data-testsupports providing multiple entries, e.g.,--data-test /path/to/A /path/to/B /path/to/C, and each entry also supports wildcards
(*,?, etc. -- The pythonglobpackage is used to parse the paths).for inference, one can use a data configuration file with differentselection/observerssections, but theinputsandlabelssections must remain unchanged.for inference, one can specify the full path of the model parameters in--model-prefix.--predict-outputsets the path for the output file. It can either be the full path (if/is contained in the path), or just the file name part (e.g.,output.root) so that the output will be written under the directory of the--model-prefix, i.e.,{model_prefix_dir}/predict_output/{predict_output}. Currently supports saving to ROOT files (use.rootextension) or parquet files containing awkward arrays (use.parquetextension).Model exportationWhen you are satisfied with the trained model, you could export it from PyTorch to ONNX format for inference (e.g., usingONNXRuntime):weaver-cdata/ak15_points_pf_sv.yaml-nnetworks/particle_net_pf_sv.py-m/path/to/models/prefix_best_epoch_state.pt--export-onnxmodel.onnxMore about data loading and processingTo cope with large datasets, the data loader inWeaverdoes not read all input files into memory, but rather loads the input events incrementally. The implementation follows thePyTorchiterable-style datasetsinterface. To speed up the data loading process,multi-process data loadingis also implemented.[Note]For small datasets that actually fit into the memory, use--in-memory --fetch-step 1to load the whole dataset (and perform the preprocessing) only once and keep it in memory for the entire run.Thedata loaderinWeaveroperates in different ways for training and prediction/inference.Training modeFor training, properly mixing events of different types (e.g., signal/background processes, different kinematic phase space, etc.) and random shuffling are often helpful for improving the performance and stability of the training.To achieve this efficiently,Weaverdivides all input files randomly intoNgroups and will load them concurrently withNworker threads (Nis set by--num-workers). Then, two data loading strategies are available at the worker thread level:[Default] The "event-based" strategy attempts to read all the input files (assigned to this worker thread) at each step in order to "maximally" mix events. To meet the memory constraint, for every step, only a small chunk of events is loaded from each input file, and then randomly shuffled before being fed to the training pipeline. The chunk size is set by--fetch-step(default is 0.01), corresponding to the fraction (i.e., 10% by default) of events to be loaded from each file in every step. This is the default strategy as, for typical HEP datasets, each individual input file originates from a specific physics process, thus contains events of only a particular type / limited phase space. Note that while this approach ensures good mixing of events, it requires a high reading throughput of the data storage (thus a fast SSD is preferred), otherwise data loading can become a bottleneck in the training speed.Note: consider setting a smaller--fetch-stepif the memory limit is exceeded.An alternative approach is the "file-based" strategy, which can be enabled with--fetch-by-files. This approach will instead read all events from every file for each step, and it will readminput files (mis set by--fetch-step) before mixing and shuffling the loaded events. This strategy is more suitable when each input file is already a mixture of all types of events (e.g., pre-processed withNNTools), otherwise it may lead to suboptimal training performance. However, a higher data loading speed can generally be achieved with this approach.[Note]If the dataset is stored on EOS and the size is not very large, it may be more efficient to transfer the files to the local storage of the worker node when submitting batch jobs. This can be done on the fly by adding the--copy-inputsoption.Prediction/Inference modeContrary to training, for prediction/inference, the events are not mixed/shuffled. Instead, the order of the events is preserved, exactly as in the input files. Therefore, only the "file-based" strategy described above is supported, and the--fetch-stepis fixed to 1.For more details on the data loader, please checkutils/dataset.py.Performance considerations on data loadingLoading data from disk can often become a bottleneck. Here are a few tips to get better data loading performance:When using ROOT files as inputs, prepare the files w/LZ4compression:f.SetCompressionAlgorithm(ROOT::kLZ4);f.SetCompressionLevel(4);Copy files to a faster disk (e.g., SSD) if possible.Enable multiprocessing for data loading. Setting--num-workersto 2 or 3 generally gives a good performance. Setting this value too high could overload the disk and degrade the performance.Note that the memory usage also increases with the number of workers. So if you are getting any memory-related errors, try reducing--num-workers.Note that the workload splitting is file-based, so make sure the number of input files is not too small (i.e., make sure each worker is able to load several files to get samplesfrom all classes).e.g., if each (signal/background) class is present in only one input file, please use--num-workers 1so that they are properly mixed for the training.