High-Precision Regressors for Particle Physics
Authors: Fady Bishara
Ayan Paul
Jennifer Dy
Venue: Submitted to ICLR 2023 (2023)
Keywords: Boosted Decision Trees
Skip Connections
Deep Neural Networks
Particle Physics
Monte Carlo Simulation
Symmetry

Abstract: Monte Carlo simulations of physics processes at particle colliders like the Large Hadron Collider at CERN take up a major fraction of the computational budget. For some simulations, a single data point takes seconds, minutes, or even hours to compute from first principles. Since the necessary number of data points per simulation is on the order of $10^9$ -- $10^{12}$, machine learning regressors can be used in place of physics simulators to significantly reduce this computational burden. However, this task requires high precision regressors that can deliver data with relative errors less than 1\% or even 0.1\% over the entire domain of the function. In this paper, we develop optimal training strategies and tune various machine learning regressors to satisfy the high-precision requirement. We leverage symmetry arguments from particle physics to optimize the performance of the regressors. Inspired by ResNets, we design a Deep Neural Network with skip connections that outperform fully connected Deep Neural Networks. We find that at lower dimensions, boosted decision trees far outperform neural networks while at higher dimensions neural networks perform better. We show that these regressors can speed up simulations by a factor of $10^3$ -- $10^6$ over the first-principles computations currently used in Monte Carlo simulations. Additionally, using symmetry arguments derived from particle physics, we reduce the number of regressors necessary for each simulation by an order of magnitude. Our work can significantly reduce the training and storage burden of Monte Carlo simulations at current and future collider experiments.

Under review as a conference paper at ICLR 2023
HIGH-PRECISION PHYSICS
REGRESSORS
FOR PARTICLE
Anonymous authors Paper under double-blind review
ABSTRACT
Monte Carlo simulations of physies processes at particle colliders like the Large Hadron Collider at CERN take up a major fraction of the computational bud- get. For some simulations, a single data point takes seconds, minutes, or even hours to compute from first principles. Since the necessary number of data points per simulation is on the order of 10 1012, machine learning regressors can be used in place of physics simulators to significantly reduce this computational bur- den. However, this task requires high sish-precision regressors that can deliver data with relative errors of less than 1% or even 0.1% over the entire domain of the function. In this paper, we develop optimal training strategies and tune various machine learning regressors satisfy the high-precision requirement. We lever- age symmetry arguments from particle physics to optimize the performance of the regressors. Inspired by ResNets, design a Deep Neural Network with skip connections that outperform fully connected Deep Neural Networks. We find that at lower dimensions, boosted decision trees far outperform neural networks while at higher dimensions neural networks perform significantly better. We show that these regressors can speed up simulations by a factor of 10 10 over the first- principles computations currently used in Monte Carlo simulations. Additionally, using symmetry arguments derived from particle physics, we reduce the number of regressors necessary for each simulation by an order of magnitude. Our work can significantly reduce the training and storage burden of Monte Carlo simula- tions at current and future collider experiments.
INTRODUCTION
Particle physics experiments like those at the Large Hadron Collider CERN, are running at pro- gressively higher energies and are collecting more data than ever before. As a result, the experimen- tal precision of the measurements they perform is continuously improving. However, to infer what these measurements mean for the interactions between the fundamental constituents of matter, they have be compared with and interpreted in light of, our current theoretical understanding. This is done by performing first-principles computations for these high energy processes order by order in a power series expansion. After the computation, the resulting function used in Monte Carlo sim- ulations. The successive terms in the power series expansion, simplistically, become progressively smaller. Schematically, this can be written as:
F(x)= fz) + a fax( + a² (fu((:) + fu2(x))
where a < is the small expansion parameter. The term of interest to our current work is the one enclosed by the curly braces in equation (I) which we will refer to as the second-order term'. The function, Fxx), must evaluated on the order of 10 - 10 times for each simulation. However, for many processes, evaluating the second-order term, specifically, fug, is computationally space- and time-intensive and could take several seconds to compute a single data point. Moreover, these samples cannot be reused leading consideration. Building surrogate models to speed up Monte Carlo simulations is highly relevant not only in particle physies but in very large of problems addressed by all branches of physics using perturbative expansion like the one in equation (1), We give a broader overview of the physics motivation and applications in appendix A.
overall high cost of computation for the entire process under
Here order refers the power of the expansion coefficient

---

Under review as a conference paper at ICLR 2023
A simple solution to speed up the computation of the functions is to build regressor using rep- resentative sample. However, to achieve the precision necessary for matching with experimental results, the regressors need to produce very-high accuracy predictions over the entire domain of the function. The requirements that we set for the regressors, and in particular what we mean by high precision, are:
High precision:
prediction error < 1% over more than 90% of the domain of the function
Speed prediction time per data point of < 10-4 seconds
Lightweight : the disk size of the regressors should be a few megabytes at the most for portability
In this work explore the following novel concepts:
With simulated data from real physics processes occurring in particle colliders, we study the error distributions over the entire input feature spaces for multi mulli-dimensional distributions when using boosted decision trees (BDT), Deep Neural Networks (DNN) and Deep Neural Networks with skip connections (sk-DNN).
We study these regressors for 2, 4, and dimensional (D) data making comparisons be- tween the performance of BDTs, DNN and sk-DNNs with the aim of reaching errors smaller than 1% 0.1% over at least 90% of the input feature space.
We outline architectural decisions, training strategies and data volume necessary for build- ing these various kinds of high-precision regressors.
In what follows, will show that part, Su (xr) + foz(ar) (defined in equation (1)), by several orders of magnitude, down from sev- eral seconds sub-millssecons without compromising the accuracy of prediction. We show that physics-motive
reduce the compute time of the most compute- intensive
normalization strategies, learning strategies, and invocation of physics symme- tries will be necessary to achieve the goal of high precision. In our experiments, the BDTs out- perform the DNNs for lower dimensions while the DNNs give comparable (for 4D) or significantly better (for 8D) accuracy at higher dimensions. DNNs with skip connections perform comparably with fully connected DNNs even with much fewer parameters and outperform DNNs of equiva- lent complexity. Moreover, DNNs and sk-DNNs meet and exceed the high-precision criteria with 8D data while BDTs fail. Our goal will be to make the most lightweight regressor for real-time prediction facilitating the speed-up of the Monte Carlo simulation.
2 RELATED WORK
Building models for the regression of amplitudes has been continued attempt in the particle physics literature in the recent past. boosted decision trees (BDTs) have been the workhorse of particle physics for a long time but mostly for performing classification of tiny signals from dominating backgrounds (Radovic et al., 2018). However, the necessity to use BDTs as a regressor for theo- retical estimates of experimental signatures has only been advocated recently (Bishara & Montull, 2019) and has been shown to achieve impressive accuracy for 2D data.
Several other machine learning algorithms have been used for speeding up sample generation for Monte Carlo simulations. Winterhalder et al. (2022) proposed the use of Normalizing Flows (Jimenez Rezende & Mohamed, 2015) with Invertible Neural Networks implement im- portance sampling (Müller et al., 2018; Ardizzone et al., 2018). Recently, neural network surrogates have been used aid Monte Carlo Simulations of collider processes (Danziger al., 2022). Bad- ger et al. (2022) used Bayesian Neural networks for regression of particle physics amplitudes with a focus on understanding error propagation and estimation. Chen et al. (2021) attempted to reach the high-precisio regime with neural networks and achieved 0.7% errors integrated over the entire input feature space. Physics-aware neural networks were studied by Maître & Truong (2021) in an attempt handle singularities in the regressed functions. In the domain of generative models, GANs (Goodfellow et al., 2014; Springenberg, 2016; Brock et al., 2018) and VAES (Brock et al., 2018) have been used for sample generation (Butter et al., 2021; Otten et al., 2021).
Similar applications have surfaced in other domains of physics where Monte Carlo simulations are used. Self-leaming Monte Carlo methods have been explored by Liu et al. (2017). Applications of
2

---

Under review as a conference paper at ICLR 2023
300
1000
2000
200
2010
300
0000
0100
200
61000
Figure 1: The effects of normalizing second-order term (x). with the zeroth-order term, foo fo) The two functions are highly correlated (p = 0.87 for 8D, 0.91 for 4D and = 0.96 for 2D) and the resulting normalized functions, f(x) [fn (x) + f=)(=)) foo(()) are much less peaked.
Boltzmann machines (Huang & Wang, 2017). deep neural networks (Shen al., 2018) and autore- gressive neural networks (Wu et al., 2021) have been seen recently. Stratis et al. (2022) use neural networks in Quantum Monte Carlo simulations learn eigenvalues of Hamiltonians and the free energy of spin configurations, an application that lies outside the domain of particle physics. How- ever, the primary goal of all these efforts has been avoid first-principles computation and, hence, reduce compute time while staying below credible error budgets that are set in a problem-specificc manner.
In contrast to prior works (Bishara & Montull, 2019; Winterhalder et al., 2022; Butter et al., 2021; Otten et al., 2021). the novelty of our contribution is that we try to attain high precision in the en- tire domain of the function being regressed with fast and efficient regressors. For that, we compare BDTs and neural networks for functions with 2D, 4D, and 8D input feature spaces. We propose a new architecture which is DNN with skip connections to avoid the problem of vanishing gra- dients for deeper neural networks and show that they perform better than fully connected DNNs. We also propose novel methods, derived from physics domain knowledge, for scaling the function being regressed with another function that computationally inexpensive to calculate and is highly correlated with the function being regressed. We leverage the symmetry properties of the physical process under consideration for the reduction of the number of regressors required to trained. The applicability of our work goes beyond the domain for which it has been developed and be used for any application that requires high precision in speeding up simulations or sample generation.
3 MODELS: DECISIONS TREES AND NEURAL NETWORKS
In this section, we will develop several methods that will enable us to achieve the requirements that we set. As a benchmark, we will use the condition: (6| < 1% over 90% of the domain of the function being regressedd. Here, ö is defined as the difference between the predicted value of the function, I' ((x prodicted. and its true value, ((x)) normalized by f(xx) or,
8 f(x) predicted
[(o)prcicctedd f(x) (x))uree
knowing priori that f(x) (rr) positive definite. Usually, the performance of a regressor and model comparison in machine learning is done using single accuracy measure which is a statistical average of the distribution for that accuracy measure over the entire test sample. This, however, does not provide complete picture of the accuracy of the regressor in high-precision applications. Using error distributions instead of a single number leads to a better criterion for model selection and enhances the interpretability of the model.
Physics informed normalization: An attempt build regressors with the raw data from the Monte Carlo simulations results in failure to meet the high-precision requirements that we have set. Hence, we have to appeal to a novel normalization method derived from the physics that governs physical processes. The functions of interest in particle physics processes at colliders often very highly peaked in one or more dimensions. This makes it quite difficult to build a regressor that will
Forr detailed explanation of the precision requirements please read appendix A
3

---

Under review as a conference paper at ICLR 2023
retain the desired high precision over the entire domain of the function. This problem cannot be ad- dressed by log scaling or standardizing to zero mean and unit variance since the peaks can be quite narrow and several orders of magnitude greater than the mean value of the function. A regressor trained on the log scaled function provides an error distribution over the entire domain which, when exponentiated, transforms to large errors around the peak. This behavior is not desirable. Normal scaling does not help since the standard deviation of the distribution is much smaller than the peak value, often being several orders of magnitude smaller, making the peak-values outliers.
As a solution, we normalized the second-order contribution with the zeroth-order contribution as defined in equation (1), i... we transform to a distribution:
(ax)) f11(1) + fa2(
foo(2))
where f (ax) is the function that will be regressed. This first-order term, (z)) also has peak sim- ilar to and is highly correlated with the second-order term, fi) + fo2()), with p ~ 0.9. Hence, this normalization yields a distribution, [(x), that is more tractable to regress. We show examples in figure 1 where one can see that both fuo(2r (æ) and [fn (x) + fuz (x)) are both very peaked and span several orders of magnitude but their ration spans only one order of magnitude as the two terms are highly correlated. Computation of the first-order term from first principles is numerically inexpen- sive and does not require regression. Furthermore, we standardize the distribution by removing the mean and scaling to unit variance.
3.1 DNN DESIGN DECISIONS
The DNN architectures that we used are fully connected layers with Leaky ReLU (Maas, 2013) activation for the hidden layers and linear activations for the output layer. We show a comparative study of activation functions in appendix C where we find that the Leaky ReLU outperforms other activation functions like ReLU (Nair & Hinton, 2010; Sun et al., 2014), softplus and ELU (Clevert et al., 2016). We do not consider any learnable activation functions in this work and leave for a future work. We use the following design decisions:
we use the mean-square-erro loss function without any regularization. While use linear relative error, d, to estimate the performance of the model over the entire feature space, our decision to use the mean meca-square-ers loss function is made so as to preferentially penalize outliers and reduce their occurrence.
Objective function:
Learning rate: It is necessary to cool down the learning rate as a minimum of the objective function approached. This is absolutely necessary to search out an optimum that allows for uniformly low error over the entire feature space. For both the DNN and the sk-DNN, we use the Adam optimizing algorithm. Kingma & Ba (2015) discuss an inverse square-root root decay rate scaling for the Adam optimizer. We do not find this optimal for this high-precision application. The square-root cooling quite rigid in its shape as it is invariant to scaling up to a multiplicative constant. Hence, we use an exponential cooling of the learning rate which has an asymptotic behavior similar to the inverse- squarc-root scaling but its shape is far more tunable. The learning rate is cooled down starting from 10-3 at the beginning of the training to 10-6 at 2500 epochs. Much of the learning is completed during the early stages of the training, i.e. within 200 epochs. The R² score at this point is about 0.5% from the final R²2 score (> 99. 99.9%). However, to attain the high-precision requirements, the final stages of the training are necessary and take about 2500 - 3000 more epochs.
Training epochs and validation: An early stopping criterion based RMSE is used determine the number of epochs the regressors is trained for with patience* set to an unusually large number, 200 epochs. We use this large patience to allow the optimizer to possibly move a better optimum while having a very small learning rate if better one exists. We first split the data into 20% test set and 80% training and validation set. The latter set further split into 60% training set and 40% validation set. This results in a 20%-48%-322 split for test, train and validation respectively. The large validation set necessary to make sure that errors are uniformly low over the entire domain of the function being regressed. For all cases, we use dataset with 10 million samples.
We define patience as the number of epochs after which the training is stopped as no reduction is seen in the RMSE computed from the validation set and the weights and biases are reset to those corresponding to the lowest RMSF.
4

---

Under review as a conference paper at ICLR 2023
3.2 DNN WITH SKIP CONNECTIONS
Ioas, lady lell anclation
The building block for a DNN with skip connections. The first two layers are fully connected with Leaky ReLU activation. The last layer has linear activation and is added to the input through the skip connection before being transformed with a non-linear Leaky ReLU function. The matrix W is weight matrix which is trainable if the input and output dimensions are different for the block and otherwise. The blocks are stacked sequentially to form the neural network.
Figure 2:
In addition to a fully connected DNN, we also experiment with DNN with skipped connections (sk-DNN) address the problem of vanishing gradients for deeper neural networks. The building block of the sk-DNN is illustrated in figure 2. Given an input æ the output of the block is
= g- (h(x) + Wx)
(4)
where ((x)) the output of the third layer with linear activation, y is non-linear function and W is trainable weight matrix of dimension i when the input dimension, i, is different from the output dimension, j. and I otherwise. The structure of this block can be derived from the Highway Network (Srivastava et al., 2015) architecture with the transform gate set to I and the carry gate set to W for dim(x) (x) * dim(y (y) and otherwise. Structurally, the sN-DNN block is similar to ResNet block (He et al., 2015) with different set of hidden layers.
We keep the normalization of the target variable and the learning rate decay schedule the same as for the DNN. We also test the Sk-DNN with the weight matrix, W fixed with random initialization of the elements and see no difference in the accuracy of the model and hence keep W trainable.
4 EXPERIMENTS
4.1 PHYSICS SIMULATIONS
The functions in question are maps, (()) Rn - R. where n E {2, 4,8) and í, E {0, 1, 2), cf. equation (1). The domain of the functions, i.c. the feature space, is mapped to the unit hypercube and populated from a uniform distribution. The corresponding datasets are generated using the particle physics simulation code VVAMP (Gehrmann et al., 2015) from first principles using building-block functions that we will refer to as form factors. Apart from the 2D dataset, which a special case of the 4D one, the same form factors were used to generate the 4D and 8D datasets. The difference between the 4D and 8D feature spaces lies in the physics of the process in question, namely the number of external particles the functions describe. The regressor of the 4D functions, Dij
(4) f
be used generate the 8D functions, f(8) after multiplying by other (exact) functions that are computationally inexpensive to calculate and summing them.
The number of resulting functions, technically called helicity amplitudes, depends on the dimension as shown in table 1. While the number of required regressors for the 4D feature space is the largest, it also offers the most flexibility for downstream physics analyses. To generate the 8D functions,
5

---

Under review as a conference paper at ICLR 2023
Symmetry properties reduce number of required functions
Dimensionality
Total functions
Independent functions
Sum physical?
2D
18
5
Yes
4D
162
25
No Yes
8D
Table 1: The number of total and independent functions that arise at 2, 4, and 8D and the number of independent functions, a minimal subset from which the other functions can be generated by re- mapping the feature space.
more details the process have to be specified during data generation which is then frozen into the regressor. Consequently, different physics analyses will require different regressors. By contrast, the 4D regressors are more general-purpose and do not contain any frozen physics parameters.
The smaller number of necessary functions in the third column of table is obtained by leveraging the symmetry properties discussed below derived from physics domain knowledge. For the data used in this analysis, it stems from the symmetries manifest in the scattering process that was simulated. The last column indicates whether summing the functions has physics meaning; in the cases where it does, i.c. 2D and 8D, only the single regressor of the sum of the functions is required.
Symmetry properties: the full set functions, p(()) for any dimension, n, is over complete. Pairs of functions be mapped into one another via partieular permutations of the external particles the process describes. This translates into linear transformation on the second coordinate, 22. independently and in combination with the permutation of the third and fourth coordinates, va and T1. in feature space. For example, in 4D, two permutations T12 + P2 and T34 := Ps + P4. where p. particle with label reduces the number of independent functions from 162 25.
Permutation
particle symmetry
coordinate symmetry
T22
P1 P2 Ps ++ P4
T2 22
T34
32 + 22 and M3 34
Computational burden of Monte Carlo simulations. Generating the 2D, 4D and 8D datasets required 144 hours on 96 AMD EPYC 7402 cores for 13 million data points per set. This had to be done twice, once for the 2D dataset and once for the 4D and 8D datasets which were generated from the same computationally intensive form factors which have to be calculated from first principles. In contrast, the regressors that we build generate a million samples in few seconds to a few minutes on any desktop computer.
4.2 BDT vs. DNN vs. SK-DNN
155
(1100
Mia.iee (130 20700000
Figure 3: Left panel: the effect of learning rate on achieving high precision with BDTs. The learn- ing rate not an important parameter for low dimensions but significant for higher dimensions. Right panel: the data volume required to train the different regressors. For lower dimensions small volumes of data (< 1M) sufficient. However, for higher dimensions, a lot more data is necessary.
We use XGBoost (Chen & Guestrin, 2016) to implement the BDTs, In varying the architecture of the regressors, we focus the max-depth of the BDT which is hyperparameter that controls the

---

Under review as a conference paper at ICLR 2023
Baskcaale 13100
10
1.5
400
200
Figure 4: The = (f(x) predicted f(x)) (r(x))tee///f ((x)) distribution for the various 2D (upper panels), 4D (middle panels), and 8D (lower panels) regressors. The labels for the DNN and sk- DNN designate depth-width (number of parameters) where depth is the number of layers for the DNN and the number of blocks for the sk- For the BDTs, the labels denote max_depth: N with max dept being the maximum depth of the trees. Detailed analyses can be found in the text.
maximum depth to which a tree grows in a boosted ensemble. figure 3 we show how changing the learning rate and the training data volume changes the accuracy of the trained BDT models. In the final version of our experiments, we use a learning rate of 0.01 and 10 million data points of which 48% is used for training, 32% is used for validation and early stopping and 20% is used for testing. More details on hyperparameter correlation and selection can be found in appendix B.
For the neural networks, we focus on the depth, width and number of trainable parameters in the regressor (denoted as width-depth (trainable parameters) in the tables and figures). The depth of the sk-DNN denotes the number of sequential sk-DNN blocks in the regressor and not the total number of layers. The width of the sk-DNNs is chosen to be half the width of the DNNs and the depth of the sk-DNN is adjusted so that they have approximately the same number of parameters as the DNNs with similar depth. An Sk-DNN with 2 blocks is an exception and has more parameters than the corresponding DNN with layers. The data strategy remains the same as for the BDTs.
To compare the performance of the regressor we use the distribution of 5 (defined in equation (2)). We focus on this distribution as is important for the high-precision requirement to identify the fraction of test data that has large errors. We will identify the following statistics:
|8| < 1%: the fraction of the test set that has less than 1% Js. the mean of the å distribution
ag: the standard deviation of the 5 distribution
To understand the efficacy of the optimization strategies that we developed, we build baseline without any optimization for BDTs, DNNs and sk-DNNs. We do not normalize the data as described in section 3, rather, we only log scale the data. We set the train-validation split to 80%- 20%. For the BDTs, we use an ensemble with max-depth 50, set the learning rate to 0.1. For the DNNs and SN-NNNs we fix the learning rate of the Adam optimizer at 10 lower the patience to 10 rounds, and use the most effective architecture chosen from amongst the gigh-precision regressors. The results are presented in table 2 and figure 4. We see that without the optimizations the regressors perform very poorly.
Baselines:
7

---

Under review as a conference paper at ICLR 2023
2D
[6] 196(%)
[6] < 0.1%((%)
(a(
(a
2-56 (3,417)
95.54
32.43
0.0114
0.52
4-56 (9,801)
99.95
75.58
-0,0005
0.13
(22.569)
0.06
baseline (8-56)
77.16
9.01
0.1361
1.83
BDT sk DNN
2-28 (4,229)
99.92
67.71
0.0001
0.14
4-28 (9,101)
99.99
87.95
0.0012
0.07
(21.281)
95.72
-0,0005
0.05
buseline (9-28)
90.79
15.31
0.0173
1.39
max-depth: 10
100.0
95.01
-0,0001
0.05
max depth: 20
100.0
99.1
0.0
0.03
mnax-dephh:
100.0
99.16
0.02
buseline (50)
99.91
94.04
-0.0045
0.1
4D)
2:72 (5,689)
99.18
34.34
0.002
0.32
DNN
4-72 (16.201)
99.97
66.42
-0.0068
0.13
(37.225)
91.58
0.07
baseline (8-72)
88.67
13.63
0.0449
1.1
DNN
2:36 (7,021)
99.96
69.23
0.0004
0.13
4-36 (1.50013)
100.0
89.24
0.0009
0.07
9.36 (34,993)
100.0
92.85
0.0007
0.06
besclino(9-36)
84.99
10.81
0,2701
1.11
max-dopth: 10
99.26
06.16
0.0014
0.22
BDT
max-dopth: 20
99.11
81.34
0.0016
0.18
max-depth: 50
99.4
83.19
0.0017
0.18
baseline (50)
95.85
27.8
0.0023
0.55
SD
2-100 (11.101)
37.2
3.95
0.1322
4.13
DNN
4-100 (31_301)
82.37
11.64
0.029
1.05
(71,701)
buselime (8 100)
31.97
3.3
0.799
4.38
DNN
2-50 (13,651)
72.76
9.31
0.049
1.5
4-50 (28.951)
90.98
15.69
0.0035
0.7
9-50 (67,201)
94.94
19.36
-0.0063
0.56
baseline (9-50)
30.91
3.23
-0.547
4.85
max depth: 10
51.68
5.95
0.0921
2.91
BDT
max-depth:
72.6
12.06
0.0577
1.85
max-depth: 50
62.15
9.61
0.1505
2.36
baseline (50)
22.33
2.3
1.3953
13.35
Table 2: Parameters extracted from the error distributions shown in figure 4. Predictions from
million test samples were used generate these statistics. The best-performing models for each of 2D, 4D, and 8D are marked in bold. More details regarding the notations are available in the text.
Key results: we present the results of the experiments in table 2 and figure 4. We show the distribu- tion errors over two variables, square root of the center-of-mase energy, /s - 21, and cos 22 in figure 5. It is clear that the BDTs far outperform the DNNs in 2D. However, at 4D and 8D the sk-DNN not only outperforms the fully connected DNNs, but also outperforms the BDTs as can be seen from the distributions in figure 4 and the numbers table 2. While at 4D the improvement of accuracy from the DNN and sk- DNN is marginal over the BDTs, at 8D the improvement of accuracy quite significant. One major disadvantage of the BDTs is that they take up significant disk space as the ensemble grows large, especially at higher dimensions, which is necessary for high-precision ap- plications but affects their portability. Hence the sk-DNNs are good solution for having portable, yet accurate regressor that meets the thresholds we set at the beginning of the work.
CONCLUSION
With Monte Carlo simulation in Physics being time and resource intensive, a distinct necessity of building regressors for speeding up the simulations exists. We carefully examine the requirements of high precision for these regressors and lay down design strategies to achieve the necessary bench- marks. We use domain knowledge from particle physics to determine normalization strategies, apply symmetry arguments to reduce the number of necessary regressors, and set benchmarks for high- precision regression.
8

---

Under review as a conference paper at ICLR 2023
1X0_0_56
H[R
107-60
0.1
0.1
0.1
110
10
10
110
0.0
0.
0.5
1.0
10-3
08
0.0
0.7
98
1.0
16-2
000
0.2
6.4
10-1
<
H1
1.1
10
10
10"
0.2
0.0
1.0
111
08
100
1.0
10
0g
02
10"
[WL 100 sn
N0T 20
0.1
0.4
64
0.4
10
10
0.5
0.2
0.0
1.0
10
0.0
10-1
6.0
0.5
10
distribution for the various 2D (upper panels), 4D (middle panels) and 8D (lower panels) regressors. The errors averaged over each bin.
Figure 5: The ( - (( (re
We show that boosted decision trees are reliable workhorses that can easily outperform DNNs at lower dimensions even when very large and complex neural networks are used. However, this edge that BDTs have over neural networks tends to fade at higher dimensions especially when DNNs with skip connections are used. fact, for 4D and 8D data, sk-DNNs outperform both DNNs and BDTs and exceed the benchmark of & < 1% over 90% of the domain of the function. Moreover, sk-DNNs are capable of outperforming DNNs of higher complexity as can be seen from table 2.
The primary disadvantage of BDTs is that for higher dimensions the ensemble of trees grows large enough to take a significant amount of disk space, often > GB, affecting the portability of regressor if it is intended to be used as part of Monte Carlo simulation package. On the other hand, the disk space occupied by a neural network stay below few megabytes, making them a lot more portable. In summary, the important conclusions of our work arc:
High precision regressors required to speed up Monte Carlo simulations by factors of 10 10 are better optimized by leveraging physics domain knowledge and symmetry argu- ments.
BDTs outperform DNNs at lower dimensions but start to make large errors in predictions in parts of the function domain at higher dimensions. While fully connected DNNs perform relatively well higher dimensions, DNN with skip connections outperforms both BDTs and fully connected DNNs at 4D and 8D.
sk-DNNs can outperform DNNs with larger number of parameters.
Compared the few seconds that it takes for single sample generation during Monte Carlo simulation, the regressors we design can provide precise predictions in milliseconds microseconds.
In this work, we aimed at reaching the desired precision but by no means have we exhausted the possibilities of achieving even higher precision. As future directions, surveying a wider gamut of activation functions, the modifications of the likelihood with possible physics constraints sym- metry arguments or further reducing the number of models by simultancously predicting a set of functions from single neural network might be directions that be explored in detail.

---

Under review as a conference paper at ICLR 2023
CONTRIBUTION TO SUSTAINABILITY
Monte Carlo simulations of physics processes leave a very large carbon footprint. It is estimated that about 50% of the energy budget of each experiment the Large Hadron Collider is consumed by such simulations. Hence, our work directly contributes to reducing the carbon footprint significantly through a much more efficient way of generating these events.
Generating the 2D, 4D and 8D datasets required 144 hours on 96 AMD EPYC 7402 cores for 13 million data points per set. This had be done twice, once for the 2D dataset and once for the 4D and 8D datasets which were generated from the same first principles computation. In contrast, the regressors that we build generate a million samples in a few seconds to tens of seconds on any desktop computer. The regressors threads and a single GPU in about a day as our focus has been to build lightweight models. No spe- cial hardware is required train test. these regressors. Given that these Monte Carlo simulations have to be done thousands of times during the life cycle of single analysis, the regressors can sig- nificantly reduce the carbon footprint from energy consumption without any significant compromise to the precision necessary for quantitative scientific research.
build can be trained personal computers with few CPU
REPRODUCIBILITY STATEMENT
The code and the data will be made publicly available after the review process to maintain the double-blind requirements.
AUTHOR CONTRIBUTIONS
ACKNOWLEDGMENTS
REFERENCES
Lynton Ardizzone, Jakob Kruse, Sebastian J. Wirkert, Daniel Rahner, Eric W. Pellegrini, Ralf S. Klessen, Lena Maier- Hein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. CoRR, /e///888.047330 2018. URL httt:://aaxiv..oo abs/ 1808. 04730.
Simon Badger, Anja Butter, Michel Luchmann, Sebastian Pitz, and Tilman Plehn. Loop Amplitudes from Precision Networks. 6 2022. URL htRL:tt/////axv.cro hwtpiii...r.//aaaaaeeaaeeeeoeeee a22/2266..222... 14831.
Fady Bishara and Marc Montull, (Machine) Learning amplitudes for faster event generation. 12 2019. URL htt: httt:://axxiv..rr//eeet.. 11111.111.11111.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. CoRR, abs/1809. 2018. URL http://arxiv..r orw. abs/ 1809. 11096.
Anja Butter, Sascha Diefenbacher, Gregor Kasieczka, Benjamin Nachman, and Tilman Plehn. GAN- plifying event samples. SciPost Phys., 10(6):139, 2021. doi: 10.2 11144///iiosttts.......... URL https: /tttts:://oi.orootee http://ooooooog////// 21468 htt.://///iPPPPPPssoooeo 1............ 139.
I-Kai Chen, Matthew D. Klimek, and Maxim Perelstein. Improved neural network Monte Carlo simulation. SciPost Phys., 10(1):023, 2021. doi: 202144/////PostPys...0
URL https:
//doi. orooi.orgg100 10lgg/10..0 214 http://rr.......////co.osss.pes.
Tianqi Chen and Carlos Guestrin. XGBoost: scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16. pp. 784-794, New York, NY, USA, 2016. ACM. ISBN 978- 778-1-4500-4232-22 doi: 10.1145/ 2939672.2939785. URL http: //ot::://do... acm.
(tt.239977222..97788 2939785.
Djork-Ame Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In Yoshua Bengio and Yann LeCun (eds.), 4th Inter- national Conference on Learning Representations, ICLR 2016. San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL WtRLhtp::// aarxiiiiv... www.ww.wwggg.........a... www.//oggg//////. 07289. httt:p:/aaxcc..org//b/////////
10

---

Under review as a conference paper at ICLR 2023
Katharina Danziger, Timo Janßen, Steffen Schumann, and Frank Siegert. Accelerating Monte Carlo event generation - rejection sampling using neural network event estimates. SciPost Phys., 12:164, 2022. 1 1448//aaPostthhs.12555555
URL http::///aattttc wrg.0.0......00.000... /
Sci SciPostPhys 1 5. 64.
Thomas Gehrmann, Andreas von Manteuffel, and Lorenzo Tancredi. The two-loop helicity ampli- tudes for qq V/222 + leptons. JHEP, 1981 2015, doi: 10.1007//HEE0092015522.. URL 11100000HHEP (2015) 123.
https: /oi. htt.://.rrg/1.111///111....
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL https:
waxxiv.rg//abss...... 26611..........
Massimiliano Grazzini, Stefan Kallweit, and Marius Wiesemann. Fully differential NNLO compu- tations with MATRIX. Eur. Phys. J. C, 78(7)55337,, 2018. doi: 10.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abs/15 512.03385, 2015. URL http: / arxiv. org/ abs 112211111111 03385.
Li Huang and Lei Wang. Accelcrated monte carlo simulations with restricted boltzmann machines. Phys. Rev. B, 95:035105, Jan 2017. doi: 00.103//hhy.ev..55..033.
URL https: :tttps://lnnk
ops. /wtg///oi//i 10. 1101 C/t./3/////....... 95. 035105.
Danilo Jimenez. Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows.
arXiv e-prints, arXiv:1505.05770, May 2015. URL https:///aaa wrt..rgg/ab..s....
05770.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Reprexentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http: //arxiv. org/ abs /all//14122 1412. 6980.
Junwei Liu, Yang Qi, Zi Yang Meng, and Liang Fu. Self-learning monte carlo method. Phys. Rev. B, 95:041101, Jan 2017. doi: 10.1103/hyyseev..55..m 100.44 URL httpt:///likkk... org/ 11000.103/// 1t1.:/.11////yyyeee.
95. 041101.
Andrew L. Maas. Rectifier nonlinearities improve neural network acoustic models. 2013. URL
http://aastttafrrrd.o...o hwt.roorrdddddeaaeeeeeee adaaaamaas/aaaaaaaaaaaaaaaaaaaaa paamaaasppprrr/elluuuaaaa firal. .dff.
cm 2013_
Daniel Maître and Henry Truong. A factorisation-awarc Matrix element emulator. JHEP, 11:066, 2021. doi: 00.1077/HEP11(20110666 URL /ttt::/:/oo...rr/00..26666/ JHEP11 (2021) 066.
Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. Neural im- portance sampling. CoRR, fbs/1808.03856, 2018. URL htttt://aaa/i org/abs/1805 03856.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. ICML'10, pp. 807-814, Madison, WI, USA, 2010. Omnipress. ISBN 9781605589077. URL https: acm. wttp//iigoo.oooo///////////////o /woi/1/....//////// 5555/555555 3104322..2222211 3104425.
Sydney Otten, Sascha Caron, Wieske de Swart, Melissa van Beekveld, Luc Hendriks, Caspar van Leeuwen, Damian Podareanu, Roberto Ruiz de Austri, and Rob Verheyen. Event Generation and Statistical Sampling for Physics with Deep Generative Models and a Density Information Buffer. Nature Commun., 12(1):2985, 2021. doi: 10.10088//4444--0122222666 URL https:
22221616-66.
Alexander Radovic, Mike Williams. David Rousseau, Michael Kagan, Daniele Bonacorsi, Alexan- der Himmel, Adam Aurisano, Kazuhiro Terao, and Taritree Wongjirad. Machine learning at the energy and intensity frontiers of partiele physics. Nature, 560(7716):41-48 Aug 2018. ISSN 1476-4687. doi: (0.10088/41158--18--3333 s41586- 018-0361--2
URL https: /ttttps://oi.or///// 1038/
11

---

Under review as a conference paper at ICLR 2023
Huitao Shen, Junwei Liu, and Liang Fu. Self-leamnig monte carlo with deep neural networks. Phys. Rev. B. 07:205140,, May 2018. doi: 10.1103/PhysRew..9222 299.00040... URL attps://limg aps. htlgg///oi/// http://....1///hyeeeeeeeee
97 2071400..
Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Reprexentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv. org/ abs/ 1et..111....0 06390.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. CoRR, abs/1505.00387, 2015, URL http:///ame wwgg hbtp//bs//tt /105. 000Ds//555.0087 00387.
Georgios Stratis, Phillip Weinberg, Tales Imbiriba, Pau Closas, and Adrian E. Feiguin. Sample gen- eration for the spin-fermion model using neural networks. arXiv e-prints, art. arXiv:2266.07753, June 2022. URLhttps: /uii wwwaaa..aawaea.......a.a..aaaaae
Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deeply learned face repressnttttinnonnnnn are sparse, selective, and robust. CoRR, abs/1412.1265. 2014. URL
1265.
Ramon Winterhalder, Vitaly Magerya, Emilio Villa, Stephen P. Jones, Matthias Kerner, Anja Butter, Gudrun Heinrich, and Tilman Plehn. Targeting multi-loop integrals with neural networks. SciPost Phys. 12(4):129, 2022. doi: 10.2 1e.114///ciPostPyyy... 12..iiiPooohys1.....2222. URL http: //oii org/ 10. 014468/$ciPoste 12....1111.. 1....22..111. 129.
Dian Wu, Riccardo Rossi, and Giuseppe Carleo. Unbiased monte carlo cluster updates with autoregressive neural networks. Phys. Rev. Research, 3:L042024, Nov 2021. doi:
(0L1083PhyyRerreesrr:..1022222 URL https:/ /ttp:///iiiiieee aps. wtt.aappp..o.ooo.ao....oooooo PhysRevResearchs L0420224.
Appendix
A THE PHYSICS BACKGROUND
The goal of this appendix is to provide the context in which the regressors discussed this paper are needed and thus elucidate the requirements laid out in the introduction. Namely,
1. that the prediction eπror relative to the exact value be < 1%.
2. that this be the case for 90% of the domain of the function,
3. that the evaluation time faster than O(10a3) seconds,
4. and, finally, that the disk size of the regressors be on the order of megabytes rather than gigabytes.
These regressors will be used as surrogate models for exact functions that are numerically slow to evaluate. As a result of their (extreme) slowness, these exact functions, which are used Monte Carlo simulations, are by far the biggest bottleneck in the simulation.
The physics context. The theoretical model that describes the fundamental particles and their in- teractions is called the Standard Model of particle physics. This model is an example of quantum field theory; what this means exactly is not crucial here. Rather, the important feature of this theory that computing observables (i.e., outcomes of experiments) cannot, in general. be done exactly because such calculations are not tractable for several reasons the explanation of which goes be- yond the scope of this work. The usual way of obtaining predictions is by expanding the theory as power series in small expansion parameter and computing higher orders in this expansion to improve the accuracy of the prediction. Such perturbative expansions are ubiquitous in physics in general since only few systems, most notably, e.g., the simple harmonic oscillator and the two- body inverse y² problem can be solved exactly. very large fraction of physics problems spanning atomic physics, nuclear physics, condensed matter physics, astrophysics, cosmology, hydrodynam- ics, electrodynamics, quantum mechanics, complex systems etc. requires the use perturbative expansions where the higher order terms very tedious and slow to compute. Hence, the methods
12

---

Under review as a conference paper at ICLR 2023
develop here are more broadly applicable in any problem where perturbative expansion is used and/or a function that requires a very large number of evaluations is very slow to evaluate and a certain threshold of precision is required.
The slow functions that are the focus of this work arise at second order in this power series ex- pansion. The number of terms produced each order rapidly increases and the complexity of the mathematical functions that appear also increases. For example, at first order in the expansion (if the zeroth order is so-called tree process), polylogarithms of at most order 2 can appear. At second or- der, higher order polylogarithms appear. On top of the fact that these functions are time-consuning to evaluate numerically, large cancellations between these functions typically exist which requires using arbitrary precision arithmetic libraries to circumvent the infamous 'catastrophic cancellation problem in numerical analysis.
For example, the time penalty for improving the accuracy of the prediction of the rate of production of four electrons by including the second-order term is a factor of 1500! For details, please see table 11 in the journal version of Ref. Grazzini et al. (2018). So here lies the logic behind requirement 3: the second-order functions typically take - 10 seconds per point to evaluate while all other functions in the Monte Carlo simulation typically take milli-seconds. Therefore, the bottleneck is removed if the surrogate model takes 10 second per point or less to evaluate.
The accuracy requirement. There are many sources of uncertainty that propagate to the final prediction. Roughly speaking, there systematic errors of order 1% that cannot be reduced at the moment and for the foreseeable future. There are also statistical errors inherent in the finite samples produced by Monte Carlo simulations. The goal is to strive to have Monte Carlo statistical errors much smaller than 1%, say 0.1%. Since the contribution of the second order functions is of order 10%, then it is sufficient for the surrogate models to be accurate to 1 1% in order for the error on the total prediction (including the zeroth and first order) to be of order 0.1%. While this precision would be good to have in the entire domain of the function it is not necessary given the error margins we aim for. Assuming that the errors in the predictions made by the model follow Gaussian distribution, we can safely set the threshold to 1% error over 90% of the function domain. We checked, after the fact, that the Gaussian assumption is approximately realized (ef. figure 4). An elaboration of this requires a discussion of specific integrals for specific scattering processes in particle physies that we shall leave for more partiele-physics-eeee prrsics- work.
Portability. In contrast with the speed and accuracy requirements (items 1, 2. & 3), the require- ment that the disk size of the models be of order megabytes (item 4) is desirable but not a strict requirement. In practice, Carlo codes, several regressors are required. Since these codes must be downloaded locally by the users, it is desirable that the disk size remain small. As shown before BDTs can reach several GB in compressed model format for higher dimensional data. Hence, in this work, we focus on building specific neural networks that are a lot more portable.
implement the surrogate models discussed in this work into Monte
B HYPERPARAMETER SURVEYS FOR BOOSTED DECISION TREES
Maximum depth of trees in the ensemble: The BDT models are trained with an early stopping condition which stops the growth of the trees once the RMSE stops decreasing after checking for its decrease for 25 rounds. This makes the hyperparameters used to train a BDT correlated certain extent. For example, decrease in the learning rate increases the number of trees grown till the optimum is reached. This can be seen from figure 6. However, as one increases the maximum depth to which each tree can grow the number of total trees grown decreases. The number of nodes of tree grows exponentially with the depth of the trees and, hence, allowing for larger maximum depth of the trees results in much larger disk size for the trained models. This is aggravated further with higher dimensional data. Therefore, when portability is a concern, BDTs cannot be used for high-precise applications for higher dimensional data.
When exploring the learning rate for the BDT models in figure 6, find that, initially, with decreasing learning rate, starting at 1, the accuracies of the trained models increase but after a point, the accuracy decreases. This is evident for shallower trees. We also note that the accuracies of the models increase with the maximum depth of the trees but
Learning rate and maximum depth of trees:
13

---

Under review as a conference paper at ICLR 2023
number of boss (dreemited naty stoppin
SD datn
fruction of test aet with M 198 (99
XD data
depth
8000
depth:
80
more depth
8000
depth
60
2000
10-2
10 10P
10a3
10a2
10
107
boarning
losning
Figure 6: Left panel: The variation of the number of trees grown in BDT ensemble increases rapidly with decreasing learning rate when using an early-stopping criterion. Note: The learning rate for BDTs with maximum depths 20 and 50 could not be reduced below 0.1 the disk size of the memory consumption while training the models with 8D data gets too large for single node in a pegh-peeformance computing cluster. Right panel: Larger maximum depth for BDT ensembles gives better accuracy up until certain value and then the accuracy falls. The optimal value for maximum depth seems to 15 or 20. Also, the learning rate has an optimal after which decreases or plateaus.
only up certain depth. In the example in the right panel of figure use the 8D data and see that the accuracy of the model increase till maximum depth of 15 and then decreases.
C DEPENDENCE ON ACTIVATION FUNCTIONS
1.1
Lonky ReL.1
1.2
ReLU
saftplus
1.0
ELU
0.8
0.6
0.4
0.2
0.0
(56)
Figure 7: A comparison between Leaky ReLU, ReLU, softplus and ELU for the 8D data using an sk-DNN with 9 blocks of width 50. The Leaky ReLU activation function outperforms any other activation function and we use it for all experiments with DNNs and sk-DNNs in our work
We performed tests for various activation functions keeping all other hyperparameters and data strategies the same. We the 8D dataset with 9-deep and 50-wide sk-DNN an exponen- tial learning rate schedule and data normalized using equation (3). We explore only non-trainable activation functions like the ReLU, Leaky ReLU, ELU and softmax activations functions. The last three were chosen as they are similar to ReL. and have shown improved learning abilities in several domains Maas (2013); Sun al. (2014); Clevert al. (2016). As the other experiments, the models were trained with an early-stopping criterion. From figure 7 activation function far outperforms all other activation function with a narrower error distribution. Hence, for all experiments in this work we use the Leaky ReLU activation function.
that the Leaky ReLU
14


Reviews:
[{"review_id": "tQMvIVpwWj", "reviewer": "Reviewer_D2vG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors leverage symmetry principles to design high-precision regressors which enable a dramatic speedup of simulations vs. traditional Monte Carlo approaches.  They demonstrate that BDT performs well in 2D and 4D, but DNN with skip connections are needed for performance in higher (i.e. 8D).  ", "summary_of_the_review": "The overall motivation of the paper is clear- to replace time consuming MC methods with faster ML methods.  However, because specific detail around the problem to be solved isn't given, it's not clear why some of the design choices are made.  The impact of dataset size is an important feature that's worth exploring.  While MC methods are slow, they do offer a \"know what you're getting\" and ability to quantify the uncertainty; it's not clear to me how certain any specific measurement under the proposed framework is without also comparing to MC at some point (i.e. how well does this generalize to different events)?", "strength_and_weaknesses": "Strengths:\nOverall the paper is clear, well motivated, and well written textually (although some clarity around the physics could be provided- see next section).\n\nThe authors compare several approaches including BDT, and two NN formulations.\n\n\nWeaknesses:\n The architectures explored/chosen are not particularly novel.  That's fine if it's not the main focus, but the emphasis seems to suggest that the use of skip connection based architectures is a key insight/contribution.  \n\nAdditional clarity around the physics would be helpful (see next section).\n\nThe authors briefly reference permutations of external processes and over-completeness in 4.1.  Would using a permutation-invariant architecture help to alleviate this problem?\n\nDemonstrating the performance of the different methods as a function of dataset size could be valuable.  \n\nHow would this extend to other calculations?  The authors have demonstrated this is a good method for this calculation by comparing to the MC based outcome, but what parameters could change to give someone implementing this method that this approach is still valid?  MC methods provide error ranges on the estimates.  How would that work here- what is the error around the physical parameter of interest to be measured and how can you \"prove\" this?\n\nOnly two NN architectures are used.  Would using a more \"sophisticated\" architecture likely lead to a better result for that approach?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "nbrQrWBGkM", "reviewer": "Reviewer_XeYb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses an important problem of building high-precision regressor for particle physics. The paper designs the sk-DNN, which uses residual layer, and with physics-informed normalization for the data as input. It compares vanilla DNN and with baseline of boosted decision trees. The paper shows that for larger dimensional input (4D, 8D), the sk-DNN shows improved performance and speeds up simulations by a factor of 10^3 – 10^6 over the first-principles computations currently used in Monte Carlo simulations.", "summary_of_the_review": "In summary, the paper addresses an important problem in particle physics, and achieves good accuracy and significant speedup compared to first-principles computations. The innovation in the ML may be limited, which may be improved with more empirical baselines.", "strength_and_weaknesses": "Strengths:\n\nThe paper addresses an important problem in particle physics. The physics-informed design of the architecture and data normalization introduces useful inductive bias and helps improving the performance. The speedup is significant which is important in the field.\n\nWeaknesses:\n\nIn terms of novelty, w.r.t. the field of machine learning for particle physics, this method may be novel and significant (I'm not an expert in particle physics). On the other hand, in terms of machine learning, the novelty is limited, since the residual connection is a commonly used architecture. The physics-informed normalization is interesting, but may be limited in its generality to more general problems. I'm OK with judging the novelty either way.\n\nThe limited novelty may be compensated by a more comprehensive empirical experiments. The authors are encouraged to explore more hyperparameters and compare with more baselines. For example, the authors may explore how activation function affect the performance, since activation is extremely important in regression tasks. In my experience, for regression tasks, ReLU does not necessarily work well since it essentially models a piecewise linear function. The activation of leakyReLU, SiLU [1], ELU [2], Rational activation [3] are typically very good candidate activation for regression tasks, especially rational activation which is able to model sharp changes.\n\n\n[1] Elfwing, Stefan, Eiji Uchibe, and Kenji Doya. \"Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.\" Neural Networks 107 (2018): 3-11.\n\n[2] Clevert, Djork-Arné, Thomas Unterthiner, and Sepp Hochreiter. \"Fast and accurate deep network learning by exponential linear units (elus).\" arXiv preprint arXiv:1511.07289 (2015).\n\n[3] Boullé, Nicolas, Yuji Nakatsukasa, and Alex Townsend. \"Rational neural networks.\" Advances in Neural Information Processing Systems 33 (2020): 14243-14253.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7n0cQX1JAqu", "reviewer": "Reviewer_XmXm", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper develops a deep neural network model for a regression problem in high-energy physics. The paper highlights the particular constraints of the application which include high precision, speed, and memory efficiency. The method is compared on dataset from physics against standard neural networks and XGBoost. ", "summary_of_the_review": "While this seems like a nice application of machine learning, there is not enough here to justify publication at ICLR.", "strength_and_weaknesses": "Strengths:\n- The authors use a large dataset generated from high-energy physics. These datasets can be quite useful in comparing ML methods.\n\nWeaknesses:\n- The introduction could be written more clearly. I believe the task is to train a surrugate model that can be used to approximate some function that is traditionally computed using Monte Carlo methods, but the description of the problem and the related work obscure this.\n- This is an example of a regression problem. The authors use this particular application as a case study in the ability to perform regression with neural networks, but ultimately any general claims will need much more supporting evidence (or theory). I don't think this application alone will be of interest to the community.\n- The intput-normaliation strategy certainly seems useful, but is particular to this data, and not a general technique that is of interest to the community. \n- The other modifications including skip connections are standard architectural design choices that are frequently optimized through hyperparameter tuning. Thus, the fact that they lead to improved performance is an interesting data point, but not enough to be of high interest in the community. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "66VUfLHLSX", "reviewer": "Reviewer_P5Fn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors aim to do regression for particle physics that has high precision over the domain of the function. The authors claim that the standard data normalization and regression models are not precise for this application. The authors introduce a normalization method and a fully connected NN with skip connection and claim that their approach is better than the standard treatment for this problem.", "summary_of_the_review": "The authors introduce three properties for a desirable regresson in the beginning of Page 2. However, they do not explain why these specific properties are needed. More particularly, it is not clear what practical implication the high precision property that says errors should be less than 1% over more than 90% of the domain of function has. Where do the numbers 1% and 90% come from? Also, why lightweight property is important?\n\nI could not make sense of the normalization introduced in the paper. I am not even sure what Figure 1 is showing. The authors also failed to show that the new introduced normalized helps improving the performance in practice.\n\nWhy not directly optimizing the relative error? Couldn’t that address the problem with normalization?\n\nFrom Figure 3, it seems that a lower learning rate than 0.01 is needed as the lower values generates better results. Why aren't values lower than 0.01 tested? Similarly from Figure 4 for BDT, it seems that a lower value for max-depth is preferred. Note that for gradient boosting in general, weaker base models such as a decision stump (decision tree with one node) is preferred. I wonder why values lower than 10 is not tested. They later use the max depth of 50 for the baseline model of BDT in Table 2. This is definitely a wrong depth value for boosting models. In the end of the day, all these hyperparameters need to be optimized using the validation set. \n\nI am a bit confused how the data are split. In Section 3.1., the authors say they used 60% for training and 40% for test. Later on the bottom of Page 6, they say they used 48% for training, 32% for validation and 20% for test. Also, I wonder how the authors came up with these specific non-standard splits. Yet again, they change their data split for the baselines in Section “Baselines” in Page 7 to 80% training and 20% test. So much inconsistencies everywhere. \n\nI am not sure why the authors did not test the same values for depth and width for both DNN and sk-DNN. It is hard to compare the results. In any case, it seems that regular DNN performs better than sk-DNN based on the results from Figure 4.\n", "strength_and_weaknesses": "Strengths:\n1-\tUtilizing domain knowledge in ML process\n2-\tMight be a problem of significance importance.\n\nWeakness:\n1-\tUnsupported claims and decisions.\n2-\tLack of novelty\n3-\tPoor experimental study\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject