Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors aim to do regression for particle physics that has high precision over the domain of the function. The authors claim that the standard data normalization and regression models are not precise for this application. The authors introduce a normalization method and a fully connected NN with skip connection and claim that their approach is better than the standard treatment for this problem.
strength_and_weaknesses: Strengths:
1-	Utilizing domain knowledge in ML process
2-	Might be a problem of significance importance.

Weakness:
1-	Unsupported claims and decisions.
2-	Lack of novelty
3-	Poor experimental study

clarity,_quality,_novelty_and_reproducibility: The paper as a whole is relatively easy to follow. However, at places it is not easy to understand the motivation or the introduced approach. The quality of experiments are very poor and I strongly believe that even though the results might be reproducible, it is not supporting the claims in the paper. I have some serious reservations when it comes to novelty. 
summary_of_the_review: The authors introduce three properties for a desirable regresson in the beginning of Page 2. However, they do not explain why these specific properties are needed. More particularly, it is not clear what practical implication the high precision property that says errors should be less than 1% over more than 90% of the domain of function has. Where do the numbers 1% and 90% come from? Also, why lightweight property is important?

I could not make sense of the normalization introduced in the paper. I am not even sure what Figure 1 is showing. The authors also failed to show that the new introduced normalized helps improving the performance in practice.

Why not directly optimizing the relative error? Couldn’t that address the problem with normalization?

From Figure 3, it seems that a lower learning rate than 0.01 is needed as the lower values generates better results. Why aren't values lower than 0.01 tested? Similarly from Figure 4 for BDT, it seems that a lower value for max-depth is preferred. Note that for gradient boosting in general, weaker base models such as a decision stump (decision tree with one node) is preferred. I wonder why values lower than 10 is not tested. They later use the max depth of 50 for the baseline model of BDT in Table 2. This is definitely a wrong depth value for boosting models. In the end of the day, all these hyperparameters need to be optimized using the validation set. 

I am a bit confused how the data are split. In Section 3.1., the authors say they used 60% for training and 40% for test. Later on the bottom of Page 6, they say they used 48% for training, 32% for validation and 20% for test. Also, I wonder how the authors came up with these specific non-standard splits. Yet again, they change their data split for the baselines in Section “Baselines” in Page 7 to 80% training and 20% test. So much inconsistencies everywhere. 

I am not sure why the authors did not test the same values for depth and width for both DNN and sk-DNN. It is hard to compare the results. In any case, it seems that regular DNN performs better than sk-DNN based on the results from Figure 4.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors leverage symmetry principles to design high-precision regressors which enable a dramatic speedup of simulations vs. traditional Monte Carlo approaches.  They demonstrate that BDT performs well in 2D and 4D, but DNN with skip connections are needed for performance in higher (i.e. 8D).  
strength_and_weaknesses: Strengths:
Overall the paper is clear, well motivated, and well written textually (although some clarity around the physics could be provided- see next section).

The authors compare several approaches including BDT, and two NN formulations.


Weaknesses:
 The architectures explored/chosen are not particularly novel.  That's fine if it's not the main focus, but the emphasis seems to suggest that the use of skip connection based architectures is a key insight/contribution.  

Additional clarity around the physics would be helpful (see next section).

The authors briefly reference permutations of external processes and over-completeness in 4.1.  Would using a permutation-invariant architecture help to alleviate this problem?

Demonstrating the performance of the different methods as a function of dataset size could be valuable.  

How would this extend to other calculations?  The authors have demonstrated this is a good method for this calculation by comparing to the MC based outcome, but what parameters could change to give someone implementing this method that this approach is still valid?  MC methods provide error ranges on the estimates.  How would that work here- what is the error around the physical parameter of interest to be measured and how can you "prove" this?

Only two NN architectures are used.  Would using a more "sophisticated" architecture likely lead to a better result for that approach?
clarity,_quality,_novelty_and_reproducibility: The intro and related works is relatively clear, although a bit vague for someone unfamiliar with particle physics expansion.  Is this solving all types of physics problems, or is it looking at a specific decay/scattering/reconstruction/etc. process?  It would perhaps help to put additional details in the appendix to further clarify the functions being expanded/regressed.  Similarly, giving a bit more detail, particularly at the end of section 2 as to how this approach is different would be helpful.  The authors reference they are proposing novel methods for scaling, but novel methods are assumed- can you more specifically state what these are here so the reader has something more tangible in mind as they move into section 3?

4.1 could use additional clarity- it's unclear what these dimensions and functions correspond to physically and whether the values in Table 1 are empirically known, set by the simulator, required by physics, etc.  Is there a choice of the number of dimensions to work in, or is this specified by the given problem (i.e. event to be modeled)?

summary_of_the_review: The overall motivation of the paper is clear- to replace time consuming MC methods with faster ML methods.  However, because specific detail around the problem to be solved isn't given, it's not clear why some of the design choices are made.  The impact of dataset size is an important feature that's worth exploring.  While MC methods are slow, they do offer a "know what you're getting" and ability to quantify the uncertainty; it's not clear to me how certain any specific measurement under the proposed framework is without also comparing to MC at some point (i.e. how well does this generalize to different events)?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses an important problem of building high-precision regressor for particle physics. The paper designs the sk-DNN, which uses residual layer, and with physics-informed normalization for the data as input. It compares vanilla DNN and with baseline of boosted decision trees. The paper shows that for larger dimensional input (4D, 8D), the sk-DNN shows improved performance and speeds up simulations by a factor of 10^3 – 10^6 over the first-principles computations currently used in Monte Carlo simulations.
strength_and_weaknesses: Strengths:

The paper addresses an important problem in particle physics. The physics-informed design of the architecture and data normalization introduces useful inductive bias and helps improving the performance. The speedup is significant which is important in the field.

Weaknesses:

In terms of novelty, w.r.t. the field of machine learning for particle physics, this method may be novel and significant (I'm not an expert in particle physics). On the other hand, in terms of machine learning, the novelty is limited, since the residual connection is a commonly used architecture. The physics-informed normalization is interesting, but may be limited in its generality to more general problems. I'm OK with judging the novelty either way.

The limited novelty may be compensated by a more comprehensive empirical experiments. The authors are encouraged to explore more hyperparameters and compare with more baselines. For example, the authors may explore how activation function affect the performance, since activation is extremely important in regression tasks. In my experience, for regression tasks, ReLU does not necessarily work well since it essentially models a piecewise linear function. The activation of leakyReLU, SiLU [1], ELU [2], Rational activation [3] are typically very good candidate activation for regression tasks, especially rational activation which is able to model sharp changes.


[1] Elfwing, Stefan, Eiji Uchibe, and Kenji Doya. "Sigmoid-weighted linear units for neural network function approximation in reinforcement learning." Neural Networks 107 (2018): 3-11.

[2] Clevert, Djork-Arné, Thomas Unterthiner, and Sepp Hochreiter. "Fast and accurate deep network learning by exponential linear units (elus)." arXiv preprint arXiv:1511.07289 (2015).

[3] Boullé, Nicolas, Yuji Nakatsukasa, and Alex Townsend. "Rational neural networks." Advances in Neural Information Processing Systems 33 (2020): 14243-14253.
clarity,_quality,_novelty_and_reproducibility: Clarity: good

Quality: fair

Originality: good in ML for particle physics, may not be novel in general ML

Reproducibility: fair
summary_of_the_review: In summary, the paper addresses an important problem in particle physics, and achieves good accuracy and significant speedup compared to first-principles computations. The innovation in the ML may be limited, which may be improved with more empirical baselines.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper develops a deep neural network model for a regression problem in high-energy physics. The paper highlights the particular constraints of the application which include high precision, speed, and memory efficiency. The method is compared on dataset from physics against standard neural networks and XGBoost. 
strength_and_weaknesses: Strengths:
- The authors use a large dataset generated from high-energy physics. These datasets can be quite useful in comparing ML methods.

Weaknesses:
- The introduction could be written more clearly. I believe the task is to train a surrugate model that can be used to approximate some function that is traditionally computed using Monte Carlo methods, but the description of the problem and the related work obscure this.
- This is an example of a regression problem. The authors use this particular application as a case study in the ability to perform regression with neural networks, but ultimately any general claims will need much more supporting evidence (or theory). I don't think this application alone will be of interest to the community.
- The intput-normaliation strategy certainly seems useful, but is particular to this data, and not a general technique that is of interest to the community. 
- The other modifications including skip connections are standard architectural design choices that are frequently optimized through hyperparameter tuning. Thus, the fact that they lead to improved performance is an interesting data point, but not enough to be of high interest in the community. 
clarity,_quality,_novelty_and_reproducibility: The clarity of the introduction could be improved. The work is original, but not particularly relevant to ICLR.
summary_of_the_review: While this seems like a nice application of machine learning, there is not enough here to justify publication at ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Reviewers either strongly recommend rejecting the paper (3-1) or are unsure and place the paper in the borderline category (5-6). This paper explores the acceleration of particle physics simulations by building surrogate regressors replacing parts (here, helicity amplitudes) of the MC simulation.  The major concern raised by the reviewers is a clear lack of novelty and significance for the ML community.  Indeed, the paper evaluates traditional regression methods (decision trees, MLPs, MLPs with skip connections) and reaches expected results: DTs perform best in low-dimensional regimes while DNNs perform better in high-dimensional regimes.  The significance in particle physics is also unclear, as the authors do not compare their results to any other previous attempts at surrogate modeling. While the experimental validation is good and optimizing this specific part of the physics simulation is certainly worth implementing, it is difficult to assess whether the approach and conclusions of the paper are generalizable to other problems in ML and science.
justification_for_why_not_higher_score: Lack of novelty and significance for ML.
Decision: Reject