Neural networks can be FLOP-efficient integrators of 1D oscillatory integrands
Authors: Anshuman Sinha
Spencer H Bryngelson
Venue: Accepted by TMLR ()

Abstract: We demonstrate that neural networks can be FLOP-efficient integrators of one-dimensional oscillatory integrands. We train a feed-forward neural network to compute integrals of highly oscillatory 1D functions. The training set is a parametric combination of functions with varying characters and oscillatory behavior degrees. Numerical examples show that these networks are FLOP-efficient for sufficiently oscillatory integrands with an average FLOP gain of $10^{3}$ FLOPs. The network calculates oscillatory integrals better than traditional quadrature methods under the same computational budget or number of floating point operations. We find that feed-forward networks of 5 hidden layers are satisfactory for a relative accuracy of $10^{-3}$. The computational burden of inference of the neural network is relatively small, even compared to inner-product pattern quadrature rules. We postulate that our result follows from learning latent patterns in the oscillatory integrands that are otherwise opaque to traditional numerical integrators.

Published in Transactions on Machine Learning Research (03/2024)
Neural networks can be FLOP-efficient integrators of 1D oscillatory integrands
A Anshuman Sinha School of Computational Science & Engineering, Georgia Institute of Technology, Atlanta, GA
anshs edu
Spencer H. Bryngelson
shb @gatech edu
School of Compulational Science & Engincering, Georgia Institute Technology, Atlanta, GA
Woodruff School of Mechanical Engineering, Georgia Institute of Technology, Atlanta, GA Guggenheim School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA
Reviewed on OpenReview: https: openreview. net/ formm? id= 5dsggEEElt
Abstract
We demonstrate that neural networks can be FLOP-efficienl integrators of one-dimensional oscillatory integrands. We train feed-forward neural network to compute integrals of highly oscillatory 1D functions, The training set is parametric combination of functions with varying characters and oscillatory behavior degrees. Numerical examples show that these networks are FI.OP-efficient for sufficiently oscillatory integrands with an average FLOP gain of 10x FLOPs. The network calculates oscillatory integrals better than traditional quadrature methods under the same computational budget or number of floating point operations. We find that feed-forward networks of hidden layers are satisfactory for relative accuracy of 10-3 The computational burden of inference of the neural network is relatively small, even compared to muner-produc pattern quadrature rules. We postulate that our result follows from learning latent patterns in the oscillatory integrands that are otherwise opaque to traditional numerical integrators.
1 Introduction
Numerical integration of highly-oscillatory functions required for problems fluid dynamics, nonlinear optics, Bose-E Eineteinstinn condensates, celestial mechanics, computer tomography, plasma transport, and more (Connor & Curtis, 1982; Tserles, 2005). Classical numerical integration schemes are based quadrature rules, like those of Newton Cotes type (e.g., trapezoidal or Simpson's rule), Romberg integration, Gauss quadrature (Davis & Rabinowitz, 2007; Milne, 2015; Hildebrand, 1987). These are unsuited for highly oscillatory integrands, requiring many quadrature points before reaching their asymptotic convergence rates. This work uses food-forward, fully connected noural networks as approximate integrators for highly oscillatory integrands. Focus is paid to the floating point operation -efliieef of different integrators: Can feed-forward neural network outperform classical integration schemes for given FLOP budget?
2 Background
Numerical integral methods crafted for highly oscillatory integrands have been developed. These are, for example, based on the stationary phase approximations (Filon, 1930; Levin & Sidi, 1981; Levin, 1996; Iserles & Nørsett, 2005; Evans & Chung, 2007; Hascelik, 2009). Each method is powerful when used appropriately but operates under relatively strict conditions, including the type of oscillatory features (c.g., sine and cosine). These methods can be generalized when more sophisticated algorithms are added to the integration technique to identify suitable basis for integration. Though the methods for this identification not well documented

---

Published in Transactions on Machine Learning Research (03/2024)
y4
A/2
1/2
y1
f(a)
81 222 423
244
82
Quadrature points
b1
by
(a)
(b)
Figure 1: (a) Newton-Cotes- method of approxinating integral with a (b) foed-forward multi-layer perceptron neural network (NN). Thee model uses inputs (x:), where the peeudo-uuadrature points 2; are fixed in the spatial domain, compute the weights and biases of the neural networks, thus approximating I.
in the literature or any open-source software, their use or evaluation of the additional computational cost associated with them is challenging to align with our objective.
More general techniques for dealing with highly oscillatory integrands are desirable. Neural networks are one such possibility for generalization. Indeed, neural networks have been used to approximate integrals before. Previous works used single hidden layer neural networks for PID controller applications (Zhe-Zhan et al., 2006), and dual neural networks with application to material modeling (Li et al., 2019). These neural networks were particularly useful in many-query settings but do not appear to generalize beyond their specific applications.
Some works have taken a tailored approach to neural network design for integration. Ying Xu Jun Li (2007) usod oscillatory basis functions for interpolation and integration, though the cosine activation function prevents approximation to broader function classes (Wu, 2009). Lloyd et al. (2020) trained single-layer networks for multidimensional integrals, focusing on parameterized, many-query problems, and showed promising results. The main limitation of Lloyd et al. (2020) is the restriction to high-dimensional many-query problems for FLOP- -.0P-efficinny. This work investigates the PLOP-efficience point operation, for integrating 1D functions via a feed-forward fully connected neural network. We aim to provide a computationally efficient solution problems that require repeated integration of an integrand with varying parameters.
integration accuracy per required floating
2.1 Problem definition
The formulation to compute the integral I of any function f(x) in a bounded domain [81, s2] which is expressed as follows,
-=
f(x)dæ.
(1)
The method of fig. 1, where the inputs of the network shown as f(x)) for fixed set of x; [s1, s2], and the outputs of the network are the integral values I of the input function.
We train the neural network model with parametrically varying samples. These neural network integrators are compared with the classical numerical integration methods like Newton -otens quadrature rule, as shown in oq. (2). We do not consider more complex integration schemes like Gauss quadrature as they do not reach asymptotic convergence behaviors for realistic numbers of quadrature points and typically become unstable before this.
2

---

Published in Transactions on Machine Learning Research (03/2024)
The imput domain [s1, s2] is divided into ny quadrature points. The integral is approximated weighted sum of function values at those quadrature points (14).
|
82 [(r)dd a
wi (x)(
(2)
Common Newton Cotes formulations include the second-order accurale trapezoidal method
[
92
f(*x) + f(0x+1) Av.
f(x)dæ
(3)
2
kml
where Ty are uniformly spaced points in the domain x E [81, and Ar = (82 -81)/ the first-order accurate is the mid--oinnt method
| f(x)dæ Σ 81 -ー //2 Ax Dr.
(4)
and the third-order accurate Simpson's method
J f(x)dæ &
(f(#2x-2) + 44((222-1) f(zz)) Ar,
(5)
3
where, again, f(+) is the integrand of interest.
2.2 Neural network method
The network's learned weights reduce computational costs. With optimized architecture, seek accurate integration using few input (quadrature) points. Improved accuracy for given number of quadrature points is feasible as the neural network has trainable weights for approximation, but classical techniques rely on fixed interpolating functions. For such methods, the more ascillatory function, the more quadrature points are required to achieve integration of the same accuracy.
In our notation, the values y of an integrand f(x) abscissa 24 (y f(x,) i I....,n) are shown in fig. 1. The paramcters
ai =
Σ wijais + b;
(6)
form each layer of fig. 1, Zij represent the information from previous layers (i.e, agz-1))) and b, the bias term in each layer. In the final layer of fig. 1, the of each input to final node (E) and the output to train the network via. back-propagation. ReLU is used as the activation function. The number of hidden layers and the number of neurons are optimized during hyperparameter optimization. Section 3.4 shows the results for the hyperparameter optimization for the model.
3 Experiments
We evaluate the proposed neural network method with several example cases. We present the employed dataset, metric of evaluation, and other details on the hyperparameter study, then the results.
3.1
Experimental dataset
3.1.1
Training data
The model inputs the function values and the bascline integral value as output in the current work. The training is done by varying the function's parameters while keeping its nature the same. While training for
3

---

Published in Transactions on Machine Learning Research (03/2024)
Train loss
Test loss
10
Normalized MSE
10-2
10
10a
10a
10
104
10
10a
10
10a
Epochs
Epochs
(a) 5 layers, 3 neurons layer
(b) 5 layers, 4 neurons per layer
Figure 2: Neural network (NN) model's performance on test and training dataset.
network configuration of the model, we keep the number of quadrature points fixed. We trained separate models for numbers of imputs ranging from 2 to 2 and evaluated their performance separate test data. These inputs (with varying quadrature points) were tested on the specific network configuration, yielding results with varying validation accuracy. For fixed number of quadrature points, the input values of the model are the function's 's values fx; at those quadrature points Z4-
3.1.2 Testing data
The model is tested by calculating the integral of a function that is the same kind as the training functions but paramctrically different and unseen. The integral is calculated within the same input space, i.c., within the same limits of the imput domain [81, 82]. The training data are divided into disjoint subsets, with an 80-10 10-10 split for train, test, and validation.
3.2
Evaluation
We evaluate the performance of the neural network model using the normalized mean squared error (Normalized MSE) metrics on the test set. The error in the model's result (īs) is computed against a represcntative exact solution to the integral (Tx), which is evaluated using the trapezoidal method with 213 quadrature points. This strategy is sufficiently accurate for our purposes, which is checked against more and fewer such baseline quadrature points. We use the surrogate truth integrand I& for the k'th sample in the test set as
Fie 211
f(z,) + +((()))) Ar. (f()))
I = Σ
(7)
2
where ny 23 for Ar (s2 1 while calculating the integral of f(x) (the integrand) between [s1, s2] using the trapezoidal rule in eq. (3).
The normalized mean square error (MSE) for the test set is calculated using the m samples of functions is
Normalized MSE 1 (Ix T₂)²
m
12
These m samples are generated by parametric variations of the same function, as described in table 1. We evaluate the performance the neural network model by comparing it to standard numerical integration methods, such as the trapezoidal and midpoint methods.
3.3 Hyperparameter optimization
The neural network's architecture was optimized according to a test-sct error value below 10 and minimizing fixed number of FLOPs. The validation data are used for hyperparameter optimization. Table shows the optimization results.
4

---

Published in Transactions on Machine Learning Research (03/2024)
Table I: Result of the neural network model on various oscillatory functions. The normalized FLOP gain for given accuracy is a, which serves as measure of performance (higher is better) and is computed with respect the classical quadrature-basee model as shown in eq. (9). All functions are 1D in æ.
Function Type
Equation
Parameter space
Result
Bessel(k, 2)
cos(kx))ob(p
v e [125, 175], & [75, 125]
6.01
Figure 4 (a)
Svan-Webster-1(6e (11, k2)
cos(k1 ([) sin (k2aa)
h1 E [5, 15), A2 [25, 75]
17.72
Figure 4 (b)
Rayleigh-Plesseee Plosset(p)
See appendix A.3
P [500, 1000)
23.46
Figure (c)
Evan-Webseee (k)
exp(x) sin(k cosh(
& [25, 75]
19.60
Figure (d)
Sine(k)
(in():))
& [5, 15]
0.91
Figure 6 (a)
Exponential(k)
exp(kx)
& [1, 5]
0.60
Figure (b)
0.04
c0s(100z))Jo(1700)
10
exp(x) sin co0h (x))
0.02
cos (100x)))o)
exp(1) (r) sin (10 cosh (x:)
0
0.02
-10
-0.04
1.2
1.4
1.8
1.5
(a) Bessel-type
(b) Svan-Websters
Figure 3: Example canonical oscillatory test functions as labeled.
The architecture was optimized heuristically with an iterative increment of network hidden layers and neurons in each layer. Each configuration was executed using the model until the training normalized MSE stagnated. An example of this convergence is shown in fig. 2, along with the lesting and training performance for two separate network configurations. The performance of each network configuration datasct, which was scparate from the training data. The iterative tests were done with increasing samples as the parameters of the network increased (Bishop, 1995). The upper limit complexity of the network is obtained via the performance of the classical integrators. With the increasing complexity of the network, we observe diminishing returns against the FLOP burden. deeper neural network can still achieve smaller integration errors. We aim achieve the highest accuracy for a given computational expense while avoiding overfitting.
evaluated validation
3.4 Results
The results of this study present the current model's applicability for efficiently computing and predicting the integrals of oscillating functions. The model is tested on oscillatory and non-oscillatory functions.
Table compares results for various functions with different oscillatory features. The degree of oscillatoriness is defined by the function's parameters, which are denoted in parentheses in the "Function Type" column of table 1. A larger coefficient corresponds to a more oscillatory function. The parameter space colurn is the range in which the function's parameters vary to create the training data. The gain in the number of FLOPs, shown in eq. (9), defines the computational advantage obtained for the number of required floating point operations while implementing the current method over the other numerical-based methods of computing the integral. The term FLOPPNN represents the number of FLOPs required by the neural network model, and FLOPsQM represents the number of FLOPs required by classical integration quadrature based model.
The number of FLOPs for the neural network method is calculated by enumerating the floating point operations involved in computing the output. of each neuron in every layer. This computation follows the
5

---

Published in Transactions on Machine Learning Research (03/2024)
Trapezoidal Midpoint
Simpson
Neural Net.:
Neural Net.: 4
10a
10
Normalized MSE
10
10-1
Bessel
(b) EW1
10-3
10-7
10a
10
Normalized MSE
10-2
10
(c) Beyyligh-Plessee
(d) EW2
10-8
mill
10 7
10
10a
10a
10a
10
10c
107
10
10a
10a
10
10a
10
107
Number FLOPs
Number of FLOPs
Figure Comparison of results of neural network (NN) model with Newton Cotes methods for the oscillatory functions of table 1. The integrands associated with panels (a) and (b) are as shown in fig. 3.
formula given in eq. (6). For an H layer fully-comneede feed-forward neural network with each layer having N neurons, each activation asy comprises N multiplications and addition operations. Given N operations per layer and II layers, the total FLOP count is (4N 2) H1 ng)- The number of FLOPs associated with the trapezoidal, mid-point, and Simpson's 's methods are 2ng + 1, 3ng + 1 and 4.5m, following eqs. (2), (4) and (5).
To study the influence of the oscillatory nature of the input function on the results, we performed integral calculations for the function while gradually increasing its oscillatory behavior. We tested the model on sinusoids with varying frequencies (increasingly oscillatory). The measure of neural network integration efficiency is the normalized ratio of FLOP gain for noural network model over traditional integration for the same MSE error, a:
FLOPsNN - FLOPSQM
FLOPSNN
A larger corresponds to a higher normalized gain by the neural network model.
fig. 3 shows the results of the model's performance by providing examples of individual functions. fig. 4 shows the test set's normalized mean square error (NMSE) in integral computation from the model as a function of the number of FLOPs required compute the individual integral. Figure shows the results of normalized MSE loss values for different numbers of quadrature points as input training data points for the model.
Table 2 shows the results for the two best perforerf network configurations (neurons hidden layers) achieved through parametric optimization. The number of FLOPs required for computing the integral increases as the number of quadrature points increases. Increasing quadrature points also increases the accuracy of the integral computation for both methods. For a normalized MSE of 10-3, the neural network strategy computes the integral using fewer FLOPs than traditional quadrature methods, making it FLOP-efficient.
We observe the opposite result for the less oscillatory functions of fig. 5. Figure shows that the neural network model requires more FLOPS to compute the same integral for given normalized MSF This result.

---

Published in Transactions on Machine Learning Research (03/2024)
10
sin(52)
exp(x)
0.5
sin( 10zz)
8
exp(2xc
6
4
00.5
2
0.4
0.6
0.8
0.2
0.8
(a) Sine-type
(b) Exponential-type
Figure 5: Example test functions tested as labeled.
Trapezoidal
Simpson
Neural Net.: x
Midpoint
Neural Net.:
10a
10
Normalized MSE
10-2
10
10
10-12
10
10
10a
10a
10a
10a
10
107
10a
10
10a
10
10a
10
107
Number FLOPs
Number FLOPs
(a) Sine-type
(b) Exponential-type
Figure 6: Results of the NN model with traditional numerical integration methods for non-oscillatory (a) sine and (b) exponential functions shown in fig. 5.
Table 2:
Model hyperparameter selection space. The optimum values are chosen based on the normalized MSE. The sample number, learning rale, and rest-train split are fixed.
Hyperparameter
Search space
Optimum value
Number hidden layer
(1,2,3,4,5)
Hidden layers
Neurons each hidden layer
(1,2,3,4,5,, 4,6,
5 Neurons
Number of samples
110?, 103, 10*)
10 Samples
Learning rate
{10-5, 10-5,1014,10 10-33
10-4
Test-train split
(0.1,0.15,02) 0.2]
0.15

---

Published in Transactions on Machine Learning Research (03/2024)
Compared
Trapezoidal
Mid-point
Simpson
10
Performance gain
10a
(a)
Evan
Webster-1
(b) Bessel
(c) Sine
10-1
50
100
150
200
0
50
100
150
200
50
100
150
200
Oscillatoriness (A)
Figure 7: Performance gain via the neural integrator for increasingly oscillatory integrands labeled which is parametrically increased according to their parameters shown in table 1.
is expectod, as these integrands can be computed accurately with only few quadrature points using low-cost. Newton Cotes methods.
Figure 7 shows that the performance gain increases rapidly with the increasing oscillation of the integrand. This indicates that the numerical integral methods require more quadrature points to evaluate the same integral. After some level of oscillation has been reached, the performance gain curve starts to plateau. Even if one adds more oscillations to the domain (by increasing the oscillatory parameters), the integral value within the sub-domain is the same. Thus, no further gain in performance is expected. For less oscillatory integrands, the values of a decay unity. The neural network integral then requires more FLOPs than a classical integrator.
The summary of the comparative study on various functions is presented in table 1. The number of FLOPs for this comparison was calculated for fixed Normalized MSE value of 10 for the loss function. This value was chosen to remain within the application limits for downstream integration use. Overall, section 3.4 shows a general trend of decrcasing normalized MSE in the integral as the complexity the network increases. The neural worw model predicts the integral result with the same accuracy but exhibits nearly two orders of magnitude better efficiency.
4 Discussion and conclusion
An approach for computing the integrals of 1D functions of highly oscillatory and non-oscillatory behaviors. We used food-forward neural network to estimate the integrals and evaluated their accuracy for a fixed FLOP budget. In a comparative study, several cases of oscillatory functions were examined. The approach of calculating integrals with neural network model outperforms existing numerical methods for fixed FLOP budget. The neural network model was an incrcasingly efficient integrator for functions with more oscillatory behavior. The proposed method does not extrapolate to qualitatively different integrands than it was trained on, mostly owing to the restriction of FLOP efficiency. Thus, our results are most applicable to cases where parametric variations an integrand must evaluated in a many-query setting, and one can forecast the integrand characteristics.
Acknowledgement
The authors appreciate discussion with Dr. Ethan Pickering at an early stage of this work. SHB acknowledges support of the US Office of Naval Research under grant no. N00014-22-2211 (PM Dr. Julie Young).
References
C. M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995.
S. H. Bryngelson and T. Colonius. Simulation of humpback whale bubble-net feeding models. Journal of the Acoustical Sociely of America, 147 1135, 2020.

---

Published in Transactions on Machine Learning Research (03/2024)
S. H. Bryngelson, R. O. Fox, and T. Colonius. Conditional moment methods for polydisperse cavitating flows. Journal of Compulational Physics,
2023.
J. N. L. Comnor and P. R. Curtis. A method for the numerical evaluation of the oscillatory integrals associated with the cuspoid catastrophes: Application to Pearcey's integral and its derivatives. Journal of Physics A: Mathematical and General, 15(4):1179, 1982.
P. J. Davis and P. Rabinowitz. Methods of Numerical Integration. Courier Corporation, 2007.
G. A. Evans and K. C. Chung. Evaluating infinite range oscillatory integrals using generalised quadrature methods. Applied Numerical Malhemalics, 57(1):73 79, 2007.
L. Filon. III. On a quadrature formula for trigonometric integrals. Proceedingss of the Royal Sociely of Edinburgh, 49:38 47, 1930.
Hascelik, A. On numerical computation of integrals with integrands the form ((z)sin(warr) on [0, 1]. Journal of Computational and Applied Mathematics, 223(1):399 408, 2009.
F. B. Hildebrand. Introduction to Numerical Analysis. Courier Corporation, 1987.
A. Iserles. On the numerical quadrature of highly-oscillating integrals 11: Irregular oscillators. IMA Journal of Numerical Analysis, 25(1):25-44, 2005.
A. Iserles and S. P. Nørsett. Efficient quadrature of highly oscillatory integrals using derivatives. Proceedings of the Royal Sociely A: Mathematical, Physical and Engineering Sciences, 461(2057: 1399, 2005.
D. Levin. Fast integration of rapidly oscillatory functions. Journal of Compulational and Applied Mathemalics, 67(1):95-108 1996.
D. Levin and A. Sidi. Two new classes of nonlinear transformations for accelerating the convergence of infinite integrals and series. Applied Mathematics and Computation, ((3):175-200 2555,, 1981.
H. Li, Y. Li, and S. Li. Dual neural network method for solving multiple definite integrals. Neural Computation, 31(1):208 232, 2019.
S. Lloyd, R. A. Irani, and M. Ahmadi. Using neural networks for fast numerical integration and optimization. IEEE Access, 8:84519 84531, 2020.
W. E. Milne. Numerical Calculus. Princeton University Press, 2015.
Milton S Plesset and Andrea Prosperetti. Bubble dynamics and cavitation. Annuull review of fluid mechanics, 9(1 9(1):145-185, 1977.
H. Wu. Global stability analysis of general class of discontinuous neural networks with linear growth activation functions. Information Sciences, 179(19))3432-3444 -4441, 2009.
L. Ying Xu and L. Jun Li. The numerical integration algorithm based neural network. In IEEE Third International Conference on Natural Computation (ICNC 2007), volume 1, pp. 325 328, 2007.
Z. Zhe-Zhao, W. Yao-Nan, and W. Hui. Numerical integration based on a neural network algorithm. Computing in Science & Engineering, 8(4):42-48, 2006.

---

Published in Transactions on Machine Learning Research (03/2024)
A Appendix
A.1 Neural network memory costs
For fully connected neural network with L layers of N neurons, each with a 4-byte floats parameterization, the memory footprint in bytes, ignoring imput layer biases, is
-L N(L -2),.
(10)
for (L layer), and 4 bytes per float. Thus, the memory footprint for 5-layer, 3-neuron network is 180 bytes, which is negligible in almost any practical computing environment.
( 1) weights, (LL-2) 2) biases (each neuron in the hidden and output layers, excluding the input
A.2 Training dataset generation
The general data generation process for training is
y(x) = f(k, x),
(11)
where y(a)) is the function generated for a set of æ values with parameter k. The total number of æ for a simulation defines the total number of samples. Here, & is the randomly selected parameter associated with the oscillatoriness of the integrated for each æ, (2),
Param.
Value
Description
Units
Ap
7670
Ambient pressure difference
Pa
p(t)
1.3 x 10 (33000))))))
Driving pressure
Pa
0.0725
Surface tension
N/m
/1
8.9 x 10-4
Dynamic viscosity
Pa
k
1.33
Polytropic exponent
Bayleigh-Plesset Table 3: Parameterization of the Rayleigh Plesset equation.
A.3 Plessigh-Plesset dynamics
The Rayleigh Plesset equation represents the oscillatory radial dynamics of gas bubbles suspended in a liquid. The equation second-order ODE and solution is highly oscillatory and dominated by nonlinear behavior in many regimes. Specifically, it can be expressed as
RR
3
dR 2) dt )
2o
4p dR
20 Ro
Ro R
3k
P
R di2
Ap p(1)
Ap
(12)
2
R
R dt
dR
R(0) = Ro.
dt (0) = 0,
where the bubble radius R is the dependent variable and time E [0, is the independent variable. Equation (12) is buttressed by initial conditions for bubble radius Ro 2.6 10-6 (all following quantities in SI units) and zero initial radial velocity of the spherical bubble. The cocfficients in eq. (12) are expressed in table 3. Elaborating on their meaning: the ambient pressure difference represents the initial discontinuily between the pressure inside the bubble and the pressure in the suspending liquid, the driving pressure how the pressure of the liquid surrounding the bubble changes time, the surface tension coefficient a standard quantity representing the proclivity for one fluid want adhere another, the dynamic viscosity is the resistance of the suspending liquid to changes in motion due to surrounding change in velocity, and the
10

---

Published in Transactions on Machine Learning Research (03/2024)
polytropic coefficient serves mediating rate for which the gas inside the bubble is able to expand contract.
We generate an arbitrarily varying set of oscillatory data by varying the ambient density of the suspending liquid as p E [500, 1000). Equation (12) is integrated over the time interval, which is accomplished using a traditional numerical differential equation solver with a suitably small time step that captures the short. time scales occurring at solutions cusps (corresponding to bubble size decay to growth). The solution is oscillatory in the regimes tested, but integrals of it required in the context of underwater hydrodynamics broadly (Bryngelson et al,, 2023; Bryngelson & Colonius, 2020; Plesset & Prosperetti, 1977).
11


Reviews:
[{"review_id": "hQJt0F4Wba", "reviewer": "Reviewer_Ua35", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- The idea is very simple but worth a careful examination. The authors take care of analyzing the benefits of their idea in terms of practical considerations, namely FLOPs for a given accuracy.\n- The paper delineates clearly some potential limitations of the approach which can help the practitioner decide which method to use.\n- The methodology is relevant and rather well-explained.\n\n**Weaknesses**:\n- There is a lack of comparison to other methods. Since the idea is particularly simple, it is important to study alternatives that have been proposed in the literature. The argument against previous methods are clear (lack of expressivity of cosine activation for example in Y. Xu and J. Li's paper). However it'll be better to have experiments justifying such claims.\n- Again while the methodology is simple, there is a also a technological opportunity to seize here that the authors could develop further. For example, it would be relevant to have access to the code of the authors to either use it as an alternative to classical integrators or to build one's own neural network to get trained integrators. \n- Overall the paper may be better formatted, see some suggestions below (the current manuscript is ok though and the suggestions below are just suggestions up to the authors.).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the efficiency of neural-network-based integrators compared to classical numerical procedures to integrate oscialtory one-dimensional functions. The idea consists simply in training multi-layer perceptrons to return accurate integrals of functions given as inputs a set of function values at given quadrature points. The authors delineate the potential of neural-network-based methods for highly oscillatory functions in terms fo flops and also point out potential limitations of the approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Desired changes**:\n- Add comparisons to earlier work.\n- While FLOPs are taken into account, a discussion about memory costs would be relevant. In particular it is not clear how to use the proposed approach if the network is trained on, say, [0, 1], and the function needs to be integrated on [10^3, 10^6].\n\n**Suggested changes**:\n- Present alternative architectures:\n  - Change the ReLU to e.g. the GeLU or other activation functions?\n  - Use convolutional layers or event transformers?\n  Overall as the idea is simple, its execution could be more extensive.\n\n**Minor presentation details**:\n- In the experiements section, the forms of the function considered should appear as close as possible to the paragraph 3.1.1 training data. \n- A full pipeline explained in e.g. pseudocode or with a figure needs to be added to understand the methodology more easily than by reading the text.\n- Use a formula in the table, or anything that speaks more than \"alpha\". Also why is alpha normalized by the performance of the network and not the performance of the classical method?", "broader_impact_concerns": "Unforeseen.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OwfytDnsOT", "reviewer": "Reviewer_Eijb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. The idea is well articulated, and the proposed approach is clearly explained.\n2. Explanation of experiments is informative.\n3. Most claims are well supported.\n\n# Weaknesses\n\n1. Motivation is not clear.\n2. It is questionable that the benchmarks are realistic.\n3. Baselines are too weak.\n4. No generalization study is available.\n5. Inductive bias of the model does not seem to be correct.\n6. Inaccuracies.\n7. Not reproducible.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose to compute integral using a fully connected neural network. More specifically, the method consists is as follows:\n1. For a selected family of functions, one gathers a dataset of the form $\\\\left\\\\{\\\\left(f_i(x_1),\\\\dots,f_i(x_N)\\\\right), I[f]\\\\right\\\\}$ where $I[f] = \\int_{a}^{b}dx\\\\,f(x)$.\n2. For this dataset, a feedforward neural network is trained to predict $I[f]$ from discrete values $\\left(f_i(x_1),\\\\dots,f_i(x_N)\\right)$\n3. The resulting network is applied to unseen functions from the family.\n\nThe approach is tested on integrals for highly oscillatory functions, and it is shown that a trained neural network can evaluate integrals more efficiently (less FLOP for a given accuracy) than classical low-order methods such as trapezoidal and midpoint rules.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Motivation is not clear.\n\nSince, for the proposed approach, one needs to collect a dataset for each new family of functions (also, see 4. below), the scope of applications is not entirely clear. We kindly ask the authors to expand the discussion of the applications and/or the formulation of the problem (e.g., loss minimization over a distribution of functions). Note that for the classical methods, the situation is clear: once they are derived, it is possible to apply them to many different problems. The situation is quite different for neural networks, so it warrants a more detailed explanation.\n\n---\n2. It is questionable that the benchmarks are realistic.\n\nIt is not entirely clear, but it seems the method is tested on four families of functions (page 5, Table 1): \"Bessel type,\" \"Evan–Webster type,\" \"Sine,\" and \"Exponential.\" The spaces of parameters for these functions are small, e.g., for \"Bessel type,\" authors listed $100$ distinct parameters, and for \"Sine,\" only $10$. It is not clear whether the families used are broad enough. It is also not clear which functions the authors actually used since Figure 3 and Table 1 (both on page 5) contain conflicting descriptions and the code is not available.\n\nI suggest authors to include:\n1. discussion on the complexity of datasets, the frequency content of the functions included\n2. tests on higher frequencies where asymptotic methods shine\n3. more accurate description of the functions and their parameters used to generate datasets\n\n---\n3. Baselines are too weak.\n\nThe authors mention in the article that classical numerical schemes are not suitable. (page 1, Introduction)\n\n> Classical numerical integration schemes are based on quadrature rules, like those of Newton–Cotes type: the trapezoidal and Simpson’s rules, Romberg integration, and Gauss quadrature (Davis & Rabinowitz, 2007; Milne, 2015; Hildebrand, 1987). These are unsuited for highly oscillatory integrands, requiring many quadrature points before reaching their asymptotic convergence rates.\n\nStill, after that, the only baselines used are the Trapezoidal and Mid-point methods. If the methods are unsuitable, why use them as baselines? I agree that special integration techniques developed for highly oscillatory functions are not generally applicable, i.e., suitable only for particular parametric integrals. Still, I see no reason to discard them as baselines. For example, [1] contains methods applicable to a broad family of highly oscillatory functions.\n\n---\n4. No generalization study is available.\n\nArguably, the method is only useful, if it works for a broad class of problems after training on a small subset of problems. The generalization study is absent from the article. The study I have in mind may include answers to the following questions:\n\n1. Suppose for particular functions we trained NN to approximate integrals of functions with oscillation frequencies $[w_1, w_2]$. What if we test the network on lower or higher frequencies? How does relative error depend on the difference between the frequency content of train and test sets?\n2. Suppose we train on one family of functions and evaluate on another family of functions. What is the relative error of integration?\n3. How does the relative error depend on the number of samples in the training set?\n\netc\n\n---\n5. Inductive bias of the model does not seem to be correct.\n\nIntegral is a linear operation, i.e., $\\\\int\\\\,dx\\\\left( f_1(x) + f_2(x)\\\\right) = \\\\int\\\\,dx f_1(x) + \\\\int\\\\,dx f_2(x)$. When a neural network is used to approximate the integral (ultimately, this is related to interpolation) there are at least two options:\n\n1. $\\\\text{NN}(f) \\simeq I[f]$\n2. $\\\\text{NN}(f) = w,\\\\,I[f] = \\sum_{I}w_i f(x_i)$\n\nOption 2. has correct inductive bias, while option 1. leads to the approximation of a linear operator with a nonlinear one. I suggest authors discuss why they use 1. and not 2. or, better, to compare the accuracy and FLOPs of 1. and 2. Note that one can enforce consistency with 2. as explained in [2] (for interpolation problem).\n\n---\n6. Inaccuracies.\n\nBelow is a list of some minor problematic parts.\n\n1. (page 3) Part below is not supported by experiments or theory.\n\n> The network’s layers will approximate an intrinsic structure in the highly oscillatory integrands. This way, it decomposes the function’s highly oscillatory and complex nature into sub-functions that can be easily interpolated. This way, we can find the best-fitting integral with fewer data points, i.e., fewer input points.\n\n2. (page 4) Normalized mean squared error is unusual. As a rule one consider average $\\\\frac{1}{m}\\\\sum\\_{k=1}^{m} (I\\_k - \\\\hat{I}\\_{k})^2\\\\big/ I_k^{2}$.\nWhat is the reason authors used  $\\\\sum\\_{k=1}^{m} (I\\_k - \\\\hat{I}\\_{k})^2\\\\big/ \\\\sum\\_{k=1}^{m}I\\_k^{2}$?\n\n3. (page 5) Figure 3 is not clear:\n\n    1. It does not agree with Table 1 in the definition of \"Bessel-type\" and \"Evan–Webster-type\" since expressions in the legends are not the same as expressions given in the table.\n\n    2. What is the $f(cos(x))$ in the legend for \"Bessel-type\"?\n\n4. (page 6) The number of FLOP for NN seems to be incorrect. If I understood the architecture, the main computations occur in the first layer since the number of points used can be up to $2^13$. This is not reflected in the expression which gives the number of operations in terms of neutrons in hidden units.\n\n---\n7. Not reproducible.\n\nCode and datasets or functions used to generate them are not available.\n\n---\n\n[1] -- S. Olver, Numerical Approximation of Highly Oscillatory Integrals.\n\n[2] -- Y. Bar-Sinai, et al, Learning data-driven discretizations for partial differential equations.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DcxAdoqOCM", "reviewer": "Reviewer_vUXA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The general question, whether standard numerical methods can be improved/complemented by novel tools from ML is interesting. Quadrature is a core task in many numerical problems, hence there might be interest in such \"learned quadrature rules\".\n\nWeaknesses\n* Variety of datasets: The authors consider only 4 very special sets of functions, and from my reading of Table 1, these classes each only contain the following number of samples: \n    - 101 (class \"Bessel\")\n    - 11*51 = 521 (class \"Evan-Webster\") \n    - 11 (class \"Sine\") \n    - 5 (class \"Exponential\")\n\n    Based on experiments with such limited datasets, it is not clear, whether/in which applications such a model could be practically useful. It is not clear whether the learned model generalizes beyond its (narrow, i.e. 1- or 2-parameter) trained class. As such, I would question whether this work is of general interest to the TMLR readership.\n* Baseline: The authors compare accuracy vs. FLOPs, to argue that neural networks can outperform standard numerical quadrature methods. However, the baseline quadrature here are the trapezoidal and mid-point rules, which are relatively low-order.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to train neural networks to predict the integral of oscillatory functions. The input of the model are grid-point values of the underlying function, the output is the value of the integral of this function.\n\nThe authors consider different families of such oscillatory functions, each parametrized by one or two parameters. An example is $sin(kx)$ with parameter $k$. They argue that (fully connected) neural networks can be flop efficient compared to standard numerical quadrature methods (Newton-Cotes).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "* For this submission to be of interest to a wider readership, the experiments would have to be considerably extended, beyond the simple function families considered in this work. For the Sine-family, Figure 5(a) suggests that the family is actually not very oscillatory. How far can one push the values of $k$?\n* An important question that is not addressed is whether such models, trained on a simple model class, can generalize beyond this model class. This would be crucial for practical applications.\n* I believe that with the specific integer-valued parameters considered by the authors, the oscillatory integrands lead to very regular integrals as a function of the parameters. The authors should include a plot of the integral $I=I(k)$ or $I=I(k_1,k_2)$ as a function of $k$ and $(k_1,k_2)$ for the various families, respectively. \n* I suggest, the authors should extend their baselines; Would the trained NN be FLOP-efficient compared to Simpson's rule?", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I agree with the reviewers that the claims are sufficiently interesting and supported well enough to justify accepting the paper for TMLR.
Decision: Accept (poster)