Smooth Mathematical Functions from Compact Neural Networks
Authors: INGI HONG
Venue: Submitted to ICLR 2023 (2023)
Keywords: regression
function approximation
mathmatical function
compact model

Abstract: This is paper for the smooth function approximation by neural networks (NN). Mathematical or physical functions can be replaced by NN models through regression. In this study, we get NNs that generate highly accurate and highly smooth function, which only comprised of a few weight parameters, through discussing a few topics about regression. First, we reinterpret inside of NNs for regression; consequently, we propose a new activation function--integrated sigmoid linear unit (ISLU). Then special charateristics of metadata for regression, which is different from other data like image or sound, is discussed for improving the performance of neural networks. Finally, the one of a simple hierarchical NN that generate models substituting mathematical function is presented, and the new batch concept ``meta-batch" which improves the performance of NN several times more is introduced. The new activation function, meta-batch method, features of numerical data, meta-augmentation with metaparameters, and a structure of NN generating a compact multi-layer perceptron(MLP) are essential in this study.

Under review as a conference paper at ICLR 2023
SMOOTH MATHEMATICAL FUNCTION FROM COMPACT NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
This is paper for the smooth function approximation by neural networks (NN). Mathematical or physical functions can be replaced by NN models through regres- sion. this study, we get NNs that generate highly accurate and highly smooth function, which only comprised of a few weight parameters, through discussing a few topics about regression. First, we reinterpret inside of NNs for regression; consequently, we propose
activation function-inte (ISLU). Then special charateristics of metadata for regression, which is different from other data like image sound, is discussed for improving the performance of neural networks. Finally, the one of a simple hierarchical NN that generate models substituting mathematical function is presented, and the new batch con- cept "meta-batch" which improves the performance of NN several times more is introduced. The new activation function, meta-batch method, features of numerical data, meta- augmentation with metaparameters, and a structure of NN generating a compact multi-layer perceptron(MLP) are essential in this study.
sigmoid linear unit
INTRODUCTION
In many fields, such as astronomy, physics, and economics, someone may want to obtain a general function that satisfies a dataset through regression from numerical data, which are fairly accurate (Ferrari & Stengel 22005); Czarnecki et al. 2017); Raissi et al. (019);; Langer (02T)). The problem of smoothly approximating and inferring general functions using neural networks (NNs) has been considered in the some literature. However, there is insufficient research on using NNs to completely replace the ideal mathematical functions of highly smooth levels, which sufficiently precise to be problem-free when a simulation is performed. This study aims to completely replace such ideal mathematical functions.
Assuming a model (X) was developed by regression on a dataset using an NN. (X) for input X can be thought of replacement of mathematical function f(X). In this study, such NN is called "neural function(NFF" as a mathematical function created by an NN, The components of an analytic mathematical function can be analyzed using series expansion or other methods, whereas is difficult for NE.
In this study, we created "highly accurate" and "highly smooth" " NFs with a "few parameters" using metadata. Particularly, we combined a new activation function, meta-batch method, and weight-generatin generating network (WGN) to realize the desired performances.
The major contributions of this study can be summarized as follows.
We dissected and interpreted the middle layers of NNs. The outputs of each layer considered basis functions for the next layer; from this interpretation, we proposed a new activation finettioninteerraeed sigmoid linear unit (IstUU)suitable for regression.
The characteristics and advantages of metadata for regression problems were investigated. A training technique with fictious metaparameters and data augmentation, which significantly improves performance, was introduced. It was also shown that for regression problems, the function values at specific locations could be used as metaparameters representing the characteristics of task.

---

Under review as a conference paper at ICLR 2023
NN structures that could generate compac NFs for each task from metaparameters were investigated, and a new batch concept- meta-bbtch -that could be used in the NFs was introduced.
NFs
2 NNS FOR REGRESSION
Let's talk about an easy but non-common interpretation about regression with a multi- layer perceptron (MLP). What do the outputs each layer of an MLP mean? They can be seen basis functions that determine the function be input to the next layer. The input Ti41 of the (i + 1)th layer can be expressed as follows:
Σ wht (x0) + by,
(1)
node
where Zo denotes the imput of the first layer, wh,a denotes the weight that connects the kth node of the ith layer to jth node of the (i + 1)th layer, and Mj denotes a model comprising the Oth ith layers and having the kth node of the ith layer as the output. This is similar to the expression f(x) = E, (x) + of the radial basis function(RBF) kernel method. Clearly, the outputs of each layer act basis functions for the next layer. Figure 2 shows the outputs of each layer an MLP that learned the dataset D (xi 11lly 0.2(x J)r(( 1.5), [-2, 2]} with the exponential linear unit (ELU) activation function.
20
Oth layer
ith layer
Figure
Perspective on MLP
54 05-
45
15
26
46
34
28
45
64
13
Inputs
Outputs Layer
Outputs Layer
Target
150
4.5
418
430
Outputs of Layer
Outputs of Layer
Outputs of Layer
Figure 2: The output graphs of each layer, trained with MLP, where the nodes of cach layer [1.30.
To efficiently extract the final function, the put functions of the intermediate layers must be well-developed. If the output functions of each layer are well-developed, the desired final NF can be compact. In addition, for the final function of NN to be infinitely differentiable, the output functions of the intermediate layers should also be infinitely differentiable.
HRU
Figure 3: The graphs ELU and ISLU(a 0.5. If the activation function is a rectified linear unit(ReL.U), the output function bends sharply after every layer. If one-dimensional regres- sion problem is modeled with simple MLP that has (k+1) layers with nodes [No Ni, Ng.. the output function will bend more than No Ny...Ng. The ELU activation function weakens such bending but does not smoothen it for the
= 1)
Comprising few parameters
2

---

Under review as a conference paper at ICLR 2023
MLP_(4L.6AN) GAN]LSSUUEE
WON 91L_001,19 1SUT0]_
MUP_A9LOON) (ONLLLO
WEN (4L64N) 1SLU MB
MLP_(4LGAA))
MLP_(AL,600) 188LLSS
IMLD (41,64)) TSLU]1]+
WEN_ (ew___4 [
(MLP_ (9011110
8-
WGN 1510_11550111111 JSLL[1SUU_U.
MB
MUP (ILLLNL
MEP (EL, 10N) JANI
WGNL(1L (65)]/ss([1a__
MLP_(1L16N)_
(41, (ON) 15)
-1
Dha MML--/LL.333 IsLUffle
(4LGAN) SortPlus(01
8
WGN (41,16N) TSLU11Ja MB
WON_(4L, (ot___ss)_s__so__...
MLP_[4L, (41,644)) SortPPus
tai (ai, Ten) 18tote]
4
(LP_(1LL,6N) Jo
48
(AL,,GN) P. (41 CAN) F (4L,1ON) sultlusto] FM. P (L1,,, 16N). FLU INLP (ML,1(4L,666) (41,166))SutPluuo SoltPlus(1]
MLP (IL, Softelus(]]
WGN, (4L_,441))) TANH. MB
MLL_(4L, soofooouu
1GN)
(LP_(41,64N) (.. SoftPlus
MI PL(4I TONOLISI uE
500
10
0.5
0.05 SCORE
SCORF
SCCRR
(a) MLP
(b) MLP with fictitious metadata
(c) WGN
Figure Scores. The numerical score table is shown in Appendix
E
Figure 5: Comparison of ELU and ISLU when training with WGN. From left to right, the Oth, sst, and 2nd derivatives of the curves with respect to time in task in the given metadatasets. Blue lines: VGN_(4L,64N] ELU MB, Red lines:
(ALL,64NJJSSUUI
first derivative. Moreover, apt attention is required when using the hyperbolic tangent function for all layers in a regression problem because the output function bends in two places after each layer.
Thus, the question is which activation function develop the intermediate basis functions well? If the activation function starts as a linear function and bends at an appropriate curvature after each layer, the final result will be good. Therefore, we propose an activation function suitable for regression, called "integrated sigmoid linear unir(SLU)).
log(a erp(3x))/8 log(1 1 a)/3.
where a and are positive numbers.
Our experiment shows that ISLU performs sufficiently well and is worth further research. It can improve the accuracy and smoothness of our experimental data. Mathematically. ISLU for a translated SoftPlus that passes the origin, but ISLU absolutely differs from SoftPlus. The purposes of their production differ, and there significant difference in their results
The experimental results are shown in Figure 5
By default, model structure is represented in the form "[the name of the model structure] number of layers]L. [the number of nodes of all hidden information (option)]." The experimental metadataset is described in Appendix A <1>, which has B, k. and m as metaparameters and the corresponding task
layers)N]] [activation function] [further [further
2 Numerical score discussion for smoothness presented in Appendix
A detailed explanation of this is presented in Appendix C
"With SofiPlus[I]. WGN could not be trained due to the divergence loss.
'n our experiment, the Swish activation function was also tested, and its performance was comparable to that of ISLU. However, for consistency,
do not discuss it in the main text; the details are presented in Appendix B
All box plots in this study are arranged in order of small scores from the top, and items wherein the box is invisible have a larger score than the shown graph range.
All experimental conditions of NNs in this study shown in Appendix D
3

---

Under review as a conference paper at ICLR 2023
dataset for L. The average of the "sum of error squares" for eight tasks among the experimental metadatasets is considered a score.
In Figure 4a consider basic MLP structure trained on one task; WGN and fMLP, which will be introduced hereinafter, in Figure 4b are models trained using metadata. Considering ISLU(0), what is in represents degree of freedom in which the activation function's 's shape
be changed. ISLU[0] is trained with a = 0.5 and 3 = 1, [SLU[1], is trained with a = 0.5 and g = var, and 0.5 and = 1 + var, where var are trainable parameters. Because
[SLU[]] trained with variables tend to be learned in a distribution with a mean near zero when training with an NN, ISLU[1], bends slightly after each layer and ISLU[1], bends certain amount and additionally adjusts its degree.
Considering the experimental results in Figure the following is observed.
(1) There is significant difference in performance between SoftPlus and ISLU.
(2) Considering MLP, there is not much difference in performance between ISLU and ELU (Figure 4ap. However, in all models trained with metadata, ISLU significantly outperforms ELU (Figure 4c)
(3) In Figure 4b
when the number of nodes is high(64N), ISLU[0] outperforms ISLU[1],
whereas when the number of nodes is low(15N,16N) ISLU[1] outperforms ISLU
(4) In Figure 4c
ISLU[1] always outperforms ISLU[O].
(5) As shown in ISLU[]]a and ISLU[]]> there are slight differences in performance depend-
ing what the shape of ISLU is based
The reason for (2) can be explained follows: setting an activation function parameter entails giving a certain bias. When given well, considerably helps predicting results; otherwise, may interfere. When using metadata, the performance is improved because biases are determined by referring to various data.
We now discuss the reasons for (3) and (4). In Figure 4b) IMLP indicates MLP structure trained with fictitious metadate for only one task. If an MLP has a lots of nodes, even if the curvature functions of all activations are set be the same, several functions can be added and combined to produce curves with the desired shapes. Meanwhile, when the nodes are few, desired curves may not be obtained well without adjusting the curvatures of the activation functions. In Figure 4c. WGN is network structure D that learns the entire metadata at once. In this case, using ISLU allows the activation shape to change between tasks, yielding better results than the exep-shaped ISLU[O].
The ISLU presented in this study is an example of an activation function for creating desired curves; better activation function can be studied.
2.1 PERSPECTIVES OF METADATA
In this study, metadata the data of datasets that sometimes the sets of task datasets, metafeatures are features of a task dataset, and metalabels metapa- rameters are parameters representing metafeatures. Con- sider case where physical system has the relation f( Z2..) and the function ff depends on the vari- ables al, a2.... For example, a pendulum's 's kinetic energy E is F. = ((). where & denotes the angle between the string and gravitational field direction, and the function f depends on the string's length / or pendulum's 's mass m.
Meta Data
Task
(X,, y
Mate Parameter
(X,,, Y,)
Mcte Farture
Tesks
(Xgy x2y
In this case, the kinetic energy E can be viewed not only f(0 m..) but also (8). The dataset D
Figure 6: Metadata structure.
Moost of the experiments in this study are done with this experimental dataset. 'The smaller the value, the closer ISLU straight line.
10 ddscribbed in 2.2
described in 3.1
4

---

Under review as a conference paper at ICLR 2023
MALL.CAL,GANNISS [O
MIL (55))
03
IMID (4) (66116)) 1R
15.
(MUL_4A (AA_LANAAANNNLN
455
ISLUEIIO
(MLP_(AL,1G) (OLL,,N))ELL
MLP_(4L,GAN) (OL,,4N)BLLL
-45
MUP [MP_(4I,TO)]_EL
MLP_(4L16N) [SLL
4.25
-20
-5
10
45
MUP (41,161) ISLVE1]6
0.2
0.4
Figure 9: Performance improve- ment when using fictitious meta- parameters. "fMLP" indicates MLP trained with using fictitious metaparameters.
Figure 7: The NN learns the sine curves even if all the task dataset's x values for any A.p.o are only at x=0.66, 1.54, 2.30, 3.84, 7.69, 8.97.
Figure Meta-angmentttions with fictitious metalabels. The meta-augmmentaions of the y- axis are displayed.
(4,, 0,, E, S(0,, m,-)) {(4, m4, D:) Dm,,d (0)) metadataset and the numerical value I, m can be considered as metaparameters.
One might want to interpret the kinetic energy as (m.) This cannot be said be wrong, and there may be various perspectives and interpretations for a numerical dataset used for regression.
2.2 ADVANTAGES OF TRAINING WITH METADATA AND META-AUGMENTATION
Consider an experiment performed with the following metadata D3 = ((x,, vi)ly; = A, * sin
I; &x), [0, 10), A E 1.5, 1.5), Pk [0.5, 1.5], Φb [0, 2mr] ]. It can be from the perspective that the tasks D = {(T), v:ll: = A * sin(p * 2; + Φ)} are given according to the metaparameters of A, p, and Φb In this case, not only x but also A. p. and á were trained training inputs, curve could be created with zero shot just by setting A, p. and a. Consequently. if metadata are used to learn, the accuracy of each task increases.
Taking a hint from the fact that metadata improve inference accuracy for each task, it can be thought that even in situation where only one task given, fictitious metadata with fictitious metalabels (or metaparameters) can be generated to learn curves. If only fictitious metalabels are used and the data remain the same, curves would be learned the direction ignoring the metalabels; therefore, some data modifications are required. For the experiment, fictitious metadata comprising 10 tasks with the metaparameter a were created by moving the y; value in parallel :=0.05 for every a = +0.02 with the original data of a = 0 for given task D {(I2, y1)). As result of using fictitious metadata, the score improved significantly (Figure D. The performance improvement was similar even when the fictitious metadata
generated by moving x;; instead of y/ according the fictitious metalabel.
We reiterate that data augmentation including ficitous meta- mnea--aramettess is required to achieve signif- icant performance improvement, otherwise there is little performance improvement. In this study, only the experimental results using MLP with fictitious metaparameters added to inputs are shown; however, further experiments show that the performance improvement due fictitious metadata occurs independent of the model structure.
2.3 LEARNING FUNCTION WITH RESTRICTED METADATA
The regression task for the numerical dataset D = ((x,, villi 0, can have a significant advantage different from the image problems, i.e., yi values at particular locations can be metapa- rameters that represent the entire lask dataset. For the set of images, if we know the RGB values at specific positions of pixels, it does not help to distinguish the features of images. However, for a of mathematical functions f(x)s such as fifth degree polynomial or sine curve sets, just knowing fix) at
MLP with inputs and the WGN 3.1
used for the experiment.
5

---

Under review as a conference paper at ICLR 2023
(MB.N
(MB.N
Weight Generator
X
Y (MB. TB.Nly)
X
Y
(MB.TB.N(k))
(MB.TB.Nly
(MB.TB.NIK])
Main Network
(MB.N (N,l))
Main Network
Figure 11: Another example of function- generating networks where meta-batch can be used.
Figure 10: WGN structure and training with meta- batch, where z denotes metaparameter, X denotes the input, Y denotes the output, and w denotes the weight of main Network.
specific x positions can let us distinguish the functions well. This can be shown the experiments
with sine curve datasets. For the tasks Dr: { yslly, AA sin(ps I, + Φx), x E [0, 10], Aş
[-1.5, 1.5], Ps E [0.5, 1.5], Φk E [0, 2m] that are not given metaparameters A, p. and ø. is possible to learn the sine curves just using the function values Y/ at six points of z; metaparameters (Figure
7 In other words, it is possible to perform few-shol learming simply without metalabels.
In addition, the relationship between the six y points and A, p. and can be learned with simple MLP that has six-dimensional inputs and three-dimensional outputs, indicating that the metaparameters A, p. and & can be completely extracted to generate sine curve using on the six points.
3 FUNCTION-GENERATINN NETWORKS
3.1 WGN
When learning metadata in regression problem, in which NF corresponding to each task is generated from corresponding meta parameters. The structure in which model is generated from variables has been studied extensively (Rusu al. (2018); Sun et al. (219)). We consider the one of the structure of a function -eeeeriingggerttingg network called weight generating network( WGN) in this study. As shown in Figure 6, WGN generates parameters such as the weight and bias of main network through a simple MLP called weight generator from metaparameters. If there trainable parameters of the activation function the main network, can also be generated from metaparameters.
think of an hierarchical NN structure
WGN is expected to generate NFs comprising a few parameters corresponding each task through the weight generator. This is because enormous data and weight generators carefully generate the parameters of the main network. Experiments showed that WGN is effective in creating the main network with excellent performance, although it comprises only few parameters.
What are the advantages of creating a NF with only a few parameters? First, because the number of times that linear input function can be bent is reduced, it may have regulation effect help create a smooth function. Second, may helpful in interpreting and analyzing the network by directly adjusting the parameters. Third, because the number of weights is small and the inference speed is fast, can be advantagcous when a fast interference speed required, such as a simulation.
3.2 META-BATCH
When training function-generatiin network, such WGN, 'one" metalabel (or metaparameter) is usually placed on the weight generator's input, and is updated with the batch of the corresponding task the main network. However, in this case, it becomes training with batch-size=1 for the metaparameters, and when is updated with backpropagation at once, the metacorrelation between tasks is not used well. From these problems, the meta-batch concept is proposed. To distinguish the

---

Under review as a conference paper at ICLR 2023
VGN_(4L,1NN]ISSLU1]
WGN_(4L,16N)ISSUIO.,)
WGN. WGN (1L61)) IS MB
WGN_(4L,16N)_E))
MLP_(4L,64NN_IISL)
WGN (41,160) (SLU[T] MB
SWGN_(4L,16N) [4N__________ 40
MLP_(4L,6NN__SUUU
(1L, ELU
MLP_ MLP_(AL,,AN)_ISUU)
WGNL(4L, (SSU) SUSSSSUU11100 ST
(LP_(4L,64N)_E)
WGN, (41, 166) 151 101
MLP_(4L,16N)_EB)
(1L, (OU0)EEUUUUM
mMLP_ (4LL,16)_ISLUPP
WON (41,04N) BLU 81
1MMP__4L,,6N)__SSL
MLP_(4L,16N)JIUUU
MLP_ (AP_(4L,16N__SLUU
0.05
0.1
SCORE
(MMP__4L,16N)_ISLUU
mMLP_ [MMLP_4LL166)__FF MUP_(4L.16NJJSLUII)
Figure 12: Comparison between using meta- batch and not using meta-batch
0.4 SCORE
Figure 13: Scores for each task of metadata from different models.
meta-batch from the conventional batch, the batch of each task corresponding to one 2; is called "task batch." " "Meta- battaa refers to both the batch metaparameters and the corresponding batch of the tasks. The training method for WGN using the meta-batch is as follows.
Suppose training metadataset )
{(Dk, 2x)]k {1..K}}} comprising task training datasets
De = {(*,, d
Nu are given, where N₂ the number of datapoints of Dj task. For index sets 1 C [1,,, N& that determines meta-batch and task batch, select the batch ]t E Imx m E M}.
M C (1,
XM = [DD 2m)(m E M) and X {(x7" "/"))("
We denote the dimensions of Tz. V/. and 2, as N[x], N(y)), and Ní2), respectively. w/y denotes the weight between the /th and (1 + 1)th layers of the WGN's main network, which has shape (N [wr], N[w++1),, where N[w::] denotes the number of nodes at the i-th layer. The inputs X of the main network are rank-3 tensors in the form of (MB, TB.N [z]), where MB and TB denote the sizes of M and T, respectively.
If 2m enters to weight generator inputs in the form of (MB,N[=)), Clwly(Zz)) generates tensor in the form (MB.N [y]] 1V[ and it reshaped (MB.N [wy), N [l)+ where Glw[g] denotes a generator that generates w/y- The outputs of the /-th layer of the main network, which has the shape (MB, TB, N [wz]), are matrix-producet
with the weights in the form (MB.N [wr)) [(w;+1)) and then becomes a tensor the form (MB, TB,N []] Finally, the outputs of the main network with shape (MB, TB.N [y]) and y are used to calculate the loss of the entire network. Conceptually, is simple shown in Figure 10
As result of the experiment, Figure 12 14 shows significant difference in performance between using and not using meta-batch, where "MB" means using meta-batch, and "ST" means training by inputting metaparameters individually without using meta-batt batch. Figure 12 also shows the difference between using WGN and just using a simple MI.P.
Meta-batch can be used in any mnnetion-generatinn network structure that generates models from variables; another example shown in Figure [T] The outputs of generators concatenate with the layers of the main network. As result of experimenting with ISLU[I] in the structure shown in Figure IT] there was a performance difference of more than four times between using and not using meta-batch.
All other parameters in the main network can he generated from weigh generators using a similar method For WGN, an MLP with three hidden layers and 40 nodes was used the weight generator.
7

---

Under review as a conference paper at ICLR 2023
WGN (41,16N) TSI JH MR
(GNL[4L, TAN)_TANIL_M
WOL (4L, 161)_
WGN (6N, ELU MB
MLP_ (AP_(4L,44N)_AAAA TAN)L
MUP_(ML@ON_ISLLLE
WGN LONLLLLN
MLP (41,64N) ELU
MUP [AP_(41,64N]_L)
MI (41, (41, 18N) ISI U[1]
MLK (AL, (ON)
MI [41 (6AN)_ 1[1 -II+
[LL_[44,64N]_TT 2A___(1,66N)_TAM
0.002 0.001
0.006
0.1
0.2
0.3
SCORE
SCORE
Figure 14: Experimental results with another
Figure 15: Experimental results with metadata Appendix (Axxxx//333 [A|3>> comprising sine and cosine functions
metadata Appendix A <2> related to atmo- spheric pressure.
Figure 13 shows the results of using WGN and meta-batch compared with those using only MLP. "sWGN" indicates WGN trained with metaparameters that the function values at 10 points of (L,t, Ø) without using original metaparameters "B, k. and m. "mMLP" indicates an MLP that trained with six-dimensional input combined with "I, I, and d" and the original metaparameters. "MLP" indicates a trained model for each task with just inputs "I,, f, and őo." This figure shows that using meta-batch, WGN outperformed MLP with fewer parameters. This also shows that WGN excels at learning all metadata and using them with only few parameters.
Figures 14 and 15 shows the results of other metadatasets, which described in Appendix A The combinations of ISLU, meta-batch, and WGN give much better performance than MLP in terms of accuracy and compactness.
CONCLUSION
In this study, we focus on creating mathematical functions with desired shapes using an NN with a few parameters. Irregular and numerous parameters are helpful for generalizations because of randomness; however, this sometimes makes it difficult interpret the network and reduces the smoothness of the functions.
In this study, we dissected NNs for regression; consequently, we proposed new activation function. We looked at the special features of regression-related metadata, such as the possibilities to extract meta-parameters immediately, and how, given only one task, we could create ficitious meta-parameters and metadata to increase performance by more than few times.
In addition, the network structures generating NFs from metaparameters were discussed and the meta-batch method was introduced and tested for the structure called WGN. WGN makes it possible to provide smooth and desired-shae -haied NFs comprised of a few parameters because carefully generates different parameters and shapes of activation functions for each task.
The findings of this study, as well as the insights obtained in the process, are significant for carning smooth and accurate functions from NNs. One of them is the perspective of obtaining desired output functions at intermediate layers from enormous data. Regarding regression problems, it will help elucidate how find the metafeature of each task and map to the corresponding metaparameter as well as how to get a smooth and compact NF of a desired shape.
8

---

Under review as a conference paper at ICLR 2023
REFERENCES
Villèvo Adanhounmè, Théophile K Dagba, and Sèmiyou Adédjouma. Neural smooth function approximation and prediction with adaptive learning rate. In Transactions on Computational Collective Intelligence VII, pp. 103-118. Springer, 2012.
Youngmin Cho. Kernel methods for deep learning. University of California, San Diego, 2012.
Wojeiech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pascanu. Sobolev training for neural networks. Advances in Neural Information Processing Systems, 30, 2017.
Silvia Ferrari and Robert Stengel. Smooth function approximation using neural networks. IEEE Transactions on Neural Networks, 16(1):24-38, 2005.
Gianni Franchi, Andrei Bursuc, Emanuel Aldea, Séverine Dubuisson, and Isabelle Bloch. Tradi: Tracking deep neural network weight distributions. In European Conference on Computer Vision, pp. 105-121. Springer, 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. arXiv preprint
2018.
Siddharth Krishna Kumar. On weight initialization in deep neural networks. arXiv preprint 1rXiv:1704.08880 2017.
Sophie Langer. Approximating smooth functions by deep neural networks with sigmoid activation function. Journal of Multivariate Analysis, 182:104696 2021.
Stéphane Lathuilière, Pablo Mesejo, Xavier Alameda-Pinede and Radu Horaud. A comprehensive analysis of deep regression. IEEE transactions on pattern analysis and machine intelligence, 42 (9):2065-2081, 2019.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint 1rXiv:1711.00165.. 2017.
Yi Loo,, Swee Kiat Lim, Gemma Roig, and Ngai-Man Cheung. Few-shot regression via learned basis functions. 2019.
Chigozie Nwankpa, Winifred Ijomah, Anthony Gachagan, and Stephen Marshall. Activation functions: Comparison of trends in practice and research for deep learning. arXiv preprint 1XXiv:8//1003777 2018.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. informe neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Janarthanan Rajendran, Alex Irpan, and Eric Jang. Meta-learning requires meta-augmentation. arXiv preprint 2000:::2007.05549, 2020.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:/710.05941, 2017.
Andrei Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960 2018.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 403-412, 2019.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence review, 18(2):77-95, 2002.
Tailin Wu, John Peurifoy, Isaac L Chuang, and Max Tegmark. Meta-learning autoencoders for few-shot prediction. arXiv preprint arXiv: arXiv:1807.000 1807.09912, 2018.

---

Under review as a conference paper at ICLR 2023
APPENDIX
A THE EXPERIMENTAL METADATASET
<|> The following dataset was generated from the formula in the physics book of graduate school The original problem and its answer are as follows.
"A spring is connected to a support at one end and has a mass m attached at the other, where the spring constant is & and the rest length is L. Neglecting the spring's mass, what is the angular position 0 of mass m under the gravitational field as a function of time t?"
answer : 0 = B cos (V
kg kl+tmm
I + é), where B, Φ are constants of integration.
The formula cos (V KL+ I + Φ) was slightly modified, and the following dataset is gener- ated.
kg + (B; - 0.3)2 -0032
D = [(Bii, kis 144 L, 2i 0,)00 V B cos (V
k kim, 4 + d.), B, 00.5, 1.5).
h; E 5). 9.8, m, E (0.5, E [1, 4) E [0.1,2), E (0,0.78)) {(Bt, kG m,, k...k.
When it was trained, all input variables B, k, m, L, e were normalized.
For each of B, k, and m, points were uniformly selected to make 1,000 metaparameter sets {(B;, k; m.)). and for each metaparameter point, task datasets DB....... [L,, Li, dx, 0.) which have 35,301 points were created by selecting 21, 41, and 41 uniform points of L, t, and ø, respectively, Among them, 100 random metaparameters were selected, and 640 points were selected for each task to be used as training metadata. The selected points are shown Figure [6
0.5
-0.5
0.5
0.
L
Figure 16: Points of training datasets.
The dataset for Figure 14
D = (( ks mi, Li, 1, di vil -myf B,k; log t, B,L, log d., B, E [1,10], k, [5,20] g = 9.8, m, E [1, 20), Li E [1,8),t E [100, 150), di e [2, 20] ((B,, K, m. D km.))
'GGolsttii,, H., Poole, C., & Safko, J. (2002). Classical mechanies.
10

---

Under review as a conference paper at ICLR 2023
<3> The dataset for Figure 13
D (B Ki, mi, Li, o. vi) ys B, (int + Bb) my cos (0:L: + m4), [-0.400.4]
& E [1, [1,1.5].g g 9.8, m, E [0.2, 11, La E 10,4.4 (0,144) E [1,155]
{(B,, ki, m,, DB, DB....m)))
B ABOUT SWISH ACTIVATION FUNCTION
MULEE 64h) BuUD
muRIE (4, (4..0000000000000 (sugg)
burk swolld))
MULDI 1OO Swlor()
DEUIEE (AL 2000 SweAE Ev)
MUUE CAL 2000 Swangg]]
PEIVE G40 Swerdd)
MUVEL 34N0_3
HONVELU
Figure 17: Swish.
Figure 18: Scores of bMLP with the amplified data.
MELIE JARYELU
DuL DAL s4n/eLu
MNIVE_/ALIION (ILL1
5036
Figure 19: Scores of fMLP with errored data.
Swish showed similar or sometimes slightly better performance than ISLU in our given experimental data. Perhaps the reason is that Swish is good for generalization, and our data fall within the smooth range of Swish activation 1).
If the data targets are in the range of (-1, 1). Swish may have to be used in range that includes two bends (inflection points), which may result in a slightly worse performance. In actual experiments, Swish underperformed ISLU when the experimental data targets had values significantly outside the range of (-1, 1). Figure 18 shows the experimental results of data obtained by increasing the target value by 20 times that of the data in Appendix A <>>, where bMLP means fMLP with the new data whose targets are amplified.
In addition, if the data mixed with some errors, ISLU tends to slightly outperform Swish. Figure 19 shows the scores when training IMLP with the experimental data obtained by adding random errors to the original data targets within 1 % This shows ISLU slightly outperforms Swish, possibly because ISLU's curve simpler and gives regularization effect.
C REASON FOR PERFORMANCE DIFFERENCE BETWEEN ISLU AND SOFTPLUS
Even when comparing ISLU[0]] with and B 1 and SoftPlus, there are differences in performance because parameters try to follow a specific distribution when an NN trained.
Particularly, for two activation functions AF and AF' = AF bb, when the parameter w is multiplied, there a difference in parallel movement by only w * because w AF"(z) = w (AF(z) + b) = AF w 6. It may be thought that the bias parameter could be adjusted produce the same performance; however, there are differences in performance because the NN parameters prefer certain distribution. Further, because IS,)(x, SoftPlus(z,3) b(3), the differences widen because is entangled with the translation part.
ML. with meta autemmmummttnnoonnn in 2.2
Data mixed with random
generated only once and applied in all cases.
11

---

Under review as a conference paper at ICLR 2023
D TRAINING CONDITIONS
The model with the structure denoted by (pL..qN) means the model where the number of hidden layers, excluding the input and output layers, is p-l, and the total number of layers is p+1.
For WGN, weight generators were configured separately for weight, bias, and activation function parameters of the main networks, all of which were MLPs with 40 nodes and hidden layers. The activation function of the generator did not significantly affect the results of either ISLU(0), ISLU[1], Swish; all of them were set to Swish for consistency. The meta-batch size 16, and the running rate was 0.88 times for every 5,000 updates starting from 0.001, with a total of 350,000 updates.
For the model structure in Figure T all settings were the same above except that the generators only generate the inputs of each layer, which are concatenated with the other inputs of each layers of the main network. Each eight gigtt-dimesiional input of the main network generated by the generators, and experiments were performed with the (4L,16N) structure.
E
SCORES FOR FIGURES
Table 1: Table for Figure 4a
Table 2: Table for Figure 4b
Model
Score
Model
Score
MLP (4L_(44,44)___ ISLU[I]b
0,0480
0.0055
ML._ (LP_(44,64N)_ISLUU
0.0650
fMLP (4___,,444)__SLLUU
0.0062
(LP_(4L,64N)_IT
0.0746
IMLP (14L,16N)_ISLUT
0,0103
MLP_(4L,64N)_EH)
0.0949
MILP_(4L,15N)IISUUT
0,0112
[LP_(4L.6AN]_ISLUII
0.1087
0.0181
(LP_(4L,16N) TANH
0.1735
IMLP (4L,115N)ISLUU)))
0.0191
MLP (4L ELU
0.2674
IMLP (4L,166)))) ISLU[O]
0.0247
MI.P (PP_(4,,,6N)_SSIUU))
0.3722
0.0418
MLP_[41,,1N)
0.3853
0.0438
MLP__4,,16N]SSooPPin
0.6513
[MLP (MLP_(4L,64N)EEL
0,0691
MLP (4L,166)
1,5230
IMLP HILP_(4,166N)_FT
0.0793
14799.9304
fMLP (4L, [ 0.1306
MLP (4L,6644))) [0 314.3878
IMLP (4L,166)) 230.7397
MLP (4_L,444_)
329.3784
Table 4: Table for Figure
Table Table for Figure 4c
Model
Score
Model
Score
0.0055
[GGN __M
0.0024
IMLP (LLP_44L, (6N),,NNN)SSUUU
0.0103
0.0024
(LLP_(4L,15N__5)
0.0112
WGN (4L.664)))) ISLU[1]a MB
0.0042
IMLP (4L,16N) ISLU[O]
0.0247
WGN (4L.16N) WGN_(4L.16N) ISLU[1]b MB
0.0058
MLP_(4L,64N) [SLLU]] ISLU[I]b
0,0479
(11_44,,15__SSSSUUUUUT MB
0.0073
MLP_(41.6AN)
0.0650
WGN (4N_SSG))
0.0074
MMLP_(4L,66) ELU
0.0691
WGN_(4L,115 [INUUUUUSSN MB
0.0078
IMLP 14L, ELU
0.0793
WGN (4L, [ANTH
0.0097
MLP (LP_(44,64N)_HH) ELU
0,0949
WGN (4L, (ISU.I [
0.0138
MLP (4L, ELU
0.2673
WGN (4L,644))) ELUMME
0.0248
MLP (4L,16N) ISLU[O]
0.3721
WGN (4L,16N) ELU MB
0.0473
MLP (4L,16N) [SLU[]]
14799.9303
12

---

Under review as a conference paper at ICLR 2023
Table 6: Table for Figure
Model
Score
WGN WGN__44,1661 [SLUU[b) MB
0.0057
Table 5: Table for Figure 12
WGN (GGN(44,16N) (SSLUO
0,0073
WGN
MB
0.0137
Model
Score
(VGN_(4L,16N) [ MB
[SLU[[]6
0.0161
WGN_(4L,166) BIUNN_EELUMMM_
0.0472
WGN WGN_ WGN_(4L,,4AN_IISII || L_M
0.0023
MLP MILP(4L..4AN
0.0479
WGN 14G__[4L,66N__SSLUI111 JMMB
0.0057
MLP (LLP
0.0650
WGN (4L,,
MB
0.0073
sWGN (4L,16N) ELU MB
0.0656
WGN (4L,64N) ELU MB
0.0248
MLP (4L,64N) ELU
0.0949
WGN (4L,64N) ISLU[I]b_SS ST
0.0330
MLP (4L,64N) ISLU[IJa
0.1086
WGN (4L,16N) LEL_ELUUM MB
0.0472
mMLP (4L,16N) ISLU[T]6
0.2243
WGN (4L,,_44),4))E LDL_ELUSST
0.0487
MLP MLP_ (4L,16N) ELU
0.2673
WGN JSLUUII) ST
0.2064
MLP (2L..446))_SSLLA
0,3721
WGN_(AL.1NN_BLU.
0.3642
mMLP MMLP_(4L,16N)
0.3780
MLP (4L,16N) ISLU[IJa
0.3852
mMLP (4L,16N) ISLU[1]a
0.5812
HMMP_(4L,161) ELU
0.8616
(A._
14799.9303
Table Table for Figure 14
Table 8: Table for Figure
Model
Score
Model
Score
WGN VVNN(44,,6N)_IIIT Ja_MB Ja_MB
0.0005
(4I,, MLP (4L.6444))) TANH
0.0032
WGN (4L,16N) TANH MB
0.0008
0.0237
WGN (4L,16N), ELU MB
0.0009
WGN (4L, ELU _B
0.0272
MLP (41,,444))_SLLLA
0.0010
(LL__(4,,44) ELU
0.0475
MLP (A.P_4L,,64N_EB
0.0013
MLP
0.1189
MLP MLP_(4L,16) (4L,16N) [SLU[]
0.0021
WGN VGN_(4L,166), [OD__TANH_MP
0.1217
MLP (6__(,,,,6)__) ELU
0.0031
MLP HILP_44,166)_LT _ANH
0.1290
MLP (A_((,,,4))_AAAA TANH
0.0031
MLP_(4L,166)_EH)
0.1387
MLP (4L,16N)__L)) TANH
0.0052
MLP (OL.16N)JISLUR
0.1903
F SMOOTHNESS SCORE
7-
3/2
2-2
V1
2-2
20
21
(a)
(b)
(c)
Figure 20: (a) shows the case where the degree of waviness in a given section can be checked. Meanwhile, cases such as (b) and (c) cannot be checked well.
In this study, the smoothness score is defined as a temporary measure, and the smoothness some tested models is examined.
The smoothness score judges "how wavy." To determine this, the smoothness of a particular period is first considered because there is need for stan- dard to determine whether case of Figure 21a or Figure 21b is smooth.
13

---

Under review as a conference paper at ICLR 2023
WON_12L 1900011111 [
wew (2) 166) is
WeR (2L oagg [stofo] MB
WON, LON]_SSUUELT MB
WANLIALL AAN
WGN (41,160) [St
WGN,OL (ON_LOL.GN) 10L uS
WON (ALTON) ISU[j]] MB
4-
(ELLTNN) EStUfoT
WGN to
HD
WEN p10_____0_____000000___ MD
WGN (4) (aM))1sHu00) MR
WEN (4) ,640 FLU MB
WORLIAL ISL
4
WGN (OL, 16ng 1stug136 MB
-9
WGN, (4L166) ISLUr114
WGN (ww.)/000.....0... [SLo[[]]..
WGN (41,G400 Lolinsa MB
WGN (4LL
WGN (AL155) ISLU[1] MB
WCN (PL (5/
WON (41,158) IS
WGN (2L JoN) [Sw[0]] Mu
1
WON_(4) 19NL:(4)11NLIISIII M6_MB
WGN (AL, JSNNEEJJ
WEN [AW__AA,,4001SSEEE11
WAN (1) 168) 1SI US) MR
[11116]]
4-
WON (2) naMg (8 Mn
WGN (AL iSLuEijo MB
WGN GANL)) GAN)_LLU BB
WON RB
WGN_i2L_BOM (20___________ (LU_MMMU
WGN (2L 166) ELU MB
WENLDPL 11____11111LMMMMMM WGN (aL, 56N) [SLvE(]+
WONL
WGN (c1,10N) BLU Mis
0.02 1. 0.08
SM2DIHINISS
SMOOTHNESS SCORE
SMOOTHNESS
(a) The smoothness score of the Oth (b) The smoothness score of the 1st (c) The smoothness score of the 2nd derivative
derivative
derivative
Figure 22: The smoothness scores with the period 0.007 in all directions for WGN models are shown.
First, the function value 9-1 1/o F1, 1/2 selected for the uniform in- terval x 2- 2 Zi, 31, F2. Let the path connecting (x-2, y-2) and (I2, 1/2) be = L(x). If the signs of !/1 L(x-1) and V/ L(x1) differ, the path is nonsmooth. For A and B, [A| + [B] (A B) is O both AandB have the same sign; otherwise. positive number greater than 0 is obtained. Therefore, using this, the degree of smoothness of the path is considered as (ly-1 I(x=11] 191 I(x1)] (y-1 [x=-1) + y1 L(x:))), and its total sum is taken as the smoothness score. In multidimensional case, the sum of smoothness scores all directions is the total smoothness score.
(a)
(b)
Figure 21: Waviness
The disadvantage of this smoothness concept is that it cannot comprise the cases where there peak and sudden bend (Figures 20b and 20c). although such cases can be checked by considering the differential of smoothness or smoothness of a smaller period.
This smoothness score has two parameters: the period, which determines the size of the section be checked, and the moving step interval. Figure 22 shows the smoothness for several WGN models with period 0.007 and step 0.007. The figure shows the following two results: (1) ISLU smoother than ELU and (2) the fewer the parameters and layers, the lower the smoothness score.
14


Reviews:
[{"review_id": "lqXAQ6ZfeG", "reviewer": "Reviewer_DP91", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "I don’t think this paper is finished and ready to be reviewed.  ", "summary_of_the_review": "I don’t think this paper is finished and ready to be reviewed. ", "strength_and_weaknesses": "Strength: Perhaps after this paper is finished there will be strengths, but currently I can’t find any.\n\nWeakness: See next section\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "f9Tx42f5eA", "reviewer": "Reviewer_Mwpn", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new activation function named integrated sigmoid linear unit (ISLU), and evaluates its performance on a regression task.", "summary_of_the_review": "See 'Strength And Weaknesses' section", "strength_and_weaknesses": "Strength\n* None\n\nWeakness\n* The paper is not well structured nor formally described. For example:\n    * What is the formal definition of 'well-developed' in p.2\n    * What is the meaning of 'good' in p.3\n    * Most figures are not cited from the main manuscript\n    * It is very hard to understand the experimental settings\n    * It is very hard to understand the definition of 'metadata'\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Kj-n5ACUjwU", "reviewer": "Reviewer_uCcd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a complex methodology of how to generate compact MLP-models that would highly accurately and smoothly represent mathematical functions. \nCore of the methodology (contributions):\n1. a novel hierarchical MLP-model called WGN (weight generating network) - one part of the model (\"weight generator\") generates parameters of the later part (\"main network\"). \n2. a novel activation function ISLU - a smooth version of ELU, concurrent to Softplus (it promises \"less wavy\" second derivatives)\n3. a meta-batch concept - meta-parameters of the represented mathematical function are used as hints during the training \n4. \"smoothness score\" - an approximate measure that evaluates how \"wavy\" a network function is\n\nExperimental evaluation comprises three relatively simple artificial datasets. It compares the MLP and WGN models with \"smoothness score\" as the tested criterion.\n\n", "summary_of_the_review": "The solved task is interesting and the presented model seems to be novel enough. \nIf the authors provide an adequate experimental evaluation to show usefulness of their approach and if they improve both description of the WGN model and settle their approach better into the related work, I would change my score towards accepting the paper.\n", "strength_and_weaknesses": "Plus\n- The paper looks at accurate regression from an interesting point of view. The authors are looking for \"unwavy\" network functions (they call them \"smooth\" which is slightly misleading according to the common sense of the word) \n- The concept of meta-parameters of the represented mathematical function that can serve as hints during training or to create meta-batches is interesting and worth further research.\n- The WGN-model seems to be novel enough, its combination with with smooth activation function and the presented meta-batch concept to achieve better results looks reasonable.\n\nMinus\n1) The experiments are not well chosen to highlight advances and usefulness of the presented model: \n- I am not sure, whether the presented model/concept isn't unnecessarily complex to succeed on the presented artificial tasks that are relatively simple. \n  I would guess, that similar compact and smooth models could be created in a much simpler way (than WGN)\n  by one (or a combination) of the classical MLP-techniques for generalization improvement and/or for reduction of model VC-dimension. \n  What about classical learning from hints - the variant where meta-parameters are used as hint-outputs during training seems promising. Also pruning, well-chosen regularization,... \n  Further experiments comprising at least learning with hint outputs could rebut such suspicion. \n- The experimental evaluation doesn't comprise any real world data. Many models work well for simple artificial tasks but have problems on real world data that is is usually less regular/noise-corrupted/not perfectly distributed etc. Could you enhance the evaluation by such data? \n- The only tested criterion is \"smoothness score\" that measures \"waviness\" of the network function. The measure is approximate and measures presence of just one type of waves. It might thus unfairly advantage one technique over others. \n   If you alter the \"smoothness score\" to e.g., identify also peaks, it might lead to more fair comparison.\n-  There is another argument, why it is not good to use \"smoothness score\" as the only criterion - best (zero) scores will be achieved by MLPs with only linear or even constant activation functions that however won't be able to learn the task at al.\n  - Therefore, why do you omit common measures, such as generalization error or sensitivity (i.e., derivative of the model outputs with respect to its inputs), as further tested criteria? \n  If the network function is \"too wavy,\" its vc-dimension is \"higher then enough\" and it is indicated also by greater generalization error. Smooth network function is also characterized by low sensitivity.\n\n2) The paper omits classical research on the topic - there are several general methods how to form compact, smooth and accurate MLP-models (i.e., regularization, pruning, learning from hints, sensitivity analysis,...). \n   A citation and discussion of such techniques would be appropriate. E.g., the concepts presented in 2.2. and 2.3 (or even 3.2) are obvious variants of learning from hints.\n3) The core of the paper - Sections 3.1 and Section 3.2 are not written comprehensibly. The details of the model and the training process are not described adequately.  \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2Mmn25yk8n", "reviewer": "Reviewer_QykB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors have worked on NNs that can generate or rather approximate accurate and smooth function mainly to solve problems related to regression. The NN uses a few weight parameters, for regression.  They have reinterpreted the outputs of NNs and have proposed a new activation function–integrated sigmoid linear unit (ISLU). \nThey have captured the essence of metadata, meta-parameters (called fictitious meta-parameters) for improving the performance of neural networks used in regression.  \t\nSome calculations of the form of the activation presented in the paper reveal the following:\nlog(alpha + exp(beta))/beta  - log(1+alpha)/beta\nlog[alpha/(alpha+1)+ exp(beta*x)/(1+ alpha)]\nlog[(alpha(1+alpha))(1 + exp(beta*x)/alpha)]\nSome constant + log(1 + exp(beta*x)/alpha) which is a minor variation of the activation function log(1 + exp(beta*x)) which is known to have a saturation problem. I don’t see a major contribution in the existing cottage industry of activations.\n", "summary_of_the_review": "The manuscript is a bit weak with respect to empirical as well as theoretical validations. Empirically, testing a model requires division of data into train and validation sets to ensure that there is no bias while experimenting. Moreover, there is just one dataset used during experimentation which is generated via a physical system of a spring attached at one end. Such physical systems (spring-mass) are now repeatedly used to describe activation functions (DiffAct, Saha et al, IJCNN 2021). The solution to the system is a polynomial where coefficients can be learned from data.\nThe NN regression should be rigorously checked on multiple datasets. The authors have not clarified the method used by the authors to generate hyper-parameters in their manuscript. Theoretical validations of the new activation function are missing as well. The authors need to show if the new AF can approximate any function with a fair degree of accuracy. \n", "strength_and_weaknesses": "An idea is presented by the authors to augment data and meta-parameters for improving the performance of NN. A novel activation function which works very close to ELU is also worth exploring. \n\nWeaknesses \nThere is no theoretical foundation of the work. When the authors say that the presence of meta parameters improves the performance of NN, is there any theory that validates the claim. If so then they must present it. Otherwise the improved performance could be just a mere coincidence on the data presented in the manuscript. The data set used by the authors is generated from the formula in the physical concept of a spring connected to a support at one end and has a mass m attached at the other end.\nAnother validation needed in the manuscript is the proof of Universal approximation theorem for the new activation function ISLU. Does ISLU satisfy UAT! \nISLU is a just modified version of ELU with additional parameters added to ELU, alpha and beta, to tune the curvature of the output function at every layer.  If there is more to it, the authors need to mention that. Like how is the AF unique? \nRegarding the architecture of NN, it is expected from authors to explore why they have chosen 4 layers for training adding/reducing layers can affect the results! Or Is it because the NN performs fine with 4 layers for that specific dataset used by authors. So the model will work fine if the dataset is changed?\nIt is important to analyse if the AF, and the concept of meta parameters and weight generating network works well for all types of datasets and on the other regression tasks. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject