Mirror Training for Input Convex Neural Network
Authors: Jiaqi Wu
Yang Weng
Venue: Submitted to ICLR 2023 (2023)
Keywords: input convex neural network
convex optimization
representation
recurrent neural network
hosting capacity

Abstract:     The input convex neural network (ICNN) aims to learn a convex function from the input to the output by using non-decreasing convex activation functions and non-negativity constraints on the weight parameters of some layers. However, in practice, it loses some representation power because of these non-negativity parameters of the hidden units, even though the design of the ``passthrough'' layer can partially address this problem. To solve issues caused by these non-negativity constraints, we use a duplication input pair trick, i.e., the negation of the original input as part of the new input in our structure. This new method will preserve the convexity of the function from the original input to the output and tackle the representation problem in training. Additionally, we design a mirror unit to address this problem further, making the network Mirror ICNN. Moreover, we propose a recurrent input convex neural network (RICNN) structure to deal with the time-series problems. The recurrent unit of the structure can be ICNN or any other convex variant of ICNN. This structure can maintain convexity by constraining the mapping from the hidden output at time step $t$ to the input of the next time step $t+1$. The experiments can support our design, including the simple numerical curve fitting, power system hosting capacity dataset regression, and the MNIST dataset classification.

Under review as a conference paper at ICLR 2023
MIRROR TRAINING FOR INPUT CONVEX NEURAL NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
The input convex neural network (ICNN) aims to learn convex function from the input to the output by using non-decreasing convex activation functions and non-negativity constraints on the weight parameters of some layers. However, in practice, it loses some representation power because of these non-negativity pa- rameters of the hidden units, even though the design of the "passthrough" layer can partially address this problem. To solve issues caused by these non- -egaaiivt constraints, we use a duplication input pair trick, i.e., the negation of the original input as part of the new input in our structure. This new method will preserve the convexity of the function from the original input to the output and tackle the rep- resentation problem in training. Additionally, we design mirror unit to address this problem further, making the network Mirror ICNN. Moreover, we propose a recurrent input convex neural network (RICNN) structure deal with the time- series problems. The recurrent unit of the structure can be ICNN or any other convex variant of ICNN. This structure can maintain convexity by constraining the mapping from the hidden output at time step t to the input of the next time step & + The experiments can support our design, including the simple nu- merical curve fitting, power system hosting capacity dataset regression, and the MNIST dataset classification.
INTRODUCTION
Convex optimization's mathematical foundations have been researched for centuries, yet numerous recent developments have sparked new interest in this topic Hindi (2004). In machine learning, convexity and optimization typically refer to the optimization of the parameters or the minimization of the loss Bengio et al. (2005). However, the input convex neural network (ICNN) in Amos et al. (2077) provides a different perspective on the convexity of the neural network, which is from the input to the output.
The input convexity of the ICNN is preserved because of the non-decreasing convex activation function, such the rectified linear unit (ReLU) Nair & Hinton] (2010); Agarap (2018), and the non-negativity constraint on some of the hidden layers. These non-negative weights can maintain the convexity from the input the output but also brings the problem of the lack of representation power. Though using the "passthrough" layers can partially provide substantial representation, this problem is still challenge during the input convex neural network training.
To tackle the challenge in representation, Chen et al. 2018) concatenates the negation of the original input with itself, making the new input of the network. This method can theoretically get more representation power because of the duplication input pair, but the actual training process does not work as expected. The reason for the bad convergence is the new non-negativity constraint on the "passthrough" layers. Therefore, convergence of the training. We prove that the non-negativity constraint on the "passthrough" layers will lead to poor training result. Moreover, the new designs in model, the negative pair of input and the mirror unit, will not break the convexity from the input to the output and improve the training performance.
proposed the modified trick, mirror training, to improve the
Because of the great convexity property of the ICNN, it has been widely used for different tasks. Chen al. (020) introduces the application of voltage regulation through the ICNN. In addition,

---

Under review as a conference paper at ICLR 2023
Chen et al. (2018) uses the ICNN for the building control task under both the single and time- series scenarios. Similarly, Kolter & Manek (2019) also extends the scope of the ICNN for dynamic models. Dynamic models sequence models are always topics that require attention. To take advantage of the convexity of the ICNN and the sequence models, such as the RNN, we construct a loop using the ICNN as the recurrent unit, which is the recurrent ICNN. The network is convex from the sequential input to output by adding the non rengteiatvv constraint on the weights of the hidden output. Our model more straightforward to modify than the other sequence structure because the recurrent unit can be any convex variant of the basic ICNN.
The remainder of this paper is organized as follows. Section 2 discusses the related works. Section 3 iillustrates the design of the mirror training and the structure of the recurrent input convex neural network. Section 4 provides the numerical validation using different datasets. Conclusion and discussions are in Section 5
RELATED WORK
Input convex neural network. The input convex neural network (ICNN) is proposed in Amos al. 2017). Given a fully connected neural network, the model can learn a convex function from the imput to the output because of the non non-negativity constraint on the weights of some hidden layers. However, this constraint reduces the representation power of the model even though the use of the "passthrough" layers can provide some additional representation. The solution provided by Chen jt al. 2018) aims to address this problem by using the duplication input pair, but the new constraint on the "passthrough" layers will cause problem in convergence. We provide the mirror training technique to tackle the problem and prove that the new structure can preserve convexity and have additional representation power.
Recurrent neural network. The recurrent neural network (RNN) is class of sequence models that are widely used for various tasks, including time-series prediction Qin et al. (2017). machine translation Cho et al.] (2014) and speech recognition Shewalkar (2019). The core design of the basic RNN is the recurrent unit, also known as an RNN "cell." where the output is connected to the input, forming a cycle. Many variants of the basic RNN achieve state-of-the-art performance by modifying the cell structure. For example, the long short-term memory (LSTM) Hochreiter
& Schmidhuber (1997) and the gated recurrent unit (GRU) Chung et al. (2014) use the concept of "gate" to forget, select, and memorize the information flowing in the model, therefore learning the time-series relationship. Our design the recurrent input convex neural network (RICNN) takes advantage of this recurrent structure. It formulates the cell convexity for time-series tasks. [Chen et al.] (2018) also proposed an input convex sequence model. The differences between this model and network are that, first, the recurrent unit of our model can be any form of a convex network, while the cell of the model in Chen et al. (2018) only has one layer of full connected neural; second, our recurrent network does not need make all weights non-negative.
basic ICNN capture the
Hosting capacity analysis in the power system. Hosting capacity analysis is a popular topic in power system research. The analysis determines how many more distributed energy resources (DERs) the power grid can host without causing technical issues Wu et al.) (022)). Traditional hosting capacity analysis can be treated as an optimization problem Yuan et al. (022). Nazir & Almassalkhi (2019) presents the hosting capacity analysis convex inner approximation of the optimal power flow problem. The data-driven hosting capacity analysis method is always formulated a time-series problem to observe the hosting capacity value changes over time Rylander et al. (018). In this paper, we use the proposed recurrent input convex neural network to consider the convexity of this analysis, meanwhile capturing the temporal correlation of the hosting capacity values.
METHOD
This section will explain how to obtain a convex mapping from the input to the output through the input convex neural network (ICNN). Moreover, design mirror training technique increase
2

---

Under review as a conference paper at ICLR 2023
the representation power of the basic ICNN. The new structure is the Mirror ICNN. With the mod- ified ICNN, we expand the work to a recurrent structure by proving that the new structure still convex in terms of the imput for the time-series tasks.
3.1 INPUT CONVEX NEURAL NETWORKS
The basic input convex neural network in Amos et al. (0177) is developed from the fully connected neural network. Given fully connected k-layer neural network, can re-construct input convex neural network shown in Fig. la The mathematical expression of the network is
21 on (U + bp),
(1)
Zi41 0, (W + U + b:),
(2)
where h, denotes the output of the i-th hidden layer in the neural network, Wi:k and Uo the parameters of the fully connected layers and the "passthrough" layers, respectively, and is the activation functions.
The convexity from the input æ to the output y is achieved following Proposition
Proposition 1 The neural network is convex from the input to the output, given that all weights in Wi are non-negafive, and all of the activation functions a (.) are convex and non-decreasing.
The proof of Proposition I follows the operations that preserve convexity mentioned in Boyd & Vandenberghe (2004). First, non-negative weighted sum of convex functions is convex. Second, for a function composition, i.e., f(x) = h(g()),, f is convex if h is convex and non-decreasing. and g is convex. Therefore, the ICNN f(x) is convex with respect to input x. By splitting the input features into different parts, i.e., and o, and only adding the non-negativity constraint on æ related weights, the partially input convex neural network (PICNN), shown in Fig. Tb can learn a function (xx o) which is convex with respect to æ.
Ug +
2,
W
-r
Zz
2g
Zk
U,
U,
U2
U2-1
(a) The structure of the fully input convex neural network.
Ug -
Z
W
23
W2
4
Z3
Ze
U.
U1
U2
U
(b) The structure the partially input convex neural network.
Figure 1: The input convex neural network.
The requirement of the non-decreasing convex activation functions not restricted. We can choose from some popular activation functions, e.g., rectified linear unit (ReLU) and exponential linear unit (ELU), which are already proven to be powerful in learning. However, the non-negativity constraint WI-8-1 will limit the representation power, leading inadequate training. The "passthrough" layers are designed to address this problem, but the linear mapping cannot compensate for all the lost representation power. In this case, we mirror training tackle this drawback fundamentally.

---

Under review as a conference paper at ICLR 2023
3.2 MIRROR TRAINING FOR ICNN
Because of the non-negativity constraint on the weights, the ICNN loses much representation power, even though the linear mapping of the "passthrough" layer is designed to address this issue. To fundamentally solve this problem, we propose using mirror training trick for the ICNN, which contains the duplication input pair and the mirror training unit for hidden layers.
First, we will introduce the duplication input pair. Besides the original input æ, we also use the negation of the input, i.c., -x, part of the input. The new structure with the duplication input
pair. - xt
is shown in Fig. la The mathematical equations can be written as
= (Ug
+
bb),
(3)
-2:
Zi+1 o. B,,,,.WWW. (Wiz, + U;
x
+ b:).
(4)
Following Proposition
2 the new input pair will not impact the input convexity.
Proposition 2 If f( is convex function, g((=)) s( - is also a convex function, i.e., the function f is also convex with respect to a.
The proof of Proposition iis simple. [oodd & Vandenberghe (2004) shows that the composition with an affine mapping will preserve the convexity. If f is convex, g(z) = f(Ax + b) is also convex.
Suppose the ICNN learns convex function g(
2 I and f(x) : R R. Using
matrix A which maps
x
[zz]. we have Proposition
Chen et al. (2018) also uses similar negation duplicate pair to retrieve the representation power of the basic ICNN. Differently, in the "passthrough" layers Uo of the model in Chen et al. (2018), the negative weights in U0.-11 of the basic ICNN structure are set zero, and their negations are set as the weights for corresponding -a. This way, the forward calculation the new structure the same result as the calculation in the original ICNN structure. For example. suppose
have input pair [21, x2] and the corresponding weights pair is [w1, W2]. where W2 < O. Following the state- ment,
weights pair [w1; 0, 0, == With the input pair [21, -11, -22], the result is (wi21 + W2222, which equals the inner product of [W1, w2] and [æ1,
get
However, in practice, this method proposed in Chen al. 2018) will not work expected. First, the network will reset weights of - after each iteration, so these weights U are only considered in the forward calculation, not in the backpropagation. Second, since all the weights of ar i.e., 1 are non-negative after each iteration, the negation of their negative value, which
the weights (--) will be small the new iteration and the final iteration. Third and most important, the "passthrough" layers are linear mapping of the input, so we can quickly know that the -o with no constraint are equivalent to any other linear transformation the network, including
the negation trick of U (+) 0 and 0:k-1
By proving that we do not need the non-negativity constraint for Uo:k-1- Proposition 2 enough to preserve the convexity from the input to the output.
we have the equation as Second, for the mirror training unit shown in Fig. 2 denoting
x
2 0, [ Z + U b,).
(5)
2 o. M, U. + b/)-
The convexity of the 2/41 can be achieved following Proposition This structure of 2,41 and 2/41 will use the output of the "passthrough" layers in a coupled way. Because of the introduced dupli- cation input pair of z and the mirror training unit, the ICNN can fundamentally gain representation power. We name the new structure mirror input convex neural network (Mirror ICNN or MICNN).

---

Under review as a conference paper at ICLR 2023
Proposition 3 24+1 is convex, given that all weights in M, are non-negative, and all of the activa- tion functions a(-) are convex and non decresssn.....
U
Mirror training
2
M|
147
2
+
K443
U1
Figure 2: The mirror training unit for the input convex neural network.
3.3 RECURRENT INPUT CONVEX NEURAL NETWORK
By the design mirror training, the input convex neural network can have convex mapping from the input to the output. However, the ICNN is not naturally designed for sequential tasks, but there are many them in practice. For example, dynamic hosting capacity analysis is popular research topic in the power system area. To capture the convexity of time-series problem, we extend the ICNN to a time-series form. In this case, we design a recurrent structure for the ICNN, i.e., the recurrent input convex neural network (Recurrent ICNN or RICNN).
Like the basic recurrent neural networks (RNNs), we construct our model using the recurrent unit, the "cell." Different from the cells in the RNN network, the recurrent unit in our RICNN network an ICNN network. Denoting the output of the ICNN at sequence (t) as (*), we have the RICNN model shown in Fig. The mathematical definition of the model can be expressed
(1+1) (U(U,(((t)) + (oy bo);
21
(.+1) o. + (++(++) H,y + b:), 2++1
(8)
where the hidden state (+) is part of the input at sequence (t + 1) and Ho:k1 the corresponding matrix in the "passthrough" layers. The convexity from the input æ to the output y of the RICNN network is maintained following Proposition 4
RICNN cell
yC+1
Ho
H
H₂
H
H
Ug - 0
Ø1
W1 0
W2 61
Xg
2v
y(
U;
U1
U2
Uk-1
Figure 3: The recurrent input convex neural network.
Proposition 4 The proposed recurrent neural network is convex from the input the output, given that all weights in W and Hoak1 are non-negative, and all of the activation functions a()) are convex and non-decreasing.
5

---

Under review as a conference paper at ICLR 2023
The proof of Proposition 4 is similar to Proposition Note that y in this network is convex function of a²()) Therefore, the matrix Ho:-- should be non-negative to maintain the convexity from a to y. The recurrent unit in the RICNN can replaced by any other convex neural network, which will not break the convexity from the input to the output.
4 EXPERIMENTS
This section will provide numerical validation the design of the mirror training trick and the recurrent imput convex neural network.
4.1 FITTING CONVEX CURVES
The imput convex neural network (ICNN) can learn convex function from the input to the output. To visualize this basic property, design fitting curve experiment to deliver direct perception of the ICNN. Given a function fx) we use basic ICNN and proposed mirror training for ICNN to this function. The results shown in Fig. 4 Both the ICNN and the Mirror ICNN can perform
well for easier tasks, such as fz) = x2 and f(x ex² However, the ICNN shows its drawback when facing a harder task with a relatively large negative bias, i.e., f(x) = e² -3000, where the proposed mirror training for ICNN can overcome the challenge.
Pun
(a) ICNN: y x²
(b) Mirror ICNN: x
Tipe Foyvate
0000
Tra Actiron
7030
000
5530
(c) ICNN: y E²
(d) Mirror ICNN: y =
Foleaae
1030
100
-1920
1100
-2000
-300
(e) ICNN: =
-3000
(I) Mirror ICNN: c
3000
Figure 4: Training results the fitting curve task using different models.

---

Under review as a conference paper at ICLR 2023
4.2 EXPERIMENTS ON THE HOSTING CAPACITY DATASET
The hosting capacity (HC) value is denoted as the maximum active power that can be injected by DERs at bus in an existing distribution grid without causing technical problems or requiring changes to power system facilities. We use the IEEE 123-bus example feeder model as our physical model. The input data is the observation of the power system data of each bus, including the real power, the reactive power, etc. The output data the hosting capacity value of the corresponding bus. The training results are shown in Fig. 5 while the mean square errors (MSE) in training are shown in T
We use three models to experiment; the first one is the basic ICNN proposed in Amos et al. (011)) the second one is the modified ICNN proposed in [Chen et al. (2018), which introduces the new non-negativity constraint on the "passthrough" layers, and the third one is the structure proposed in this paper with the mirror training. The results show that our model can reach the smallest MSE in smoother training process.
30000
KBN with comsinatto sendireggh
10000
40000
40000
30000
30002
30000
20000
10000
30002
10000
0
10000
20300
30002
40000
20000
10200)
20250
30100
40030
50003
10300
20000
20000
ab0c0
50020
Boratin
to:tion
(a) ICNN
(b) ICNN with new constraint
(c) Mitror ICNN
Figure 5: Training results of the hosting capacity dataset using different models.
Table 1: Training MSE of the hosting capacity dataset using different models.
Model
Training MSE
ICNN
713.26
ICNN with new constraint
9222.87
Mirror ICNN
509.98
4.3 EXPERIMENTS ON MNIST DATASET
MNIST is a dataset of handwritten digits LeCun 1998), which is a classic classification task in machine learning. We the 3-layer neural network as our baseline. The used ICNN and Mirror ICNN have the same quantity of hidden neurals as the baseline.
By observing the result, as shown in Table they all have similar testing accuracy on this dataset. When zooming into the first 100 iterations of the training, we can see that the Mirror ICNN has both faster and better performance, as shown in Fig.
Table 2: Testing accuracy of MNIST dataset using different models.
Model
Testing accuracy
NN
97.70% + 0.06% 97.45% + 0.06% 97.96% + 0.07%
ICNN
Mirror ICNN
7

---

Under review as a conference paper at ICLR 2023
2.5
ICNN
Mirror ICNN
2.0
NN
Training less
1.5
20
40
100
Figure 6: The first 100 iterations of the training using different models.
4.4 RECURRENT PREDICTION ON THE HOSTING CAPACITY DATASET
To evaluate the performance of the recurrent input convex neural network, we design an experiment on the hosting capacity dataset. Currently, many researchs show the value of dynamic hosting ca- pacity analysis, i.e., hosting capacity number change over time. The IEEE 123-bus example feeder model is our physical model. The input data is the time-series observation of the power system data of each bus. The output data is the hosting capacity value of the corresponding bus.
We choose the LSTM model as our baseline. As shown Table the mean square losses on both models are very close. However, the Recurrent ICNN model has a faster convergence speed, as shown in Fig.
Table 3: Training results of time-series hosting capacity dataset using different sequence models.
Model
Training MSE
LSTM
1358.09 1327.65
RICNN
RCCNN
120000
LSTM
40000
20000
10000
20000
30000
40000
50000
Reeation
Figure 7: The first 50000 iterations of the training using different models.
8

---

Under review as a conference paper at ICLR 2023
5 CONCLUSION
The input convex neural network (ICNN) learns convex function from the input to the output. However, because of the non-negativity constraint on the hidden layers, loses some representation power. This paper proposes a mirror training technique to address this problem for the ICNN. This mirror training, namely mirror input convex neural network (Mirror ICNN or MICNN), includes the duplication input pair and the mirror training unit for the ICNN. The results of different datasets show that this technique will improve training performance. Morcover, to extend the ICNN to time- series tasks, design the recurrent input convex neural network (Recurrent ICNN or RICNN). The recurrent unit, which can be any convex variant of the basic ICNN, takes the output of the last time part of the input.
During the training. we found that though the proposed new structure of the mirror unit will have a better performance, the training cost is relatively high compared the other model. In our future work, we plan to find a strategy to reduce the training cost. Additionally, we believe the learned convex function can be composed with other structures as the physical enhancement of the learning. Therefore, we plan to explore more application scenarios for the input convex neural network.
REFERENCES
Abien Fred Agarap, Deep learning using rectified arXiv:1803.08375. 2018.
linear units (relu).
arXiv preprint
Brandon Amos, Lei Xu, and Zico Kolter. Input convex neural networks. International Confer- ence on Machine Learning, pp. 155-155. PMLR, 2017.
Yoshua Bengio, Nicolas Roux, Pascal Vincent, Olivier Delalleau, and Patrice Marcotte. Convex neural networks. Advances in neural information processing systems, 18, 2005.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Yize Chen, Yuanyuan Shi, and Baosen Zhang. Optimal control via neural networks: A convex approach. arXiv preprint arXiv: 1835, 2018.
Yize Chen, Yuanyuan Shi, and Baosen Zhang. Input convex neural networks for optimal voltage regulation. arXiv preprint arXiv:2002.08684 2020.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using mnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406. 1078, 2014.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Haitham Hindi. A tutorial convex optimization. In Proceedings of the 2004 American Control Conference, volume 4, pp. 0252-3265. IEEE, 2004.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
J Zico Kolter and Gaurav Manek. Learning stable deep dynamics models. Advances neural information processing systems, 32, 2019.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist// 1998.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
Nawaf Nazir and Mads Almassalkhi. Convex inner approximation of the feeder hosting capacity limits on dispatchable demand. In 2019 IEEE 58th Conference on Decision and Control (CDC), pp. 4858 4864. IEEE, 2019.

---

Under review as a conference paper at ICLR 2023
Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. A dual-stage attention- based recurrent neural network for time series prediction. arXiv preprint arXiv:1704.02971, 2017.
M Rylander, J Smith, and L Rogers. Impact factors, methods, and considerations for calculating and applying hosting capacity. Electric Power Research Institute, 300201 1009, Jan, 2018.
Apeksha Shewalkar. Performance evaluation of deep neural networks applied to speech recognition: Rnn, Istm and gru. Journal of Artificial Intelligence and Soft Computing Research, 9(4):235 2019.
Jiaqi Wu, Jingyi Yuan, Yang Weng, and Raja Ayyanar. Spatial-temporal deep learning for hosting capacity analysis distribution grids. IEEE Transactions on Smart Grid, 2022.
Jingyi Yuan, Yang Weng, and Chin- Woo Tan. Determining maximum hosting capacity for sys- tems in distribution grids. International Journal of Electrical Power & Energy Systems, 135: 107342, 2022.
10


Reviews:
[{"review_id": "vfHd7UrHrd", "reviewer": "Reviewer_Ngme", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the representation problem of the input convex neural network (ICNN). It uses a duplication input pair trick from the work of Chen et al. (2018). Notability, the paper presents a mirror unit and a mirror training trick for ICNN. It also extends a recurrent structure for the ICNN to handle time-series problems. Experiments test the effectiveness of the proposed method.", "summary_of_the_review": "The input convex neural network (ICNN) has the representation problem because of these non-negativity parameters of the hidden units. The paper presents the mirror training trick with the duplication input pair to address this problem. The idea is simple and clear. However, the experiments need more elaborations, which is the main weakness. ", "strength_and_weaknesses": "Strength:\nThe paper is well-written. The idea is simple and clear. The proposed mirror training with a duplication input pair is very practical to address the representation problem of ICNN.  \n\nWeaknesses:\nThe claims are not well-supported by the current experiments. The detailed comments are below.\n1.\tThe mirror ICNN uses a duplication input pair. So its weights U_i and W_i are larger than those of the basic ICNN. What is a fair comparison between ICNN and mirror ICNN? It needs to give more detailed descriptions on the issue. At the end of the first paragraph in subsection 4.3, it claims that it uses the same quantity of hidden neurons. What does it mean?\n2.\tIt is better to include ICNN, modified ICNN by Chen et al. 2018, and the proposed mirror ICNN in all the comparisons. For example, it needs to compare with the modified ICNN by Chen et al. 2018 in Figure 4.\n3.\tThe compared neural network are all shallow. It is better to test with deep networks.\n4.\tWhich activation function is used in the experiment? Does the used activation function affect the results? \n5.\tIn Figure 5 and Table 1, why the modified ICNN proposed by Chen et al. (2018) performs so poor? It is known that the modified ICNN gets more representation power than the basic ICNN.\n6.\tThere are four propositions in the paper. It should distinguish between what propositions proposed in this paper and what propositions in others.\n7.\tIt is better to use three-line tables. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "S_pdQsYuyF", "reviewer": "Reviewer_m7TB", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a modified input convex neural network structure and demonstrated the performance of the proposed structure in learning single-input single-output convex functions, hosting capacity dataset, and MNIST. ", "summary_of_the_review": "This paper proposes a modified ICNN structure, with empirical results in learning convex functions, power systems and MNIST.", "strength_and_weaknesses": "Strength:\n\nThe topic of studying input convex neural networks is interesting and applies to many real-world applications.\n\nWeakness:\n\n1. The paper is poorly written and many statements are wrong. For example, One of the big motivations for this paper as mentioned in the Introduction part of the paper is \"These non-negative weights can maintain the convexity from the input to the output but also brings the problem of the lack of representation power... To tackle the challenge in representation, Chen et al. (2018) concatenate the negation of the original input with itself, making it the new input of the network. This method can theoretically get more representation power because of the duplication input pair, but the actual training process does not work as expected. \" <-- this statement is wrong. \n\nThe original input convex neural network paper [Amos et al. 2017] and the modified ICNN structure paper [Chen et al. 2018] both have the representation power to represent all convex functions. As mentioned by [Chen et al. 2018] page 4, \"our construction of ICNN in Proposition 1 is motivated by [12] but modified to be more suitable to control of dynamical systems... Our construction achieves the exact representation as [12]...\" [Chen et al. 2018] introduced the duplication trick because they not only want the function to be convex, but they need the composition of ICNN functions to be still convex - which is important for dynamical system modeling where the network needs to be “rolled out in time”.  \n\nIf the objective of ICNN is not for dynamical system control - like in all the experiments shown in this paper, from fitting convex curves, fitting the hosting capacity, and MNIST. I don't see a need for not using the original ICNN structure in [Amos et al. 2017]; also the proposed structure does not seem to provide any better representation-wise or training-wise benefits compared to [Amos et al. 2017].\n\n2. The Experiment section feels pretty sloppy and rushed. All experiments are toy-scale problems not meeting the bar of a top AI conference. \n\n3. The experiment results are also mysterious and not convincing. For example, For the MNIST experiment 4.3, why NN has a worse performance than ICNN? NN has much better representation power and a simple search on Google will give you a trained NN that can obtain 99% accuracy in the test set. Also, if you look at Fig 6, does the training even converge? Same for Figure 7, the LSTM curve - the training loss is still fast decreasing.\n\n4. Also, the major content in the Method parts are from previous papers, e.g., Proposition 1 is from [Amos et al. 2017], and the negation method is from [Chen et al. 2018]. The technical contribution of the paper seems quite marginal. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Czup7sl9qHD", "reviewer": "Reviewer_p9y3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce and study a novel architecture for input convex neural networks (i.e. neural networks that are designed to be convex as a function of the inputs). The traditional architecture of ICNNs is improved through the addition of a \"mirroring\" step, appending to the original input, the negation of this input and using the resulting concatenation as an input to each of the layers (which is slightly different from previous work where the concatenation was used solely as an input to the first layer). The paper proves convexity of the resulting architecture and then goes on to use this architecture in the setting of recurrent neural networks.", "summary_of_the_review": "\nGeneral comments\n- I understand why convexity w.r.t. the weights is useful but I don’t understand why convexity w.r.t the input is interesting\n\nDetailed comments \n\npage 1\n- I would remove the proof of proposition 1 (honestly you can just keep one line (if you really want) recalling that the composition of convex functions is still a convex function and that non negative combination maintains convexity)\n\npage 4\n- Same comment as for proposition 1. You can drop the proof and just add a link to Boyd and Vandenberghe? In fact I would remove the proof of proposition 2 and expand on the paragraph “However, in practice, this method proposed in Chen et al. (2018) will not work as expected ” which is unclear (see my comments below)\n- At the end of the page, you write “By proving that we do not need the non-negativity constraint for U0:k−1, Proposition 2 is enough to preserve the convexity from the input to the output”. \n- The notation z_i’ is unclear. the expression of both z_{i+1} and z_{i+1}’ depend on the previous z_i but z_{i+1} seems to never be reused. This is also unclear in your Figure 2 as this figure only represents one step\n\npage 5\n- In (8) when you write z_i, I guess you mean z_i^{(t+1)}? and you do not define y^{(t)} which from the diagram should be given as z_{K}^{(t)}. I would recommend adding this to (7) and (8)\n- In your diagram (Fig 3) for the recurrent neural network, I think the y^{(t+1)} should be changed into a y^{(t)}. I.e. I find it confusing that your y^{(t)} suddenly becomes a y^{(t+1)} in the feedback\n\npage \n- Fig. 4. You only compare the prediction from your architecture with the simple ICNN. How about the ICNN in Chen et al where the mirroring step is already used (albeit only in the first layer)?\n\npage 7\n- “The used ICNN and Mirror ICNN have the same quantity of hidden neurals “ —> do you mean “hiden neurons” ?\n\npage 8\n- For the hosting capacity dataset, it would be interesting to understand the difference between the minimizers returned by the training of the LSTM and RICNN networks. Are those minimizers comparable in terms of generalization? if not, how do they differ?\n\nTypos:\n- “The requirement of the non-decreasing convex activation functions is not restricted” —> “is not restrictive”\n- on page 2, after the proof of proposition 2, “uses a similar negation duplicate pair” —> perhaps what you mean is “use a duplicate of x” ? or “add the nagation of the original inputs”?\n- again after proposition 2, I would replace “Differently, ” by “on the contrary” or “however”\n- page 8, first paragraph of section 4.4. “Currently, many researchs show the value” —> “many/several results”\n", "strength_and_weaknesses": "The core material of the paper appears relatively light/incremental (at this stage) with respect to the results in Amos et al. and Chen et al. (As an illustration of this, compare Figures 1 and 2 in Amos et al.  to Fig. 1 in the present paper)\n\nThe work is still at an early stage (although there are some interesting foundations such as Fig. 4(e) and (f)). The exposition gives an impression of dilution and lacks substance. As a few examples of this, a whole page is used to discuss previous work, another page is used to give intuition on the proof of proposition 1 which can be found in Amos et al. 2017. \n\nThe paper lacks a clear (theoretical or even more elaborate numerical) comparison between the expressive power of mirror networks and simple input convex networks (or networks with the mirroring step restricted to the first layer). I.e It would be great if you could provide theoretical guarantees for a class of functions that cannot be captured by vanilla input convex neural networks but which can be expressed by your new mirrored architecture. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5PRtFhFF91n", "reviewer": "Reviewer_72S5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an input-convex neural network (ICNN) variant that considers negating the input to increase the capacity (Section 3.2/Figure 2) as well as a recurrent variant (Section 3.3/Figure 3). Section 4 experimentally compares these methods for fitting synthetic curve fitting tasks (Fig 4), a power system hosting capacity task (Section 4.2/4.4), and MNIST classification (Section 4.3).", "summary_of_the_review": "I recommend to reject the paper because the new ICNN variant being proposed is not evaluated on any established settings using ICNNs, which make the contribution difficult to understand.", "strength_and_weaknesses": "Strengths\n+ The non-negativity constraints of the ICNN is harmful to the representational capacity and this paper tries to improve upon this\n+ The focus on improving power systems is a strong application area with large potential to improve\n\nWeaknesses\n+ The main weakness is that experimental results are unfortunately not well-contextualized and make it difficult to understand where the real contribution is coming from. There are many of established experimental settings using ICNNs (cited within the paper), but the paper does not explicitly quantitatively build or improve upon any of them. Because the contribution of the paper is meant to be a new model that should improve ICNNs, it is important to show how the model performs in the many existing settings.\n+ The hosting capacity task in Section 4.2 seems similar to the results in [Section VI of Chen et al. 2020](https://arxiv.org/pdf/2002.08684.pdf) as they both use the IEEE 123-bus test feeder, but this connection is not mentioned. None of the experimental results refer back to the results originally published in Chen et al. 2020. For example, it would be extremely convincing to take Table 1 from Chen et al. 2020 and add a \"Mirror ICNN\" column that improves upon their results.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject