SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Input-convex neural networks. Input-Convex Neural Network (ICNN) were originally designed in the context of energy models (Amos et al., 2017). We focus on the fully convex ICNNs variant and use gradient descent for optimisation instead of the proposed bundle-entropy method. In their implementation, Amos et al. (2017) use projection methods to keep weights positive, i.e. negative values are set to zero after every update. However, also other projection methods can be used to keep the weights positive (e.g. Sivaprasad et al., 2021). Instead of projecting the weights onto the non-negative reals after every update, it is also possible to use a reparameterisation of the weights. E.g. Nesterov et al. (2022) square the weights in the forward pass. Note that a reparameterisation can have an effect on the learning dynamics because it is an inherent part of the forward pass. Another alternative is to use regularisation to impose a soft non-negativity constraint on the weights (Makkuva et al., 2020). Similar to (Sivaprasad et al., 2021), we directly train an ICNN, but we do not observe the same generalisation benefits. On the other hand, Sankaranarayanan and Rengaswamy (2022) point out that restricting the network to be convex decreases their capacity. They propose to use the difference of two ICNNs to allow modelling non-convex functions (c.f. Yuille and Rangarajan, 2001). However, we did not find the convexity restriction to cause any problems in our experiments. Nesterov et al. (2022) use ICNNs to simplify the computation of level sets to learn invariances. We adopt this experiment setting to highlight the potential of ICNNs. Optimization. While training the parameters of an ICNN is unrelated to convex optimisation, Bengio et al. (2005) showed how training a neural network together with the number of hidden neurons can be seen as a convex optimisation problem. This work was continued by Bach (2017), who established a connection between this convex optimisation and automatic feature selection. Signal propagation theory. The study of signal propagation is concerned with tracking statistics of the data throughout multiple
layers of the network. In his thesis, Neal (1995) computed how the first two moments propagate through a two-layer network to study networks as Gaussian processes. This is similar to how modern initialisation strategies have been derived (e.g. LeCun et al., 1998; He et al., 2015). A similar approach has been taken to incorporate the propagation of gradients for initialisation (Glorot and Bengio, 2010). The idea to explicitly account for the effects of activation functions can be attributed to (Saxe et al., 2014). All of these analyses assume that weights are initialised from a zero-mean distribution, which is not possible in ICNNs. Mishkin and Matas (2016) empirically compute the variance for the weights. This approach is more robust to deviations, but nevertheless requires adaptations to account for weights with non-zero mean. There is also a significant body of work that studies signal propagation in terms of mean field theory (Poole et al., 2016). In these works, the correlation between samples is included in the analysis (Poole et al., 2016). In our work, we find that the correlation between features plays an important role. Schoenholz et al. (2017) applied the mean field analysis to the backward dynamics and introduced the concept of depth scales to quantify how deep signals can propagate. The mean field theory was also applied to convolutional networks to derive the delta-orthogonal initialisation (Xiao et al., 2018). We refer to (Martens et al., 2021) for a general overview of signal propagation.
We evaluate our principled initialisation strategy by training ICNNs on three sets of experiments. In our first experiments, we investigate the effect of initialisation on learning dynamics and generalisation in ICNNs on multiple permuted image datasets. We also include non-convex networks in these experiments to illustrate that ICNNs with our principled initialisation can be trained as well as regular networks. However, we would like to stress that non-convex networks are expected to outperform ICNNs because they are not constrained to convex decision boundaries. Moreover, we assess whether ICNNs using our initialisation are able to match the performance of regular fully-connected networks in toxicity prediction without the need for skip-connections. Finally, we explore ICNNs as a tool for more controlled latent space exploration of a molecule auto-encoder system. This illustrates a setting where regular networks can no longer be used. Details on the computing budget are provided in the Appendix.
5.1 Computer vision benchmarking datasets
As an illustrative first set of experiments, we use computer vision benchmarks to assess the effects of our principled initialisation strategy. Concretely, we trained fully-connected ICNNs on MNIST (Bottou et al., 1994), CIFAR10 and CIFAR100 (Krizhevsky, 2009), to which we refer as permuted image benchmarks (cf. Goodfellow et al., 2014).
In our comparison, we consider four different types of methods: (1) a classical non-convex network without skip-connections and default initialisation (cf. He et al., 2015). (2) an ICNN, "ICNN", without skip connections and default initialisation (3) an ICNN, "ICNN + skip", with skip connections and default initialisation (4) an ICNN, "ICNN + init", without skip connections and our proposed initialisation. All networks use ReLU activation functions. The non-negativity constraint in ICNNs is implemented by setting negative weights to zero after every update (cf. Amos et al., 2017). Note that skip-connections in ICNNs introduce additional parameters to allow for additional representational power (cf. Amos et al., 2017).
Training dynamics. Settings. In the first experiment, we analyze the training dynamics during the first ten epochs of training. Similar to Chang et al. (2020), we fixed the number of neurons in each layer to the input size, and the number of hidden layers to five. We refer to Appendix C.2 for results with different depths. The learning rate for the Adam optimiser was obtained by manually tuning on the non-convex baseline. Results. We found that ICNNs with our principled initialisation exhibit similar learning behaviour as the non-convex baseline, while ICNNs without our initialisation strategy can not decrease the loss to the level of the baseline on CIFAR10 and CIFAR100. Figure 2 shows the learning curves for our different methods on three permuted image benchmarks.
Generalisation. Settings. In our next experiment, the effects of our principled initialisation on the generalisation capabilities of ICNNs is investigated. To this end, we train our three ICNN variants and the non-convex baseline on the permuted image benchmarks again, but focus on the test performance this time. To this end we perform a grid-search to find the hyper-parameters that attain the
Figure 2: Training loss curves of ICNN variants with the same architecture on the MNIST, CIFAR10 and CIFAR100 datasets. "ICNN" input-convex network with default initialisation. "ICNN + skip": same settings, but with skip connections. "ICNN + init" our principled initialisation for ICNNs w/o skip connections. "non-convex": a regular non-convex network. Each curve represents the median performance over ten runs and shaded regions indicate the inter-quartile range. The inset figures provide a view of the loss curves zoomed in. Note that ICNN losses do decrease before the plateau.
Figure 3: Test set metrics of compared methods on the (a) MNIST and (b) CIFAR10 datasets. Each curve displays the median performance over ten runs. Shaded regions represent the inter-quartile range over the ten runs. On MNIST, all methods can be successfully trained and exhibit similar performance. On CIFAR 10, ICNNs with our proposed initialisation outperform other ICNNs variants.
best accuracy after 25 epochs of training on a random validation split, for each of the four compared methods. The grid consists of multiple architectures with at least one hidden layer, learning rate for Adam,  $L_{2}$ -regularisation and whitening pre-processing transforms (details in Appendix C.2). Results. On the MNIST dataset, all variants reach similar accuracy values, which we attribute to the simplicity of the prediction problem. However, on CIFAR10 the initialisation strategy leads to better generalisation already in early epochs and — compared to the respective ICNNs variants without our initialisation — improves generalisation overall (see Figure 3). The ICNN with our initialisation almost matches the accuracy values of non-convex nets.
5.2 Toxicity Prediction
To test ICNNs in a real-world setting, in a different application domain, we consider the binary multi-task problem of toxicity prediction in drug discovery. More specifically, we train ICNNs on the Tox21 challenge data (Huang et al., 2016; Mayr et al., 2016; Klambauer et al., 2017). The input data consists of so-called SMILES representations of small molecules. The target variables are binary labels for twelve different measurements or assays indicating whether a molecule induces a particular toxic effect on human cells. During pre-processing, SMILES strings were converted to Continuous and Data-Driven Descriptors (CDDDs) (Winter et al., 2019), which are 512-dimensional numerical representations of the molecules. The choice of these particular descriptors is motivated by CDDDs properties of being decodeable into valid molecules (see Section 5.3). Hyper-parameters were selected by a manual search on the non-convex baseline. We ended up using a fully-connected network with two hidden layers of 128 neurons and ReLU activations. The network was regularised with fifty percent dropout after every hidden layer, as well as seventy percent dropout of the inputs. The results in Table 1 show that our initialisation significantly outperforms ICNNs with standard initialization ( $p$ -value 2.6e-13, binomial test) and ICNNs with skip connections ( $p$ -value 5.5e-14). Furthermore, results are close to the performance of traditional, non-convex networks.
Table 1: Area under the ROC curve on the test set for each of the twelve tasks in the Tox21 data. Each value represents the median performance over 10 runs and error bars is the maximum distance to the boundary of the interval defined by the (0.05, 0.95) quantiles. Of all variants of ICNNs, the variant with our proposed initialisation, "ICNN+init (ours)", performs best matching almost the predictive quality of traditional non-convex neural networks.
NR.AhR	NR.AR	NR.AR.LBD	NR.Aromatase	NR.ER	NR.ER.LBD
ICNN+init (ours)	91.29 ± 0.58%	81.84 ± 2.43%	81.36 ± 3.58%	82.40 ± 1.09%	78.37 ± 0.80%	77.85 ± 0.90%
ICNN	90.56 ± 0.63%	81.28 ± 4.82%	78.14 ± 3.15%	80.46 ± 1.74%	77.30 ± 0.78%	78.15 ± 1.02%
ICNN+skip	89.83 ± 0.21%	68.12 ± 1.74%	74.17 ± 2.20%	78.95 ± 0.45%	76.95 ± 0.48%	81.92 ± 1.16%
non-convex	91.01 ± 0.65%	79.21 ± 1.73%	86.19 ± 2.50%	83.63 ± 0.34%	77.88 ± 1.13%	74.32 ± 1.59%
SR.ATAD5	SR.HSE	SR.MMP	SR.p53	NR.PPARγ	SR.ARE	AVG
ICNN+init (ours)	77.01 ± 1.67%	80.05 ± 1.36%	93.69 ± 0.27%	81.10 ± 0.46%	77.46 ± 2.50%	76.80 ± 0.33%	80.57 ± 11.84%
ICNN	74.25 ± 3.26%	78.64 ± 1.42%	93.19 ± 0.59%	81.32 ± 0.43%	75.00 ± 1.73%	76.11 ± 0.82%	78.33 ± 13.42%
ICNN+skip	75.93 ± 1.26%	78.23 ± 0.50%	75.40 ± 0.88%	78.25 ± 0.93%	92.09 ± 0.41%	79.32 ± 0.97%	78.29 ± 12.56%
non-convex	78.93 ± 1.68%	80.12 ± 2.57%	93.99 ± 0.39%	82.17 ± 0.96%	82.70 ± 1.21%	78.30 ± 0.90%	81.31 ± 11.03%
Figure 4: Example of two level set trajectories of a Tox21 model. The leftmost molecule represents the starting molecule with low toxicity (top: "NR.AR", bottom: "NR.ER.LBD"). The rightmost molecule represents the target molecule. All intermediary molecules are samples on the level set between the starting molecule and the target molecule and exhibit different drug-likeness ("QED"). In this way, one predicted molecular property can be kept fixed while another property is optimized.
5.3 Latent space exploration of molecular spaces
In this experiment, we exploit the intrinsic property of ICNNs that level sets, i.e. the set of all inputs that map to the same output, can be parameterised (Nesterov et al., 2022). These level sets provide an opportunity in drug discovery to keep one molecular property, such as low toxicity, fixed while optimizing another property, such as drug-likeness. To demonstrate this application, we randomly chose reference molecules with low toxicity and followed the according level set in the direction of another randomly chosen target molecules. Since the input space of the ICNN that predicts toxicity (see above) possesses a decoder model from (Winter et al., 2019), the numeric representations of the molecules on the level set can be decoded into molecular structures. The obtained trajectories in Figure 4 demonstrate that the predicted toxicity of the molecules remain constant, while different QED scores, which measure drug-likeness, are obtained by traversing the level set from the reference molecule to the target molecule.
We demonstrated how signal propagation theory can be generalised to include weights without zero mean. By controlling the propagation of correlation through the network, we derived an initialisation strategy that promotes a stable propagation of signals. We empirically verified that this steady propagation leads to improved learning. Finally, we showed that ICNNs can be trained to a comparable level as regular networks on the Tox21 data using CDDDs.
Our initialisation approximates the distribution of pre-activations with Gaussians, which falls short of agreement with the observed data, but has been sufficient to improve the initialisation of ICNNs. A possible way forward is to use the central limit theorem for weakly dependent variables. In Ap
pendix C.3, we also observed that random bias initialisation leads to pre-activations with a more Gaussian distribution. Appendix A.5 also includes an analysis for convolutional layers, but proficient empirical results with input-convex convolutional networks are still to be made. We conjecture that this is due to the more complicated covariance structure in convolutional layers. We also included a signal propagation analysis for back-propagation in Appendix A.4. However, incorporating insights from this analysis into an initialisation method is also left for future work. We envision that our initialisation strategy will make it easier to incorporate ICNNs or other networks with non-negative weights.
We thank Sepp Hochreiter for pointing us in the direction of input-convex networks and their interesting properties. Special thanks go to Niklas Schmidinger for his help with experiments on convolutional networks.
The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects AI-MOTION (LIT-2018-6-YOU-212), DeepFlood (LIT-2019-8-YOU-213), Medical Cognitive Computing Center (MC3), INCONTROL-RL (FFG-881064), PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF-36235), AI4GreenHeatingGrids(FFG-899943), INTEGRATE (FFG-892418), ELISE (H2020-ICT-2019-3 ID: 951847), Stars4Waters (HORIZON-CL6-2021-CLIMATE-01-01). We thank Audi.JKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo) Software Competence Center Hagenberg GmbH, TÜV Austria, Frauscher Sensonic, TRUMPF and the NVIDIA Corporation.
Amos, B., Xu, L., and Kolter, J. Z. (2017). Input convex neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 146-155. PMLR.
Bach, F. (2017). Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research, 18(19):1-53.
Bengio, Y., Roux, N., Vincent, P., Delalleau, O., and Marcotte, P. (2005). Convex neural networks. In Advances in Neural Information Processing Systems, volume 18, pages 123-130. MIT Press.
Bottou, L., Cortes, C., Denker, J., Drucker, H., Guyon, I., Jackel, L., LeCun, Y., Muller, U., Sackinger, E., Simard, P., and Vapnik, V. (1994). Comparison of classifier methods: a case study in handwritten digit recognition. In Proceedings of the 12th IAPR International Conference on Pattern Recognition, volume 2, pages 77-82, Jerusalem, Israel. IEEE.
Brock, A., De, S., and Smith, S. L. (2021). Characterizing signal propagation to close the performance gap in unnormalized ResNets. In International Conference on Learning Representations, volume 9, virtual. arXiv: 2101.08692.
Chang, O., Flokas, L., and Lipson, H. (2020). Principled weight initialization for hypernetworks. In International Conference on Learning Representations, volume 8.
Cho, Y. and Saul, L. (2009). Kernel methods for deep learning. Advances in neural information processing systems, 22.
Clevert, D.-A., Unterthiner, T., and Hochreiter, S. (2016). Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). In International Conference on Learning Representations, volume 4, San Juan, Puerto Rico. arXiv: 1511.07289.
Daniely, A., Frostig, R., and Singer, Y. (2016). Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity. In Advances in Neural Information Processing Systems, volume 29, pages 2253-2261, Barcelona, Spain. Curran Associates, Inc.
Nair, V. and Hinton, G. E. (2010). Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning, pages 807-814, Madison, WI, USA. Omnipress.
Neal, R. M. (1995). Bayesian Learning for Neural Networks. PhD thesis, University of Toronto.
Nesterov, V., Torres, F. A., Nagy-Huber, M., Samarin, M., and Roth, V. (2022). Learning invariances with generalised input-convex neural networks.
Poole, B., Lahiri, S., Raghu, M., Sohl-Dickstein, J., and Ganguli, S. (2016). Exponential expressivity in deep neural networks through transient chaos. In Advances in Neural Information Processing Systems, volume 29, pages 3360-3368.
Sankaranarayanan, P. and Rengaswamy, R. (2022). CDiNN - convex difference neural networks. Neurocomputing, 495:153-168.
Saxe, A., McClelland, J., and Ganguli, S. (2014). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations, volume 2.
Schoenholz, S. S., Gilmer, J., Ganguli, S., and Sohl-Dickstein, J. (2017). Deep information propagation. In International Conference on Learning Representations.
Sivaprasad, S., Singh, A., Manwani, N., and Gandhi, V. (2021). The curious case of convex neural networks. In Oliver, N., Pérez-Cruz, F., Kramer, S., Read, J., and Lozano, J. A., editors, Machine Learning and Knowledge Discovery in Databases. Research Track, Lecture Notes in Computer Science, pages 738-754. Springer International Publishing.
Wang, Z. and Wu, Z. (2023). Input convex LSTM: A convex approach for fast Lyapunov-based model predictive control. arXiv preprint arXiv:2311.07202.
Winter, R., Montanari, F., Noé, F., and Clevert, D.-A. (2019). Learning continuous and data-driven molecular descriptors by translating equivalent chemical representations. Chemical Science, 10(6):1692-1701. Publisher: The Royal Society of Chemistry.
Xiao, L., Bahri, Y., Sohl-Dickstein, J., Schoenholz, S., and Pennington, J. (2018). Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 5393-5402. PMLR.
Yang, G. and Schoenholz, S. (2017). Mean field residual networks: On the edge of chaos. Advances in neural information processing systems, 30.
Yuille, A. L. and Rangarajan, A. (2001). The concave-convex procedure (cccp). In Advances in Neural Information Processing Systems, volume 13, pages 1033-1040.
Zhang, G., Botev, A., and Martens, J. (2022). Deep learning without shortcuts: Shaping the kernel with tailored rectifiers. In International Conference on Learning Representations, volume 10, virtual.