Kati8/ICLR_2023_2025 huggingface dataset:
authors: Hok Shing Wong
authors: Matthias J Ehrhardt
authors: Subhadip Mukherjee
reviews: 
Comments: In this work the authors consider a primal-dual algorithm for solving composite optimisation problems where the regularisation term is parameterised by a non-smooth ICNN. In fact, the authors first present a more general input-convex type reformulation of the regularisation which is more general than the standard ones and then, using epigraphical projection reformulation, rewrite in a convex way the training problem, solving it with a tailored primal-dual algorithm. Numerical tests are showed.
Strengths: The paper is very nicely written, concise and well-structured.
It provides a good overview of the state-of-the-art approaches existing in the fields of image reconstruction with network-parametrised regularisation models, a good presentation of the fremework of ICNN and the nice idea of convexifying the (in principle) more general reformulation introduced by the authors via epigraphical projections, which is, essentially, a lifting procedure making the relaxed problem convex and amenable to be solved by, e.g., primal-dual algorithms.
Quite a few numerical results are presented.
Weaknesses: My main concern is the only comparison with subgradient methods in terms of computational efficiency and the resulting claim that the proposed algorithm improves upon it. This is clearly the case, but the reason is that subgradient method is well-known to be very slow ( O(1/\sqrt{k}) w.r.t. to the function values! I think that something that the authors should do is including a comparison with smooth solvers applied to slightly smoothed versions of their architectures. That would allow you to use gradient-type algorithms and make comparisons w.r.t. to O(1/k) convergent approaches, which I believe is more representative of the speed you obtain.
The other weakness of this work si that the authors introduce a more general theoretical framework in Section 2  which, in practice, is not really used later on in the experiments. The framework proposed, indeed, is nice but hardly related to the standard linear + non-linear nature of NNs, so the authors indeed simply apply it to ICNN, as expected. I am not sure then that this relies provides anything interesting from a theoretical view point.
The other major limitation is that while nice from a theoretical viewpoint, ICNNs perform worse than less-structured networks. It would be nice to show a baseline comparison w.r.t. to non-convex or, better, weakly-convex regularisers on which, I believe, analogous considerations could be made but with much better performance.
Also, while the authors declare explicitly their interest into imaging applications starting from the title, I think that some further tests to more ML-related applications showing the interest of ICNN there (if any) would be appreciated.
Recommendation: 5
Reviewer confidence: 5
Soundness/correctness: 4
Clarity: 3
reviews: 
Comments: In this paper, the authors propose a novel, more general, formulation of input-convex neural networks (ICNN). The main contribution to the paper consists in relaxing the usual constraint that layerwise functions should be increasing with a mere convexity constraint. This more general framework encompasses the traditional case, but allows for more general neural network architectures. The authors showcase their new architecture on various imaging problems: salt-and-pepper denoising, image inpainting and CT reconstruction.
Strengths: Overall, this is a well written paper that is both original and interesting. 

1. The paper is original. It goes against the current trend of always larger architectures, and makes interesting links with convex optimization.
2. The paper is mathematically sound.
3. The paper is well written and very clear.
Weaknesses: The main weakness of the paper is the experimental evaluation. As such, it is difficult to put the work in context of other works due to the lack of relevant baselines (see details in the "questions" section).

1. There are no baselines to which the authors compare their approach in the experimental section.
2. The constraints on the architecture seem quite strong (weight positivity, see comments below).
3. Differences with respect to existing works (PnP, ICNNs) could be further clarified (see comments below).
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The paper addresses the optimization problem in a data-driven variational reconstruction framework where a neural network-based regularizer, specifically an input-convex neural network (ICNN), is used. The authors propose a reformulation of the problem that removes the nested nature of a network structure. By linking this reformulation to epigraphical projections of activation functions, they convert the problem into a convex optimization format that is suitable for efficient solution via a primal-dual algorithm.
Strengths: - The authors propose a reformulation of the problem that removes the nested nature of a network structure. By linking this reformulation to epigraphical projections of activation functions, they convert the problem into a convex optimization format that is suitable for efficient solution via a primal-dual algorithm.

- The authors claim to remove the assumption that the activation function does not need to be non-decreasing.

- Numerical validation is presented using the learned convex regularizers.
Weaknesses: - The authors claim that their framework does not require the activation function to be non-decreasing; however, the experiments only involve activation functions that are both non-decreasing and convex. Validation with activation functions that are convex but decreasing would strengthen this claim. For instance, how might the framework perform with functions like the negative exponential or negative logarithm functions? These are provided as illustrative examples.

- Additionally, a plot showing the landscape or level curves of the learned regularizer (neural network) would provide valuable visual insight. For instance, the authors can make use of 2D contour plots or 3D surface plots of the regularizer output for different input features. Also, analyzing how the landscape changes during training would help to visually validate the convexity of the neural network.
		
- Based on my previous comments some discussion about the limitations of the proposed framework is needed. For instance, would be interesting to discuss about the computational complexity for larger networks, potential issues with certain types of imaging problems.

**Some minor comments**

- I suggest the authors to change the notation across the manuscript. Vectors in boldface and matrices in capital boldface. Aligning with this standard notation will improve clarity and make the manuscript more consistent with conventions commonly used in the literature.
		
- I suggest the authors to define $\theta$ immediately after equation (EQ) to ease the reading.
		
- Since the authors prove the convexity of the built neural networks would be interesting and needed to plot the landscape/curve levels of the learned regularizer to have a visual understanding of it. Maybe using the reference below.

*Li, H., Xu, Z., Taylor, G., Studer, C., Goldstein, T. (2018). Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31.*
Recommendation: 5
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 3