Kati8/ICLR_2023_2025 huggingface dataset:
authors: Cheng Kang
authors: Xujing Yao
reviews: 
Comments: This paper presents a control system perspective of the ANNs training process and views the training algorithms as controllers. In addition, the authors analyzed several optimization methods (SGD, SGDm, AdaM, PID, LPF-SGD, HPF-SGD, Fuzzy PID) on different types of ANNs (CNN, FFNN, GAN, CycleGAN, ResNet). Through experimental studies, the authors conclude that different ANNs should use different training algorithms to achieve their best performance.
Strengths: The control system perspective on the ANN training process is nice, although it is not new. Combining PID controller and fuzzy logic is novel to the best of my knowledge. There is a potential that the Fuzzy PID controller can actually improve the training process.
Weaknesses: The presentation quality of the current manuscript can be further improved. 

The novelty of this paper is weak. The idea of interpreting ANN training as controller design is not new. The idea of Fuzzy PID is new but incremental compared to [Wang2020]. In addition, the authors did not show the advantage of Fuzzy PID compared to other optimizers in the numerical section. It would be great if the authors could provide a rigorous proof of the convergence of the training algorithms through the lens of stability theory in control literature.  

The dataset used in the numerical section is limited, and the model used in this part is too simple and unrepresentative. The results obtained in the main paper are inconsistent with those in the appendix (Figure 2 vs. Figure 15).
Recommendation: 3: reject, not good enough
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 1 poor
Clarity: 1 poor
reviews: 
Comments: This paper interprets the training of neural networks as a feedback controller on the parameter of the network. Several commonly used training algorithms are interpreted as proportional or PI controllers.
Strengths: + Neural network training, or understanding why neural network training works is an interesting question. 
+ The interpretation in the paper is interesting and uses control theoretic ideas that are well-known.
Weaknesses: - The technical problem considered in the paper is too simple to represent training of neural networks in my opinion. In particular, it assumes there is a single equilibrium, $\theta^*$, and the feedback is in the error between $\theta$ and $\theta^*$. A better model would be that we have the output $y(\theta)$, for example, the loss on the training set. But there are many $\theta$'s that give the same loss. For example, suppose that we can make the training loss 0, and $y(\theta_1*)=y(\theta_2*)=...=y(\theta_n^*)=0$ for some $n$. The challenge is that $n$ maybe very large, we don't know any of the $\theta_i^*$, and the feedback is on the error in $y$. The system is actually nonlinear, with many equilibriums, and convergence is much harder to understand.
Recommendation: 1: strong reject
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: The paper investigates the point of view of considering optimization neural networks as a control problem. They then investigate different known optimizers and controllers on a set of tasks.  I had great difficulty understanding the logic of this paper, my best guess is that the goal is to understand why certain optimizers are chosen for specific tasks in deep learning.
Strengths: - I like that more recent neural network architectures such as  forward-forward neural network are considered
- In general the view of neural network training from the perspective of control and the respective methodology (like PID and fuzzy controllers) is of interest
- In general the paper tries to adress different domains, such as GAN-like training, vision, and feedforward neural networks
Weaknesses: - The paper is written very poorly. It is very difficult to understand what the intentions of the the authors are. There are countless examples, starting with the title: "Based on What We Can Control Artificial Neural Networks"   Other examples include for instance "To analyze the learning progress of most ANNs, for example, CNN using backpropagation algorithm, FFNN using forward-forward algorithm, and GAN such a generative model using random noise to generate sample" (page 2)

- I don't see how the experiments make sense given the research question. It look to me that standard classification tasks are considered, with standard architectures and evaluations.
Recommendation: 1: strong reject
Reviewer confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 1 poor
Clarity: 1 poor
reviews: 
Comments: This work frames the learning process of neural networks through the lens of feedback control theory, where various standard optimization methods are characterized as proportional or PI control systems. While the core concept has merit, the mathematical formulation employed is overly simplistic for capturing the complexity of neural network optimization. The model assumes a unique equilibrium point θ* with feedback based directly on parameter deviation, which does not adequately represent realistic training scenarios. A more appropriate framework would incorporate the actual network output y(θ), such as training loss, which exhibits multiple equivalent solutions. For instance, if training loss can reach zero with θ₁*, θ₂*, ..., θₙ* all yielding y = 0, the true challenge becomes apparent: the number of such solutions may be enormous, their locations are unknown a priori, and feedback operates on output error rather than parameter error. The inherent nonlinearity and multiplicity of equilibrium points make convergence analysis substantially more challenging than the current treatment suggests.
Strengths: + Examining neural network training and elucidating the mechanisms underlying successful neural network learning constitutes a valuable research direction. + The proposed control-theoretic perspective is conceptually appealing and leverages established control theory frameworks.
Weaknesses: - The mathematical formulation is insufficiently complex to adequately capture the nature of practical neural network training. Specifically, the assumption of a single unique equilibrium θ* with feedback measured as the parameter error is unrealistic. A superior approach would center on the observable output y(θ), such as empirical loss, which inherently admits many different parameter configurations achieving identical loss values. Consider the scenario where training loss achieves zero for multiple distinct parameter configurations θ₁*, θ₂*, ..., θₙ*; this multiplicity could be arbitrarily large, the exact locations are a priori unknown, and feedback operates on output error rather than parameter perturbation. The true system exhibits significant nonlinearity with numerous equilibrium states, making the convergence problem substantially more involved than the current framework acknowledges.
Recommendation: 1: strong reject
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair