Title: Online Learning of Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Online learning

Neural networks

Deep learning
Abstract: We study online learning of feedforward neural networks with the sign activation function that implement functions from the unit ball in $\mathbb{R}^d$ to a finite label set $\mathcal{Y} = \{1, \ldots, Y \}$.
First, we characterize a margin condition that is sufficient and in some cases necessary for online learnability of a neural network: Every neuron in the first hidden layer classifies all instances with some margin $\gamma$ bounded away from zero. Quantitatively, we prove that for any net, the optimal mistake bound is at most approximately $\mathtt{TS}(d,\gamma)$, which is the $(d,\gamma)$-totally-separable-packing number, a more restricted variation of the standard $(d,\gamma)$-packing number. We complement this result by constructing a net on which any learner makes $\mathtt{TS}(d,\gamma)$ many mistakes. We also give a quantitative lower bound of approximately $\mathtt{TS}(d,\gamma) \geq \max\{1/(\gamma \sqrt{d})^d, d\}$ when $\gamma \geq 1/2$, implying that for some nets and input sequences every learner will err for $\exp(d)$ many times, and that a dimension-free mistake bound is almost always impossible.
To remedy this inevitable dependence on $d$, it is natural to seek additional natural restrictions to be placed on the network, so that the dependence on $d$ is removed. We study two such restrictions. The first is the multi-index model, in which the function computed by the net depends only on $s \ll d$ orthonormal directions. We prove a mistake bound of approximately $(1.5/\gamma)^{s + 2}$ in this model.
The second is the extended margin assumption. In this setting, we assume that all neurons (in all layers) in the network classify every ingoing input from previous layer with margin $\gamma$ bounded away from zero. In this model, we prove a mistake bound of approximately  $(\log Y)/ \gamma^{O(L)}$, where L is the depth of the network.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proves lower and upper bounds on the problem of online learning neural networks with the sign activation function in the realizable setting. These bounds relate the problem to the \"totally separable packing number\", a geometric quantity. Bounds are improved for two special cases (e.g., prediction depends only on low-dimensional subspace). Finally, a known reduction from the literature is applied to also say something for the agnostic case.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- problem studied is fundamental and important, and seems to be not well-studied so far (even though I don't know the literature and trust the authors on that).\n- the paper is well-written and seems to contribute quite a few non-trivial technical results to this fundamental and important problem.\n- I like that the authors put significant effort in high-level overviews of their proofs.\n\nWeaknesses:\n- I'm not sure how meaningful Thm. 2.1 is, as it mainly relate some complicated quantity (bounds on online learning) to another complicated quantity (TS-packing number). It is not immediately clear why this is a useful insight.\n- Thm. 2.2 doesn't seem very surprising. It basically says \"if we are in a k-dimensional subspace of R^d, then everything behaves as if we were in R^k\", which seems mathematically trivial. Am I missing something here?\n- For Thm. 2.3, it is not clear how realistic the assumption of \"large margins everywhere\" is. Is this a theortical borderline case, or is there some justification why this is interesting to study?\n- Section 2.4 sounds like a fancy additional technical contribution, but in fact it is only there to remedy a weakness that was swept under the rug before, namely that gamma and b need to be known for the results in Sections 2.1 to 2.3. This should be mentioned immediately when stating the results!\n- Prop. 2.5 is seems to be strong and general, but is only a minor technical contribution, as it basically follows from Hanneke et al. (2023a).\n- sign activation function is not much used in practice (but I recognize it is still a fundamental case that deserves to be studied, as well-argued by the authors).\n\nMinor (no need to discuss in rebuttal, these don't influence my evaluation):\n- line 28: recieves --> receives\n- line 32: Is the architecture of Phi* fixed? Known to Lrn?\n- line 46: typo in \"converasions\"\n- line 97: I don't understand what the distance between \"an input point\" and \"a neuron\" is. Also assumptions should not be \"hidden\" in footnotes. Later, in line 104, I again do not understand what this distance should be: (w,b) is a vector in R^{d+1}, but x_k is in R^d. Do you only compare inputs to the weight vectors, ignoring the bias? In both cases? Please make this clear.\n- line 110: \"Note that the difference between the upper and lower bounds is roughly quadratic in the worst case.\" --> what does this mean? make it more clear. does it mean that one is the square of the other (at most)?\n- line 114: \"not catastrophic\" is very dramatic and judging wording. Can you put this into more scientific terms and justify why it is \"not catastrophic\"?\n- line 150: this definition in prosa is quite unclear. Please provide a more rigorous definition of gamma.\n- line 165: where does the b suddenly come from? Can you explain this better?\n- line 193: randomized --> randomize (remove d)\n- line 342: improvmnt --> improvement (insert e)\n\nQUESTIONS:\n- How difficult is it to compute or approximate the TS-packing number? From a computational complexity perspective?\n- Can Lrn use unbounded computational ressources? It seems so. Is this correct?\n- The paper [Khalife, S., Cheng, H., Basu A. (2023). Neural networks with linear threshold activations: structure and algorithms (Extended version). Mathematical Programming, 2023.] shows that all linear threshold networks (which should be the same as the sign activation networks in this paper, as far as I understand) can be equivalently represented as networks with only two hidden layers. Can this be leveraged to improve Thm. 2.3, which depends exponentially on the depth? Or any other result of this paper?\n- More generally, the division of a neural network into a \"pipeline with two stages\" reminds me of the proof strategy of the result I mention in the previous point. So maybe you should read it and potentially cite it, even if it does not yield an improvement for your results.\n\nLIMITATIONS:\nadequately addressed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online learning for the class of feedforward neural networks of a fixed architecture (that maps the unit ball to $Y=\\{\\pm 1\\}^k$ for some finite $k$) with the sign activation function. In the realizable setting, where the sequence presented by an adversary respects the labels assigned by some neural network in the class, the first result in the paper (upper bound in Theorem 2.1) shows that there exists a learning algorithm, that makes at most $\\tilde{O}(TS(d, \\gamma_1)/\\gamma_1^2)$ mistakes on any sequence $S=\\{x_1,\\dots,x_T\\}$ labeled by some target network $N$; here, $\\gamma_1$ is the smallest absolute value (margin) of a neuron (pre-activation) in the first layer of $N$ over all the examples in $S$, $d$ is the input dimension, and $TS(d, \\epsilon)$ denotes the totally-separable packing number of the unit ball, which is the size of the largest set such that there exists a hyperplane that separates every pair of points in the set, and is also at least $\\epsilon$ away from every point in the set. Complementary to this, the lower bound in Theorem 2.1 shows that for every learner, there exists a network and sequence of examples for which the number of mistakes is at least $\\Omega(TS(d, \\gamma_1)+(1/\\gamma_1^2))$. \n\n$TS(d, \\epsilon)$ depends poorly on the dimension $d$ (sometimes being exponential in $d$ for small $\\epsilon$)--- thus, to obtain dimension-free bounds, the authors consider two restrictions of the learning setting. The first setting considers the class of sign activation neural networks where the overall function computed by the neural network can be written as some function of the linear projection of the input onto a $k$-dimensional subspace. For this class, denoted as the multi-index model, the authors show (Theorem 2.2) a mistake upper bound of $\\tilde{O}(TS(k, \\gamma_1)/\\gamma_1^2)$. The authors show that $TS(k, \\gamma_1) \\le (1.5/\\gamma_1)^k$, so for small values of $k$, this mistake bound is $1/poly(\\gamma_1)$.\n\nThe second setting that the authors consider is that where every neuron pre-activation in the *entire* neural network has absolute value at least $\\gamma$ over all the examples in the sequence $S$. For this setting, the authors show an upper bound of $\\tilde{O}(\\log(|Y|)/\\gamma^{\\Theta(L)})$, where recall that $Y$ is the label space.\n\nThe algorithms above assume knowledge of $\\gamma_1$ or $\\gamma$: the next results in the paper consider the setting in absence of this knowledge. Theorem 2.4 shows the existence of an algorithm that does not require this knowledge, and attains a mistake bound that is only polynomially-worse than the mistake bound of the algorithm that knows this information. Lastly, the authors touch upon the agnostic setting (Propostiion 2.5)---using a reduction from agnostic to realizable learning for multiclass online classification established by Hanneke et al. 2023a.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is generally very well written, and does a good job of motivating the problem and also situating the work amidst relevant literature on online learning of neural networks. The proof sketches provide sufficient insights with regards to the technical tools used. The mistake bounds presented in the paper in the general setting are near-optimal. The algorithm used for the general setting is a nice adaptation of the standard multiplicative weights algorithm with experts; the class of experts constructed in the paper, and the essential arguments that make the experts algorithm work---namely 1) class of experts is small and 2) some expert is accurate--- are interesting to see. The techniques required to make the general-purpose algorithm work in the more restrictive settings are non-trivial, especially the setting that assumes margin over all the neurons. Lastly, the upper bounds can be be viewed as instantiations of the network-pruning/lottery hypothesis principle, which is quite a curious result. \n\nAs for weaknesses: it would be great if the definition of the precise online learning game considered in the paper (and the definition of the class of neural networks being considered) is included in the main paper, right before stating the results, in a preliminaries section. I understand the page limit restrictions, but hopefully, the extra page after paper decisions will help with this. As written right now, in order to fully understand all the terms in the statement of Theorem 2.1, I had to glance through Appendix A to understand some of the definitions and the precise problem setting.\n\nQUESTIONS:\n1) Maybe I am missing something obvious, but with regards to the footnote at the bottom of page 4, why does the lower bound from Theorem 2.1 trivially hold in the multi-index setting? Is it clear that the lower bound instance from Theorem 2.1 is a multi-index model?\n\n2) With regards to pruning: Is the following conclusion accurate given the results in your paper: For any neural network, there exists a small subset of neurons in the first layer, such that, if we retain only these neurons and get rid of all the other parameters of the network, then one can get a near-optimal mistake bound for the online learning game played on examples labeled by the unpruned network?\n\n3) I was trying to parse through the proof of the upper bound Theorem 2.1 in Appendix A, and the final form of the labelling function $L_G$ that the learner uses for the expert in Figure 3 was not clear to me. Could you please spell this out, and also point me to where it gets defined in the proof in Appendix A?\n\n4) Consider the class of neural networks of a fixed architecture (say with sign activations, and a single output neuron)---this class has some Littlestone dimension $d$. Standard realizable online learning says that $d$ is the optimal mistake bound for realizable online learning for deterministic algorithms. Is it fair to say that the upper and lower bounds in Theorem 2.1 are in some sense, *instance-specific* bounds for every target function from the class? That is, is it clear that the upper bound in Theorem 2.1 is uniformly always smaller than the Littlestone dimension of the class? Also, it must be the case that the lower bound is smaller than the Littlestone dimension (because of SOA achieving this bound always), right? Please let me know if what I am asking does not make sense.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gives mistake upper and lower bounds for the problem of online learning of neural networks that have sign functions. It seems that the important parameter is the totally separable packing number (TS) which depends on the dimension of input and $\\gamma_1$ which is the margin of the neuron’s of the first layer from the sample. The authors demonstrate a mistake bound that scales with the $TS(d, \\gamma_1)$. This dependence can be exponential in the dimension $d$. Given this, they consider a different multi-index setting, where the labeling of data only depends on some $k <<d$ unknown dimensions. In this case, they show the learner enjoys a mistake bound that only scales with $TS(k,\\gamma_1)$. Another restricted setting is where every neurone in every layer has margin $\\gamma$. The authors show that in this setting the dependence on the dimension can be removed but an exponential dependence on the depth appears. I like the idea used here where in the space of first layer’s neurons each sample is considered as a hypothesis that gives a value to the neurons. Then it is possible to define a probability distribution over the neurons which is proportional to the weights with which the neurons are connected to the next layer/output. Using uniform convergence we can guarantee a small set of neutrons exists that approximates the output with an accuracy that does not change the sign, thus the output remains the same.\n\nFor the mistake bound a lower bound is also provided which is not tight an has a gap of approximately $1/\\gamma_1^2$ in some cases.\n\nThe authors also make progress in relaxing the knowledge of the margin parameter $\\gamma_1$ and the dimension. I think this is an important part that is worth exploring further because $\\gamma_1$ is a value that strongly depends on the entire sample.\n\nAll the upper bounds are derived for the agnostic case using known agnostic to realizable reduction techniques.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well-written, clear and concise. All the ideas are clearly explained and elaborated as much as possible. For the technical parts, please see the summary. In general, I enjoyed the technical part and the attempts to make progress with different assumptions.\n\nI think the question of improving the exponential dependence on depth is an important question because in the supervised classification setting, the depth usually appears linearly. It seems to me that the exponential dependence here still does not come from the nature of the problem, rather than from the technique to approximate a function, i.e., neural network, with a different function that depends on less parameters and is easier to learn. So it is really not clear if the dependence is necessary, as pointed out by the authors too.\n\nI also think knowing having the knowledge of the $\\gamma$ is a very important challenge. I understand in a lot of online learning settings, it is easy in to relax this notion by simulating the learner and doubling the parameters iteratively. In this setting I also think it is a very important parameter because of its dependence on the sample. I however think this may require more efforts and can be addressed in future work. I also think that providing a bound that removes this requirement, although at an exponential price, is still important, which the authors have provided.\n\nQUESTIONS:\nThe results are considered for the input lying in the unit ball. What is the dependence on the radius of the input? Similar to the linear classifiers where we get a mistake of $R^2/\\gamma^2$ for radius $R$ and margin $\\gamma$, is it possible that the dependence comes in the form of a value $O(R/\\gamma_1)$ in the packing number? Does generalizing the unit ball require different approaches or do the authors  focus on the unit ball for the simplicity of the notation?\n\nHow would one attemp other activation functions? Like if we have ReLU, what happens to the regions? There is a bit of a mention in related work that we can replace the activation by Lipschitz activations with a Lipschitz parameter that dependens on the margin everywhere parameter. But what if we have some layers with smooth activation functions, probably up to last layer, and then have sign functions. Can we reason about how the combined layers separate the space in regions and try to find the set of experts we need? It seems that still the parameter $\\gamma$ could be controlled by the weight of the layers and the Lipschitzness of the layers. Would this then be different than the prior work mentioned or will it give new understandings? This will potentially depend on the smoothness and weight of different layers and the margin of the first layers. To me it seems that we can kind of separate the first layers and then conclude a margin for it, which depends on its depth and weight of the vectors. Then the same analysis for the rest and output layer can be applied.\n\nLIMITATIONS:\nI believe the main important limitations are already mentioned in the paper. Please see Questions and Strengths and Weaknesses section for more comments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors study deriving mistake bounds of NNs with sign activations in the online learning setting.\nThe study considers both the realizable and agnostic cases, and introduces multiple bounds with varying assumptions. Specifically, authors first start with acceptably tight bounds that depend on $TS(d, \\epsilon)$ terms that are exponential in $d$ (input dimensionality) when $\\epsilon$ is small. Then, they move away from $d$ by projecting the input to a $k<<d$ dimensional space, and operate on the lower dimensional manifold that input is assumed to be in. This helps the authors to recover a bound with $TS(k, \\gamma_1) \\leq (1.5 / \\gamma_1)^k$ that is $poly(1/\\gamma_1)$. Then, the authors extend the $\\gamma_1$ treatment to all neurons and derive a logarithmic bound that has an exponential dependency to the NN depth.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a strong paper that studies a fairly underexplored are from a theoretical perspective. It was as refreshing read as many existing works on online learning w/ NNs are highly empirical and they rely on questionable heuristics. Characterization of the problem and the use of pruning seem to be on point, and being able to move away (w/ assumptions) from depending to input dimension seems novel. The findings also align with empirical analyses and common insights on NN learning dynamics as the 3rd bound is exponential in L. The paper is also well-written and easy to follow - I especially appreciate the inclusion of Section 4 to the main body.\n\nMost of my feedback is around organization and emphasis of the points. As the authors also emphasize, the findings do not immediately translate to practice as the study mainly focuses on sign activated NNs. The authors briefly mention the learnings would generalize to popular activation functions, but I believe it needs to be expanded upon. I also like the implied connections to pruning and autoregressive learning, however, it would be great if the authors provide more insights on those connections and propose how their findings can benefit these existing techniques. Finally, given the high quality of the rest of the sections (and appendix), open questions section needs to be rewritten to convey the authors' vision on the domain better.\n\nQUESTIONS:\nI have 2 main questions, both stem from my comments above. Regardless, I believe the work is strong and my questions are more towards gaining insights from the authors. If needs to be deprioritized to focus on critical questions from the other reviewers, I don't plan to change my ratings.\n\n1) Can you briefly expand on how well your analyses would generalize to say a ReLU network? What would need to change in your approach to acquire similar bounds, and why wasn't it studied in this work?\n\n2) A crude way to interpret these findings is that NNs can be learned online with reasonable regret when either intrinsic dimensionality of data is low /pattern can be learned by a pruned NN / or the NN is sufficiently shallow (which is somehow complementary to low data complexity). At what point these limitations become too strict that fitting linear model with logarithmic regret is more favorable?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}