Multiscale Training of Convolutional Neural Networks

Authors: Niloufar Zakariaei, Shadab Ahamed, Eldad Haber, Moshe Eliasof
Primary area: optimization
Keywords: Multilevel Stochastic Gradient Descent, Multiscale Training, Mesh Free Convolutions

Abstract:
Convolutional Neural Networks (CNNs) are the backbone of many deep learning methods, but optimizing them remains computationally expensive. To address this, we explore multiscale training frameworks and mathematically identify key challenges, particularly when dealing with noisy inputs. Our analysis reveals that in the presence of noise, the gradient of standard CNNs in multiscale training may fail to converge as the mesh-size approaches to $0$, undermining the optimization process. This insight drives the development of Mesh-Free Convolutions (MFCs), which are independent of input scale and avoid the pitfalls of traditional convolution kernels. We demonstrate that MFCs, with their robust gradient behavior, ensure convergence even with noisy inputs, enabling more efficient neural network optimization in multiscale settings. To validate the generality and effectiveness of our multiscale training approach, we show that (i) MFCs can theoretically deliver substantial computational speedups without sacrificing performance in practice, and (ii) standard convolutions benefit from our multiscale training framework in practice.

Decision: reject
Reviewer scores: 6, 8, 3, 6

Reviews and discussion:
<thread>
creation date:
1730465669364

thread_type:
official_review

<thread_object>
creator: reviewer_5ub1

<content>
summary:
The paper presents a training framework designed to optimize CNNs more efficiently by employing multiscale training strategies. This approach targets the significant computational challenges that arise when training CNNs at high resolutions, especially when handling noisy input data. By introducing Multiscale-SGD and the MFCs, the framework reduces computational costs and ensures stable gradient behavior across varying scales.

strengths:
1.	The paper presents a good framework that integrates multiscale training with convolutional neural networks (CNNs).
2.	The writing of this paper is easy to follow.
3.	The experiments validate the effectiveness of the proposed methods, showcasing practical improvements of the proposed method.
4.	This paper offers thorough mathematical analysis, including proofs and lemmas.

weaknesses:
1.	A significant limitation is the evaluation performed with shallow CNNs, all models containing fewer than four layers. Given that practical applications often leverage deeper architectures like ResNet with over 18 layers, it raises concerns about the scalability and real-world applicability of the proposed multiscale framework. Demonstrating results with deeper networks could help validate its utility for larger and more complex tasks.
2.	In the context of modern architectures, transformers such as Vision Transformers (ViT) are becoming more prominent for various applications. The paper does not explore whether the proposed method could be adapted or extended to transformer-based architectures. This is particularly relevant as transformers may handle multiscale data differently from CNNs. Discussing or experimenting with how the proposed method might extend to these architectures would strengthen the paper's broader applicability
3.	While the paper presents experimental validation of the proposed method's effectiveness, the experiments are mostly focused on simpler tasks. The applicability of the method to more complex and challenging applications, like object detection, video understanding tasks, remains untested. 
4.	While the paper claims significant computational advantages with the multiscale approach, it lacks a thorough comparative analysis of computational costs with existing training methods. For instance, how this method compares in terms of hardware resource use, memory consumption, or training speed against some techniques like mixed-precision training is not clearly outlined.
5.	The paper could benefit from a deeper exploration and further discussion of how the training dynamics change with the proposed method compared to standard methods.

questions:
See the above weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730584120269

thread_type:
official_review

<thread_object>
creator: reviewer_tn3w

<content>
summary:
This work begins with an error analysis of the the standard training procedure for convolutional networks when training at multiple resolutions, and (using some clever re-interpretations by varying mesh size and sample size) derive new multiscale training algorithms that require fewer network evaluations at the finest mesh size. They further present mesh-free convolutions that can be thought of as limits of progressively refined mesh-dependent convolutions. Finally, experimental results are presented to compare the training algorithms and network variants against deep learning standards like SGD and U-Nets/ResNets.

strengths:
1) The primary research questions in the paper are well-motivated.
2) The technical analysis seems to be sound, and the authors are able to propose interesting, non-obvious training algorithms from them (i.e., this goes far beyond "multiscale training is useful for learning convolutional kernels"). I also found the further step of mesh-free convolutions to be very interesting from a theoretical perspective.

weaknesses:
My major problems with the paper pertain to the experimental results, and fall into two main categories:

1) The experiments done are on extremely small networks and datasets. There are very standard collections of computer vision experiments that could be used to show the benefits of the proposed multiscale-SGD training algorithms (see, for example, the benchmarks used for experiments in any landmark CV papers from the past few years, like CLIP or Segment Anything). I understand that compute access could be a factor here, but many of these datasets are small enough that access to even a single GPU should be sufficient to perform the necessary experiments.

2) Unclear presentation of results. It's difficult to understand the relative compute requirements of different algorithms based on the work unit representation that is used in the experiments. It would be much clearer to either present the FLOPs required per iteration or backward pass, or best of all would be to simply show the amount of time taken to train the network in each case.

3) Non-competitive baselines. In practice, multiscale training is implicitly accounted for in most deep learning training procedures via data augmentations in the form of varying-sized crops of images (see the widely-used RandomResizedCrop transform in torchvision). To me, this is the actual baseline to which we should compare to, but it is not mentioned or compared against at all in your work.

Based on these, I find it difficult to be convinced by the presented experiments that the proposed multiscale algorithms would be better to use than standard SGD on usually-sized computer vision trianing tasks.

questions:
1) Why are there no experiments on CV prediction tasks like classification, segmentation, object detection, etc.? Are the proposed algorithms not expected to work well in these contexts?

2) Corresponding to the data augmentation comment above: It was not clear to me if you use data augmentations like RandomResizedCrop or similar in any of your experiments; do you? If not, would you be able to show what happens when these are added to the single-scale SGD training procedure?

3) Corresponding to the comment on presentation of results comment above: could you add (even approximate) FLOPs computations or raw timings to your experiments?

4) Are there situations in which it would be preferable to use the Multiscale-SGD algorithm that you proposed instead of the Full-Multiscale-SGD algorithm? If not, it would be better to focus more of the experiments on the Full-Multiscale algorithm, and leave the Multiscale algorithm to an ablation study.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730752525747

thread_type:
official_review

<thread_object>
creator: reviewer_5fgj

<content>
summary:
Training convolutional neural networks (CNNs) on high-resolution images can be computationally demanding. One way to alleviate the computational burden is by including downsampled versions at different scales during training. This paper identifies shortcomings of standard CNNs in such multiscale approaches and introduces “Mesh-Free Convolutions” (MFCs) to address these limitations. The paper presents two novel multiscale training algorithms, “Multiscale-SGD” and “Full-Multiscale-SGD”, which estimate high-resolution parameters using coarser-resolution samples, thus avoiding costly training on high-resolution data. For such multiscale training schemes, gradients of conventional CNNs for noisy data might diverge with decreasing resolution compared to “smooth data”. Inspired by differential operators, MFCs offer a resolution-independent generalisation of standard convolutions. The empirical validation demonstrates that these techniques can improve training time and ensure consistent gradient behaviour, as shown for standard UNet and ResNet architectures and image blur estimation, image denoising, and image super-resolution tasks.

strengths:
I believe this paper provides several important contributions and extends the existing literature in different significant ways:

__S1.__  The authors propose new techniques for training CNNs within a multiscale resolution framework, which improve computational efficiency while maintaining test performance. This alone can be a valuable contribution, as shown by the “non-MFC” empirical results in Tables 3, 4, and 5. 

__S2.__ As far as I can tell, MFCs are an original and promising alternative to traditional convolutional layers (though some clarifications may be necessary; see W2 below) that can alleviate some of the shortcomings of traditional CNNs in multiscale approaches. Given their potential application across various high-resolution tasks, this can be a significant contribution, too.

__S3.__ The quality of the paper is high, with claims well-supported through mathematical derivations, proofs, and empirical evidence. In particular, the experimental results in Tables 3, 4, and 5 highlight that significant speed-ups might be feasible (as measured by #WU; however, see Q2 below).

For these reasons, I consider this a good paper.

weaknesses:
Some clarity aspects could be addressed to strengthen the paper. In more detail, I currently see the following weaknesses:

__W1.__ Section 3.1 on “Mesh-Free Convolutions” is relatively dense and challenging to follow. For instance, some notations typically used with parabolic PDEs, like indices denoting partial derivatives, should be more clearly introduced. More importantly, the connection between $u$, $v$, $\tilde{v}$, and $\mathcal{C}$ is not very clear and should be made more explicit.

__W2.__ Expanding on the previous point, the paper would benefit from a more intuitive explanation or illustration of the connection between standard convolutions and mesh-free convolutions. Currently, the connection is difficult to assess. In that regard, Figure 3 lacks clarity and might require additional explanation. 

__W3.__ _Computational considerations_: As the authors highlight in lines 334-338, the required Fast Fourier Transform dominates the overall computational cost. To provide the complete picture, additional consideration of running time and optimisation steps in the results in section 5 on “Experimental Results” would strengthen the contribution. This would allow a more comprehensive assessment of MFC's computational feasibility. 

Minor remarks:
- Line 231: “[…] without compromising on its efficiently.” should likely be “[…] on its _efficiency_.”
- Line 293: “[…] let $v = \mathcal{C}(\xi)u$ be the mesh-free convolution is parameterized by $\xi$“ needs to be rewriten.
- Line 296: Equation “(17a)” likely should just read “(17)”.

My rating is currently “borderline accept” due to the aspects raised in W1 to W3. However, I am willing to adjust my rating if some of these points and the questions below can be addressed.

questions:
__Q1.__ To illustrate the difference between conventional CNNs and MFC, would it be possible to compare feature maps between these two approaches?

__Q2.__ A more formal definition of the _work unit_ metric #WU might be beneficial. As I understand it, #WU measures how many evaluations of the highest resolution are required, with evaluations at lower resolutions being weighted by the corresponding “downsampling factor”. Is this correct? If so, should this not lead to fractional values of #WU?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730760074222

thread_type:
official_review

<thread_object>
creator: reviewer_35kc

<content>
summary:
This paper introduces an approach to improving the computational efficiency of CNNs through a multiscale training framework.
It first presents Multiscale Stochastic Gradient Descent (Multiscale-SGD), which leverages the Multilevel Monte Carlo method to reduce computational costs by approximating gradients across resolutions.
However, the authors identified that noisy or high-frequency inputs can lead to unbounded gradients across mesh resolutions, complicating the convergence process.
To address this, they propose Mesh-Free Convolutions (MFCs), which are independent of specific input scales and ensure consistent gradient convergence across resolutions, even with noisy inputs. This mesh-independence of MFCs overcomes the convergence limitations observed in standard CNNs during multiscale training.

strengths:
This paper introduces an innovative approach to multiscale CNN training through Multiscale-SGD and Mesh-Free Convolutions (MFCs).
All the developments are backed by mathematically reasoning.
And the whole paper is logically organized.

weaknesses:
1. The experimental results are really limited, and you should also compare the performance with the fixed computational budget.
2. The experimental comparison with existing multiscale or Fourier-based CNN methods is not presented
3. The mathematical foundation for Mesh-Free Convolutions, particularly the differential operator theory, could be challenging for readers less familiar with this domain. Adding a visual explanation or intuitive analogies could make the theory more accessible.
4. In the experiments, the network seems really shallow, could you train deeper networks, to see whether the methods works?

questions:
1. All the results show mixed performance bettween the regular SGD and the two Multiscale-SGD. If you could train longer for the multiscale-SGD, could you improve all the performance comparing to SGD?
2. The MSE results are strange, with the two UNet have huge gap. What is the reason behind it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731980451458

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# General Response to all Reviewers:
We would like to express our sincere gratitude to all the reviewers for their valuable and constructive feedback.
Overall, the reviewers acknowledged the novelty and contributions of our proposed multiscale training framework. Reviewer **35KC** highlighted the innovative aspects of Multiscale-SGD and Mesh-Free Convolutions (MFCs), noting that the paper is "logically organized" and supported by "mathematical reasoning." Reviewer **5FgJ** recognized the framework's ability to "improve computational efficiency while maintaining test performance" and described MFCs as "an original and promising alternative to traditional convolutional layers." They also praised the high quality of the paper, commending the "mathematical derivations, proofs, and empirical evidence" that support our claims. Similarly, Reviewer **TN3W** found the technical analysis to be "sound" and appreciated the novelty of our multiscale training algorithms and mesh-free convolutions, describing the latter as "very interesting from a theoretical perspective." Reviewer **5Ub1** acknowledged the "thorough mathematical analysis" in our work and the practical improvements demonstrated in the experiments, further noting that the writing is "easy to follow."
We are also pleased to see that multiple reviewers (**5FgJ, TN3W, 5Ub1**) appreciated the strong theoretical foundations of our method, with **5FgJ** and **5Ub1** emphasizing the value of our proofs and lemmas. Reviewer **35KC** further noted that our contributions address "convergence limitations" effectively.
Your thoughtful comments and suggestions have greatly contributed to improving our paper, particularly in addressing aspects related to experimental depth, clarity, and broader applicability. We hope that our responses and revisions sufficiently address your concerns and that you will consider revising your scores accordingly. As always, we remain open to any additional questions or suggestions and are happy to engage in further discussions.

**General comment on our work.** The primary objective of this paper is to explore multiscale training and optimization strategies, rather than introducing new architectures for specific tasks such as segmentation or denoising. Our focus is on demonstrating how the proposed approach enables more effective utilization of multiscale structures for training. While Mesh-Free Convolutions are theoretically better suited for this purpose, our experiments show that the proposed ideas also perform surprisingly well with standard convolutions and Fourier Neural Operators (FNOs), even in cases where no theoretical justification exists for regular convolutions. Notably, for FNOs, the theoretical foundations align with the observed performance. In the revised paper, we have included new experiments based on the reviewers' feedback and refined the presentation to clarify these points. We hope these updates enhance the clarity of our contributions and the implications of our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732771903413

thread_type:
official_comment

<thread_object>
creator: area_chair_zxwb

<content>
comment:
Dear Reviewers,

If you have not responded to author's rebuttal, please kindly do so as soon as possible. The deadline is Dec 2, but the authors can potentially further clarify questions if you respond earlier. Thanks!

Best, AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: a novel multiscale training framework for CNNs with multiscale-SGD and mesh-free convolutions (MFCs). This enables more efficient optimization and consistent gradient behavior across resolutions.

Strengths: strong mathematical grounding; improved efficiency.

Weaknesses: limited comparisons with existing multiscale methods, including data augmentation; small datasets, outdated architectures (no ViT or modern ConvNets); complex presentations; unclear practical gains.

Reasons for decision: this is a borderline paper. However, the limited empirical evidence in modern tasks and practical benefits overshadow the strong contributions in theory.