Speeding Up Image Classifiers with Little Companions

## Abstract

Abstract Scaling up neural networks has been a key recipe to the success of large language and vision models. However, in practice, up-scaled models can be disproportionately costly in terms of computations, providing only marginal improvements in performance; for example, EfficientViT-L3-384 achieves <2% improvement on ImageNet-1K accuracy over the base L1-224 model, while requiring 14 × 14\times 14 × more multiply–accumulate operations (MACs). In this paper, we investigate scaling properties of popular families of neural networks for image classification, and find that scaled-up models mostly help with “difficult” samples. Decomposing the samples by difficulty, we develop a simple model-agnostic two-pass Little-Big algorithm that first uses a light-weight “little” model to make predictions of all samples, and only passes the difficult ones for the “big” model to solve. Good little companions achieve drastic MACs reduction for a wide variety of model families and scales. Without loss of accuracy or modification of existing models, our Little-Big models achieve MACs reductions of 76% for EfficientViT-L3-384, 81% for EfficientNet-B7-600, 71% for DeiT3-L-384 on ImageNet-1K. Little-Big also speeds up the InternImage-G-512 model by 62% while achieving 90% ImageNet-1K top-1 accuracy, serving both as a strong baseline and as a simple practical method for large model compression.

## 1 Introduction

Advances in parallel computing hardware, such as GPUs, have made end-to-end single-pass parallel processing standard in computer vision models. Large vision datasets like ImageNet-1K [] made it possible for such deep vision models (e.g.Alexnet [], ResNet [] and ViT []) to learn general visual features at scale. While vision models surpassed human performance on ImageNet-1K a decade ago [], researchers are in the perpetual pursuit of achieving improved performance by using a combination of two strategies: 1) developing more performant models and training techniques for a given compute budget, and 2) scaling up the models.
While improved models from the former approach are often preferred in compute or memory-constrained applications, the latter has become increasingly popular, thanks to its success in large language models (LLMs) []. However, despite architectural improvements, scaling up models remains expensive; we are often trading exponential compute cost for marginal gains in model accuracy (Table 1).

In this work, we show that much of the inefficiency comes from our implicit preference for single-pass models and propose an embarrassingly simple two-pass algorithm to drastically speed up models with little companions.

We summarize our work in response to two critical questions around model scaling and compression.

- ⟨Q1⟩
Given a pair of Little-Big models in the same model family, which incorrect predictions made by the Little model are fixed using the Big model?

- •
Using ImageNet-1K as the test bed and binning the Little model’s predictions by confidence, we find that, very often, mistakes made by the Little model correspond to low confidence (measured via maximum softmax probability)

- ⟨Q2⟩
Without compromising accuracy, can we speed up a Big model by using a Little model to preprocess a proportion of samples in a distribution?

- •
We propose a two-pass Little-Big protocol where a light-weight Little model is used to make predictions (class and confidence) on all samples in the first pass, and a Big model performs a second pass on samples with low confidence from the first pass, achieving significant reduction in inference compute costs without compromising accuracy.

- •
Without any modification to existing models, we prescribe Little-Big pairs that significantly reduce compute costs of models across types and scales, while not compromising accuracy: Little-Big models achieves MACs reductions of 76% for EfficientViT-L3-384, 81% for EfficientNet-B7-600, 71% for DeiT3-L-384 on ImageNet-1K. Little-Big also speeds up the very large InternImage-G-512 model by 62% while achieving 90% ImageNet-1K top-1 accuracy, serving both as a strong baseline and as a practical approach for efficient deployment of large models.

**Table 1: Scaling up is expensive. Scaling up model size is a popular way to improve performance without redesigning neural architectures or training recipes. Popular practices often involve sparing compound scaling in input resolution $H$, model width $w$ and depth $l$ over the base model (characterized by $H_{0},w_{0},l_{0}$). However Equation 2 shows that model size and inference cost quickly blow up by $10-100\times$, with only marginal performance gains over small base models of the same family.**
| Model Family | Size | $H/H_{0}$ | $w/w_{0}$ | $l/l_{0}$ | ImageNet-1K Val | GMACs |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Accuracy (%) | $\Delta$ | Absolute | $\Delta$ |  |  |  |  |  |
| EfficientNet [] | B0-224 | 1 | 1 | 1 | $77.65$ | $--$ | $0.39$ | $--$ |
| B2-280 | 1.3 | 1.1 | 1.2 | $80.56$ | $+2.91$ | $1.09$ | $+1.8\times$ |  |
| B4-380 | 1.7 | 1.4 | 1.8 | $83.45$ | $+5.80$ | $4.39$ | $+10.3\times$ |  |
| B7-600 | 2.7 | 2.0 | 3.1 | $84.11$ | $+6.45$ | $37.8$ | $+95.8\times$ |  |
| EfficientViT [] | L1-224 | 1 | 1 | 1 | $84.39$ | $--$ | $5.3$ | $--$ |
| L2-288 | 1.3 | 1 | 1.4 | $85.60$ | $+1.21$ | $11.0$ | $+1.1\times$ |  |
| L3-384 | 1.7 | 2 | 1.4 | $86.34$ | $+1.95$ | $81.0$ | $+14.3\times$ |  |
| DeiT3 [] | S-224 | 1 | 1 | 1 | $83.05$ | $--$ | $4.6$ | $--$ |
| B-224 | 1 | 2 | 1 | $85.60$ | $+2.55$ | $17.6$ | $+1.1\times$ |  |
| L-384 | 1.7 | 2.7 | 2 | $87.73$ | $+4.68$ | $191.2$ | $+40.6\times$ |  |

## 2 Related Work

### 2.1 Computer Vision Models

Since Alexnet [], single-pass neural classifiers trained end-to-end have dominated leaderboards of various vision tasks from image classification to video segmentation [, , , , ]. Most neural models originates from two families of neural architectures: convolutional neural networks (CNNs) and transformers. Core to CNNs are “convolutions” which apply the same compute across locations on a feature map. On the other hand, transformers, which first found success in sequence-to-sequence language models [] and subsequently in vision [], embed a sequence of tokens (e.g., image patches) and utilize attention mechanism to model intra- and inter-token interactions. Finally, hybrid models like Swin [] combine CNN priors and attention mechanisms to achieve good performance.

### 2.2 Scaling in Vision

Many neural classifiers can be expressed as a composition of layers [, , , ]:

$$ $y=F(x)=f_{w_{l}}\cdot...\cdot f_{w_{1}}\cdot f_{w_{0}}(x),$ (1) $$

where $x\in\mathbb{R}^{C\times H\times H}$ denotes an input image (using square images for simplicity) and $y\in(0,1)^{N}$ denotes a $N$-dimensional softmax confidence score. To get a class prediction, one finds the class $n$ with the highest confidence. $f_{w_{j}}$ denotes the function of layer $j$ with its characteristic width $w_{j}$. The inference cost of a single sample $x$ with such a model $F(x)$ can be expressed as:

$$ $MACs[F(x)]\approx C_{F}*H^{2}*w^{2}*l$ (2) $$

where $C_{F}$ is a scaling coefficient that is a function of the model family. In turn, the average inference cost per sample over a (finite) distribution $D$ is given by:

$$ $\operatorname{\mathbb{E}}_{D}\bigg{(}MACs[F]\bigg{)}=\frac{1}{|D|}\sum_{x\in D }MACs[{F}(x)]\approx C_{F}*H^{2}*w^{2}*l$ (3) $$

Complementing innovations in model architecture $F$ that make models more compute efficient, a straightforward way of improving performance is to scale up the model. Thanks to architectural improvements like the skip connections [], normalization layers [], as well as better initialization and parameterization[], scaling up a model by orders of magnitude has been made feasible. Furthermore, efficient scaling strategies such as compound scaling in model width and depth, as well as input resolution [] have also emerged. However, Equation 2 imposes a fundamental limitation on the prohibitive cost of scaling: doubling in $H$, $w$, and $l$ leads to $2^{5}-1=31\times$ increase in model compute cost. The marginal accuracy gains associated with model scaling shown in Table 1 further make it unappealing for many practical use cases with limited compute budget.

### 2.3 Model Compression

It is well known that modern neural networks have significant redundancy, thus impacting both the model size (often measured by the number of parameters) and inference compute cost (often measured by MACs). This has motivated the design of compression strategies to reduce redundancy in the model. Many lines of research focus on improving the inference time [], reducing both model size and compute simultaneously [], or trading model size for speed [].

Pruning is a popular way to reduce model compute cost by ablating weights that are deemed less essential for the prediction. For example, WDpruning [] reduces widths based on saliency scores, and X-pruner [] measures a unit’s importance by its contribution to predicting each target class. The implicit assumption is that the redundancy in size and compute are coupled, and one reduces compute by removing the units.

Another way to achieve model speedup is through adaptive computation mechanisms that ideally spend just enough compute on each sample. They often adopt some form of “early exit” mechanisms that reduce $l$ in Equation 2. DynamicViT [] downsamples the number of tokens adaptively to reduce compute cost, A-ViT [] introduces learned token halting for ViT models so that not all tokens are processed by the full depth of the model, thus effectively reducing the average depth of the compute graph during inference.

### 2.4 Human Vision

While parallel processing plays an essential role in making it possible to ingest gigabits/s of raw visual information and compress it to tens of bits/s to guide our behavior, human vision is not a single-pass process. Human eyes have two distinct information processing pathways that originate from two types of photoreceptors called rods and cones. While the visual acuity in the cone-rich fovea is $\sim 1$ arcmin, it only covers $\sim 2$ degrees, or $\sim 0.01\%$ of the visual field []. The rest of the $\sim 99.9\%$ of the visual field is mostly dominated by rods which provides much lower visual acuity ($\sim 10$ arcmin). A given small patch in the visual field either only gets processed in a single-pass by the low-acuity rod pathway, or followed by additional passes with high-acuity foveal vision if needed as directed by the saccade. Studies [, ] have shown that human visual classification performance adapts to variable compute budget.

## 3 Scaling Up Helps with “Hard” Samples

Figure: Figure 2: Using the EfficientNet family as an example, we first show that confidence of individual models correlate well with prediction accuracy (left), which allows us to approximate a “hardness” axis with prediction confidence. Mistakes made by Little models can be categorized as correctable (solid bars, $\times\rightarrow\checkmark$) and non-correctable (shaded bars, $\times\rightarrow\times$) by the Big model. Breaking down the mistakes of Little models by hardness in the right panel, we find that very few ($<10\%$) correctable mistakes are easy (confidence > 0.5-0.7). This motivates the idea of a two-pass Little-Big algorithm enabled by decomposing the samples by confidence thresholds. More examples can be found in Appendix Figure 4.
Refer to caption: x2.png

In answer to ⟨Q1⟩, we first define an axis of “hardness” along which we can break down the predictions of the Little models. In lieu of an objective notion of hardness, we use the model confidence $max(F_{small}(x_{i}))$ as a surrogate since it reflects model calibration, i.e., higher prediction confidence correspond to higher accuracy (Figure 2 left).

Intuitively, there are 3 simple hypotheses on how scaled up “Big” models help correct the mistakes of base “Little” models:

- ⟨H1⟩
Big models uniformly help samples across difficulty,

- ⟨H2⟩
Big models preferentially help with “hard” samples,

- ⟨H3⟩
Big models preferentially help with “easy” samples.

Using the Pytorch [] pretrained EffcientNet family [] as an example, the right panel in Figure 2 visualizes prediction mistakes by Little model confidence (B0, B2, and B4) on the ImageNet-1K validation set. The mistakes made by Little models are divided into two categories, correctable (solid bars) and non-correctable (shaded bars) by the Big model. The full height of each bar (solid + shaded parts) sums up to the total number mistakes in the corresponding bin.

Quantitatively, the average confidence of correctable mistakes for EfficientNet-B0+B7, EfficientNet-B2+B7, EfficientNet-B4+B7 pairs are $0.38$, $0.41$, and $0.30$ respectively. In fact, $90\%$ of correctable mistakes fall under confidence thresholds of $0.65$, $0.67$, $0.47$, respectively. This suggests ⟨H2⟩ is the likely to be true and motivates the two-pass algorithm in the next section.

## 4 Two-Pass Little-Big Algorithm

### 4.1 Speeding Up Big Models

To answer ⟨Q2⟩, when we allow a sample to be solved with more than one pass like human vision, a simple way is to have a Little-Big pair $G_{F_{Little},F_{Big}}(x,T)$ or shortly $G(x,T)$ for simplicity:

$$ $G_{F_{Little},F_{Big}}(x,T)=\begin{cases}F_{Little}(x),\text{if }max(F_{Little }(x))\geq T\\ F_{Big}(x).\\ \end{cases}\qquad\text{\bf Little-Big}$ (4) $$

The essence of this Little-Big algorithm is to use a light-weight model to pre-screen samples and only pass hard samples to the Big model (see Algorithm). The average per-sample cost of inference over a dataset $D$ can then be expressed as:

$$ $\displaystyle\operatorname{\mathbb{E}}_{D}(MACs[G(x,T)])$ $\displaystyle=\frac{1}{|D|}\bigg{[}\sum_{x\sim D}MACs[F_{Little}(x)]+\sum_{x \sim D^{*}}MACs[F_{Big}(x)]\bigg{]}$ $\displaystyle=MACs[F_{Little}(x)]+\frac{|D^{*}|}{|D|}MACs[F_{Big}(x)],$ (5) $$

where $D^{*}\subseteq D$ is defined as the set of $x$ where $max(F_{Little}(x))<T,\forall x\in D$.

Using EfficientNet-B7 [] as an example Big model to speed up, the top left panel in Figure 3 shows how the relative size $|D^{*}|/|D|$ varies as a function of threshold $T$ with Little models ranging from EfficientNet-B0 to B6. The shape of the curves correspond to the cumulative distribution of prediction confidence for each Little model. Equation 4.1 further links $|D^{*}|/|D|$ to the relative compute cost $\operatorname{\mathbb{E}}_{D}(MACs[G(x,T)])/\operatorname{\mathbb{E}}_{D}(MACs
[F_{Big}(x)])$: the higher the threshold the more samples that get passed to the big model, thus increasing MACs. Since $|D^{*}|/|D|\leq 1$, the upper bound of relative MACs overhead in the worst case scenario is $MACs[F_{Little}(x)]/MACs[F_{Big}(x)]$, which is usually no greater than $1$ with proper choices of the Little model. As shown in the top middle panel of Figure 3, the net effect of adding a pre-screening Little model to the Big model leads to significant reduction in compute cost for a wide range of $T$ across difference choices of Little models.

Figure: Figure 3: Speeding up EfficientNet-B7 with smaller EfficientNets. For using Little-Big, one needs to choose what small model to use and set a threshold $T$ based on an accuracy or MACs target. 50 evenly spaced $T$ in the range of 0 to 1 are sampled to generate each curve. “$\star$” marks the accuracy-MACs tradeoff of the Big EfficientNet-B7. “$\star$” indicates the optimal Little-Big pair without any loss of accuracy on ImageNet-1K, achieving 81% of MACs reduction. The same fixed $T$ performs well on both ReaL and ImageNet-V2 as well. Importantly, the optimal pair achieves both better accuracy and lower MACs than simply scaling down B7 to B6 or B5.
Refer to caption: x3.png

### 4.2 Speeding Up Without Losing Accuracy

Figure: Little-Big Algorithm: Pytorch pseudo code of Little-Big for single image inference. Separate pre-processing image transforms are included as Big and Little models may require different input resolution and/or interpolation. The implementation keeps both models in the memory, Big and Little models can be loaded/unloaded to avoid overhead in max memory usage.

#### Accuracy-MACs trade-off

In addition to MACs, one is interested in how the accuracy of $G(x,T)$ changes as a function of $T$. While it is not surprising that the accuracy generally decreases with smaller $T$, eventually degenerating to the accuracy of the Little model at $T=0$, certain choices of $G(x,T)$ achieve very little accuracy drop or even slight accuracy boost across a wide range of $T$. The bottom left panel of Figure 3 visualizes the accuracy-MACs trade-off on ImageNet-1K. One might immediately notice that any point on the curves is a valid $G(x,T)$ model, and adjusting $T$ allows traversal along the full curve. Many points on certain curves, such as $G_{B4,B7}(x,T)$, achieves new Pareto optima. A convenient way to pick an optimal operating point of $G(x,T)$ is to set up a target accuracy either in the form of absolute accuracy or tolerable accuracy loss $\Delta Acc$. Setting $\Delta Acc>=0$, one can easily draw a horizontal line on the accuracy-MACs plot (dashed line in bottom row of Figure 3) and find the leftmost intersection point (blue star, $G_{B4,B7}(x,T=0.24)$), yielding the Little-Big pair with the least compute cost while satisfying the accuracy target, speeding up B7 by $81\%$.

#### Generalization beyond ImageNet-1K

To test whether the optimal $G$ determined on ImageNet-1K generalizes well, individual models as well as Little-Big pairs are evaluated two additional datasets: 1) ImageNet-ReaL [] where the ground truth labels of the ImageNet-1K validation set are reassessed with an improved labeling protocol ((Figure 3 bottom middle), and 2) ImageNet-V2 [] where slightly “harder” validation images are collected (Figure 3 bottom right). Comparing accuracy-MACs curves across the three datasets, one may notice that although the absolute accuracies and MACs varies, the shapes match qualitatively. Quantitatively, we find the optimal $G_{B4,B7}(x,T=0.24)$ found on ImageNet-1K performs well on ReaL and V2, with only marginal accuracy losses of $0.04\%$ and $0.07\%$, respectively, validating the generalizability of the optimal $G$.

Evidently, the key to determining the optimal $G$ is estimating the accuracy-MACs trade-off curves on the target distribution $D$. In practice, however, one might only have access to a small subset of $D$ or a set $D^{\prime}$ close to to $D$. We simulate this case by determining an optimal $G(x,T)$ on the smaller V2 (10000 samples) and test generalization performance on ImageNet-1K and ReaL (50000 samples). Following the aforementioned process of determining optimal $G(x,T)$ on V2 (Appendix Figure 5) yields a similar optimal pair $G^{\prime}_{B4,B7}(x,T=0.28)$), speeding up B7 by a similar $78\%$ on ImageNet-1K. This validates the robustness of such a process.

### 4.3 Speeding Up Models Across Types And Scales

**Table 2: The simple Little-Big algorithm achieves strong MACs reduction across model types and scales. Thresholds ($T$) are determined as the minimum value achieving a preset accuracy loss tolerance on ImageNet-1K (IN-1K). All but InterImage-G-512 are compressed without loss in IN-1K accuracy. Models in blue denote the Little model is from a different model family as the Big model. Configurations with lowest MACs are in bold.**
| Model | Params | Top@1 Accuracy ($\%$) | GMACs |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| IN-1k | ReaL | V2 | IN-1k&ReaL | IN-V2 |  |  |
| Efficientnet-B2-288 [] | $9.1M$ | $80.56$ | $86.31$ | $68.95$ | $1.09$ | $1.09$ |
| +B0-224 (T=0.66) | $+3.5M_{(+58\%)}$ | $80.59_{(+0.03)}$ | $86.35_{(+0.04)}$ | $68.99_{(+0.04)}$ | $0.78_{(-28\%)}$ | $0.91_{(-16\%)}$ |
| +EfficientViT-B1-224 (T=0.58) | $+9.1M_{(+100\%)}$ | $80.57_{(+0.01)}$ | $86.35_{(+0.04)}$ | $68.92_{(-0.03)}$ | $\bm{0.73_{(-33\%)}}$ | $\bm{0.83_{(-24\%)}}$ |
| Efficientnet-B3-300 [] | $12.2M$ | $82.01$ | $87.28$ | $71.16$ | $1.83$ | $1.83$ |
| +B1-240 (T=0.66) | $+7.8M_{(+64\%)}$ | $82.01_{(+0.00)}$ | $87.35_{(+0.07)}$ | $71.15_{(-0.01)}$ | $1.36_{(-26\%)}$ | $1.56_{(-15\%)}$ |
| +EfficientViT-B1-224 (T=0.78) | $+9.1M_{(+74\%)}$ | $82.01_{(+0.00)}$ | $87.31_{(+0.03)}$ | $70.94_{(-0.22)}$ | $\bm{1.07_{(-42\%)}}$ | $\bm{1.27_{(-31\%)}}$ |
| Efficientnet-B4-380 [] | $19.3M$ | $83.45$ | $88.43$ | $73.27$ | $4.6$ | $4.6$ |
| +B3-300 (T=0.50) | $+12.2M_{(+63\%)}$ | $83.46_{(+0.01)}$ | $88.42_{(-0.01)}$ | $73.39_{(+0.12)}$ | $2.7_{(-38\%)}$ | $\bm{3.1_{(-29\%)}}$ |
| +B2-288 (T=0.72) | $+9.1M_{(+47\%)}$ | $83.47_{(+0.02)}$ | $88.42_{(-0.01)}$ | $73.28_{(+0.01)}$ | $\bm{2.7_{(-39\%)}}$ | $3.2_{(-27\%)}$ |
| +B1-240 (T=0.86) | $+7.8M_{(+40\%)}$ | $83.46_{(+0.01)}$ | $88.44_{(+0.01)}$ | $73.27_{(+0.00)}$ | $4.0_{(-10\%)}$ | $4.3_{(-3\%)}$ |
| +B0-224 (T=0.94) | $+5.3M_{(+27\%)}$ | $83.45_{(+0.00)}$ | $88.43_{(+0.00)}$ | $73.29_{(+0.02)}$ | $3.9_{(-10\%)}$ | $4.2_{(-4\%)}$ |
| EfficientViT-B3-288 [] | $48.6M$ | $84.13$ | $88.49$ | $74.12$ | $6.5$ |  |
| +B3-224 (T=0.60) | $+48.6M_{(+100\%)}$ | $84.14_{(+0.01)}$ | $88.49_{(+0.00)}$ | $73.98_{(-0.14)}$ | $4.7_{(-28\%)}$ | $5.1_{(-22\%)}$ |
| +B2-288 (T=0.76) | $+24.3M_{(+50\%)}$ | $84.13_{(+0.00)}$ | $88.55_{(+0.06)}$ | $73.82_{(-0.30)}$ | $\bm{3.8_{(-42\%)}}$ | $\bm{4.3_{(-33\%)}}$ |
| +B2-224 (T=0.94) | $+24.3M_{(+50\%)}$ | $84.14_{(+0.01)}$ | $88.52_{(+0.03)}$ | $74.11_{(-0.01)}$ | $\bm{3.8_{(-42\%)}}$ | $4.5_{(-30\%)}$ |
| ConvNext-L-224 [] | $197.8M$ | $84.39$ | $88.75$ | $74.34$ | $34.3$ |  |
| +S-224 (T=0.52) | $+50.2M_{(+25\%)}$ | $84.39_{(+0.00)}$ | $88.75_{(+0.00)}$ | $74.35_{(+0.01)}$ | $15.7_{(-54\%)}$ | $19.2_{(-44\%)}$ |
| Efficientnet-B7-600 [] | $66.3M$ | $84.11$ | $88.84$ | $74.39$ | $37.8$ |  |
| +B6-528 (T=0.24) | $+43.0M_{(+65\%)}$ | $84.13_{(+0.02)}$ | $88.90_{(+0.06)}$ | $74.50_{(+0.11)}$ | $20.1_{(-47\%)}$ | $21.4_{(-43\%)}$ |
| +B5-456 (T=0.38) | $+30.4M_{(+46\%)}$ | $84.12_{(+0.01)}$ | $88.78_{(-0.06)}$ | $74.65_{(+0.26)}$ | $13.2_{(-65\%)}$ | $15.4_{(-59\%)}$ |
| +B4-380 (T=0.24) | $+19.3M_{(+29\%)}$ | $84.12_{(+0.01)}$ | $88.80_{(-0.04)}$ | $74.32_{(-0.07)}$ | $\bm{7.1_{(-81\%)}}$ | $\bm{9.5_{(-75\%)}}$ |
| +B3-300 (T=0.66) | $+12.2M_{(+18\%)}$ | $84.13_{(+0.02)}$ | $88.88_{(+0.04)}$ | $74.41_{(+0.02)}$ | $14.6_{(-61\%)}$ | $18.9_{(-50\%)}$ |
| +B2-288 (T=0.74) | $+9.1M_{(+14\%)}$ | $84.11_{(+0.00)}$ | $88.83_{(-0.01)}$ | $74.39_{(+0.00)}$ | $15.6_{(-59\%)}$ | $20.1_{(-47\%)}$ |
| +B1-240 (T=0.90) | $+7.8M_{(+12\%)}$ | $84.12_{(+0.01)}$ | $88.85_{(+0.01)}$ | $74.34_{(-0.05)}$ | $32.9_{(-13\%)}$ | $34.6_{(-8\%)}$ |
| +B0-224 (T=0.92) | $+5.3M_{(+8\%)}$ | $84.12_{(+0.01)}$ | $88.84_{(+0.00)}$ | $74.40_{(+0.01)}$ | $28.4_{(-25\%)}$ | $31.1_{(-18\%)}$ |
| EfficientViT-L3-384 [] | $246.0M$ | $86.34$ | $89.66$ | $77.35$ | $81.1$ |  |
| +L3-256 (T=0.52) | $+246.0M_{(+100\%)}$ | $86.35_{(+0.01)}$ | $89.71_{(+0.05)}$ | $77.36_{(+0.01)}$ | $40.4_{(-50\%)}$ | $44.3_{(-45\%)}$ |
| +L2-384 (T=0.60) | $+63.7M_{(+26\%)}$ | $86.34_{(+0.00)}$ | $89.83_{(+0.17)}$ | $77.55_{(+0.20)}$ | $27.0_{(-67\%)}$ | $31.7_{(-61\%)}$ |
| +L2-288 (T=0.66) | $+63.7M_{(+26\%)}$ | $86.35_{(+0.01)}$ | $89.86_{(+0.20)}$ | $77.37_{(+0.02)}$ | $\bm{19.8_{(-76\%)}}$ | $\bm{25.0_{(-69\%)}}$ |
| DeiT3-L-384 [] | $304.8M$ | $87.73$ | $90.24$ | $79.34$ | $191.2$ |  |
| +B-224 (T=0.82) | $+86.6M_{(+28\%)}$ | $87.73_{(+0.00)}$ | $90.23_{(-0.01)}$ | $79.36_{(+0.02)}$ | $71.2_{(-63\%)}$ | $90.3_{(-53\%)}$ |
| +EfficientViT-L2-288 (T=0.90) | $+63.7M_{(+21\%)}$ | $87.73_{(+0.00)}$ | $90.37_{(+0.13)}$ | $79.43_{(+0.09)}$ | $\bm{54.7_{(-71\%)}}$ | $\bm{74.0_{(-61\%)}}$ |
| ViT-H-14-518 [] | $633.5$ | $88.55$ | $90.51$ | $81.12$ | $1016$ |  |
| +L-16-512 (T=0.46) | $+305M_{(+48\%)}$ | $88.59_{(+0.04)}$ | $90.89_{(+0.38)}$ | $81.04_{(-0.08)}$ | $430_{(-58\%)}$ | $488_{(-52\%)}$ |
| +DeiT3-L-384 (T=0.70) | $+305M_{(+10\%)}$ | $88.56_{(+0.01)}$ | $90.64_{(+0.13)}$ | $81.06_{(-0.06)}$ | $\bm{204_{(-80\%)}}$ | $\bm{276_{(-73\%)}}$ |
| InternImage-G-512 [] | $3.076B$ | $90.05$ | $90.97$ | $83.04$ | $2700$ |  |
| +XL-384 (T=0.84) | $+335M_{(+11\%)}$ | $90.01_{(-0.04)}$ | $90.98_{(+0.01)}$ | $82.99_{(-0.05)}$ | $1436_{(-47\%)}$ | $1781_{(-34\%)}$ |
| +DeiT3-L-384 (T=0.90) | $+305M_{(+10\%)}$ | $90.03_{(-0.02)}$ | $90.99_{(+0.02)}$ | $82.95_{(-0.09)}$ | $\bm{1035_{(-62\%)}}$ | $\bm{1335_{(-51\%)}}$ |

Table 2 shows more examples the strong MACs compression with Little-Big on a variety of model families including CNNs, transformers, and hybrid models, across scales from $1$ to $2700$ GMACs. Thanks to the model agnostic nature of Little-Big , it allows pairing up models of different families (models in blue in Table 2). Large modes such as EfficientVit-L3-384 [], DeiT3-L3-384 [], ViT-H-14-518 [] can be compressed without any loss of accuracy by $70\%$ to $80\%$.

It is worth noting that the roles of Big and Little in the Little-Big pair are relative. For example, DeiT3-L-384 performs well as Little model, efficiently speeding up the 3B-parameter InternImage-G-512 [] and 600M-parameter ViT-H-14-518 by 62% and 80%, respectively. However, it does not imply that DeiT3-L-384 itself cannot act as the Big model and be sped up by an even smaller model. In fact, Table 2 shows that it can be sped up by the EfficientViT-L2-288 by 71%. Similarly, EfficientNet-B4 can serve as a strong Little model for compressing EfficientNet-B7, as well as a Big model to be compressed by the smaller EfficientNet-B2. This enables the generalization of Little-Big to a $K$-pass framework where $K>1$ with additional performance gain (Appendix Section A.1). Another important observation is that the speed up depends on the distribution $D$: Relative MACs reduction are consistently lower on the slightly harder ImageNet-V2 than on ImageNet-1K. This suggests that one may measure distribution shift by measuring average MACs with Little-Big.

### 4.4 Comparison with Prior Art

**Table 3: Little-Big outperforms a variety of pruning methods and adaptive compute models (†). Models in blue denote the Little model is from a different model family as the Big model. Thresholds in red indicate the Little model can simply replace the Big model, achieving model compression and better accuracy. “+disl.” denotes additional distillation training.**
| Model | Method | ImageNet-1K | GMACs |  |  |
| --- | --- | --- | --- | --- | --- |
| Accuracy (%) | $\Delta$ | Remaining | $\Delta$ |  |  |
| DeiT-S-224 [] | Baseline [] | $78.9$ | $--$ | $4.6$ | $--$ |
| DynamicViT-DeiT-S† [] | $78.3$ | $-0.6$ | $3.4$ | $-26\%$ |  |
| A-ViT-S† [] | $78.6$ | $-0.3$ | $3.6$ | $-22\%$ |  |
| A-ViT-S† + disl. [] | $80.7$ | $+1.8$ | $3.6$ | $-22\%$ |  |
| eTPS-DeiT-S [] | $79.7$ | $+0.8$ | $3.0$ | $-35\%$ |  |
| dTPS-DeiT-S [] | $80.1$ | $+1.2$ | $3.0$ | $-35\%$ |  |
| SPViT-DeiT-S [] | $78.3$ | $-0.6$ | $3.3$ | $-28\%$ |  |
| SPViT-DeiT-S + disl. [] | $80.3$ | $+1.4$ | $3.3$ | $-28\%$ |  |
| Our Baseline | $79.81$ | $+0.9$ | $4.6$ | $--$ |  |
| +EfficientViT-B1-224(T=0.44) | $79.81$ | $+0.9$ | $\bm{1.1}$ | $\bm{-77\%}$ |  |
| +EfficientNet-B2-288(T=0.00) | $80.56$ | $+1.7$ | $1.1$ | $-76\%$ |  |
| DeiT3-S Baseline [] | $83.05$ | $+4.1$ | $4.6$ | $--$ |  |
| +EfficientViT-B1-224(T=0.54) | $\bm{83.11}$ | $\bm{+4.2}$ | $2.1$ | $-54\%$ |  |
| +EfficientNet-B2-288(T=0.52) | $83.06$ | $+4.1$ | $2.0$ | $-56\%$ |  |
| DeiT-B-224 [] | Baseline | $81.81$ | $--$ | $17.6$ | $--$ |
| DynamicViT-DeiT-B† [] | $81.4$ | $-0.4$ | $11.4$ | $-35\%$ |  |
| SCOP [] | $79.7$ | $-2.1$ | $10.2$ | $-42\%$ |  |
| UVC [] | $80.57$ | $-1.24$ | $8.0$ | $-55\%$ |  |
| WDPruning [] | $80.76$ | $-1.05$ | $9.9$ | $-44\%$ |  |
| X-Pruner [] | $81.02$ | $-0.99$ | $8.5$ | $-52\%$ |  |
| eTPS-DeiT-B [] | $81.1$ | $-0.7$ | $11.4$ | $-35\%$ |  |
| dTPS-DeiT-B [] | $81.2$ | $-0.6$ | $11.4$ | $-35\%$ |  |
| SPViT-DeiT-B [] | $81.5$ | $-0.4$ | $8.4$ | $-52\%$ |  |
| SPViT-DeiT-B + disl. [] | $82.4$ | $+0.6$ | $8.4$ | $-52\%$ |  |
| +DeiT-S (T=0.40) | $81.10$ | $-0.71$ | $6.5$ | $-63\%$ |  |
| +DeiT-S (T=0.60) | $81.83$ | $+0.02$ | $9.0$ | $-49\%$ |  |
| DeiT3-B Baseline [] | $85.75$ | $+3.94$ | $17.6$ | $--$ |  |
| +DeiT3-S (T=0.00) | $83.05$ | $+1.24$ | $\bm{4.6}$ | $\bm{-74\%}$ |  |
| +DeiT3-S(T=0.60) | $85.75$ | $+3.94$ | $11.4$ | $-35\%$ |  |
| +EfficientViT-B2-288 (T=0.60) | $\bm{85.77}$ | $\bm{+3.96}$ | $7.8$ | $-56\%$ |  |
| Swin-S-224 [] | Baseline | $83.17$ | $--$ | $8.7$ | $--$ |
| STEP [] | $79.6$ | $-3.6$ | $6.3$ | $-28\%$ |  |
| WDPruning [] | $81.8$ | $-1.4$ | $6.3$ | $-28\%$ |  |
| X-Pruner [] | $82.0$ | $-1.2$ | $6.0$ | $-31\%$ |  |
| SPViT-Swin-S [] | $82.4$ | $-0.6$ | $6.1$ | $-30\%$ |  |
| SPViT-Swin-S + disl. [] | $83.0$ | $-0.2$ | $6.1$ | $-30\%$ |  |
| +Swin-T (T=0.28) | $82.03$ | $-1.14$ | $4.9$ | $-44\%$ |  |
| +Swin-T (T=0.68) | $\bm{83.18}$ | $\bm{+0.01}$ | $7.0$ | $-20\%$ |  |
| +EfficientViT-B2-224 (T=0.58) | $\bm{83.18}$ | $\bm{+0.01}$ | $\bm{2.5}$ | $\bm{-71\%}$ |  |

As discussed in Section 2.3, many methods have been developed for model compression in various axis. We compare accuracy-MACs tradeoff of Little-Big with many such methods in Table 3. Typical pruning methods such as WDPruning [] and X-Pruner [] selectively removes units that are less important to model accuracy, methods like SPViT (2024) [] “prune” some attention layers into convolutional layers, effecitvely changing the model type. Addtionally, distillation is used to retrain the network to achieve better performance. We show that even with tricks that effectively retrained models, many pruning methods are not competitive, especially compared with modern baselines such as DeiT3 [], which in essence are better trained ViTs. For example, the best performing SPViT-DeiT-B with distillation failed to outperform a better trained DeiT3-S baseline in both accuracy, model size, and MACs. It remains to be seen whether these methods work with the improved baseline models. In contrast, the process of choosing $T$ for lossless compression with Little-Big in Section 4.2 will yield $T=0$ which automatically suggests replacement of the Big with the more performant Little model. Adaptive compute may still be an interesting direction, however popular models such as A-ViT [] also fail to match the performance of better trained baseline models or show that their adaptive models are really better than simply scaling down the model moderately to match the MACs of the adaptive counterparts.

## 5 Discussion

A large corpus of literature in modern computer vision [, , , , , , ] has followed the norm of developing single-pass solutions trained end-to-end for a wide variety of vision tasks ranging from image/video classification to dense prediction.

While many multi-pass test-time augmentations (TTAs) [] that aggregate predictions of several augmented views of the same sample have been developed as a post-hoc add-on to improve performance, these methods go in the opposite direction of Little-Big . Fundamentally, $K$-pass TTAs are multiplicative methods that trades $K\times$ inference cost for slight improvement of accuracy. Little-Big is a subtractive multi-pass algorithm that relies on a good decomposition of problems and solve each part with the least compute, not unlike Speculative Decoding in languange modelling []. What TTA and Little-Big share in common is that both are post-hoc methods that do not require any modification to the original models. It is actually possible to combine both in the same inference pipeline much like human vision to make predictions adaptively.

A potentially important contributing factor to the sub-optimal performance of popular adaptive compute models such as DynamicViT and A-ViT lies in their end-to-end training protocol. As the number of tokens decrease over depth, the deeper layers effectively “see” less pixels during training. This implicit coupling between the need to reduce compute cost at inference time and at training time due to the end-to-end training protocol may hinder the potential of adaptive methods. In contrast, Little-Big decouples inference and training compute costs, and uses models individually trained on all pixels on full datasets.

#### Limitations

It is worth noting that, while Little-Big achieves drastic MACs reduction across model families and scales, the speed up is not free: since an additional Little model is needed to make the Little-Big pair, the storage overhead is usually a small fraction of the storage requirement of the Big model. However, the storage overhead does not necessarily translate to memory overhead. Although the example Pytorch pseudo code keeps both Big and Little models in memory at the same time to minimize latency while increasing maximum memory usage, one can alternatively only load one model into the memory at a time so that there is no overhead on maximum memory usage. Batching predictions can further reduce the memory I/O overhead per sample.

#### Extensions

It possible to extend the same principles of Little-Big to other tasks by re-examining and modifying Equation 1. For video classification,
where the input $x$ is updated as a sequence of images with an added time dimension $t$, $x\in\mathbb{R}^{C\times W\times H\times t}$. The inference cost of a neural video classifier is given by:

$$ $MACs[F(x)]\approx C_{F}*H^{2}*t*w^{2}*l,$ (6) $$

which makes brute-force upscaling more costly because of the additional $t$ dimension. However, decomposition based on Equations 4 and 4.1 will most likely generalize to this task provided video classifiers are well calibrated and video samples are decomposable by some simple metrics like confidence. For dense image prediction tasks like semantic segmentation, the prediction $y$ becomes a map, where
$y\in[0,1]^{W\times H\times N}$.
In principle, one can continue perform decomposition on a per sample basis, but it may be more efficient to perform decomposition on a per pixel or per patch level.

## 6 Conclusion

In summary, we investigate how scaled-up models help base models correct their mistakes, show that the former preferentially help with samples with low confidence predictions. Inspired by that, we propose a simple two-pass Little-Big algorithm that selectively pass “difficult” samples to large models, achieving drastic MACs reduction of up to $80\%$ without sacrificing accuracy for a wide range of model families and sizes.

The inefficiency in model scale-up and the effectiveness of Little-Big in compressing the scaled-up models are two sides of the same coin, with effective decomposition of samples in a dataset being the key to connecting the two sides. Despite being embarrassingly simple and surprisingly effective, Little-Big is considered as a lower bound on how much a model can be compressed without losing accuracy. More sophisticated ways to decompose the data distribution and better ways to use the Little models output are promising directions to further improve the performance of such subtractive multi-pass algorithms.

## 7 Acknowledgement

This work was performed under the auspices of the U.S. Department of Energy by the Lawrence Livermore National Laboratory under Contract No. DE-AC52-07NA27344. Supported by the LDRD Program under project 22-ERD-006. LLNL-JRNL-864664.

## Appendix A Appendix

### A.1 Additional Experimental Details

Implementation The three datasets used in this work, ImageNet-1K,
ImageNet-ReaL, and ImageNet-V2 are released with ImageNet license [https://www.image-net.org/download.php](https://www.image-net.org/download.php), Apache-2.0 license, and MIT license, respectively. All models are implemented in Pytorch []. EfficientNet [], Swin [], ConvNext [], and ViT [] checkpoints are loaded from public Torchvision pretrained models with BSD-3 license. Pretrained models including EfficientViT [], DeiT [], DeiT3 [] are accessed on Timm [] under Apache-2.0 license. The official code and weight of InternImage models [] are accessed under MIT licence.

Model inference are conducted on an NVIDIA RTX 3090 with 24 GB vRAM, taking  1 minute to  4 hours to finsh evaluation on the ImageNet-1K validation set. The main results are based on measurements of single pretrained models. To estimate the error bar on single model accuracy, we train a small EfficientNet-B0 on ImageNet-1K from scratch for 300 epochs on 32 NVIDIA V100 GPUs with cosine learning rate decay [], TrivialAugment [], input resolution of 224, batch size of 2048, AdamW [] with initial learning rate of 0.003, and weight decay of 0.05. Three models are independently trained from scratch and achieve top-1 accuracies of $76.76\%$, $76.43\%$, $76.56\%$, with a standard deviation of $0.14\%$.

As shown in Section 4.2, the dataset used in choosing $T$ can affect the optimal $T$ as well as relative MACs reduction. By choosing $T$ over ImageNet-1K, Real, V2 for the same model pair, we estimate MACs reduction reported in Tables 2 and 3 to be $\sim 2\%$, which is an order of magnitude smaller than the effect size of $50\%$ throughout the paper, validating the statistical significance of our speedup.

**Table 4: To simulate errors stemming from noise in determining the optimal $T$, we choose $T$ for each Little-Big pair on ImageNet-1K, Real, and V2 and compute mean and standard deviation of accuracy change and MACs reduction on ImageNet-1K.**
| Choose $T$ on: | IN-1K | ReaL | $V2$ |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | $\Delta Acc_{1K}$ | $\Delta GMACs_{1K}$ | $\Delta Acc_{1K}$ | $\Delta GMACs_{1K}$ | $\Delta Acc_{1K}$ | $\Delta GMACs_{IN}$ | $\Delta Acc_{1K}$ | $\Delta GMACs_{IN}$ | $\Delta Acc_{1K}$ | $\Delta GMACs_{IN}$ |
| Big Model | Little Model |  |  |  |  |  |  | Mean | SD |  |  |
| EfficientNet-B7-600 | B6-528 | $+0.02$ | $-47\%$ | $-0.01$ | $-48\%$ | $-0.11$ | $-49\%$ | $-0.03$ | $-48\%$ | $0.06$ | $1\%$ |
| B5-456 | $+0.01$ | $-65\%$ | $+0.09$ | $-62\%$ | $-0.20$ | $-68\%$ | $-0.03$ | $-65\%$ | $0.12$ | $2\%$ |  |
| B4-380 | $+0.01$ | $-81\%$ | $+0.05$ | $-80\%$ | $+0.09$ | $-78\%$ | $+0.05$ | $-80\%$ | $0.03$ | $1\%$ |  |
| B3-300 | $+0.02$ | $-61\%$ | $-0.02$ | $-65\%$ | $-0.01$ | $-69\%$ | $-0.00$ | $-65\%$ | $0.02$ | $3\%$ |  |
| B2-288 | $+0.00$ | $-59\%$ | $+0.01$ | $-53\%$ | $-0.16$ | $-67\%$ | $-0.05$ | $-60\%$ | $0.08$ | $6\%$ |  |
| B1-240 | $+0.01$ | $-13\%$ | $-0.01$ | $-18\%$ | $-0.12$ | $-51\%$ | $-0.04$ | $-27\%$ | $0.06$ | $17\%$ |  |
| B0-224 | $+0.01$ | $-25\%$ | $+0.01$ | $-25\%$ | $-0.03$ | $-44\%$ | $-0.01$ | $-31\%$ | $0.02$ | $9\%$ |  |

As discussed in Section 2.3, the roles of Big and Little in the Little-Big pair are relative. For example, DeiT3-L-384 performs well as Little model, efficiently speeding up the 3B-parameter InternImage-G-512 [] and 600M-parameter ViT-H-14-518 by 62% and 80%, respectively. However, it does not imply that DeiT3-L-384 itself cannot act as the Big model and be sped up by an even smaller model. In fact, Table 2 shows that it can be sped up by the EfficientViT-L2-288 by 71%. Similarly, EfficientNet-B4 can serve as a strong Little model for compressing EfficientNet-B7, as well as a Big model to be compressed by the smaller EfficientNet-B2. This enables the generalization of Little-Big to a $K$-pass framework where $K>1$ with additional performance gain.

One can extend Little-Big to a 3-pass Tiny-Little-Big model $G_{F_{Tiny},F_{Little},F_{Big}}(x,T_{1},T_{2})$ or shortly $G(x,T_{1},T_{2})$ for simplicity:

$$ $G_{F_{Tiny},F_{Little},F_{Big}}(x,T_{1},T_{2})=\begin{cases}F_{Tiny}(x),\text{ if }max(F_{Tiny}(x))\geq T_{1}\\ F_{Little}(x),\text{if }max(F_{Tiny}(x))<T_{1},max(F_{Little}(x))\geq T_{2}\\ F_{Big}(x).\\ \end{cases}$ (7) $$

A 3-pass model $G_{B2,B4,B7}(x,T_{1}=0.74,T_{2}=0.26)$) further compresses EfficientNet-B7, achieving 85% MACs reduction while preserving the Big model performance.

### A.2 Additional Data

Figure: Figure 4: More examples with ConvNext, EfficientViT, and DeiT3. Prediction confidence of individual models correspond well with prediction accuracy (left), which allows us to approximate a “difficulty” axis with prediction confidence. Breaking down the mistakes of little models by difficulty, we find that big models disproportionally “correct” mistakes that are difficult to the little models.
Refer to caption: x4.png

Figure: Figure 5: Speeding up EfficientNet-B7 with the EfficientNet family. Different from Figure 3 where $T$ is chosen on ImageNet-1K, the optimal pair (blue star) is chosen on the smaller V2 set. This yields a similar optimal $T=0.28$ achieving 78% of MACs reduction.
Refer to caption: x7.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: (1) The authors could also plot a figure for top-1 accuracy vs. latency.

(2) How long does it take to load and unload models compared with the inference latency for each batch? I'm wondering whether the method can be used for online inference.
soundness: 3
strengths: (1) The motivation and method of Little-Big is very simple and straightforward. 

(2) It seems that Little-Big is very easy to implement. In addition, Little-Big is model-agnostic which can be applied to models with different scales and architectures.

(3) Little-Big can accelerate a pre-trained model without introducing additional training cost.
summary: The paper proposes a method named Little-Big to accelerate image classification with neural networks. Little-Big uses a light-weight model to quickly classify all of the samples and selects the "hard" samples which get low confidence behind the threshold. Then, it uses a large model to update the prediction for each hard sample. Little-Big can significantly reduce the inference cost and latency for many advanced large classification models without sacrificing the accuracy. The authors provide many experiments with different pairs of large and small models to validate the effectiveness of Little-Big.
weaknesses: (1) Lack of novelty. As the authors say, Little-Big is an embarrassingly simple method, which adopts a large model and a light-weight model for image classification. It's the major advantage but also the major disadvantage of Little-Big. Many previous works share the similar motivation with Little-Big which uses different networks for accelerating, such as early existing and speculative decoding as you mentioned in the paper. While these works mostly include specific and delicate designs. I understand that Little-Big is very simple, but i don't think it's novel.

(2) The proposed method only focuses on the classification tasks. While the authors provide the example about how to extend it to video classification, it's hard to directly apply the method for other popular tasks (e.g., object detection and segmentation), which limits its use.

(3) The authors could include more classification tasks to further prove the generation ability of Little-Big, such as multi-label classification, binary classification.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: - Please clarify the relationship between "hardness" and model confidence. Is low confidence always indicative of a hard sample, or are there cases where this assumption does not hold?
- Can you provide results where the threshold T is determined using only the training set or a held-out portion of the validation set? This would help to assess the potential for overfitting to the validation set and the generalizability of the method.
soundness: 2
strengths: - The proposed Little-Big algorithm is conceptually straightforward and easy to implement. It requires minimal modifications to existing models and training pipelines.
- The paper demonstrates significant MACs reduction across a range of model architectures (CNNs, transformers, hybrids) and scales, suggesting broad applicability.
- Experiments are conducted on multiple datasets (ImageNet-1K, ImageNet-ReaL, ImageNet-V2) to evaluate the robustness and generalizability of the method.
- The Little-Big approach addresses a critical issue in deploying large vision models: their high computational cost. The proposed method offers a practical solution for model compression without retraining or complex modifications.
summary: This paper proposes a simple yet effective two-pass algorithm called "Little-Big" to speed up image classifiers.  The core idea is to leverage a smaller, less computationally expensive "Little" model to pre-screen input samples. Only samples for which the Little model exhibits low confidence are then passed to a larger, more accurate "Big" model.  The paper claims that this approach significantly reduces the computational cost (measured by Multiply-Accumulate operations or MACs) for a variety of model architectures and scales, without sacrificing accuracy on ImageNet-1K and other datasets. The authors demonstrate MACs reductions of up to 80% while maintaining or even improving accuracy compared to the single Big model baseline. They also argue that this approach is more effective than existing model compression and adaptive computation methods.
weaknesses: # Major
- The method seems to rely on finding an optimal threshold T on the test set (Imagenet validation set) to determine which samples are passed to the Big model. This raises concerns about potential overfitting to the validation set and its impact on generalization performance. Results should be provided using a threshold determined on the training or a held-out portion of the validation set to address this concern.
- The paper could benefit from a more comprehensive discussion of related work, particularly in areas like cascade models and dynamic inference methods.  Specifically, work on early-exit models [1] and confidence-based dynamic routing [2] appears closely related and should be discussed. This would help to better contextualize the novelty and contributions of the proposed approach. UPDATE: {See my response below.} 
- Experiments solely focuses on ImageNet dataset. More experiments needed to understand the robustness of the proposed method. 
- I also find it difficult to parse the results presented in huge tables: specifically Table 3: there are multiple baselines for DeiT models. Are you comparing results with different baseline accuracies? 

[1] https://github.com/txsun1997/awesome-early-exiting?tab=readme-ov-file
[2] https://arxiv.org/pdf/2102.04906

# Minor
- [L132] I can't see the definition of "w and l".
- [Section 2.3] Quantization is a key method for compression and not mentioned here. Also Mixture of Depths (https://arxiv.org/abs/2404.02258)
- The paper's use of "hardness" and its relationship to model confidence is not always clear. In some sections, low confidence is equated with hardness, while in others, the opposite is implied. This needs clarification. For example [L199] "which allows us to approximate a "hardness" axis with prediction confidence." hardness means low confidence, no?

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: Q1) Can you develop better ways to explain your idea’s novelty, or do you have any ideas to further enhance the novelty?

Q2) Can this idea be applied to language models as well? Particularly, I'm interested to see how it compares with speculative decoding.
soundness: 3
strengths: S1) The proposed method is practical. It is easy to implement and does not require any modification or additional training of existing models. 

S2) It is widely applicable. For any classification problem, it’s readily available. We could apply it to other tasks as well if we could come up with confidence estimation methods for them.

S3) Extensive experimental results show that the proposed method is robustly performing well in the ImageNet classification task.
summary: This paper proposes a technique called the Little-Big algorithm. It combines a small and large pre-trained model to improve the trade-off between cost and accuracy. It first applies the small model to a given sample. When the confidence is high, the prediction is returned. Otherwise, the large model is applied, and its prediction is used.

In experiments, the authors focused on the ImageNet-1K image classification task. They demonstrated that the proposed method boosts efficiency for various pairs of models.
weaknesses: W1) The proposed approach lacks novelty. The idea of using multiple models with different cost-accuracy tradeoffs is highly common, to name a few, such as speculative decoding for language models and cascade ranking systems for recommendation and information retrieval. 

W2) The experiments are weak. All the experiments are about ImageNet-1k image classification task, so it is quite uncertain whether this method works well for other tasks as well.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: My main suggestion is regarding the fairness of comparisons with baselines (mentioned above).

I also would like to understand the justification for the choice of Little model (mentioned above).
soundness: 2
strengths: - The paper studies an important topic - efficiency of visual recognition models.
- The speedup claimed by the paper is substantial. At a fixed accuracy, their method improves speed by 30%-80% (Figure 1).
- The paper is written clearly and is easily understandable. The investigation presented studies the natural questions that arise with threshold tuning for the Little model's confidence. Figure 3 clearly demonstrates how the accuracy and efficiency change as a function of the threshold.
- The paper accounts for generalization across different datasets by fixing the threshold on ImageNet and analyzing results on ImageNetv2 and ImageNetReal. This is an important aspect of the investigation, as choosing the threshold based on the validation set can create bias.
summary: The paper discusses improving the speed of visual recognition systems using a "Little-Big" model setup. The Little model is a smaller architecture that processes examples first. If the confidence is below a predefined threshold, the sample is reprocessed by a "Big" model. This simple setup improves the speed of visual recognition systems on ImageNet, ImageNetv2, and ImageNetReal significantly (without loss in accuracy). The authors experiment with both CNNs and Transformer models. They also study the fraction of examples processed by the Little and the Big network, showing the accuracy as a function of threshold.
weaknesses: My main concern is the comparison with prior art. As line 437-439 states, "even with tricks that effectively retrained models, many pruning methods are not competitive... with modern baselines... which in essence are better trained ViTs". It seems that the Little-Big method is evaluated using modern architectures and training recipes, whereas other baselines (pruning, etc.) are using older architectures or training recipes. I'm worried that the gains of this method are primarily attributable to the use of newer architectures or training recipes. A fair comparison would use the same architectures as previous works.
- For example, Table 3 shows a datapoint with T=0, meaning the Big architecture is never used.
- Additionally, the baseline architecture in Table 3 ("Our Baseline") is significantly more accurate than previous work's baseline (Yin et al.).

The choice of "Little" network seems arbitrary in some cases. In Table 2, EfficientNet-B2-288 uses EfficientViT as a little network, but most other EfficientNet variants do not. And EfficientNet-B2-288 is not used with EfficientNet-B1, but most other variants are. I have similar thoughts on most of the rows in Table 2. Can you please justify the choice of Little architecture?

Line 132: Equation 2 should have a reference, and there should be some more specific qualifications as to what types of models this equation applies to. Similarly, the characteristic width w_j is not well defined and doesn't have a reference.

Line 150: I recommend also discussing quantization briefly here.

Line 172-173: "ingest gigabits/s of raw visual information and compress it to tens of bits/s" <- this needs a reference

In Table 3, it would help the reader if you mark the baselines by their general approach (e.g. which ones are pruning, etc.).

Line 202: "confidence > 0.5-0.7" <- what does this mean? How can a confidence be greater than a range? Did you instead mean "0.5 < confidence < 0.7"?