From Clutter to Clarity: Visual Recognition through Foveated Object-Centric Learning (FocL)
Authors: Amitangshu Mukherjee
Deepak Ravikumar
Kaushik Roy
Venue: Accepted by TMLR ()

Abstract: Human active vision integrates spatial attention (dorsal) and object recognition (ventral) as distinct information processing pathways. Rapid eye movements focus perception on task-relevant regions while filtering out background clutter. Mimicking this ventral specialization, we introduce FocL (Foveated Object-Centric Learning), a training strategy that biases image classification models toward label-consistent object regions by replacing full images with foveated crops. Standard training often relies on spurious correlation between label and background, increasing memorization of hard examples in the tail of the difficulty distribution. FocL simulates saccades by jittering fixation points and extracting foveated glimpses from annotated bounding boxes. This object-first restructuring reduces non-foreground contamination and lowers mean training loss. FocL reduces memorization, lowering mean cumulative sample loss by approximately 65 % and making nearly all high-memorization samples (top 1 %) easier to learn. It also increases the mean $\ell_2$ adversarial perturbation distance required to flip predictions by approximately 62 %. On ImageNet-V1, FocL achieves up to 11 % higher accuracy on oracle crops. When paired with the Segment Anything Model (SAM) as a dorsal proposal generator, FocL provides around an 7 % gain on ImageNet-V1 and up to 8 % under natural distribution shift (ImageNet-V2). Extending this setup to COCO, FocL improves cross-domain mAP by 3--4 points without any target-domain training. Finally, given object localization (bounding boxes), FocL reaches higher accuracy using roughly 56\% fewer training images, offering a simple path to more robust and efficient visual recognition.

Published in Transactions on Machine Learning Research (02/2026)
From Clutter to Clarity: Visual Recognition through Foveated Object-Centric Learning (FocL)
Amitangshu Mukherjee
makher44 @pprrdue. edu.
Flmore Family School of Electrical and Computer Engineering
Purdue University
Deepak Ravikumar
druvikum Spurduc. edu
Elmore Family School of Electrical and Compuler Engineering Purdue University
Kaushik Roy Elmore Pamoly School of Electrical and Computer Engineering Purduc Universily
kaushikk Spurdue edu
Reviewed on OpenReview: https: // openreiiee. net/ forum? WV=7
Abstract
Human active vision integrates spatial attention (dorsal) and object recognition (ventral) distinct information processing pathways. Rapid eye movements focus perception on task-relevant regions while filtering out background clutter. Mimicking this ventral spe- cialization, we introduce FocI. (Foveated Object-Centric Learning), training strategy that biases image classification models toward label-consistent object regions by replacing full images with loveated crops. Standard training often relies on spurious correlation between label and background, increasing memorization of hard examples the tail of the dif- ficulty distribution. FocL simulates saccades by jittcring fixation points and extracting loveated glimpses from annotated bounding boxes. This object-first restructuring reduces non-foreground contamination and lowers mean training loss. FoeL reduces memorization, lowering mean cumulative sample loss by approximately 65% and making nearly all high- memorization samples (top 1%) easier to learn. It also increases the mean & adversarial perturbation distance required to flip predictions by approximately 62%. On ImageNet-V1, FoeL achieves up to 11% higher accuracy on oracle crops. When paired with the Segment Anything Model (SAM) as a dorsal proposal generator, FocL provides up 7% gain on ImageNet- V1 and up to 8% under natural distribution shift (ImageNet- V2). Extending this setup to COCO, FoeL improves cross-domain mAP by 3-4 points without any target-domain training. Finally, given object localization (bounding boxes), FocL reaches higher accuracy using roughly 56% fewer training images, offering a simple path to more robust and efficient visual recognition.
1 Introduction
Deep neural networks often achieve high performance by relying spurious correlations between labels and irrelevant background features (Bayat et al. 2025 Geirhos et al. 2020), rather than learning robust object-centric represontations. This hinders generalization on hard examples in the tail of the sample-level difficulty distribution, even when class frequencics are balanced (Arpit et al. 2017 Usynin et al. 2024 TTe code available GitHub

---

Published in Transactions on Machine Learning Research (02/2026)
Hard samples cluster in the tail of the sample difficulty distribution
Spurious co-oceurrence leads to semantic contamination
Smoothed Long Distribution (Sample Perspective)
turtle
1000
Easy samples
Hard samples
Unlabeled human presence in ImageNet.
Co-occurring human features with unrelated objects
Effeeber Stnule Count
Swimming cap
600
Tench
Label ambiguity from multiple objects
Labrador [208]
200
Norwegian elkhound [174] Pomeranian [259]
2000 Sorted Sample Index (Kasiest to Hardest)
4033
5003
Shortcut Learning Overfitting to spurious features
Easy
(a)
Measured by a proxy
Hard
Poor generalization memorization
Figure 1: Figure illustrates key challenges that drive memorization and hinder generalization in visual recognition. (Left) A conceptual long-tail curve of sample-level difficulty, with harder examples concentrated in the tail and difficulty measured via proxies such sample loss. (Right) Two major sources of sample- level hardness: (a) Spurious correlations from unlabeled co-occurring entities (c.g., humans) cause models to overfit to background context; (b) Label ambiguity from multi-object images (e.g., a "Labrador" sample also containing other dog breeds) introduces confusion. These effects weaken object-label consistency and promote reliance on shortcuts.
An example of sample-level difficulty is the sample's training loss (or its gradient norm), which quantifies how challenging it is to learn. The left sub-panel of Figure Tauureeillsttnnn Iillustrates this: harder examples concentrate in the tail under difficulty measures (e.g., loss or curvature) (Garg et al. 2024 Ravikumar et al. 2024 2025).
These instances often lead memorization, where models overfit to background context, dataset artifacts, unrelated co-occurring objects instead of focusing the labeled foreground object (Brown et al. 2021:
Feldman & Zhang, 2020 The right sub-panel of Figure iillssrates common failure sources. These failures include unlabeled distractors like humans, and label ambiguily from multiple objects in a single annolated image, for example, a "Labrador" image that also contains other dog breeds.
Existing methods mitigate learning challenges in long-tailed distributions through class re- colbaaanning or data augmentation strategies (Kang el al. 2020 Ren el al.) 2020 Tan et al.] 2021 Yun al.] 2019 Zhang el al. 2018 2023). However, these approaches primarily target class-level imbalance rather than sample-level difficulty. In contrast, focus on individual examples that challenge model even when class frequencies are balanced. This leads to natural question: can generalization be improved by presenting object-centris foveated views analogous to how humans focus on informative regions while filtering out irrelevant. and spurious features?
To explore this, we draw on insights from biological vision, where focusing on task-relevant regions offers natural analogy for object-centric learning. As illustrated in Figure a(a), human perception operates through an active vision system that combines goal-directed sampling with object-centric encoding. The initial visual input is captured via peripheral vision, which provides coarse information across the scene. Based on this, saccadic eye movements shift the fovea, the high-acuity center of the retina, toward salient targets. According to the two-stream hypothesis CClrk 2013 Goodale & Milner 2004 Milner & Goodale, 1992 Mishkin et al. 1983 Sakuraba el al.) 2012 Ungerleider & Haxby] 1994), the dorsal stream computes where to look by identifying spatially informative regions. In parallel, the ventral stream processes the high- resolution fovcated input (Eckstein) 2011 Shao al. 2024g to determine what is being observed, extracting semantic features such as shape and identity. This foveated mechanism allows humans to extract consistent, object-centered representations across varied contexts, forming the basis for robust generalization.
2

---

Published in Transactions on Machine Learning Research (02/2026)
Separate processing pathways
Foveated vision reduces variability and enhances consistency
object
Box turtle
Swimming
Tench
Ventral Stream What the object
Box turtle
Point
Information
Foveated Region
(b)
(c)
(d)
Preserves intra-class consisteney samples
Reduces interference co-occurring features
Preserves
object-label consistency
Fovented input learning the object, suppressing background noise.
FoeL disentangles classes feature space (t-SNE)
Standard supervised model
FoeL supervised model
Ethouetter 0:35
cent-dist 13.32
waker duuriaoo
828
hippopatamue
Spurious
background
(water)
proximity lends to entangled representations
glipbpotamut
Suppressing spurious context (water) improves cluster separation
Figure 2: FocL emulates human foveated vision to improve generalization by suppressing spu- rious context. (a) FoeI. uses object-centric glimpses inspired by human visual streams to focus on relevant features. (b d) This object-centric bias leads to more robust learning outcomes (intra-class consistency, reduced interference, object-label alignment). (Boltom) I-SNE: FoeL (right) achieves better class separation (silhouette +0.19, avg. centroid dist. +1.07), unlike standard models (left) that confuse distinet classes (c.g., hippopotamus/ buffalo) by relying shared water backgrounds.
Inspired by these principles of biological vision, we introduce FocL, which trains networks on foveated, object-centric crops that isolate the foreground to simplify learning and improve generalization. We emulate saccades by jittering bounding-box centers to generate multiple object-foeused glimpses. By suppressing background clutter and isolating task-relevant regions, FoeL reduces sample complezity, shifting hard in- stonces from the tail toward the mode of the difficulty distribution. Rather than requiring models learn from visually complex scenes, FoeL restructures the input space to emphasize object-label consistency, reframing image classification alization on loveated inpuls and reduces memorization. FoeL models require larger adversarial perturbations and converge faster, enabling learning with less data. Figure where FoeL produces tighter semantic clusters. In contrast, a standard model entangles distinct classes, such as hippopotamus and water buffalo, by relying on shared background context (e.g., water). By focusing on loveated object regions, FoeT. learns cleaner semantic boundaries and more robust representations.
simpler, more targeted task. This object-centric strategy improves gener-
2 visualizes this effect in a t-SNE projection,
Our contributions as follows:
FocL: Object-centric training strategy. We introduce FoeL, training method that generates object-centric glimpses by jittering ground-truth boxes, encouraging models focus on foreground features. (Sec)
3

---

Published in Transactions on Machine Learning Research (02/2026)
Reduced memorization and more stable features. FocL mitigates overfitting to non-robust contextual cues by restructuring visual input around object-centric crops. Tt reduces mean cumu- lative sample loss by approximately 63%, indicating a shift toward easier and more stable learning dynamics. Using Feldman Maadann-Zhang memorization scores intersected with our training split, FocL low- ers cumulative sample loss on virtually all (>99%) of the ligh--memorization samples. FoeL also increases the mean &2 perturbation distance required to flip predictions by approximately 62%, rein- forcing the connection between reduced memorization and more stable feature representations. (Sec 1.1
Improved generalization object features. The reduction in memorization leads directly to stronger object-focused generalization. FocL improves Top-1 accuracy by up to 11% over baseline classifiers when each model is evaluated using its appropriate input modality. When paired with SAM as a dorsal proposal generator, Foel. further provides an approximately 7 pp gain over baselines on fixed validation subset considered under the Any-correct metric, which serves as an empirical performance ceiling for the Classifier system. (Sec 4.2)
Transfer to ImageNet- V2 and COCO. Extending the dorsal ventral selup, pairing SAM with FoeL improves recognition under natural distribution shift on ImageNet- V2, yielding an approxi- mately 8 pp gain over SAM paired with a standard classifier. Evaluated on a 1.5 K-image subset from COCO, the same combination achieves 3-4 pp higher mAP at IoU 0.5, demonstrating cross-dataset generalization in multi-object scenes despite no COCO-specifie training. (Sec 4.2
Enhanced learning dynamics and sample efficiency. The simplified, object-centric supervision in FocL yields smoother optimization (approximately 46% lower mean gradient norms) and faster convergence, enabling comparable or higher accuracy with roughly 56% fewer training images under object localization (bounding boxes or proposals) than the standard baseline. (Sec 4.3
Our focus is on systematically demonstrating that a foveated, object- input design yiclds consistent gains across learning dynamics, generalization, and cross-domain transfer. Our Foel classifier acts as a specialist module that enhances standard backbones and interfaces naturally with foundation models such SAM. This demonstrates that biologically inspired design principles can strengthen conventional classifiers without modifying their architecture.
2 Related Work
We provide here a compact yet comprehensive survey of work most relevant to FocI; an expanded version is provided in Appendix A.I
Object-centric and foreground-focused learning. Unsupervised methods such as MONet (Burgess ct al. 2019) and Slot Attention [Locatello et al. 2020) aim to disentangle objects, whereas attention add-ons
(e.g., CBAM (Woo et al. 2018) and discovery pipelines like CutLER (Wang et al. 2023) modulate full-image features or mask foregrounds after the fact. A related thread learns where to look through iterative policies, exemplified by RANett (Mnih et al.) 2014), Saccader-style models (Elsayed et al. 2019), GFNet (Wang
ct al. 2020), FABLE (Ibrayev et al. 2024a). and FALcon (Ibrayev et al. 2024b). FocL instead trains on explicitly supervised foveated crops, aligning the foreground with its label and suppressing background interference. Object localization can be delegated external detectors (c.g., SAM (Kirillov et al. 2023
Ravi et. al. 2024) or FALcon (Ibrayev et al. 2024b)) inference, while our focus remains on the learning benefits of an object-first bias.
Memorization in long-tailed learning. Networks typically fit frequent patterns before memorizing rarc, noisy, or atypical tail instances (Arpit el al. 2017 Feldman & Zhang 2020). Theory and evidence suggest such memorization can sometimes be necessary for accuracy under skewed data [Bownn et al., 2021 Usynin
ct al. 2024), yet it raises fairness, robustness, and privacy concerns (Li ct al., (025). Recent analyses propose proxies like Cumulative Sample Loss (CSL) RRaiikumar et al. (025) and link high input-loss curvature
to memorized long-tail samples (Garg et al. 2024 Ravikumar et. al. 2024 While our work focuses
4

---

Published in Transactions on Machine Learning Research (02/2026)
Standard
FocL supervise model
Tench [0]
Tench
Classifier (Ventral Stream)
Classifler (Ventral Stream)
Data
IP
Augmentation On foveated
Augmentation On full inputs
inputs
Fovented input Centered at X" Fov (X.p)
Standard input
Annotated isput (X.p)
(8)
(b)
Random crops standard input
Random crops on foveated input
(c)
(d)
Figure 3: (a) Standard training uses the full image. (b) FocL replaces the raw input with foveated crop/crops centered on the annotated object. (c-d) Effect of Bandoo-RReizee--Coe Each row shows the original image (left) followed by five crops seen across training epochs. In (c), full-image augmentation often captures irrelevant background (e-g., a fisherman's 's jacket just water), encouraging spu- rious correlations. In contrast, (d) applies the same augmentations to foveated crops, yielding centric views that preserve foreground features. These cleaner views lead to more disentangled, object-aligned rep- resentations (see t-SNE, Figure 2
augmentation under both pipelines.
supervised ImageNet settings, similar unintended "déjà-vu" memorization effects have been reported in self- supervised models (Meehan al. 2023 Kokhlikyan et al., 2024) and vision-language models (Jayaraman)
et al.
2024). Complementary to studies analyzing these memorization dynamics, FocL restructures inputs
remove background clutter, simplifying hard examples and reducing reliance on brittle shorteut cues (Geirhos
et al.
2020 While methods such as Mixup, CutMix, logit adjustment (Zhang et al. 2018 Yun et al.
2019
mitigate class-level imbalance, FocL addresses instance-level difficulty directly.
Fovcation, robustncss, and our contribution. Recent robustness-oriented work blurs or down-samples the periphery, such as R-Blur for adversarial defense (Shah et al.] 2023), textural encodings for IID gains (Gant al. 2021), and active-vision systems that integrate multiple glimpses against transferable attacks (Mukherjee al. 2025). These methods still retain substantial background context, and the robustness- memooiizainn relationship remains delicate; for instance, adversarial training can induce robust overfitting (Dong et al. 2022 Foel, adopts different stance: it restricts the visual field to supervised object-centric crops, substantially reducing peripheral and background information while preserving label- consistent context. The observed increase in mean adversarial distance and the substantial drop in CSL are natural consequences of FocL simplifying each learning instance, rather than outcomes of explicit robust- ness optimization. This improved learnability is accompanied by smoother convergence and more stable, generalizable representations.
3 Methodology
In this soction, we introduce FocL, our multi-glimpse fovcated learning framework for visual recognition. We begin by briefly reviewing standard supervised learning and highlight how data augmentation behaves differently when applied to global versus foveated inputs. We then describe FoeT. in detail.
5

---

Published in Transactions on Machine Learning Research (02/2026)
FocL: Multi glimpse emulating small saccadic shifts
Step 1
Step 2
Step
Step
Constrained Crop and resize glimpses
Seleet One or training
Generate Fixation points around (p)
Sample at most fixation points
Glimpse
Tench
Glimpse 2
Data Augmentation foveated inputs
Foveated Input Centered x" (X.p)
Glimpse 3
Classifler
(Ventral Stream)
Figure 4: FocL with structured glimpse variation (Steps 1-4) simulates small saccadic shifts by jittering the fixation point and extracting up three constrained crops around the object. Each glimpse is resized the input resolution and used individually or jointly during training. These object-centric views reduce the influence of background clutter and encourage the network to focus label-relevant foreground features, promoting stronger alignment between object structure and class semantics.
Standard Supervised Training: conventional supervised pipelines (red panel, Figure ʃ is trained directly on full-resolution images using the slandard cross-ent.rop loss:
31), the classifier
Loup = E(x,))< [f([()), w)],
where (z, v) denotes image-label pair and l is the classification objective. Since the entire image æ serves as input, data augmentations such random resized cropping, horizontal flipping, and color jitter are applied globally across both foreground and background regions. This global augmentation strategy can introduce semantic drift (i.e., misalignment between label semantics and the visual content of augmented crops): the network may passively learn background features that are unrelated to the object label. As shown in Figure 3r, random crops observed during training emphasize irrelevant context, such as the fisherman's 's jacket instead of the tench (top row), or mostly water instead of the red-breasted merganser (bottom row). Such misaligned augmentations promote spurious correlations between background and label, causing the model lo overfit to incidental context rather than learning object-centrie, generalizable representations.
3.1
FocL: Foveated Object- Cenect-Centtic Learning
Given labeled image (I, y), define the annotated bounding box b = (Xmins Vmins Tmax Imax), and let its geometric center define surrogate saccadic fixation point p E TR2. All pixels within b are treated as the effective foreground region representing the object associated with label hence, the fixation is supervised and object-aligned. While biological foveation involves gradual spatial falloff and peripheral blur, we ap- proximate it using hard foveated glimpse obtained by cropping around p to retain the labeled foreground and suppress most surrounding context. Because bounding boxes typically include some peripheral pixcls, the glimpse may contain limited background, but remains substantially more object-focused than full-image crops. This setup is visualized in Figure 3p.
Using this formulation, we instantiate FocL, strategy that generates multiple object-focused glimpses (up to three per image) by applying small, rolled spatial and scale jitter around the initial fixation point -15% of box width/height and +5% scale range; details in Appendix). These jittered glimpses serve relax tight bounding boxes, emulate human-like saccadic sequences, introduce mild viewpoint variations, and mitigate geometric distortions from resizing. By primarily exposing the model these varied object- centric views, FocL encourages strong inductive bias towards foreground features over background clutter.

---

Published in Transactions on Machine Learning Research (02/2026)
Consequently, even when standard augmentations are applied, these glimpses maintain semantic consistency and preserve object identity (yellow panel, Fig. 31).
For each image, we extract up to k = 3 foveated glimpses (a tunable parameter) and treat them as individual training examples that share the same label. During training, these glimpses are inchuded in the same mini- batch (i.e., not shuffled across images) diversity. This setup enables the model to jointly process multiple views of the same object, reinforcing consistent foreground-label mappings. The per-sample loss is computed by averaging over all glimpses follows:
preserve intra-object coherence while still benefiting from batch
FooL = (x,
Here, (2v pi) denotes the 1th foveated crop generated around distinct jittered fixation point ps, sampled from a small neighborhood of the annotated object center p. Although each crop uses its own offset Pt, we write Fov, p) to indicate that all glimpses are relative to the same base annotation. The generation of these object-focused glimpses (illustrated in Figure 4 first samples fixation centers within a small neighborhood around the annotated point. Candidate centers are then adjusted to respect image boundaries, followed by distortion-aware cropping that preserves object aspect ratio during resizing. This design encourages robust learning primarily from foreground regions while maintaining mild variability in position and scale, A detailed algorithm, including jitter magnitude and selection criteria, is provided in Appendix A..2] We apply BatchNorm in the standard way over the concatenated crop batch (across all images and glimpses); while crops from the same image are correlated, each mini-batch still contains many distinct images and independently augmented crops, so correlation mainly enforces intra-object coherence rather than dominating BN statistics.
4 Experiments
We evaluate FocL across three dimensions: robustness to memorization, gencralization under foveated inputs, and sample efficiency. Details regarding hyperparameters and reproducibility are provided Appendix A.3
4.1 Does FocL reduce memorization?
We address this question through two complementary analyses. First, we examine Curullatie Somple Loss (CSL) introducod by Raaikumar et al. (202)), a proxy for sample-level learning difficulty, to quantify how FocL reshapes training dynamies. The CSL distribution shifts from the tail toward the mode, indicating that previously hard examples become easier to learn. To relate this effect to memorization, we align CST values with pre-computed scores from Feldman & Zhang (2020) for our training set's 's ImageNet indices. By identifying the top 1% most memorized samples, we isolate cohort of 820 verifiably memorized examples. FoeT is exceptionally effective for this group, making 99.88% of these samples easier to learn (lower CSL). Finally, we evaluate adversarial resistance to assess whether this reduction in memorization also yiclds more stable and robust features.
Setup All memorization analyses are conducted on the 100K ImageNet partition (details in Appendix A.3 using 85K samples for training, 15K for validation. FocL is evaluated in its singlo-glimpse configuration (k=1), while the standard model is trained on full images. Both use identical hyperparameters and training protocols, ensuring that any observed differences arise solely from the input representation.
Learning Difficulty and Memorization. To investigate how FoeL mitigates memorization, we analyze CSL, a proxy for learning difficulty where higher values correspond to harder-to-learn samples. On an aggregate level, FocL substantially reshapes the CSL distribution (Fig. 5, left), roducing the mean CSL from 206.66 to 72.30. To understand the source of this improvement, we conduct a targeted analysis using pre-computed memorization scores from Feldman & Zhang (2020). By intersecting the indices of the top
7

---

Published in Transactions on Machine Learning Research (02/2026)
CSI Hatagiam foc. Stardard
Standard Input
Confibens Telecary
Standard (n9an=226/655, medion193. 1830
300
Fec. Bingle Crapt Im1an=72.335 med 6n 600)
Foveated Input
Conbiiene Veleeaary
50
80
Carrlctive Sample Less (ESL
(a) Distribution over entire training set
(b) Loss and confidence trajectories for sample
Figure 5: Cumulative sample loss (CSL) memorization proxy analysis. Left: Shift from tail mode. FoeL exhibits significantly lower values, indicating that samples become easier to learn due to object-centric inputs and reduced contextual interference. Right: Example of a high-CSL sample from the Llama class. In the standard model, back- ground elements like the human rntroduce semantic contamination, leading to noisy loss and confidence trajectories. With FoeL, foveated input enables more stable learning, reflected in the smoother trajectories and a large CSL drop from 388.18 to 104.66.
and median CST, and the distribution is tightly concentrated toward lower
FG: 0.023 F2: 0.9960
FG: 0.282 F2: 0,9942
FG: 0.013 F2: 0.9927
FS: 6.056 FZ: 0.9600
FG: 0.079 FZ: 0.9677
Standard CSL 299.5 NNL 33.9
Standar 2041 POOL 92.0
Standard 237.3 CSL 99.2
Starcard 254-5 FOOL COL 97.
Suandard CSL: 120
5
Dining
Littee
Barbell
Stancord Col 294.3 Pool. CSL 25-2
Stardard 231.7 (PeLL CSL 48.5
Standard CoL. 203 4 FoeL COL: $6.5
Surcert CAC 245.7 LSL-433
Soandar: CRL 265 29.27 CSL (A.)
Figure 6: Example visualizations from the top 1% most memorized ImageNet samples (Feldman & Zhang scores). We show representative examples that often include surrounding context beyond the target object; FG is reported only descriptively in these figures (box-area / imaagearea) (e.g., the 'Soccer Ball' example has FG 0.013). By isolating the object region, FocL reduces learning difficulty, reflected by large per-sample drops in CSL.
1% most memorized ImageNet samples with our 85K training set, we identify a cohort of 820 verifiably memorized examples. For this group, FoeL is exceptionally effective, making 99.88% of these samples casier learn (p < 0.001). We assess significance with a one-sided paired sign (exact binomial) test over the 820 samples, where sample is counted improved if CSL < CSLsu: full hypotheses and the exact

---

Published in Transactions on Machine Learning Research (02/2026)
Robust Accuracy vs. Perturbation Magnitude
Robust Accuracy vs. Perturbation Magnitude
100
100
100
Robust, Accuracy (%)
Robust Accuracy (%)
80
60
60
40
40
20
Slandard
20
Slandard
FoeL
FocL
0.10
0.25
0.50
0.75
1.00
0.10
0.25
0.50
0.75
1.00
(L, PGD-10)
(L2 PGD-10)
(a) Train set
(b) Validation set
Figure 7: Robust accuracy of standard and FoeL models against l2 PGD-10 perturbations. At e = 0.25, FoeL achieves 89.59% on the training set versus 57.05% for the standard model, a gap of 32.54 percentage points. On the validation set, the gap is 25.82 percentage points (75.24% vs. 49.42%). The standard model's accuracy drops sharply at small perturbation levels (c 0.25), with many predictions flipped by minimal adversarial budgets. This behavior, especially evident on the training set, suggests memorization and reliance on brittle and non non-generalizable features. The degradation is less pronounced in FocL, likely due to its robust object-centric feature learning that reduces background interference.
p-value are reported in Appendix A.6. We additionally include example visualizations from the top-1% FZ set (Figs. 6 and 12) as qualitative examples of memorized samples; in these figures, we report FG as the ground-truth bounding-box area divided by the full image area. Overall, this targeted analysis shows that FocL's aggregate gains are driven by reducing learning difficulty on such problematic instances, shifting mass from the tail of the difficulty distribution toward its mode.
Adversarial Resistance To further probe the stability of learned representations rather than explicit sraetted-grdieen--dee
adversarial robustness, we apply balanced ImageNet subset. For fairness, each model is attacked its native input type-full images for the standard baseline and foveated crops for FoeL. The analysis indicates that FoeL-trained representations are harder to perturb, reflecting increased stability. The mean adversarial distance, defined as the average l perturbation required to alter a model's prediction, increases by 62% under FoeL training (d = 0.6169 vs. 0.3806 for the standard model), showing that substantially stronger perturbations are needed to induce misclassification. Moreover, Foel. maintains consistently higher accuracy across increasing perturbation magnitudes (e), as shown in Figure 7 In contrast, the standard model exhibits a steep accuracy drop even at small c, suggesting that it relics brittle, non-generalizable fcatures often linked to memorization (Carllni et al. 2019 This evidence complements the CSL analysis, reinforcing that FocL reduces memorization by promoting more stable and semantically aligned features.
(PGD) &a perturbations (adry al., 2018 on
Key Takeaways. FoeL primarily reduces memorization by simplifying learning at the sample level. It markedly lowers mean Cumulative Sample Loss, shifting hard examples from the tail toward the mode of the difficulty distribution, and alleviates background-driven memorization observed the most challenging samples. This improvement in learnability directly translates into greater feature stability: when attacking each method its native input to match its training regime (both use 224x224 inputs: full images for the standard baseline and foveated crops for FoeL; resizing within the attack loop), FocL requires significantly larger & perturbations to flip predictions. We note that cropping changes the input content distribution (e.g., reduced background clutter), which can make perturbations harder to "hide" in background regions; accordingly, these distances should be interpreted as stability of the deployed end-to-end object-centric input representation plus classifier, consistent with more robust, object-centered features. This an intended property of foveated perception, since the goal is suppress background-driver cues and focus decisions on object-relevant evidence. Additional results are provided in Appendix A.6 and A.5
9

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Full-image, Oracle, and active localization (FALcon) performance on ImageNet-Vi (2
K subset). Top-1 / Top-5 accuracy (%) for Standard and FocL classifiers under full-image input, oracle (GT) crops, and FALcon multi-glimpse aggregation.
Single-glimpse setting
Standard (Full Image)
Standard (Oracle)
FocL (Oracle)
Top-1 (%)
60.90 + 1.10
59.83 + 1.35
72.23 0.66
Top-5 (%)
83.05 + 0.74
82.05 + 0.48
90.82 + 0.42
Multi-glimpse (FALcon)
Avg
Voting
Weighted
Standard (Top-1)
61.25 + 0.47
60.10 = 0.92
60.52 + 0.94
FoeL (Top-1)
61.45 + 0.11
60.68 + 0.37
61.37 + 0.24
4.2 Does FocL improve generalization under foveated inputs?
Experimental Sctup. We use the ImageNet subset with bounding box annotations similar to Deng et al. );009); Russakovsky et al. (2015), containing 482K images. All models adopt the standard ResNet50 backbone (He al.) (2015); the FoeL variant employs foveated glimpse(s) as described in Sec. 3 (single-crop or multi-crop, as specified per experiment). FoeL and standard models are trained with identical settings to ensure fair comparison. Our classifier does not take segmentation as an additional input channel; ground- truth boxes (masks when used), are used only to define object-centric inputs (crop or masked image), and the ResNet-50 backbone architecture is unchanged.
We evaluate generalization under three complementary settings: (1) oracle bounding boz evaluation using a single ground truth crop per image, (2) multi glimpse aggregation with FALcon introduced by Tbrayev let al. (2024b), and (3) an empirical high performance dorsal ventral evaluation using Segment Anything (SAM) developed by Ravi al.] (2024). For the first. setting, each model is evaluated in its respective input domain, using full images for the standard classifier and bounding box crops for FocL. In addition, the standard model is also tested on oracle crops as a diagnostic to assess its reliance on background context. For FALcon and SAM, both models receive identical localized crops from the respective localizer, enabling a fair comparison. All ImageNet. V1 experiments, including oracle and FALcon evaluations, use the same 2,000 validation images for a fair comparison. For shifted distribution testing, we evaluate on 2,000 images from the ImageNet V2 (MatchedFroqueney)) set. To further isolate whether the gains arise merely from background/ horroonnddcnttetttteeeeoo ccontext removal from our foveated training, we provide additional intermediate baselines MaarrMMas/MMMeaPPP and RolAlign) in Appendix A.10-13. Finally, we extend this framework to cross-domain generalization on COCO, where SAM and both classifiers are tested without any COCO specific training.
Oracle Bounding Box Evaluation. We first evaluate performance under ideal foveation using oracle bounding box crops (Table first two rows). Each model is tested in its native domain, with the standard model evaluated on full images and FoeL on its corresponding foveated crops. To assess context dependonce, the standard model is also tested on oracle crops as a diagnostic. Its Top accuracy drops by 1.07 pp, indicating reliance on background cues. When compared across their respective inputs, Fool outperforms the standard model by 11.33 pp in Top and 7.77 pp Top 5 accuracy, highlighting its stronger generalization under object centered supervision. Additional ablations are provided in Appendix A.4 Please refer to Appendix A. (Table 11 where we compare against HardMask and RolAlign under oracle crops and observe that FocL's advantage prevails over both baselines. This highlights the benefit of FocL's glimpse- based, jittered object-aligned training beyond simple mean-padding based context suppression or Rol-style feature pooling.
Multi Glimpse Aggregation with FALcon. We next evaluate both models within an active vision setup using FALcon, which provides object centric glimpses for classification (Table bottom rows). Unlike the oracle setting, FALcon introduces viewpoint changes and uses automatically localized bounding boxes that are not always precise, leading to a small drop from ideal foveation but offering a more practical scenario. The standard model shows minor variation across aggregation schemes, achieving 61. 25%, 60.10%,
10

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Generalization under foveated inputs on ImageNet-Vi (in-distribution) and ImageNet- V2 cout-of-distiibution) subsets. Top-1 accuracy (%) for three inference pipelines: Standard classifier only, SAM + Standard classifier, and SAM + FoeL classifier. FoeL consistently outperforms the standard model across both domains, while the dorsal like SAM proposals illustrate the high performance potential of the active vision system.
Dataset
Standard only
SAM + Standard
SAM + FocL
ImageNet--V ImageNet- V2
60.90 + 1.10
68.08 + 0.97
75.82 = 0.23
49.77 = 0.94
57.50 + 1.92
65.80 = 0.55
Table 3: Cross-domain generalization setup on COCO. mAP at IoU = 0.3 and 0.5 for two SAM proposal regimes. Values show performance improvement of the overall SAM + FocL system relative to the SAM + Standard bascline.
(a) Max 20 proposals image
(b) Max 300 proposals
image
IoU
SAM + Standard - SAM + FocL
IoU
SAM + Standard - SAM + FocL
0.3
24.39 + 28.22 (+3.83 pp / +16%)
0.3
28.19 + 31.76 (+3.57 pp +12.7%)
0.5
13.10 + 16.22 (+3.12 pp / +24%)
0.5
14.29 + 17.24 (+2.95 pp +20.6%)
and 60.52% Top 1 accuracy for Average, Voting, and Weighted aggregation respectively. FoeL maintains consistently higher performance with 61.45%, 60.68%, and 61.37% Top 1 accuracy, demonstrating improved stability and better generalization under multi glimpse viewpoint variations.
Evaluation with SAM a Dorsal Localizer. We next integrate the classifiers with the Segment Any- thing Model (SAM) developed by Ravi et al. (2024), whose hyperparameters are provided in the Appendix. SAM serves as a strong dorsal module that can segment and generale object proposals with high precision across a wide range of object sizes. This expressive foundation model complements our specialist FoeT. classifier, together forming a dorsal ventral system that offers a high performance reference for evaluating classification accuracy under the relaxed improves performance for both classifiers. On ImageNet V1, pairing SAM with the standard classifier yields 68.08% Top 1 accuracy, while the SAM and FoeL combination achieves 75.82%, marking 7.7 pp gain. This advantage further increases on ImageNet V2 (Recht et al. 2019), where SAM and FoeL reach 65.80%, outper- forming the baseline by more than 8 pp. While the monolithic standard classifier tends to capture spurious background statistics specific to ImageNet V1, FoeL learns object aligned representations that generalize effectively to out of distribution data, yielding more stable behavior under natural shifts. Feature-pooling baselines such as RoTAlign exhibit a similar pattern under SAM proposals, performing competilively on ImageNet-Ve V1 but dropping more sharply on ImageNet-V2, as shown in Table [2] Appendix A.13 provides a much more detailed comparative analysis. Details rogarding SAM hyperparameters and sctup are provided in Appendix A.4 D and A.
criterion. As shown in Table 2 SAM significantly
cross-domain generalization from ImageNet to COCO. To examine cross-domain generalization, we evaluate the SAM and classifier pipelines on the COCO dataset without any fine tuning. This setup is not intended as a competitive benchmark but as a diagnostic test of transferability isolating whether the model has learned robust object features merely dataset-specific background statistics statisties-coosiiteen
consistent with
prior studies that evaluate ImageNet trained models on COCO style tasks (He et al. 2018 Shin et all 2023 Public frameworks such as Image by Lo (2020) similarly map overlapping ImageNet and COCO categories to enable cross-domain analysis. We follow this mapping, where 65 of the 80 COCO categories align directly with ImageNet classes, and use 1.5K images from the COCO training set for evaluation since these samples are generally more cluttered and challenging than validation images. SAM serves as the dorsal localizer, producing up to 50 proposals per image, while the ImageNet trained Standard and FocL classifiers predict object labels for each proposal. Mean average precision (mAP) is computed at ToU thresholds of
11

---

Published in Transactions on Machine Learning Research (02/2026)
Max vs 300 proposals @ IoU 0.5 from right
1.
2.
(a)
(b)
Extreme cluttered scenes (Max 300 proposals @ loU 0.5)
Figure Cross-domain generalization in a zero-shot transfer setup (IoU = 0.5). SAM (blue) provides region proposals to the ventral classifiers (FocL + SAM in green, Standard + SAM red). Rows 1-2 show identical COCO images under two proposal regimes: (a) 20 proposals and (b) 300 proposals per image. FoeL maintains stable object-centered detections and suppresses background confusion proposal density increases. Row 3 depicts extreme clutter (300 proposals), where Foel yields lighter and semantically consistent predictions while the Standard model mislabels overlapping objects, demonstrating improved cross-domain transfer without any COCO-spocific training.
0.3 and 0.5. We include the relaxed IoU-0.3 threshold to account for potential localization drift when transferring to an unseen domain.
As shown in Table 3. the SAM and FoeL pipeline consistently outperforms the SAM and Standard setup across ToU thresholds and proposal counts. With 20 proposals per image, FocL improves mAP by 3.83 pp at IoU 0.3 and 3.12 pp at IoU 0.5 (16% and 24% relative gains). Increasing the proposals to 300 further raises performance 31.76 and 17.24, respectively. Qualitative examples Fig. illustrate cleaner, object centered detections and fewer spurious boxes. In highly cluttered scenes (Row 3, second image), the SAM and FocL system correctly identifies three distinct objects, while the SAM and Standard combination misses all, highlighting FocL's stronger object awareness and cross-domain robustness. More details are provided in Appendix A.4 (C, F, G, H).
4.3 Does FocL enhance learning efficiency and stability?
To enable fair comparisons, experiments in this section the FocL single-crop variant against standard full-image training. We assess FocL's 's impact on optimization stability, and data efficiency. Unless other- wise stated, all results in Sec. 4.3 compare FoeL (single-crop) against standard full-image training using an identical, fixed ImageNet. training recipe (no method-specific hyperparameter tuning or setting-specific schedule adjustments). Training details are provided in Appendix A.3.
Smoother Optimization via Gradient Norm Reduction. Analysis of le gradient norms during train- ing on the 85K subset further reveals FocL's stabilizing effect. Foel exhibited consistently lower gradient
12

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Comparison across dataset scales and FocL variants on ImageNet. Top-1 / Top-5 accuracy (%) for Standard ResNet-50 trained on the full 1.03M ImageNet set and the 482K subset, and for FocL single- crop (SC) and multi-crop (MC) models trained on the same 482K subset.
Model Type
1.03M Standard
482K Standard
FocL SC
FocL MC
Full Img
BBox
Full Img
BBox
BBox
BBox
Top-1
70.65
67.10
60.90
59.83
71.05
72.23
Top-5
89.35
85.70
83.05
82.05
89.80
90.82
magnitudes. Specifically, the mean gradient norm per parameter (normalized by model size) was reduced by approximately 45.8% (from 1.49 x 10-3 for standard to 8.08 10 for FocL). This substantial drop suggests FocL creates a simpler optimization landscape with less gradient noise. (Absolute mean gradient norms: Standard 3.81 10 FocL 2.07 x 10*)
Sample Efficient Learning. FocL's simplified learning paradigm leads to strong sample efficiency (Ta- When trained on the 482K annotated ImageNet subset, the FocL multi crop model achieves 72.23% Top and 90.82% Top 5 accuracy when evaluated using oracle bounding boxes. This performance surpasses the standard model trained an 80% partition of the full 1.28M ImageNet dataset. (=1.02M images), which attains 70.65% and 89.35% Top 1 and Top 5 accuracy respectively, despite using more than twice the data. These results show that FocL achieves competitive generalization with significantly fewer training samples. We emphasize that this efficiency is measured in the number of training images conditional on access to object localization (bounding bozes or proposals), and does not claim reduced haman annotation cost relative to image-level labels. In practice, such localization can come from existing metadata or outomated proposal methods (e.g., SAM), as studied in our active-vision pipeline. Importantly, FocL not coresct or data pruning method; its efficiency gains arise from foreground aligned input restructuring rather than selecting discarding samples. Additional analyses are provided in the Appendix A.8.
ble
5 Conclusion
In this paper, we introduced FocL, a multi-glimpse training strategy that encourages models to learn object- centric features by reducing background clutter. By directing learning toward label-consistent object regions, FocL reduces memorization and yields more stable representations. A reduction of approximately 65% in mean cumulative sample loss and an approximately 62% incrcase in the & perturbation required to flip predictions both indicate that FoeL learns less brittle and more generalizable features. Under ideal foveated crops, FoeL improves Top-1 accuracy by up to 11 pp compared to standard and intermediate baselines, showing that minimizing background context simplifies learning. When paired with SAM dorsal proposal generator, FocL. further improves performance by about 2-7 pp on ImageNet-Vi V1 and about 6-8 under natural distribution shift to Ima This object-centric bias also transfers across domains: on COCO, FocL yields over 12% relative gains in cross-domain transfer without any target-domall training. Finally, FoeL reaches comparable or higher accuracy using roughly 56% less training data while maintaining smoother gradients, reflecting more efficient and stable optimization. Overall, foveated learning provides a simple and biologically grounded path toward models that memorize less, generalize better, and transfer more effectively across visual domains.
Acknowledgments
This work was supported in part by, the Center for the Co-Design of Cognitive Systems (CoCoSys), a DARPA-sponsored JUMP 2.0 center, the Semiconductor Research Corporation (SRC), and the National Science Foundation.
13

---

Published in Transactions on Machine Learning Research (02/2026)
References
Ananthu Aniraj, Cássio F. Dantas, Dino Tenco, and Diego Marcos. Masking strategies for background bias removal in computer vision models. In IEEE/CC /VV International Conference on Computer Vision, 1CCVW 2023 Workshops, Paris, France, October 2-6, 2023, 2023.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kan- wal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A Closer Look at Memorization in Deep Networks. In Proceedings of the 34th International Conference on Machine Learning. PMLR, 2017.
Reza Bayat, Mohammad Pezeshki, Elvis Dohmatob, David Lopez-Paz, and Pascal Vincent. The pitfalls of memorization: When memorization hurts generalization. In The Thirteenth International Conference on Learning Represenlations, 2025. URL htttp::/ wttp://ewe.....oomom//oo/////aaa
Gavin R Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-aceuracy learning? In ICML Workshop, 2021. URL https:
7/arxiv. wtt/p/dg///eoeooooooooo/oeeeer// /2/12...02221211111 pdf
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew M.
Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. CoRR, abs/1901.11390 2019.
Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arx preprint 1r100:1910. 1910.13477 2019.
A Clark. Whatever next? predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3):181-204, 2013.
Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James DiCarlo. Simulat- ing a primary visual cortex at the front of cnns improves robustness to image perturbations. In Advances in Neural Information Processing Systems, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In JEFE Conference on Computer Vision and Pattern Recognition, 2009.
Yinpeng Dong, Ke Xu, Xiao Yang, Tianyu Pang, Zhijie Deng, Hang Su, and Jun Zhu. Exploring mem- orization in adversarial training. In International Conference on Learning Representations, 2022. URL
Miguel P Eckstein. Visual search: A retrospective. Journal of vision, 11(5):14-14 14, 2011.
Gamaleldin F. Elsayed, Simon Kornblith, and Quoe V. Le. Saccader: Improving accuracy of hard attention models for vision. In Neural Information Processing Systems, 2019.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In NewrIPS, 2020. URL httpp:://arxiv.orr///ttt /2000 03703, pdf
Jonathan M Gant, Andrzej Banburski, and Arturo Deza. Evaluating the adversarial robustness of a foveated texture transform module in CNN. In SVRHM 2021 Workshop @ NeurIPS, 2021.
Isha Garg, Deepak Ravikumar, and Kaushik Roy. Memorization through the lens of curvature of loss function around samples. In Proceedings of the 4ist International Conference on Machine Learning. PMLR, 2024.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shorteut learning in deep neural networks. CoRR, abs/2004.07780, 2020. URL httpp://aaxxi.oorr///////0000/00 ort////b///0004077730
Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Keepangment: A simple information- preserving data augmentation approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), 2021.
14

---

Published in Transactions on Machine Learning Research (02/2026)
M A Goodale and A D Milner. Sight unseen: An exploration of conscious and unconscious vision. Oxford University Press, 2004.
Anne Harrington and Arturo Deza. Finding biological plausibility for adversarially robust features via metameric tasks. In International Conference on Learning Representations, 2022.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognation (CVPR), 2015.
Kaiming He, Ross B. Girshick, and Piotr Dollár. Rethinking imagenet pre-training. 2019 IEEE/CVF
International Conference on Computer Vision (ICCV), pp. 4917-4926, 2018. URL https://api.
semanticscholar. org/ /ttp//oosssSD:33337711 Stt.
Shaoli Huang, Xinchao Wang, and Dacheng Tao. Snapmix: Semantically proportional mixing for augmenting fino-grained data. Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Timur Ibrayev, Amitangshu Mukherjee, Sai Aparna Aketi, and Kaushik Roy, Towards two-stream foveation- based active vision learning. IEEE Transactions on Cognitive and Developmental Systems, 2024a.
Timur Ibrayev, Manish Nagaraj, Amitangshu Mukherjee, and Kaushik Roy. Exploring foveation and saccade for improved weakly-superrised localization. In Proceedings of The 2nd Gaze Meets ML workshop, volume 226 of Proccedings of Machine Learning Research, pp. 61-89. PML.R, 16 Dec 2024b. URI. https:// proceedings mlr.
html
Bargav Jayaraman, Chuan Guo, and Kamalika Chaudhuri. Déjà vu memorization in vision models. In Aduances in Newral Information Processing Systems, 2024.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalan- tidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020.
Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 2020.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023. doi: (LL10///CCC55110022233.000
Narine Kokhlikyan, Bargav Jayaraman, Florian Bordes, Chuan Guo, and Kamalika Chaudhuri. Mcasuring dejavu memorization efficiently. In The Thirly-eightt Annual Conference on Neural Information Processing Systems, 2024. URL https://openrevvi/ wttp://wwww...e...ooo///deeeeee.
Qiongxiu Li, Xiaoyu Luo, Yiyi Chen, and Johannes Bjerva. Trustworthy machine learning via. memorization and the granular long-tail: A survey on interactions, tradeoffs, and beyond, 2025. URL /tttps///arxiv. org/abs/2503 07501
Howard Y. C. Lo. Imagenet2ooco:
Cross-dataset evaluation toolkit for zero-shot transfer from imagenet to COC0. https://github http:ttt...coooooyttyeeee hww.oo/wwwlllooo//eeeee//// 2020. Accessed: 2025-11-14.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems, 2020.
Yan Luo, Xavier Boix, Gemma Roig, Tomaso A. Poggio, and Qi Zhao. Foveation-based mechanisms alleviate adversarial examples. ArXiv, abs/1511.06292, 2015.
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu, Towards deep learning models resistant to adversarial attacks. Tn ICLR,, 2018.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Casey Meehan, Florian Bordes, Pascal Vincent, Kamalika Chaudhuri, and Chuan Guo. Do SSL models have déjà vu? a case of unintended memorization in sell-superviedd learning. In Thirly-seventh Conference Newral Information Processing Systems, 2023. URL https://opennevvee htt.pppwwwweeee....oooooororoaaa met:/oorruuiic111BByyTTo
A Milner and M A Goodale. The visual brain in action. Oxford University Press, 1992.
Mishkin, LG Ungerleider, and KA Macko. Object vision and spatial vision: Two cortical pathways. Trends in Neurosciences, 6:414 417, 1983.
Volodymyr Mnih, Nicolas Manfred Otto Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. In Newral Information Processing Systems, 2014.
Amitangshu Mukherjee, Timur Tbrayev, and Kaushik Roy. On inherent adversarial robustness of active vision systems. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://
openreriew.net/forumnf 2tdd/foorummdd--11111335
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam Segment anything in images and videos, 2024. URL https://arxiv. http:///arrr..oorgaaa//////0.0/ 00714
Deepak Ravikumar, Eistathia Soufleri, Abolfazl Hashemi, and Kaushik Roy. Unveiling privacy, memorization, and input curvature links. In ICML, 2024. URL
Deepak Ravikumar, Efstathia Soufleri, Abolfazl Hashemi, and Kaushik Roy. Memorization and the orders of loss: A learning dynamics perspective, 2025. URL
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. -380-5389. PMTR, 2019.
Jiawei Ren, Cunjun Yu, shunan sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and hongsheng Li. Balanced meta-softmax for taigl visual recognition. In Advances in Neural Information Processing Systems, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei- Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252 2015.
S Sakuraba, S Sakai, K Yamanaka, K Yokosawa, and K Hirayama. Does the human dorsal stream really
process category for tools? Journal of Neuroscience, 32(21):7721-778) 2012.
Muhammad A Shah, Aqsa Kashaf, and Bhiksha Raj. Training on foveated images improves robustness to adversarial attacks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Z Shao, L Ma, B Li, and D M Bock. Leveraging the human ventral visual stream to improve neural network robustnes: Xixiv preprinl ar Xix:22240 022.0064., 2024.
Gyungin Shin, Samuel Albanie, and Weidi Xie. Zero-shot unsupervised transfer instance segmentation. 2023 IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4848 4858, 2023. URL htt.:tttt//ppp..semtt.cotoooooa org/Corruu 258352774
Jingru Tan, Xin Lu, Gang Zhang, Changqing Yin, and Quanquan Li. Equalization loss v2: A new gradi- ent balance approach for long-tailed object detection. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1685-1694, June 2021.
A F M Shahab Uddin, Mst. Sirazam Monira, Wheemyung Shin, TaeChoong Chung, and Sung-Ho Bac. Saliencymix: A saliency guided data augmentation strategy for better regularization. In International Conference Learning Representations, 2021. URL https://ppenneees....ee/rtrrrrre
16

---

Published in Transactions on Machine Learning Research (02/2026)
L G Ungerleider and J V Haxby, 'what' and 'where' in the human brain. Current Opinion in Neurobioloyy, 4(2):157-1665 1994.
Dmitrii Usynin, Moritz Knolle, and Georgios Kaissis. Memorisation in machine learning: A survey of results. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL. https://openrevier.
net/forrunidd-HWWDDbb
Manish Reddy Vuyyuru, Andrzej Banburski, Nishka Pant, and Tomaso Poggio. Biologically inspired mech-
anisms for adversarial robustness. In Advances in Neural Information Processing Systems, 2020.
Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2023.
Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang, and Gao Huang. Glance and focus: a dynamic approach to reducing spatial redundancy image classification. In Advances in Neural Information Processing Systems, 2020.
Sanghyun Woo, Jongchan Park, Joon- Lee, and In So Kweon. Cbaa: Convolutional block attention module. In Proceedings of the Europeon Conference on Computer Vision (ECCV), 2018.
S. Yun, D. Han, S. Chun, S. Oh, Y. Yoo, and J. Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri-
cal risk minimization. In International Conference on Learning Representations, 2018. URL https:
//openreview wwt.:/ooooondddrdoir......
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. doi: 10.1109/TPAMI. 2023.3268118.
A Appendix
A.1
Expanded Related Work
This section expands on the related work most relevant to FocL, organized into three main areas: (1) object- centric and foroground-focused lcarning, (2) memorization and generalization in long-tailed scttings, and (3)
foveation-based methods for robustness and efficient learning.
Object-Centric and Foreground-Focusee Learning Unsupervised object-centric models such MONet (Burgess et al. 2019) and Slot Attention (Locatello et al. 2020) aim to decompose scenes into discrete object representations, but often struggle complex natural images. Attention modules like CBAM (Woo) et al. 2018) reweight spatial and channel-wise features post hoc, while pipelines like CutLER (Wang et al. 2023 attempt to discover and mask foregrounds, still operating over full-image inputs. A related class of models learns dynamic visual attention through iterative glimpses. RANet (Mnih et al. 2014) uses a recur- rent attention network to focus on different image regions over time, while Saccader (Elsayed et al. 2019) and GFNet (Wang al., 2020) emulate saccadic movements and process glimpses within computational budgets. FABLE (lbrayev et al. 2024a) models a dersal-ventral system using reinforcement learning to locate objects, and FALcon (Ibrayev et al. 2024b) further introduces saccades and foveation, enabling active multi-object detection even from single-object training. These models mimic human vision by sequentially sampling high-resolution glimpses, discarding background via lask-adaptive attention. FocL adopts dif- ferent paradigm. Rather than learning fixation policies, uses supervised bounding boxes to directly crop foreground objects, fully removing background prior to training. This object- label alignment reduces contextual bias and simplifies training, focusing on the impact of this transfornation generalization, memorization, and convergence.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Memorization in Long-Tailed Learning Deep networks tend to memorize rare, noisy, or atypical ex- amples after first fitting frequent and simpler patterns (Arpit et al. 2017 Feldman & Zhang. 2020). Arpit
et al. (2017) show that during training, networks prioritize learning generalizable patterns but eventually begin memorizing outliers and noisy data. Feldman & Zhang (2020) further argue that memorization is not just. incidental but sometimes essential lor accurate predictions on tail samples, especially when such examples are underrepresented or conflict with dominant patterns in the data. Building on this, [Brown et al. (2021) h provide theoretical insights into why high-accuracy learners may be forced memorize substantial information about training data in natural, long-tailed settings. [synin et al. (2024) offer comprehensive survey of memorization across multiple regimes, categorizing its benefits and drawbacks with respect to generalization and privacy. Li et al. (2025) take a systems-level view, framing memorization as central to the trustworthiness of machine learning systems. They explore its role across fairness, robustness, and data privacy, and propose a taxonomy to reason about these interactions based on data granularity such as class imbalance, noise, and atypicality. To characterize memorization quantitatively, RRavikumar et al. (2025) introduce the Cumulative Sample Loss (CSL), which tracks the cumulative training loss per sample. They show that hard-to-learn and noisy samples consistently exhibit higher CSL, providing strong signal of mem- orization. Complementary to this, Garg et al. (004)) and Raviiumar et al. (2024) explore the curvature of the loss surface. Their results show that memorized examples lie in sharper regions of the landscape i.c., with higher curvature. This often indicate less robust. generalization and more brittle learning dynamics. Recent studies also demonstrate that memorization is not limited to supervised learning. [Meehan et al. (2023) uncover "déjå vu" memorization in self-supervised models, where training samples are memorized even without explicit labels. Kokhlikyan et al. (2024) refine the measurement of this phenomenon, offering efficient evaluation tools for memorization in large SSL models. Similar memorization behavior is observed in vision-language models (Jayaraman et al. 2024 , where individual image or object information is retained by the model even beyond its intended abstraction level. FocL offers an input-level simplification by suppressing background clutter entirely, reducing reliance on spurious correlations and short- cut cucs (Geirhos et al.) 2020). By restructuring the input itself, FoeL shifts the learning task to focus on object-relevant features from the outset. Unlike techniques such as Mixup (Zhang et al. 2018), CutMix (Yun) et al. ) or ldgit-adjustment methods, which alter training dynamics via label smoothing, augmentation, reweighting, FocL tackles instanco-level difficulty directly by improving input-label consistency through foveated, object-aligned supervision.
Foveation, Robustness, and Efficient Learning Foveation-inspired methods have been explored as mechanisms for improving robustness. R-Blur (Shah et al.) 2023) applies adaptive Gaussian blurring simulate peripheral vision, improving resistance to adversarial attacks. Gant et al. (2021) use Foveated Texture Transform to enhance both IID generalization and robustness. Active-vision systems (Mukherjee
ett al. 2025) formulate a deep learning-basell dorsal-ventral architecture by building on prior works such as FALcon [Ibaaev et al.] 2024b) and GFNet (Wang et al. 2020). and demonstrate improved robustness in black-box transfer attack scenarios. By processing sequential glimpses at multiple fixation points, the approach enhances adversarial resilience for both CNNs and transformer- based ventral networks, particularly under natural and transferable adversarial inputs. [uo et al. (2015) apply CNNs to foveated regions, achieving strong robustness to perturbations. R-Warp (Vuyyuru et al. 2020) and VOneBlock (Dapello) et al. 2020) embed cortical and retinal processing into CNNs. Harrington & Deza 2022) show how robust models align with texture-based peripheral vision, and Shah et al. (2023) simulate peripheral degradation for robustness gains. FoeL introduces simplified mechanism: complete background cut-off via supervised crops. This restructuring results in cleaner, more learnable samples and exhibits a coupled effect; higher adversarial perturbation energy required to flip predictions and lower Cumulative Sample Loss (CSL). Both serving as indicators of reduced memorization.
Saliency guided data augmentation methods to reduce background Saliency-guided data augmen- tation provides a simple, aulomated way to reduce background dependence by preferentially preserving or mixing sengroondddiscrimmnate than sampling cut-and-paste areas uniformly at random. For example, Uddin et al. (2021) uses a saliency map to select an informative patch and paste it onto another image, mixing labels based on the pasted region. Similarly, Kim et al. (2020) exploits saliency and local image statistics to optimize the mixing
regions, thereby discouraging reliance on spurious background cues rather
18

---

Published in Transactions on Machine Learning Research (02/2026)
mask, producing more semantically meaningful composites for mixup-style training. Along the same line, Huang et al. (2021) | leverages class activation maps to estimate semantic composition and set mixing pro- portions, reducing label noise in mixed samples (especially in fine- tiee-graiedd recognition). Goog et al. (2021) propose KeepAugment, which uses saliency maps to identify informative regions and then constrains standard augmentations to preserve those regions (e.g., avoiding cutting salient areas or pasting them back after trans- formation). Aniraj al.] (2023) study background-biar mitigation via masking, comparing early masking (removing background at the imput image level) and late masking (masking nackground-corresponddgg spatial features), and show these strategies improve generalization under OOD background shifts. In contrast, FocL uses bounding-box-drree supervision than heuristic saliency) and explicitly evaluates reduced memorization. It further extends to an active-vision pipeline where SAM acts as an external proposal nelwork to generate foveated glimpses and demonstrate cross-dataset generalization.
foweated jitter to enforce consistent, label-aligned object coverage (i.e., stronger
FoeL thus bridges perceptual inspiration with practical gains in generalization, memorization reduction, and efficient learning without requiring specialized architectures costly training procedures.
A.2 FocL glimpse generation algorithm details
Overview. The FocL framework generates up to three object-centric glimpses per image, centered around supervised fixation point derived from the annotated bounding box. These glimpses simulate small saccadic shifts near the object and reduce background clutter while proserving semantic alignment with the label. While the main paper outlines the high-level steps Figure 4, this section details the underlying algorithm and implementation used in our experiments.
Step-by-step Procedure. Given an annotated image (x, v) with bounding box (Imins Ymin, maxx Umax), we define the center p E IR² the box as the base fixation point. Glimpses are then constructed as follows:
Step 1:
Sampling fixation candidates. Around p, we sample up to keand candidate centers using a uniform offset in both spatial directions. The maximum offset is set to fraction a of the bounding box width/height, i.e.,
Dr, Ay ~ u(-aww,warm aw), where w = Tmax Tmin
These jittered candidates simulate parafoveal fixations while remaining near the object center. Con- cretely, this defines a square jitter window around the center of the bounding box, within which candidate fixation points p. =p+ (Az, Ay) are sampled.
Valid fixation selection. For each candidate pt, we compute a crop region whose aspect ratio and scale are randomly jittered using multiplicative factors Bz, By u(1-1,1 -,,1 3). We retain up to & 3 valid crops whose regions lie entirely within image bounds. This ensures all glimpses are valid, foregroumd-aligned views.
Step 2:
Step 3: Distortion-aware cropping. For cach selectod Pa, the crop is resized to the modcl's input resolution. If the required resizing scale exceeds a threshold computed via an inverse crop ratio ") 1/(1 max_ meax_crop_rete (rop_ratio), we first expand the crop window proportionally around its center (without crossing image bounds). This reduces geometric distortion when handling small or thin boxes.
Step 4: Aggregation. Each image yields k foveated crops (Fov,(z, p:)) These are treated as label-consistent training samples and either randomly subsampled (k = 1) stacked into correlated mini-batch. Glimpses from the same image are never shuffled across batches, preserving the coherence of multi-view supervision.
Implementation Notes. parameters:
The algorithm is implemented as a configurable pipeline which exposes key
19

---

Published in Transactions on Machine Learning Research (02/2026)
Offset_ fraction 0.2: sets a, the maximum offset for sampling.
Scale_jitter 0.1: sets B, the jitter range for scale and aspect ratio.
cer__propraaiooooo copprraioooo 0.2: defines threshold 7 to trigger crop expansion. The max crop ratio is threshold parameter that controls how much crop is allowed to be resized relative to the original bounding box before geometric distortion is considered too high.
Max_crop
Area_threshin thresholdd = 0.2: used to activate distortion-aware expansion for small objects.
flag: True, all k glimpses are returned together; if False, one random crop is sampled per epoch.
Augmentation mode entails { ("conservative", "medium", "aggressive"): scales the above hyperpa-
rameters accordingly.
This design ensures that glimpses maintain semantic alignment while providing spatial diversity around the object. The same framework supports single-glimpse (I: = 1) and multi-glimpse (k 1) supervision via a unified pipeline.
Table 5: FocL dataset and cropping hyperparameters.
Parameter
Value
Offiset fraction (a)
0.2
caae//sppctt// jitter (3)
0.1
Max crop ralio
0.2
Area threshold (for distortion-aware fallback)
0.2
Number of glimpses k
1 or 3
Multi-crop batching
Enabled for k > 1
Batch size
128 (k=1), 64 (k=3)
Input resolution
224 x 224
Augmentation
Medium
A.3 Training Details and Reproducibility
Dataset Preparation. Following the setup in Meehan et al. 2023 we sample and curate our annotated dataset from ImageNet using the official codebase available at https: //t::://gttttbttt http://ggtuubbc//teoeee comm/facebooreeearchh
DejaVu All dataset checks, bounding box extraction, and curation pipelines were built on top of this repository. We adapt their utilities to generate the subset used for FoeL, ensuring consistency in annotation quality and reproducibility of bounding box metadata..
We evaluate FocL across multiple ImageNet subsets with bounding box annotations. Our experiments use the following curated partitions:
Full-scale split. We use the complete curated bounding- subset of ImageNet-1K (2012), totaling 482,187 images. We apply a 94/6 train train-validation split, yielding 453,254 training images and 28,933 validation images. This full-scale setup used for all generalization experiments presented Section 4.2.
For comparison, Section 4.3 also includes results from a standard ResNet-50 trained on the full ImageNet- classification dataset (approximately 1.3M images) using an 80/20 train-vatt validation split, producing roughly 1.03M training examples. This model is included solely show that increasing the volume of training data does not improve generalization on foveated (oracle-crop) inputs, the standard model continues to rely heavily non-foreground contextual cues.
20

---

Published in Transactions on Machine Learning Research (02/2026)
Controlled low-data splits. We also construct two disjoint 100K ImageNet subsets, referred to Partition A and Partition B, each split into 85K training and 15K validation images. Partition A is used for all controlled analyses in reduced-data settings:
The 85K training split of Partition is used to evaluate adversarial robustness (PGD per- turbation distance), gradient norms, and memorization via cumulative sample loss (CSL), as discussed in Section 4.1.
The 15K validation split of Partition B used for validation- adversarial robustness eval- uation, enabling clean separation between training and evaluation subsets.
Each image is preprocessed to extract either one up to three foveated crops using the method described in Section 3.1. All crops are resized to 224 x 224 resolution. Inputs are normalized using the standard ImageNet mcan and standard deviation.
Model Architecture We use standard ResNet-50 He al. 2015 architecture across all experiments, with no architectural differences between Standard and FocL models.
Training Configuration. All models are trained for 90 epochs using SGD with momentum 0.9 and weight decay x 10-4 The initial learning rate is set to 0.1 and decayed by a factor 0.1 every 30 epochs. We use batch size of 64 per worker, which is flattened across multiple glimpses during multi-crop training (e.g., k glimpses per image). We use batch size of 128 for glimpse per image. Each training sample is augmented using standard ImageNet transforms: random resized crop, horizontal flip, and color jitter. All experiments are tracked using Weights & Biases.
Optimization and Learning Rate Schedule. We use the standard cross-entropy loss as the training objective. Optimization is performed using stochastic gradient descent (SGD) with momentum set to 0.9 and weight decay of 1 x 10-4 The initial learning rate is 0.1, decayed by a factor of 0.1 every 30 epochs using StepLR scheduler. All models are trained with mixed precision using PyTorch's GradScaler for improved stability and efficiency.
Reproducibility and Statistical Significance We ensure statistical rigor by repeating key experiments across multiple random seeds and reporting mean and standard deviation where applicable:
Generalization experiments (Section 4.2): All models evaluated using both oracle bounding box inference, FALeon and SAM inference are trained across 3 random seeds. We report the mean Top-1 accuracy along with standard deviation in the main paper.
Data-efficient learning (Section 4.3): To assess consistency in low-data settings, we train models on Partition A (100K subset) across different random soeds. The required table with standard deviation provided in the Appendix.
CSL and adversarial robustness (Section 4.1): Cumulative sample loss is computed by logging per-sample training loss across all 90 epochs on the 100K subset. We also evaluate PGD-based adversarial distance across 5 different & budgets (c) on the same partition.
Compute and Environment. All models are trained on NVIDIA A40 GPUs with 48GB memory per device. We follow the same training hyperparameters and optimization settings for both Standard and FocL models. The full training pipeline, configuration scripts, and an environment file are included in the code submission.
An environment file named requirements. txt is included in the supplementary materials ensure full repro- ducibilily.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Table 6: Ablation under oracle bounding box inference on 2,000 ImageNet-Vi validation sam- ples. Top-1 and Top-5 accuracy (%) for three variants of the FoeL training pipeline: the base foveated crop, aistortion-aware single-crop variant (SC), and the full multi-glimpss model (MC).
Metric
FoeL Base
FocL SC
FocL MC
Top-1 (%)
69.05
71.05
72.23
Top-5 (%)
89.50
89.80
90.82
Table 7: Ablation under SAM-2 proposals on ImageNet-Vi (in-distribution) and ImageNet- V2 (out-of-distribution)) "Any" Top-1 accuracy (%) is reported, where an image is counted correct if ony SAM-generated crop is classified correctly.
Dataset
FoeL Base
FocL SC
FocL MC
ImageNet-Vi
70.30
75.05 64.80
75.82
ImageNet-V2
61.50
65.80
A.4 Generalization Results
A. Ablation: Role of Glimpse Diversity and Distortion-Aware Cropping. We evaluate the con- tribution of each component of FoeL on the same 2,000 ImageNet-V1 samples used in the main paper. All ablation experiments are reported using a single seed for clarity. The FocL Base model uses a single resized bounding-box crop with no jitter, distortion-aware expansion, or parafoveal variation. The FocL Single-Crop (SC) model corresponds the full FoeL algorithm with k == introducing scale jitter, mild positional shifts, and distortionn-wware cropping. The FoeL Multi-Crop (MC) model uses jittered glimpses, adding structured multi-view diversity.
As shown in Tables 6 and i performance improves consistently from Base to SC to MC across all evaluation settings. Under oracle bounding-uee evaluation, SC already improves upon the Base model by +2.00 pp in Top-1 accuracy (69.05% + 71.05%), and MC yields a total gain of +3.18 pp (72.23%). These gains occur even under idcalized cropping, where the input contains minimal background contamination.
Under the more realistic SAM-2 proposal setting, where crops may be noisy incomplete, the improvements become substantially larger. On ImageNet-V1, SC improves Base by +4.75 pp (70.30% 75.05%) and MC yields total gain of +5.52 pp. On ImageNet-Ve V2, an out-of-distribution benchmark, SC improves Base by +3.30 pp, and MC by +4.30 pp. These trends demonstrate that both distortion-aware cropping (SC) and structured glimpse diversity (MC) play complementary roles: the former increases robustness to imperfect box geometry, while the latter improves stability under viewpoint variation and proposal noise.
B. Diagnostic analysis of background dependence. To better understand the failure modes of the Standard ResNet-50, we visualize a set of images that are correctly classified when evaluated on the full ImageNet input, but misclassified when the model is evaluated only on the ground-truth object crop. Figure g shows representative examples across diverse categories. In many cases, the full image prediction is driven by strong scene-level correlations-as such characteristic backgrounds (e.g., water, vegetation), co-occurring objects (e.g., humans, tools), or global geometry and scale, which are removed when only the object bounding box is provided. The resulting tight crops often preserve only ambiguous local texture or partial shape, leading to confident but incorrect predictions. These qualitatively illustrate that the standard model does not acquire robust object-centric features and instead leans heavily on background cues.
C. Additional COCO cross-domain results. To supplement the COCO experiments in the main paper, we provide additional qualitative predictions from the Classifier + SAM-2 system in Figure I0 These examples span range of proposal budgets and scene complexities, including occlusion and dense clutter. Consistent with the main findings, the FoeL classifier selects the correct SAM proposal substantially more reliably than the Standard model, which frequently relies background correlations when transferred to
22

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 9: Representative failure cases showing the dependence of Standard ResNet-50 on background context. Each pair shows the full image (left) and the corresponding ground-truth bounding- box crop (right). While the model predicts the full image correctly (greon), removing background features oflen eliminates global scene cues such as habilat, co-occurring objects, scale, or pose, causing confident misclassifiaations on the cropped object (red). These examples indicate that standard ImageNet training does not yield robust objoet-centric representations.
Table 8: Core SAM-2 hyperparameters for ImageNet proposal generation (20 proposals/Imago)
Parameter
Value
Purpose
SAM version
SAM-2.1 (Hicra-L)
Mask proposal backbone
lred_iou_thie thresh
0.5
Retain lower-confidence masks for diversity
stability__core_theee thresh
0.5
Relaxed region filtering
aa_mask_reegin area
25 px
Allow small objects
NMS IoU threshold
0.9
Keep diverse proposals
Max proposals Fallback
20
Top proposals used for classification
Full image
Used if SAM yields no valid masks
COCO. These qualitative results reinforce FoeL's improved cross-domain generalization and its ability to operate robustly under noisy object proposals.
D. SAM-2 hyperparameters for ImageNet proposal generation For the ImageNet and ImageNet- V2 cross-domain experiments using the Classifier + SAM system, we generate object proposals with SAM- 2.1 (Hiera-Large) in automatic mask generation mode. To encourage diverse candidate regions suitable for object-centric classification, we use relaxed confidence thresholds and retain the top 20 proposals per image. Masks with extremely small foreground support (area ratio < 10-1) or bounding boxes smaller than 5 pixels are filtered out. A high NMS IoU threshold (0.9) is applied remove only near-duplicate proposals, ensuring that viewpoint-, scale-, and boundary-level variations are preserved. If no valid proposal remains after filtering, we fall back using the full image single proposal.
The resulting proposal distribution matches the operating regime used in the main paper, where the Standard and FoeL classifiers are evaluated on ImageNet-V V1 and ImageNet-Vi V2 using 20 SAM-2 proposals per image. Table summarizes the key hyperparameters.
E. Classifter + SAM evaluation (Any prediction) For the Classifier + SAM experiments on ImageNet-V1 and ImageNet- V2, we evaluate the models using an "Any" prediction metric built on top of the SAM-2 proposals described above. For each image, SAM-2 provides up 20 candidato bounding boxes, which we crop from the original resolution, resize 224 x 224, normalize with ImageNet statistics, and classify in single batch. The classifier produces Top-1 prediction for every crop, and image is
23

---

Published in Transactions on Machine Learning Research (02/2026)
Max 20 vs 300 proposals loU 0.5 from left right
2.
(a)
(b)
Extreme cluttered scenes (Max 300 proposals @ IoU = 00.5)
3.
Figure 10: Additional COCO cross-domain generalization examples using the Classifier + SAM system. Following the analysis in the main paper, we visualize further cases where an ImageNet-trained classifier paired with SAM-2 proposals and evaluated on COCO images. Across both low-proposal (20) and high-proposal (300) settings, Foel. reliably selects the correct object-centric proposal even in the presence of clutter heavy proposal noise, whereas the Standard classifier often fires on contextual or background- driven regions. These examples complement the quantitative cross-domain results reported in the main text and further highlight the robustness of FoeL to domain shift and proposal imperfections.
Table 9: SAM-2 settings for COCO proposal generation. Both regimes use the same mask-generation thresholds; only the number of retained proposals differs. Multiscale in inference is applied for the 300-proposal setting to increase spatial and scale diversity.
Parameter
COCO-20
COCO-300
SAM version
SAM-2.1 (Hiera-L)
SAM-2.1 (Hiera-L)
Pred. IoU threshold
0.70
0.70
Stability threshold
0.80
0.80
Min. region area
100 px
100 px
NMS IoU threshold
0.65
0.65
Multiscale inference
No
Yes (0.85, 1.0, 1.15)
Max proposals per image
20
300
counted as correct under the Any metric if any crop-level Top-1 prediction matches the ground-truth Ima- geNet label. Because this metric assumes ideal proposal selection, it serves as an empirical high performance ceiling on the performance of the Classifier + SAM system. Images for which SAM-2 yields zero valid pro- posals are kept in the denominator and counted as incorrect. This evaluation is applied identically to the Standard and FocL classifiers, ensuring a fair comparison, and the resulting Any accuracies correspond the values reported in Table and in the main paper.
24

---

Published in Transactions on Machine Learning Research (02/2026)
F. SAM-2 hyperparameters for COCO proposal generation For COCO cross-domain experiments, we generate proposals using SAM-2.1 (Hiera-T.) in automatic mask generation mode. We consider two oper- ating points that correspond to the settings used in the main paper: a 20-proposal regime for light evaluation and a 300-proposal regime which emphasizes coverage in cluttered scenes. Both scttings use the same SAM-2 configuration, dilfering only in the maximum number of retained proposals.
To ensure comparability with the ImageNet experiments, we use consistent mask-generation thresholds and filter out only extremely small connected components. For each image, all SAM-2 masks are converted to bounding boxes, merged using loU-based non-maximum suppression, and then ranked by SAM's predicted IoU score. We keep the top N (20, 300} proposals. When multiscale inference is enabled (for the 300- proposal setting), proposals from three resized versions of the image (0.85x, 1.0x, 1.15x) are merged before final selection. No ground-truth information is used at any stage.
G. Cross-domain generalization on COCO with the Classifier + SAM system. To assess the cross-domain behavior of the Classifier SAM system, we evaluate ImageNNel-trained Standard and Foel models on COCO using pre-computed SAM-2 proposals (20-50 per image). Following the mapping protocol described in the main paper, restrict evaluation to COCO categories that have either direct or super- category correspondence with ImageNet. For each proposal, we crop the region, apply standard ImageNet preprocessing, and assign a COCO category by taking the maximum softmax score over all ImageNet classes mapped to that category. We then apply a fixed confidence threshold (0.1) and per-category non-maximum suppression (IoU 0.5). Detection performance is measured using the standard 11-point interpolated average precision at IoU thresholds 0.3 and 0.5. No COCO labels or bounding boxes are used for training, tuning, proposal filtering. This evaluation isolates how well cach ImageNet classifier transfers to COCO under identical SAM proposals, providing clean measure of cross-domain generalization.
H. Note on NMS thresholds. For proposal generation, we apply a relaxed NMS loU threshold of 0.65 to preserve spatial and scale diversity among SAM-2 candidates. During COCO evaluation, however, detections are post-processed using the standard per-category NMS with an loU threshold of 0.5, consistent with conventional detection protocols.
A.5 Adversarial Robustness
Setup We evaluate adversarial resistance by computing the minimum l perturbation required to flip model predictions using PGD attacks (Madry et al., 2018). All experiments are conducted on a balancod ImageNet subset with 100 samples per class and an 85/15 train-validation split. We use a PGD-12 atlack with 10 steps, random initialization, and random restarts enabled. The step size is set to = e/10, and we sweep the perturbation budget c e (0.0, 0.25, 0.5, 0.75, 1.0).
To construct a clean and balanced evaluation protocol, we select 15,000 correctly predicted samples from the training set (Partition A) and 15,000 correctly predicted samples Partition B. This forms the validation set results for Partition A (unseen). This ensures that the evaluation is based on semantically aligned, clean samples and keeps the number of inputs consistent across training and validation settings. We compute both the robustness curves and the mean adversarial distance on these subsets, allowing for a fair comparison between FoeL and standard models.
Mean Adversarial Distance To quantify robustness, we compute the average adversarial distance:
1 N N
d
Σ |11.112 where f(z; + 8.) $ ya
Here, &; denotes the smallest perturbation (in & norm) found via PGD that causes misclassification. We find that the standard model has = 0.3806, while FoeL achieves d = 0.6169 a 62% increase. This gap reflects a substantial improvement robustness. Higher adversarial distance implies that more energy is required to change the model's decision, suggesting more stable and semantically aligned representation. These results support the argument that standard models overfit incidental background cues, while FocL focuses learning on foreground-relevantt features that are inherently harder to perturb.
25

---

Published in Transactions on Machine Learning Research (02/2026)
Llama
1
Peacock
2
Beaver
3
BIAATT COTT
Figure 11: Sample visualization of CSL dynamics across three classes Llama, Peacock, and Beaver. Each compares full-image (left, red box) vs. FoeI-based cropped inputs (right, green box). Across classes, FocL leads to faster convergence (loss trajectory), more confident predictions (confidence trajectory), and substantially lower cunulative sample loss (CSL). These patterns consistent with aggregate statistics shown in CSL distributions, vraidvaaii loss curves, and gradient norm plots.
26

---

Published in Transactions on Machine Learning Research (02/2026)
FG: 0.105 FZI 0.9992
FG: 0.068 FZ: 0.9959
FG: 0:43.4 FZ: 0.9929
FG: 0.220 F2: 0.0930
Labeland
Matchstick
Standars CSL: 186.2 PooL CSL 50.5
Sancard 207-307.1 FOEL CSL 76.4
CAL 183.5 Foct,
0.9930
F2:
Winkor
Engine
Standard CSL 296.6 FOOL CSL 45.3
Staidard CSL 230 3 F90. CS. 79.7
Standerd CAL 2 Feet CSL 65.3
Surcert CR. 2003 PooL CSL: $33
Saaddard 100 Foct 117.0
Figure 12: Additional examples from the Feldman Zhang (FZ) memorization cohort. Shown are samples identified as highly memorized according to the F7. score, together with their bounding-box annotations and CSL values for the Standard and FoeL models. Some exhibit severe background clutter or ambignous multi-object scenes. We report FG only as a descriptive attribute (box-area imaag-area), not as evidence for memorization. As in the main paper, FoeL consistently reduces CSL on these difficult samples, indicating substantially lower memorization.
Saadaan-Zhang
A.6 Cumulative Sample Loss (CSL) as a proxy for learning difficulty
Setup We evaluate cumulative sample loss (CSL) as a proxy for sample difficulty and memorization. The setup follows the same 85K/15K vrainvvaliattion split used in the robustness analysis. CSL quantifies how difficult sample is to learn by accumulating its training loss over epochs. Formally, for training sample (I, v), the cumulative sample loss over T' epochs is defined as:
CSL(2) = - C(for.))
where denotes the cross-cntropy loss, fo. is the model at epoch t, and 2 is the training sample. For the FoeL model, the sample is represented as Fov(r, y), denoting a foveated crop centered on the object. High CSL values correspond samples that remain difficult across multiple epochs and are more likely to be memorized rather than learned robustly. For fair comparison, evaluate CSL for the FocL single-crop variant to match the standard model's single-view training.
In Figure II we provide a per-sample analysis demonstrating how FoeL facilitates easier learning. For each class (Llama, Peacock, Beaver), the green-boxed examples (FocL) consistently show lower cumulative sample loss (CSL) compared to their red-boxed full-image counterparts. This shift in CSL values explains the leftward shift the aggregate CSL distribution observed the main paper, supporting our claim that FoeL improves learning stability and efficiency.
Additional FZ memorization examples. To complement the analysis in Section 4.1, provide further qualitative examples from the Feldman-Zhang memorization cohort in Figure 12 These examples often include small object exlent and substantial surrounding context; they are provided for illustration of the FZ-selected cohort. A notable case the fifth image in the first row, where a drake and boz turtle co-occur within the same scene despite ImageNet assigning only single label. This is a canonical instance of the label ambiguity highlighted in our first figure in the main text, where the Standard model overfits correlated background structure. Across all shown cases, FoeL markedly reduces CSL relative to the Standard model,
27

---

Published in Transactions on Machine Learning Research (02/2026)
including challenging cases with cluttered ambiguous scenes. These additional examples reinforce the central implication of our FZ analysis: FoeI. systematically reduces CSL. on challenging, cluttered, and label-ambiguous samples in the F'Z-ellccedd cohort, shifting memorized instances closer to the mode of the difficulty distribution.
Clarifying the p-value in Sec. 4.1 (memorization reduction). We define a sample as "easier to learn" if its cumulative sample loss is lower under FoeL than under Standard for the same image (paired by sample index the same training split), ie., CSL. < CSLsa In the top-1% Feldman-Zhang memorization cohort overlapping our Partition-A training set (n = 820), 819/820 samples improve (99.88%). We assess this with one-sided exact binomial (sign) test under the null that improvement is equally likely as degradation (Ho = 0.5 for CSL < CSLsaa,, which yields 1.17 x 10-244, As a complementary magnitude- based summary, we also report a paired t-test on CSL differences (mean CSL drops from 204.37 to 72.22; p == 1.09 x 10-20 10-276).
FG ratio diagnostic. FG denotes the foreground area fraction (ground-truth bounding- box area divided by full image area). We report FG only as descriptive metric to contextualize the memorized cohort; it is not used as causal proxy for memorization. On the 85K training split, the baseline mean FG is 0.470, while the memorized cohort mean is lower (0.457), this is a small difference. Furthermore, the fraction of samples with FG < 0,5 differs by only 1.33 absolute percentage points between the memorized cohort and the full 85K split. These statistics demonstrate that FG is a weak separator and does not explain cohort membership. Consequently, our memorization conclusions rely strictly on the identified cohort and the significant CSL reductions. Qualitative visualizations are included to illustrate the contextual complexity of memorized samples, not as evidence that FG drives memorization.
A.7 Gradient Norm Analysis
Setup To probe the optimization dynamics of FocL, we analyze the magnitude of gradients during training, Specifically, compute the & norm of gradients with respect to all model weights on the training set of Partition A (85K samples from the 100K ImageNet subset). Gradient norms are logged throughout training for both the standard model and the FoeL single-crop variant. This analysis provides insight into training stability and the case of optimization under different input regimes.
Foel exhibits consistently smaller gradient magnitudes compared to standard training, suggesting smoother optimization landscape. The standard model records a mcan gradient norm of 3.81 10 with a standard deviation of 2.26 104, while FoeL reports lower mean of 2.07 x 10a and a standard deviation of 1.28 10 When normalized by the total number of ResNet-50 parameters (~ 2.56 x 10²), the per- rer-aarmetter gradient norm drops from 1.49 x 10-3 (standard) to 8.08 10-4 (FocL)-a relative reduction of approximately 45.8%. This substantial drop suggests that FocL's object-centric inputs result in less gradient noise and more stable optimization, aligning with our findings on faster convergence and lower memorization.
A.8 Data efficiency
Setup for Low-Data Regime To evaluate data efficiency, we train all models on Partition A of the 100K balanced ImageNet subset. The standard baseline uses 85 training samples per class, while low-data models are trained with 50 samples per class. These 50-per-class subsets are derived from five random data partitions of Partition A (i.e., five distinct data seeds). All models are trained for 90 epochs using SGD with momentum and step learning rate scheduler (decay at epochs 30 and 60), with otherwise identical hyperparameters.
Evaluation is performed on a fixed 50K test set from Partition B. The standard model is evaluated on full-resolution images, while FoeL models are evaluated using bounding box-aligne aligned crops. As shown in Scction 4.1.1 of the main paper, full-image models underperform when evaluated on oracle bounding boxes. Therefore, we report results using their respective optimal evaluation inputs. For FoeL, we use the single-crop variant, consistent with the setup in Section 4.3.
28

---

Published in Transactions on Machine Learning Research (02/2026)
Table 10: Evaluation the low-data regime using the 50K test set from Partition B. With 41.18% fewer training samples (50 vs. 85 per class), the FocL Single Crop model achieves comparable or better performance than the standard model trained on the full set. All results report mean + standard deviation across 5 random data partitions.
Tested on
Dataset Size (K)
Top-1
Top-5
Full Image (Standard) Full Image (Standard) Bounding Box (FocL SC)
85
44.30
68.56
50
26.91 + 1.20
49.36 = 1.17
50
45.04 + 0.93
70.30 = 0.85
Training and Validation Accuracy Compartson
Training and Validation Loss Comparison
Faci, Train Loss
Focl. Val Acc
Fooi Val Loss
0.8
Standard Train Acc
Standard
Standard Val Acc
Standard Val Loss
20
60 80
Soach
Bocch
Figure 13: Training and validation accuracy and loss curves for FoeL and standard models over 90 epochs. FoeL converges faster and achieves lower training and validation loss throughout.
Analysis. FoeL demonstrates superior data efficiency in the low-data regime as well. With only 50 training samples per class (41.18% fewer than the standard bascline), it achieves Top-1 accuracy of 45.02% and Top-5 of 70.30%, outperforming the standard model trained on 85 samples/che cmppee/class (Top-1: 44.30%, Top-5: 68.56%). This margin holds consistently across 5 random data partitions. These results are statistically consistent across data splits, highlighting FocL's robust ability to leverage object-centric signals even in lower data regime.
A.9 Training Dynamics and Convergence.
Learning curves on the 482K ImageNet subset (Figure 13 highlight the impact of foveated inputs on opti- mization behavior. Across the entire training trajectory, FoeT. converges faster, reaches lower training and validation losses, and achieves higher accuracy than the Standard model at matched epochs. This indicates that the foveated representation prior reduces the effective complexity the learning problem, enabling the optimizer to make more stable progress with fewer samples. The gap emerges early in training, within the first few epochs, and remains consistent throughout, suggesting that FoeL benefits from smoother loss landscape and more informative gradients. Overall, these trends support the view that object-centric inputs lead to improved sample efficiency, more predictable optimization dynamics, and better generalization from the same amount of data.
A.10 Comparison with Feature Pooling TTwo-Stage Detection Baseline)
To validate the motivation behind the FocL "foreground-only" strategy, compare it against "localize- then-classify" baseline modeled after the second stage of modern two-stage object detectors (c.g., Faster R- R-CNN). This baseline employs feature pooling rather than raw image cropping to isolate object information.
Dataset:
We utilize the curated ImageNet subset (482K images) consistent with the main pa- per. This ensures that the model exposed to the same object-centric instances and ground-truth bounding box distributions as the Foel and Standard models.
29

---

Published in Transactions on Machine Learning Research (02/2026)
2. Baseline Method: We implement ResNet-50 backbone modified for feature pooling. The architecture processes the full-resolution image up to the third residual block (Layer 3), re- sulting in feature map with a spatial stride of 16. We then apply an Rol Align layer wttp:risiiinoos.r........ (torchvision, ops.roi. ops.roi_aligm with a 7 output spatial size and a spatial scale of 1/16. This allows the model to extract region-specific features directly from the high-dimensional feature map based on ground-truth coordinates. The pooled features are passod through global average pooling layer and final linear classifier.
Evaluation: same prrer-parameters as the standard and FoeL models.
This comparison isolates the effect of the imput representation (hard foveated crops vs. pooled feature maps). While feature pooling is the standard for detection, our results demonstrate that providing the model with righ-resolution "hard" crops (FocL) leads to superior classification accuracy and reduced memorization of background noise compared pooling from a lower-resolution backbone feature map.
A.11 Evaluation of Background Removal without Resolution Enhancement
To evaluate whether the performance gains in FoeL stem solely from the removal background clutter, we compare it against a "HardMask" bascline. This model processes images where the background is strictly neutralized while the object remains al its original image-space scale, thereby isolating context removal from foveated resolution enhancement.
Dataset: We utilize the curated ImageNet subset (482K images) consistent with the main paper. This ensures the model is trained on identical instances and ground-truth bounding box distributions as the FoeL and Standard models.
We implement a 'Mean-Pad" masking strategy within a custom dataset wrapper (HardMaskDatasstt). For each image, the ground- toonn bounding box is parsed from ImageNet XML annotations. A composite image is then created at the original resolution: pixels inside the bounding box are preserved, while all pixels outside the box are replaced with solid color representing the ImageNet dataset mean (RGB: 124, 116, 104). This masked moage-containing the object at its original scale- is then passed through a standard augmentation pipeline (e.g., RandomResizzeCroop) and a standard ResNet-50 backbonc.
Bascline Method:
3. Evaluation: Same hyper-parameters as the standard and Foel models.
This baseline strictly isolates the impact of background removal. By maintaining the object at its original resolution within the full image frame (no zooming or foveation), this setup tests whether performance gains observed in FoeL are merely result of filtering background noise if the foveated resolution enhancement provided by cropping and resizing is necessary for the observed improvements.
A.12 Evaluation under Idealized Localization (Oracle GT)
The primary motivation for this experiment is to assess the inherent classification capability of each training strategy when provided with perfect localization information. By utilizing oracle ground-truth bounding boxes, we isolate the performance of the ventral classifier from any potential errors introduced by a dorsal proposal generator.
Inference Methodologies:
Standard: Inference is performed on the full-resolution image using standard center-cropping, representing traditional classification pipeline.
RoI Align: The model processes the full 224 224 image, and features are extracted from the ground-truth region via Rol Align pooling on the Layer feature map, exactly replicating its training mechanism.
30

---

Published in Transactions on Machine Learning Research (02/2026)
Table 11: Classification performance comparison on the 2K ImageNet-VV subset using Oracle ground-trute (GT) boxes. All models are evaluated their optimal inference domain to assess the inherent capability of each training strategy.
Method
Inference Strategy
Top-1 (%)
Top-5 (%)
Standard
Full image (standard classifier inference)
60.90
83.05
RoI Align
Full image + GT box (RoIAlign pooling on GT boxes)
63.55
84.65
HardMask
GT box + mean-pad outside (HardMask inference)
62.95
85.95
FoeL (SC)
GT box cropping inference
71.05
89.80
FoeL (MC)
GT box cropping inference
72.23
90.82
HardMask: Ground-truth regions are preserved while the background is masked with the dataset mean color at the original resolution, followed by a standard validation transform, matching the HardMask training protocol.
FocL (SC/MC): The ground-truth bounding box is used extract foveated crop which is resized to the input resolution for classification.
Results The empirical results in Table 11 demonstrate significant performance disparities across the eval- uated strategies. While the Rol Align and HardMask baselines improve upon the Standard model by 2.65 and 2.05 percentage points in Top-1 accuracy, respectively, they remain substantially inferior to the FocL models. Specifically, FoeT. (SC) achieves 71 71.05% Top-1 accuracy, outperforming the Rol Align baseline by 7.5 percentage points and the Standard model by 10.15 percentage points. FocL (MC) further extends this lead to a 72.23% Top-1 accuracy.
Explanations The modest gains in MardMask and RoLAlign are expected, since both leverage localization to emphasize the foreground and reduce reliance on spurious background correlations. However, the RoIAlign baseline pools from a backbone feature map computed on the full image; because these features have large receptive fields (and are influenced by global context through convolution catttiionpaddigg////ooe soaddingnnommalzzatoon the re- sulting Rol features are not strictly object-only and can still carry contextual signals. In contrast, HardMask removes background in pixel space but does not increase the object's effective resolution, since the object remains at its original scale within the resized frame. FoeL addresses both issues by explicitly foveating in image space (crop + resize), providing higher-resolution object evidence to the classifier, which accounts for the remaining performance gap beyond context removal alone.
A.13 Evaluation under Realistic Localization (SAM-2 Proposals)
The primary motivation for this experiment is to evaluate the transferability and robustness of the ventral classifiers within a realistic active vision pipeline. By pairing the classifiers with the Segment Anything Model (SAM-2) as dorsal proposal generator, we test their capability handle real-world proposal noise and natural distribution shifts on ImageNet-V1 and ImageNet-V2. This evaluation utilizes the " "Any Accuracy" metric, which serves as empirical performance ceiling for the overall system by considering sample correct if the ground-truth label is predicted for any of the top 20 proposals.
Following the identical evaluation setup and hyperparameters outlined in Appendix A.4.D and E, all models are evaluated using the same set SAM-2 generated proposals to ensure a fair comparison. For the Standard and FoeL models, the inference strategy follows the localized object crops pipeline. To provide a rigorous baseline comparison, the RoI Align and HardMask models are evaluated using both their native training- matched strategies and the localized object crops strategy.
Inference Methodologies:
The inference strategy follows the localized object crops pipeline where each SAM proposal region is cropped from the original image and resized to the input resolution.
Standard:
31

---

Published in Transactions on Machine Learning Research (02/2026)
Table 12: Classification performance under the SAM-2 proposal regime using the "Any Accuracy" metric. This evaluates the robustness of the classifiers when paired with an external dorsal localizer on ImageNet- V1 and ImageNet-V2.
Method
Inference Strategy
VI Acc (Any)
V2 Acc (Any)
Standard ResNet
Localized Object Crops
68.08%
57.50%
HardMask
Proposal Masking (Hardmask)
62.30%
50.90%
HardMask
Localized Object Crops
60.00%
48.90%
Rol Align
Rol Pooling
73.05%
58.75%
RoI Align
Localized Object Crops
59.45%
49.25%
FocL (SC)
Localized Object Crops
75.05%
64.80%
FoeI (MC)
Localized Object Crops
75.82%
65.80%
Rol Align:
This baseline is evaluated using (1) Rol Pooling, which processes the full 224 x 224 image and extracts features for each SAM proposal via Rol Align pooling on the Layer 3 feature map to replicate its training mechanism, and (2) Localized Object Crops.
This baseline is evaluated using (1) Proposal Masking, which ncutralizes the back- ground of the full original image with the dataset mean color around each SAM box (followed by a standard validation transform without foveated zoom), and (2) Localized Object Crops.
HardMask:
FocL (SC/MC): The inference strategy follows the localized object crops pipeline where each proposal region is cropped from the original image and resized to the input resolution.
Results
The empirical results in Table 12 demonstrate that FoeL significantly outperforms all baselines under realistic proposal conditions, establishing its superiority in active vision tasks. While the Rol Align model achieves competitive performance using its native Rol Pooling strategy ImageNet-VI V1 (73.05%). closing the gap to FocI. (SC) to within 2%, its accuracy collapse on ImageNet-V2 is much more pronounced, with the gap extending to 6.05%. Similarly, when evaluated using Localized Object Crops, the performance of the Rol Align model drops to 39.45% on V1 and 49.25% on V2, indicating that its features are heavily coupled to the global context and fixed spatial resolution of the full-image feature map.
Explanation FocL learns robust object-centric features that transfer superiorly across datasets because it is trained to identify objects in a scale-invariant, foveated input domain. By forcing the ventral stream classify high-rosolution, cropped instances, FoeL prevents the model from relying spurious background correlations or the specific spatial stride of a backbone feature map. Consequently, while fealure-pooling base- lines like Rol Align suffer when faced with the distribution shifts and proposal noise inherent in ImageNet- V2, FoeL maintains its lead by leveraging high-acuity object evidence that remains consistent across different environments.
32


Reviews:
[{"review_id": "MwZYNT0YpE", "reviewer": "Reviewer_BEdk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a supervised training strategy called FocL that is inspired by the human visual system's separation of dorsal (localization) and ventral (identification) pathways. The authors identify that standard deep learning models often rely on \"spurious correlations\" between labels and background clutter leading to memorizing hard examples rather than learning robust features. To mitigate this, FocL uses crops from foreground bounding boxes instead of full-image as training inputs, which they call foveated glimpses. The method simulates biological saccades by \"jittering\" the fixation point to generate multiple, slightly shifted views of the object, which are processed in the same mini-batch to enforce intra-class consistency.\n\nStrengths:\n\n* The paper provides compelling evidence for spurious correlations by showing that standard models perform worse when given ground-truth oracle crops, proving their reliance on background context.\n\n* The method works with SAM which opens up opportunities for integrating it with foundation models in general for object proposals.\n\n* The paper includes empirical results from extensive experiments in several dimensions such as learning dynamics, robustness, and generalization.\n\nWeaknesses:\n\n* The paper claims \"learning with less data\" but it doesn't account for higher cost of bounding boxes compared to simple labels for images.\n\n* The authors ran baselines on full images and their method on crops. It's not a fair comparison. At least the authors didn't point out how much advantage they already have by starting from crops..\n\n* Again, empirical results are between full images and crops. We're missing intermediate baselines like saliency-based cropping or copy-paste augmentation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors's main areas of claims are well supported.\n\n* Memorization: They use cumulative sample loss and the Feldman & Zhang memorization score to show that hard samples become easier to learn when background clutter is removed.\n\n* Generalization: The paper shows that the proposed method achieved significant gains in top-1 accuracy on ImageNet-V2 and COCO.\n\n* The paper also claims smoother optimization and faster convergence which are supported by results from tests with ImageNet, in 4.3.\n\n* The paper also claims that their FocL learns \"more stable features\". It's supported by evidence such as tighter t-SNE clusters in Figure 2 and larger $l2$ adversarial distances in 4.1.", "audience_yn": "Yes", "audience_explanation": "The paper addresses core challenges of deep neural networks, such as robustness, memorization, and data efficiency. These are central and interesting to the TMLR community. The proposed method combines specialist classifier with a foundation model, SAM, is also relevant to current trends.", "requested_changes": "About the efficiency claims. Authors claim that the proposed method uses \"56% less training data\" and thus offers \"efficient visual recognition\". This is misleading because it ignores the high cost of bounding box annotations. The authors should acknowledge that although they can be more efficient by using fewer samples, the annotation efficiency is actually going lower compared to standard class-label training.\n\nAbout related work: The literature review in the paper contrasts FocL with complex RL policies or unsupervised discovery. The paper should mention saliency-based data augmentation which is a simpler and also automated method for breaking background dependence.\n\nAbout adversarial evaluation in 4.1. The authors attack the standard model on full images and FocL on crops. Since cropping out the background pixels making it harder to hide perturbations. The paper should acknowledge that cropping itself is a contributing factor to the increased perturbation distance, in addition to the improved feature stability, unless the authors don't agree with me.\n\nAbout batch normalization. In section 3, the paper goes that glimpses from the same image are in the same mini-batch, which doesn't fit a typical assumption that samples in the same mini-batch are i.i.d. We need an explanation on whether this correlation affected batch normalization statistics, either positively or negatively.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SUhROYMoro", "reviewer": "Reviewer_bs49", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper, inspired by the human visual system, investigates training paradigms for object recognition from foveated crops. The effectiveness of this approach is validated from multiple perspectives, including: 1) reduced model memorization requirements; and 2) enhanced robustness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper provides highly comprehensive and multi-faceted experimental validation.\n1. Fig. 1 provides an intuitive visualization of the impact of holistic learning.\n2. Fig. 5, 6 effectively validate that the FocL training paradigm helps reduce model memorization.\n3. Fig. 7 effectively demonstrates the contribution of the FocL training paradigm to model robustness.\n4. Table 1 and 2 clearly demonstrate the effectiveness of this training method.", "audience_yn": "Yes", "audience_explanation": "Yes, this paper explores a key issue in the field of object recognition. \n\nIts research hypothesis—that objects need to learn mainly from the foreground to be robust and ignore the influence of spurious background information—is intriguing.\n\nAnd the validation experiments are sufficiently thorough.", "requested_changes": "While the validation is extensive, a comparison with standard two-stage object detectors is necessary to validate your motivation, and proof that your \"foreground-only\" (hard crop) strategy is superior to the \"localize-then-classify\" (feature pooling) approach used in detection.\n\n1. Dataset: ImageNet subset used in your paper, or the COCO dataset,\n2. Baseline Method: Faster R-CNN or other methods your like, only if it is fair.\n3. Evaluation: Compare your trained classifier with the classifier trained in these two-stage object detection methods (To ensure fairness, please evaluate the two-stage object detector's classification ability using the same ground-truth bounding boxes)", "broader_impact_concerns": "This work will not give rise to ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iaAzOUH3Oe", "reviewer": "Reviewer_vWJm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new method, called FocL, for learning a classifier from images with pre-annotated segmentations of the main object(s). The method consists of taking one or many crops of varying size centered near the central object's bounding box, and then using those crops directly as training samples in the same batch in the standard classification pipeline. The authors conduct experiments to show that this method improves over classifiers trained using the standard pipeline in several ways: training stability, reducing memorization, cleaner embedding spaces as measured by t-SNE projection, robustness to $\\ell_{p}$ input perturbations, and sample efficiency. When segmentations are not available at inference-time, the authors show that FocL-trained classifiers can also work with a trained segmentation model such as SAM.\n\nMain strengths:\n- The FocL method is simple and intuitive: reducing background context leads to smoother training and fewer outliers, leading to most or all of the above mentioned benefits.\n- Many of the above benefits (see below) are relatively significant on image data where a pre-annotated segmentation exists.\n\nMain weakness:\n- The method needs a pre-annotated bounding box around the relevant object(s), at least during training (as far as I can see, the use of a model to do segmentation during _training_ time has not been explored). This makes a comparison between FocL and a standard classification pipeline not very fair, because you are essentially comparing a model which maps image -> class to a model which maps (image, segmentation) -> class (or, more relevantly for comparing training stability, the training steps map (image, class) -> loss vs (image, class, segmentation) -> loss). There can be some simple baselines for a comparison in this sense, e.g., feeding in an embedding of the coordinates along with the whole image (if the authors were using a transformer this could be done using cross-attention or similar; the situation is slightly more delicate for the ResNet backbone that they do use).", "claims_and_evidence_yn": "Yes", "claims_explanation": "There are no theoretical claims. The empirical claims are mostly supported, with some caveats, which I will  go through in order of their presentation:\n- FocL reduces memorization (4.1): The authors claim that FoCL makes 99.88% of the \"hardest samples\" (defined more precisely in the paper) easier to learn, then cites a $p$-value of $< 0.001$. What is this $p$ value? What is the underlying probability distribution, etc? Also, in the same paragraph, they cite that such hardest samples are hard because they have foreground to image area ratio (FG ratio) relatively large, saying it is equal to 0.457 in expectation. What is the baseline FG ratio?\n- FocL improves generalization under foveated inputs (4.2): The authors claim that FoCL-trained models with a pretrained SAM to do the bounding box annotation perform better at in-domain image classification (training set ImageNet V1, testing set ImageNet V1) and out-of-domain image classification (training set ImageNet V1, testing set ImageNet V2 or COCO). The claim here is a bit tricky since SAM has surely been trained on many kinds of images including ImageNet V2 and COCO. However, using SAM with standard image classifiers still demonstrates the gap, so I think it should be OK. In my opinion, it would be better to also do some experiments with a possibly weaker segmentation model, where you know the training data exactly and can test on some truly OOD data.\n- FocL enhances learning efficiency and stability (4.3): In instances where the authors vary the training hyperparameters (e.g. number of steps or learning rate schedule), they do not specify whether the hyperparameters are optimized (by hand or otherwise) for this setting. E.g. a suboptimally chosen learning rate schedule may explain the results of 4.3. If hyperparameters are only optimized for one case (like the main case) they can cause bad training dynamics in other cases. It might be worthwhile to clarify this, as I didn't see this in a read-through of the main body and appendix.\n\nMore broadly, a major focus of the paper is that FocL improves on the standard baseline classification _because_ it does not feed in background context into the model. This is not sufficiently proven, in my opinion. This is related to the weakness discussed above, and can be resolved in the same way: try some other simple baselines which incorporate segmentation information into the model and yet include the whole input image to the network. This would reveal whether the effect is due to the additional segmentation information or the loss of irrelevant information.", "audience_yn": "Yes", "audience_explanation": "The problem is to learn an image classifier, a standard task in machine learning and one in which many members of TMLR's audience may be interested. This paper argues that there is a substantial benefit to the image classifier when using segmentation maps during training and inference, at least using their method. If true this could potentially improve practical classification pipelines when there is segmentation data already available.", "requested_changes": "These changes derive from the above weakness and analysis of the main claims.\n1) This has been suggested twice above: train some baselines which incorporate segmentation information to demonstrate that the improvements of FocL really do come from removing the irrelevant context. Unless I am missing something in my reading, this issue seems the most critical to me (and therefore for my recommendation for acceptance).\n\n2) It would be good to clear up the numbered caveats pointed out in the analysis of the paper's claims. Namely, the first and third caveats, related to sections 4.1 and 4.3 respectively, can be done without any additional experiments and just clarified in the paper, if the results are as implied by the relevant sections of the paper. The second caveat, related to section 4.2, would be a nice-to-have-resolved but is not necessary for my acceptance due to the marginal value; it would also require extra effort to run such experiments with a new segmentation model (especially one trained just by the authors).", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)