Contrastive Visual Data Augmentation
Authors: Yu Zhou
Bingxuan Li
Tang Mohan
Xiaomeng Jin
Te-Lin Wu
Kuan-Hao Huang
Heng Ji
Kai-Wei Chang
Nanyun Peng
Venue: ICML 2025 poster (2025)
Keywords: Data Augmentation
Text-to-Image Generation
Feature Extraction
Primary area: applications->computer_vision
TL;DR: We introduce the Contrastive Visual Data Augmentation strategy and the NovelSpecies dataset.

Abstract: Large multimodal models (LMMs) often struggle to recognize novel concepts, as they rely on pre-trained knowledge and have limited ability to capture subtle visual details. Domain-specific knowledge gaps in training also make them prone to confusing visually similar, commonly misrepresented, or low-resource concepts. To help LMMs better align nuanced visual features with language, improving their ability to recognize and reason about novel or rare concepts, we propose a Contrastive visual Data Augmentation (CoDA) strategy. CoDA extracts key contrastive textual and visual features of target concepts against the known concepts they are misrecognized as, and then uses multimodal generative models to produce targeted synthetic data. Automatic filtering of extracted features and augmented images is implemented to guarantee their quality, as verified by human annotators. We show the effectiveness and efficiency of CoDA on low-resource concept and diverse scene recognition datasets including INaturalist and SUN. We additionally collect NovelSpecies, a benchmark dataset consisting of newly discovered animal species that are guaranteed to be unseen by LMMs. LLaVA-1.6 1-shot updating results on these three datasets show CoDA significantly improves SOTA visual data augmentation strategies by 12.3% (NovelSpecies), 5.1% (SUN), and 6.0% (iNat) absolute gains in accuracy.

Contrastive Visual Data Augmentation
Yu Zhou Bingxuan Li Mohan Tang" 1* Xiaomeng Jin Te-Lin Wu
Kuan-Hao Huang" 23
Heng Ji
Kai-Wei Chang
Nanyun Peng'
Abstract
(a) Novel Concept Clouded Tiger Cat
(b) Confusing Concept
Resupply Base
Large multimodal models (LMMs) often strug- gle to recognize novel concepts, they rely on pre-trained knowledge and have limited ability to capture subtle visual details. Domain-specific knowledge gaps in training also make them prone to confusing visually similar, commonly mis- represented, or low-resource concepts. To help LMMs better align nuanced visual features with language, improving their ability to recognize and reason about novel Contrastive visual Data Augmentation (CoDA) strategy. CoDA extracts key contrastive textual and visual features of target concepts against the known concepts they are misrecognized as, and then uses multimodal generative models to pro- duce targeted synthetic data. Automatic filtering of extracted features and augmented images is im- plemented to guarantee their quality, verified by human annotators. We show the effectiveness of CoDA on low low-resource concept and diverse scene recognition datasets including INaturalist and SUN, We additionally collect NovelSpecies, a benchmark dataset consisting of newly discovered animal species that are guaranteed be unseen LMMs.. VL. 1-shot updating results these three datasets show CoDA significantly im- proves SOTA visual data augmentation strategies by 12.3% (NovelSpecies), 5.1 (SUN), and 6.0% (iNat) absolute gains in accuracy. Code and data at contrastive-visual-dan -ataaaaammmettatiinnnnn github.io
African Leopard
Wholesale Store
X
concepts, propose
Bushhy ta
Rosette Patterns,
Plain décar, Cardhoard boves,
Calorful diicos
Elongited silm body.
Spotted
Textual
Carts,
People everyday cloth,
Round ears
build.
People uniifomm,
Visual
CoDA
Generstive Data Aug
Clouded Tiger Cat
Resupply Base
Figure I. CoDA uses diffusion diffusion-generatte synthetic data help LMMs recognize novel and confusing concepts in wild. The "Clouded Tiger Cat (L., pardinoides)" is animal species first described in April 2024, while "Resupply Base" is an example of a confusing concept for LMMs. Based model failures (collected from GPT4o-2024 contrastive visual and textual features to generate synthetic image data for model updating.
and LLaVA -NXT 34B), CoDA extracts
1. Introduction
remains challenge for current state-of of-tte-artttt proprietary and open-source models to robustly recognize novel visual concepts (e.g. "Clouded Tiger Cat" Figure la) and confusing low-resource commonly misrepresented visual concepts (c.g. "Resupply Base" Figure lb).
Recent advancements in multimodal pre-training (OpenAI, 2023; Google, 2023; Hurst al., 2024) and visual instruc- tion tuning (Liu et al., 2023b;a; 2024c) have enabled impres- sive LMM abilities. However, shown in Figure still
Equal contribution, interchangeable ordering. 'UCLA PVIUC 'TAMU. Correspondence to: Zhou <yuzhou@ cs. Eyuzhou@cs.ucla.cooom ucla.
In order help models better acquire new visual concepts and distinguish confusable concepts, existing approaches straightforwardly 1). Fine-tune text decoder corpora to expand the concept base; and 2). Fine-tune both vision and text components on new web image-text pairs
textual
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Contrastive Visual Data Augmentation
for visual concept acquisition. These approaches are inef- fective due data scareity for certain concepts and data inefficiency caused by not knowing what precisely confused the models (3.1). As depicted in Figure 1, it can be difficult to obtain ample high-quality real images for novel concepts such as new animal species. While for confusing concepts, the problem usually lies with biased concept representa- tion in web image-text data. For example: online images of "Resupply Base" mostly only consist of exterior views of the architecture without the interior details, which may cause the models to confuse it with a "Wholesale Store" that shares some interior features.
Our key contributions include:
CoDA, simple plug-and-play contrastive visual data augmentation method that can be used to effectively and efficiently improve LMMs' ability to recognize novel and confusing concepts. CoDA is also the first widely suc- cessful method using text-to-image generation for visual data augmentation.
NovelSpecies, a new benchmark dataset of novel animal species discovered in recent years, providing an ideal benchmark for novel concept recognition. NovelSpecies currently consists of 2240 annotated images and will continue to be updated with future discoveries.
To help LMMs recognize and reason about novel and con- fusing concepts more robustly and efficiently, we propose CoDA, a Contrastive Visual Data Augmentation technique. For each target concept, CoDA first identifies "confusable concept" that the LMMs finds most similar to the target. Then, it extracts contrastive textual and visual features of the target concept with respect to the confusable concept. The extracted features go through a filtering process based on discriminability and generability to make sure that: 1). The features are possessed by the target concept but not the confusable concept; and 2). The feature can be reliably generated by the text-to-image generative model and recog- nized by the LMMs. Afterwards, the features are passed to the text-to-image generative model to produce augmented visual instances of the target concept. To make sure that the features are indeed generated and recognizable by the LMMs, CoDA again uses the LMMs' zero-shot inference to rank and filter the augmented images. Finally, the resulting augmented images can be used to update the LMMs via low-rank adaptation, basic fine-tuning, in-context learning, or any other method of choice.
2. Related Works
Few-shot image recognition is a long-standing problem in the vision community. Early works this area focused on improving traditional image classifiers on classifying exist- ing concepts (Vinyals et al., 2016; Finn et al., 2017; Nichol, 2018; 8; Dhillon et al., 2019; Tian et al., 2020; Bhagat et al., 2023; Afrasiyabi et al., 2022). On the other hand, while re- cent advancements in the training of vision language models (VLMs) and large multimodal models (LMMs) (OpenAI, 2023; Google, 2023; Hurst al., 2024; Liu et al., 2023b;a; 2024c) have shown great promise and extensibility, they still severely lag behind traditional models in image classi- fication, especially for low-resource, novel, and confusing concepts (Zhang et al., 2024; Cooper et al., 2024; Wu et al., 2023b; Yang et al., 2024; Ha et al., 2025).
While commonly used text-side VLM data augmentation strategies (Yuksekgonul et al., 2022: Yang et al., 2023; Liu et al., 2024e; Sharifzadeh et al., 2024) have little effect this issue, a more promising technique solve this is through visual data augmentation. This includes basic visual manipulations such as cropping, flipping, and rotation (Yang et al., 2022; Kumar et al., 2024); and more advanced model- based augmentation such as style transfer (Zheng et al., 2019; Chun & Park, 2021) and image mixing (Uddin et al., 2020; Xie et al., 2021; Hao al., 2023). More recently, with the rise of controllable and promptable visual generative models, knowledge and feature editing-based augmentation methods (Liu et al., 2022; Wu et al., 2023a; Jin et al., 2024) have gained in popularity. Such methods generally focus on using multimodal data and general knowledge bases to guide image-editing models in creating augmented visual data based on existing images.
In addition to evaluating on existing datasets INaturalist and SUN, we create NovelSpecies, an annotated image dataset of newly discovered animal species in recent years. Novel- Species allows the simple selection of species discovered after any model's latest knowledge cutoff date, ensuring the selected species were never seen by the model. Therefore, NovelSpecies is the perfect testbed for methods aimed at improving LMMs' novel concept recognition ability.
Comprehensive experiments with V.LAVA-NeTT on the 3 datasets show CoDA performs surprisingly well in teach- ing LMMs novel and confusing concepts, significantly im- proving data efficiency compared to existing methods. In additional experiments, we show that CoDA is also able to improve novel concept recognition for traditional classi- fiers like VIT and proprictary LMMs such as GPT4o-mint Finally, ablation experiments show that CoDA can be sig- nificantly improved by simply replacing its off-the-shelf components such as the text-to-image generation model with superior versions of similar models.
One main issue with current methods is that the augmented images they produce must be closely based on existing real images, which makes them unhelpful for novel concepts where real images are extremely rare, and mis-represented concepts where existing real images do not accurately depict the concept. Additionally, due to their close connection
2

---

Contrastive Visual Data Augmentation
to existing images, such augmented images usually lack visual frame structure and view variation. In contrast, our method CoDA can extract accurate and meaningful features from extremely limited multimodal data, and use text-to- image generative models to produce diverse high-quality augmented data for LMM updating.
search. However, we believe currently the advantages brought by these methods do not out-weigh their complexity overhead, thus we opt for simplicity.
Visual Feature Extraction While textual feature extrac- tion generally works well for pre- pre-existing and non-hyper- comain--ppccffc concepts that are prevalent in textual data sources, it tends to fail when either of the conditions are not met. For example, large language model with knowledge cutoff prior to June 2023 would not be able to provide mean- ingful features regarding the Apple Vision Pro device an- nounced in July, or the new animal species "Clouded Tiger Cat (L. pardinoides)" first described by scientists in April 2024 (Figure 1). In addition this weakness, LLM-based textual feature extraction is also unreliable when asked to provide detailed information regarding hyper-domain- specific concepts like the "Mazda MX-5 Miata RF" or the "Lear's Macaw (Anodorhynchus Leari)". In practice, we observe that for novel and syper-domain-specfii concepts, most of the LLM extracted textual features end up being filtered out by our automatic feature filtering module.
3. Methods
As shown in Figure 2, CoDA consists of 4 major steps including contrastive textual and visual feature extraction, feature filtering, feature-controlled image generation, and augmented image filtering. Together these steps ensure CoDA reliably generates informative and high-quality aug- mented images that help LMMs recognize novel and con- fusing concepts.
3.1. Feature Extraction
Textual Feature Extraction In our exploratory experi- ments, we find that significant mis-recognition errors occur on low-resource commonly mis- mis-represented concepts in vision-language instruction fine-tuning and multimodal pre- training datasets, which the LMMs are trained on. For exam- ple, the LLaVA 1.6 (34B) model (Liu et al., 2024c), mainly tuned on LAION-GPT-4Y(L.M GLA-ONLLIIO 2024) and ShareGPT- 4V (Chen et al., 2023) datasets, has a strong tendency to mis- recognize interior images of "Resupply Base" as "Wholesale Store" (Figure 1). Unsurprisingly. we find that all related ref- erences of "Resupply Base" across the 3 instruction- -tuning datasets only depict exterior views of the concept rather than interior views. While the concept itself is not a low-resource concept in existing text corpora, it is severely low-resource and also commonly mis-represented in vision-language in- struction fine-tuning datasets.
To address this weakness, we implement an additional vi- sual feature extraction module based on VLMs. Given a single image of the target concept, the VLM is asked to extract its key visual features. When there is more than one image containing the target concept available, we use a LM to de-duplicate and summarize the combined extracted visual features from all images. For simplicity and cost- efficiency, we use the GPT4o-mini model for both visual feature extraction and feature de-duplication.
In contrast to textual feature extraction, visual feature extrac- tion is most effective for hyper-domain-specfffi and novel concepts that are very rare or non-existent in textual cor- pora but have a limited number of visual examples. Thus, it well-complements textual feature extraction. Similarly, we do not attempt to classify which concepts fall under this criterion; instead, we apply this step to all concepts and rely on automatic filtering (3.2) to remove low-quality features.
To address this issue, we prompt LLMs to directly generate feature attributes of the target concept based on their existing knowledge, focusing on visual appearance, and avoiding hallucination for unfamiliar concepts. For this task, we use the cost-efficient GPT40-mini model with chain of thought reasoning. Generally, textual feature extraction is most applicable for concepts that are high-resource in existing textual corpora, yet low-resource and/or commonly mis- represented in vision-language instruction-tuning and pre- training datasets. Here we do not try to classify which concepts fall under this criteria, but rather apply this step for all concepts. To ensure extracted feature quality and filter out hallucinated and/or visuulll-recoonizzaae features, we pass all extracted features through an automatic filtering step, described in 3.2.
Contrastive Feature Extraction While basic textual and visual feature extraction both aim exhaustively list iden- tifying features of the target concept, this is essentially an intractable task for complex concepts as it usually requires huge number of features to fully describe them. For novel or low-resource concepts the LMM has likely never seen before, it is extremely difficult to teach the LLM the new concept using an incomplete description.
There are two potential solutions to this problem: (1). Lever- aging hierarchical information to narrow down concept cat- egory and reduce descriptional features. (2). Illustrating the new concept based on contrastive differences from a similar existing concept the LMM already understands. Pre- vious works in language and visual data augmentation (Jin et al., 2024) tend to use solution (1). However, its feasibility
We also considered other methods for textual feature ex- traction, including using knowledge bases (Jin et al., 2024), retrieval augmented generation, and LLMs with internet
3

---

Contrastive Visual Data Augmentation
Target Concept
Feature Extraction
Feature-controlled Augmentation
Model Updating
Anodorhynchus Leari (Lear's Macaw)
Cüff-nesting, Faster flight,
Native to Brazil
Shorter life, Lighter,
LoRA, Fine-tuning, ICL, VL-Adaptor,
Visual
Anodorhynchus Leari (Lear's Macaw)
Cyanopsitta Spixii (Spix's 's Macaw)
yellow eye rings,
Coal black curved beak
Light facial skin,
Zero-Shot Inference
Augmented Image Filtering
Anodorhynchus Leari (Lear's Macaw)
Feature Filtering
Confusable Concept
FYellow eye ring"
"Cosl black curved beak"
eYellow cheek skin"
Discriminsbiliyy 0.45 Generability; 0.93
Cyanopsitta Spixii (Spix's Macaw)
DDep cobalt blue"
X
"Bright Yellow ring"
Discriminabiliyy 0.98
"Yellow eye ring"
X
Generability: 0.86
"Yellow cheek skin"
X
"Native to Brazil"
DDeep cobalt blue"
Cyanopsitta Spixii (Spix's 's Macaw)
Discriminability: 0.48
x
Genembilitty 0.14
Figure 2. The CoDA method. Including Feature Extraction, Feature Filtering, Feature-comtrolled Augmentation, and Augmented Image Filtering. The target concept and misidentified concept are highlighted respectively. Specific feature filtering scores are for illustration only. Here the example concepts Anodorhynchus Leari (Lear's Macaw) and Cyanopsitta Spixii (Spix's Macaw) are from the iNaturalist (Van Horn et al., 2018) dataset, and augmented images are produced by the Recraft V3 model (Recraft.Al, 2024).
contingent on the existence of a comprehensive textual knowledge base or tree-like structure that already includes the target concept. As discussed in Section 3.1, this is of- electronic
To calculate the Discriminability of a feature f given the tar- get concept CT and misidentified concept CM,, we compute the likelihood that CLIP (Radford et al., 2021) associates this feature with real images of the target concept compared to the likelihood that it is associated with real images of the misidentified class:
the case for novel concepts such products (e.g. Apple Vision Pro) new animal species (eg. Clouded Tiger Cat).
CLIP CC &T CLIP
To enable the handling of novel concepts and remove the need for external databases, we adopt solution (2) and per- form contrastive multimodal feature extraction for target concepts. First, we use the LMM's zero-shot inference on the target concept Cy to obtain the misidentified concept Cм- Then, we perform contrastive textual and visual fea- ture extraction by querying LLMs and VLMs for visually identifying features that belong to Cr but not C.M
D(S, Cr, Ca)) =
Σ CLIP IEI
Caa
Here we use an equal number of images of the target and misidentified concepts. A score below 0.5 indicates that the feature is more likely to be associated with the misidentified class rather than the target class. To ensure that selected features are more strongly associated with the target class, we filter out all features with Discriminability below 0.6. This method avoids the CLIP score bias against smaller features by only comparing feature association with the two classes and not relying the absolute CLIP score.
3.2. Feature Filtering
Automatic Feature Filtering After obtaining visually identifying features from contrastive textual and visual fea- ture extraction, we filter them based on two key criteria:
Generability is calculated for all features that pass the Dis- criminability threshold. We prompt the T2I generative model to generate synthetic images of the target con- cept that contains the feature f. and then compare the aver- age CLIP similarity between and those generated images against the average CLIP similarity between and real.
1. Discriminability (D(f,Cr Car)): measures whether feature indeed differentiates the target class CT from the misidentified concept CM (check whether f is valid feature of CT but not C.))
2. Generability (G(f.C CM))) measures whether a feature f can be properly generated by the text-to- image generative model.
syntbotic 'C, CLIP synthetic, + CLIP (E &r
CLIP
G(f.Cr, CM.
Σ iET

---

Contrastive Visual Data Augmentation
To verify the reliability of our feature filtering and aug- mented image verification modules, conduct human eval- uation on a subset of iNaturalist and the novel animal species dataset. For target concepts, we select 100 image-feature pairs for both real and augmented synthetic images. We also select 100 image-feature pairs for real images of misiden- tified concepts. 3 external human annotators are asked to label whether they believe the given feature belongs to the concept in the corresponding image.
Here we rank all remaining features by their Generability score and select the top 5 features to be passed to the text-to- image generative model (as current diffusion models usually have limited text encoder attention span). This step identifies features that not only help distinguish the target concept, but also can be effectively rendered by the text-to-image generative model in synthetic images, which is critical to the success of synthetic data augmentation.
Our automatic feature filtering module based on Discrim- inability and Generability ensures feature quality and limits the information loss between features and the generated augmented images. The remaining features are used for image generation and improving in-context recognition abil- ity in inference prompts. We further verify the quality of remaining features with human evaluation in Sec.3.4.
Results in Table 1 show human annotators overwhelmingly agree that the final extracted features belong to the target concept (92%) but not the misidentified concept (14%). The augmented synthetic images of the target concept also likely contains the desired features (83%), though as expected, there some information loss between the text-to-image generation step. In addition, the three independent annota- tors generally agreed in their response (>0.8 IAA).
3.3. Image Generation and Verification
3.5. In-Context Inference for Enhanced Recognition
Image Generation After feature extraction and filtering based on Discriminability and Generability, we pass the selected features to a text-to-image generative model to generate augmented visual data. We experiment with both SOTA open-weights (Esser et al., 2024; Stability AI, 2024) and proprietary (Recraft.AI, 2024) models.
In addition to updating the LMM with augmented data, we can further boost performance by integrating the extracted features into the inference prompt. For each query, we can append a concise list of the most discriminative and gen- erable features of the target and confusable classes. These features serve as an in-context guide, focusing the LMM's attention on critical distinguishing attributes. By explicitly highlighting what to look for (and what not to mistake it for), the model more reliably identifies the correct concept.
Verification To ensure final images for augmentation con- tain our extracted and filtered target concept features, we propose a simple automatic verification metric that checks whether desired features are recognized in the augmented images by the LMM we want to update: Given the vanilla LMM M, a set of features F, and an augmented image gyymthetic the feature satisfaction rate S ((gyymhetic F, F) for each augmented image: Siyymthetic
4. NovelSpecies Dataset
Proprictary LMMs like GPT4o (Hurst et al., 2024) and Gemini (Google, 2023) are trained on vast online text-image data and proprietary data, both non-public and impossible to inspect. Some open-source and open-data LMMs such as LLaVA (Liu et al., 2024b;d) are trained on publicly available image-text datasets. However, the text encoders used by such models are often not open-data, for example L.LaVA- 1.6 34B uses the closed-data Yi-34B model as its language backbone, Even in the rare cases where both image-text training data and text encoder training data are publicly available, it is still difficult to ascertain whether concepts in your benchmark were seen by your LMM through indirect data leakage (i.e. partial / paraphrased mentions). Due to the above issues, it is difficult to evaluate true novel concept recognition ability with existing datasets.
Fenytheic,F,M, F, = Ejer IMMS, (synthetic))
|F
Here M(f, (yythetttt)))) }} returns true if the feature f is recog- nized in the image jsymibetic Afterwards, we filter out all im- ages with S(jyyihetic, F,M) <1.0, keeping only augmented images that fully match all target concept features.
3.4. Human Evaluation
Image Type
Target Concept (%)
Misidentified Concept (%)
Inter-Annotator Agreement (x)
One way to bypass this problem with 100% guaranteed suc- cess is to use a dataset that only contains concepts created discovered after the LMM's knowledge cutoff, i.e. the latest knowledge cutoff date among all of its textual / visual sub- components, Based on this idea, we curate NovelSpecies, dataset of novel animal species discovered in each recent
Real 92.51
14.32
0.87
Synthetic 83.97
0.82
Table 1. Human eval of extracted features and augmented im- ages. 3 external annotators asked to answer (yes/no) whether the extracted and filtered features are present in the corresponding real and synthetic images. IAA based on Fleiss' Kappa.
5

---

Contrastive Visual Data Augmentation
year, starting with 2023 and 2024. We provide detailed infor- mation for each species, including time of discovery, latin
here for direct comparison with CoDA and other exist- ing feature-based augmentation methods.
name, common name, family calegory, textual description, and more. Data will be released upon publication.
3. ARMADA (Jin al., 2024) the current state-of-the- art feature-based visual data augmentation strategy for concept recognition and image classification.
To create NovelSpecies, we start by collecting the list of species first described in each year by Wikidata (Wikidata, 2024). Then, to make sure we can curate visual benchmark of novel species, we manually annotate and filter out extinct species and species with too few publicly available images. After filtering, we end up with dataset of 64 new species, each consisting of 35 human-verifien image instances, thus a total of 2240 images. The images are split into training, validation, and test sets. For each species, there are 5 train- ing images, 15 validation images, and 15 test images. This data split is consistent with our goal of creating a benchmark dataset for novel concept recognition, where the maximum number of training instances for a completely unseen con- cept can range from 1 to 5.
In addition to these 3 baselines, we also include ablations of CoDA with non-contrastive textual and visual features, i.e. w/o contrastive guidance from confusable concepts (3.1) nor discriminability-bine based feature filtering (3.2).
5.2. Main Experiment
For our main experiment, we consider two different resource settings that correspond to common real-world scenarios:
Fixed Real Data Under the fixed real data setting, we only have access to real images for each concept. Each data augmentation strategy may generate 1-5 synthetic images. Then, the model is LoRA-adapted and synthetic images. This setting simulates real-world scenarios, where there isn't 't sufficient real training data for certain concepts. This is common for novel concepts, hyper- Nommann-spccifii concepts, and long-tail distributed datasets. In these scenarios, the quality and effectiveness of synthetic augmented data is especially instrumental to the updated model's performance.
the combined real
5. Experiments
5.1. Datasets and Baselines
To evaluate CoDA's ability to improve novel and confusing concept recognition in LMMs, we experimented with CoDA and other relevant baselines three different datasets:
1. The iNaturalist Dataset (Van Horn et al., 2018) is challenging natural world concept recognition bench- mark for LMMs due to its extensive highly domain- specific and fine-graine fine-grained species categories and inclu- sion of rare and low-resource species classes.
Experiment results across the 3 datasets show that CoDA consistently outperforms existing traditional and feature- based data augmentation methods in the Fixed Real Data setting. When augmenting the training set with just a single synthetic image, CoDA able to achieve 11.8% (Novel- Species), 10.0% (SUN), and 17.8% (iNat) absolute gains in accuracy compared to using all real images. It further out- performs the best existing baseline augmentation methods by 5- 12% absolute gains. We also observe that the ablated performance of CoDA (w/o contrastive) is still significantly above traditional and image-cditing- bbsed augmentation baselines while being almost consistently below CoDA's 's performance. This shows the benefits of text-to-image gener- ative augmentation methods compared to existing methods, as well as the benefits of fine-graines textual features dur- ing inference. This also highlights the need for contrastive feature selection and discriminability- -bsed feature filtering, We find that increasing the number of augmented synthetic images does not necessarily improve updated model perfor- mance: this may be attributed the fact that all generated images are ranked and selected from the same pool, with the first image being of the highest quality. Finally, the largest improvement over existing baselines can be seen in Nov- elSpecies, where CoDA methods involving visual features achieve the highest performance. This makes sense as the visual feature extraction method is designed to be robust to novel concepts with little textual documentation.
2. The SUN Dataset (Xiao et al., 2010) is a widely used large-scale scene recognition dataset that contains rich and confusing visual scenes. Correctly recognizing the scenes requires fine- mne-grained visual reasoning and understanding of the scenes.
3. NovelSpecies Dataset (Sec.4) is our new dataset con- sisting only of novel animal species concepts that LMMs are guaranteed to have never encountered in their training or fine-tuning.
For each dataset, we use an automatic data selection strat- egy A.2 to find subset of challenging concepts that the model fails to recognize. Then, we apply CoDA along with 3 other visual data augmentation baselines:
1. All Real uses an all real augmented image set. In the Fixed Real Data setting, this means using the 5 real images provided. In the Fixed Compute setting, this means using unlimited real images to match the total number of real + synthetic images in other settings.
2. Cropping and Flipping are widely used traditional visual data augmentation strategies. We include them
6

---

Contrastive Visual Data Augmentation
Augmentation Method
Feature Type
Fixed Real Data (Real:Syn)
Fixed Compute (Real:Syn)
Dataset
5:0
5:1
5:3
5:5
20:0
10:10
0:20
All Real
61.2
Cropping
60.4
60.4
59.5
Basclines
Flipping
60.7
62.9
60.1
ARMADA
60.7
60.2
61.2
NovelSpecies (Sec.4)
Textual
69.1
68.6
70.5
CoDA (w/o
Visual
71.8
72.6
71.7
contrastive)
T+V
70.3
65.1
70.1
Textual
72.0
69.2
70.3
CoDA
Visual
73.0
72.8
71.8
T+V
70.1
72.6
73.0
All Real
73.4
74.3
Cropping
78.3
75.8
76.3
77.3
76.4
Baselines
Flipping
75.7
78.4
74.8
75.2
76.1
ARMADA
75.9
78.3
77.6
76.2
76.8
SUN (Xiao et al., 2010)
Textual
80.6
79.7
79.4
81.3
80.8
CoDA (w/o
Visual
81.3
81.6
79.3
80.0
80.8
contrastive)
T+V
82.7
80.7
80.4
82.8
82.1
Textual
79.2
83.2
82.3
82.8
82.1
CoDA
Visual
82.3
81.7
82.2
81.8
83.1
T+V
83.4
81.7
82.6
83.3
82.1
All Real
49.2
64.3
Cropping
59.7
58.8
62.2
61.4
63.9
Basclines
Flipping
61.0
61.1
62.3
62.1
62.7
ARMADA
60.1
60.7
61.1
61.6
58.5
iNaturalist (Van Horn et al., 2018)
Textual
63.9
64.6
66.5
65.6
63.2
CoDA (w/o contrastive)
Visual
65.0
64.7
64.3
65.6
63.2
T+V
62.8
64.4
62.3
64.4
63.4
Textual
63.9
67.8
62.6
65.0
64.9
CoDA
Visual
67.0
66.0
65.1
62.5
60.9
T+V
63.5
65.0
64.6
67.0
64.1
Table 2. Main experiment results on INaturalist, SUN, and NovelSpecies under Fixed Real Data and Fixed Compute settings: Experiments are defined by the number Real:Synthetic images used. example, means model uses 5 real images and synthetic image for each concept class training time. All results are in terms of LLaVA-1.6 34B concept recognition accuracy (%). Best performance scores for each setting and scores using all real data are highlighted in Red and Green, respectively.
Fixed Compute Under the fixed compute setting, we as- sume access unlimited real and synthetic images. How- ever, the fine-tuning budget only support total of 20 images, allowing different percentages of real and synthetic images, from 0% synthetic (20:0) to 100% synthetic (0:20). This setting simulates real-world scenarios, where there is abundant real data. In such cases, the question is whether to just use all real data to update the model, or to include non- trivial amount of augmented synthetic data. Traditionally, real data is always preferred due to perceived higher-quality. However, CoDA's 's effectiveness in the Fixed Real Data set- ting prompts to test the possibility of it being beneficial to include synthetic data even when real data is abundant. This
hypothesis is tested by whether any of the models fine-tuned with mixed real/synthetic data can outperform the model fine-tuned with all real data.
Experiments iNaturalist show diverging results between CoDA and other baseline augmentation methods: While including synthetic images generated by baseline methods generally led to lower performance, using CoDA augmented images can actually lead to improvements over using all real data. Furthermore, 50-50 real- syntheyt data mix gener- ally outperforms all real all synthetic data. We attribute the success of mixing synthetic and real data to the fact that CoDA generated synthetic data is aimed to highlight

---

Contrastive Visual Data Augmentation
Augmentation
Feature Type
LLa VA-
GPT4o-mini
VIT
Method
5:0
5:1
5:3
5:5
5:0
5:1
5:3
5:5
5:0
5:1
5:3
5:5
All Real
61.2
84.3
75.4
Cropping
60.4
60.4
59.5
84.8
86.3
85.9
78.3
77.6
79.6
Baselines
Flipping
60.7
62.9
60.1
83.2
83.5
84.3
76.9
77.9
78.2
ARMADA
60.7
60.2
61.2
84.1
84.3
83.9
76.3
76.4
78.6
Textual
74.8
75.1
74.7
87.6
87.2
87.0
82.5
84.5
84.7
CoDA (w/o contrastive)
Visual
76.5
77.9
76.2
88.3
89.6
88.2
82.5
83.0
82.6
T+V
77.6
78.9
78.8
89.5
91.2
87.9
84.3
84.9
82.5
Textual
76.4
75.9
76.8
87.1
87.9
87.4
84.6
85.0
84.5
CoDA
Visual
77.5
78.1
77.9
91.3
90.8
92.6
85,5
84.6
85.7
T+V
78.8
78.7
79.2
91.6
90.8
91.4
85.3
85.8
86.3
Table Experiments on NovelSpecies with open-weight VLM (LLa proprietary LMM (GPT4o-mini), and traditional classifier (VTT) under the Fixed Real Data sefting. Results in terms of accuracy (%). Synthetic image data generated by Recraft V3. Best performance scores for each setting and scores using all real data are highlighted in Red and Green, respectively.
discriminable features of the confusing novel concepts, making them more prominent and visible compared to real images. On the other hand, real images provide valuable style information and is more accurate reflection of the test- time distribution, helping to "ground" the updated model.
iNaturalist, their performance significantly degrades on NovelSpecies due to having never encountered the novel concepts. To test whether CoDA can effectively improve novel concept recognition performance for such proprictary LMMs, we fine-tune the gpt-4o-mini 2itiiii-22--7--1111 18 model us- ing CoDA and relevant augmentation baselines. Results in Table 3 demonstrate significant performance gain (9.5%) for GPT40-mini after being fine-tuned on CoDA augmented synthetic images. While this improvement not as signifi- cant compared to the LLLVA--1.6 model (20.3%), it is due to GPT4o-mini's s better base performance.
5.3. Additional Experiments
For additional experiments, we focus on NovelSpecies as it most closely resembles real-world scenarios, where over time, models are required to learn novel concepts without access to sufficient real training data.
Traditional Classifier on LMMs which take image-text input and produce text output, we also test whether it can help traditional image classifiers recognize novel concepts. We run the widely- used ViT classifier (Alexey, 2020) on NovelSpecies with CoDA and other augmentation baselines. Results in Table 3 show that CoDA is able to achieve a consistent performance gain over existing baselines for ViT-base (9. 1% for single- shot augmentation). The VIT classifier provides stronger base performance compared general VLMs, thus offering less room for improvement. However, we note here that our main focus improving LMMs instead of such traditional classifiers stems from LMMs' superior extensibility and generalizability to other related tasks such as recognition- based reasoning and explanation.
addition to evaluating CoDA
Advanced T2I Model As explained in Sec.3, off-the-shelf model components used in CoDA can be easily swapped for superior versions of similar models to improve perfor- mance. To demonstrate this, we replace the open-weight Stable Diffusion 3.5 Large Turbo model (Stability AI, 2024) with the SOTA proprietary Recraft Model (Recraft.Al. 2024) and run the same LLa Vpdupating experiments as in Table 2. Here note that Recraft V3 has better instruction- following ability as well as better image generation quality compared to Stable Diffusion 3.5 Large Turbo. More details on these differences can be found in Sec.6. Our experiment results Table show a significant performance boost when LoRA fine-tuning LLaVA with Recraft V3 produced synthetic images compared fine-tuning on all-real data (28.7%) and also compared fine-tuning on Stable Diffu- sion 3.5 Large Turbo produced synthetic data (7.9%). This demonstrates the potential increase of CoDA's 's effective- ness along with improvements in Text-to-Iman Image generative models. We believe it is also possible to achieve similar improvements by replacing the LLM/VLM components of CoDA with superior models in the future.
6. Discussions
In Figure 3, we compare example synthetic images gener- ated by CoDA and baseline visual data augmentation meth- ods including Cropping and ARMADA (Jin et al., 2024). While providing localized feature emphasis, Cropping of- ten results in the loss of crucial visual details necessary for concept identification. For instance, for Phyllobates Sam- peri, cropping occludes the black spots on the frog's s skin,
Proprietary LMM While proprietary LMMs like GPT40- mini (Hurst et al., 2024) tend to have relatively strong 0- shot performance on existing datasets such as SUN and
8

---

Contrastive Visual Data Augmentation
CoDA Features: Prominent orange stripes, Dark shiny coloration, Black spots on legs,
Phyllobates
Samperi
CoDA (SD-3.5)
CoDA (Recraft V3)
Real Images
Cropping
ARMADA
Tail-Spot
Wrasse
CoDA Features: Vibrant horizontal stripes along body, Greenish-yellow cyan accents,
Figure 3. Qualitative Comparison of CoDA and baseline visual data augmentation methods. Phyllobates Samperi and Tail-Spot Wrasse are example concepts from the NovelSpecies dataset. All CoDA images are generated using contrastive textual + visual features.
an essential distinguishing feature. Without such essential distinguishing features, the cropped images provide less helpful learning signals compared to other methods.
ability to accurately generate multiple feature details in a single image. For example, while CoDA-Recraft-V? is able to accurately generate all three extracted features including "prominent orange stripes", "dark shiny coloration", and "black spots legs"; CoDA-SD--35 only able to generate the first two features while failing capture "black spots on legs". With such limitations in mind, we give CoDA an extremely modularized design. This allows each pre-traee -reitraieed model component in CoDA to be easily replaced for newer and stronger versions of similar models, including more perceptive VLMs and T2V generative models with stronger instruction-followinn ability and higher generation quality.
Unlike Cropping, ARMADA successfully retains some structural features of the target concept, using WikiData text features to guide its image-editing backbone (Brooks et al., 2023). However, this setup also induces two signif- icant issues: (1) Leveraging only existing textual features present in WikiData leads to an incomplete feature set, es- pecially for novel concepts. This is apparent in images generated for Phyllobates Samperi. where generated images contained "black spots" but failed specify their location on the legs of the frog instead of the body. In addition, the model completely failed to generate Phyllobates Samperi's s iconic "orange stripes" due to the feature not being recorded in WikiData. (2) Image-editing models are not as strong depicting precise details compared text-to-image gen- erative models. In the case of the Tail-Spot Wrasse, the ARMADA generated image fails to accurately depict "vi- brant horizontal stripes along the body," leading to visually inconsistent and less biologically accurate representation.
7. Conclusion
In this work, we propose CoDA, contrastive visual data augmentation approach that helps LMMs recognize novel, confusing, and low-resource concepts through efficient and effective model updating. CoDA is a plug-and-play method which utilizes off-the-shelf models for contrastive feature extraction, feature filtering, text-to-image generation, and image filtering. We evaluate CoDA against four existing baselines and self-ablations on three datasets: INaturalist, SUN, and NovelSpecies, which we created in this work. Consisting only of animal species discovered in recent years, NovelSpecies offers an ideal testbed for LMMs' novel con- cept recognition. We provide comprehensive additional experiments demonstrating CoDA's 's effectiveness for tradi- tional classifiers and proprietary LMMs. Finally, we show that CoDA can be casily improved by replacing off-the-shelf components, such as text-to-image generation model with superior versions of similar models in the future.
In contrast to existing baselines, Figure 3 shows that CoDA is much better at generating high quality synthetic images of the target novel concept that depict accurate and realistic details. Both versions of CoDA using different backbone models (Esser al., 2024; Recraft.AI, 2024) are able to produce significantly more realistic images compared to the two baselines, maintaining general biological consistency. However, we should note that CoDA's performance is in- herently bounded by the instructionffolloiin ability of its image generation backbone model, more specifically the

---

Contrastive Visual Data Augmentation
Acknowledgment
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
This material is based on research supported by the ECOLE program under Cooperative Agreement HR001 12390060 with the US Defense Advanced Research Projects Agency (DARPA), award from Office of Naval Research with -000014-23-1-7780 Apple Research Award, and Amazon AGI Research Award. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing DARPA, or the U.S. Govern-
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
ment.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. Interna- tional conference on machine learning, pp. 1126-1135. PMLR, 2017.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Gidaris, and Komodakis, N. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4367-4375, 2018.
References
Google. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Afrasiyabi, A., Larochelle, H., Lalonde, J.-F., and Gagné, C. Matching feature sets for few-shot image classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9014-9024, 2022.
Ha, H., Jin, X., Kim, J., Liu, J., Wang. Z., Nguyen, K. D., Blume, A., Peng, N., Chang, K.-W,, and Ji, H. Synthia: Novel concept design with affordance composition. arXiv preprint arXiv:2502.17793. 2025.
Alexey, D. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929. 2020.
Hao, X., Zhu, Y., Appalaraju, S., Zhang, A., Zhang, W., Li, B., and Li, M. Mixgen: A new multi-modal data augmentation. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pp. 379- 389, 2023.
Bhagat, S., Stepputtis, S., Campbell, J., and Sycara, K. Sample-efficient learning of novel visual concepts. In Conference on Lifelong Learning Agents, pp. 637-657. PMLR, 2023.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning follow image editing instructions. In Pro- ceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 18392-18402, 2023.
Jin, X., Kim, J., Zhou, Y., Huang, K.-H., Wu, T.-L., Peng, N., and Ji, H. Armada: Attribute- based multimodal data augmentation. arXiv preprint arXiv:2408. 10086, 2024.
Chen, I.,, Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12799 2023.
Kumar, T., Brennan, R., Mileo, A., and Bendechache, M. Image data augmentation approaches: A comprehensive survey and future directions. IEEE Access, 2024.
Chun, S. and Park, S. Styleaugment: Learning texture de-biased representations by style augmentation without pre-defined textures. arXiv preprint wrXiv:218.100593 2021.
LAION.
Laion-gpt-4v, 2024.
URL https:
/ banggingface.com htt.:////aaassss/aaatas. gpt 4v dataset.
Cooper, A., Kato, K., Shih, C.-H., Yamane, H., Vinken. K., Takemoto, K., Sunagawa, T., Yeh, H.-W., Yamanaka, J., Mason, I., et al. Rethinking vlms and lims for image classification. arXiv preprint arXiv:2410.. 14690, 2024.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- technical report. arXiv preprint urXiv:2412. 19437, 2024a.
Dhillon, G. S., Chaudhari, P., Ravichandran, A., and Soatto, S. A baseline for few-shot image classification. arXiv preprint /909 wtt..0099.9.09....... 2019.
Liu, H. Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning, 2023a.
10

---

Contrastive Visual Data Augmentation
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tuning, 2023b.
Tian, Y., Wang, Y., Krishnan, D., Tenenbaum, B., and Isola, P. Rethinking few-shot image classification: good embedding is all you need? In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part XIV 16, pp. 266-282. Springer, 2020.
Liu, H., Li, C., Li, Y., and Lee, Y. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024b.
Uddin, A., Monira, M., Shin, W. Chung, T., Bae, S.-H., et al. Saliencymix: A saliency guided data augmenta- tion strategy for better regularization. arXiv preprint arXiv:2006.01799, 2020.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee. Y. J. Llava-next: Improved reason- ing, oer, and world knowledge, January 2024c. URL https://llavaa wwwprawww.gii.t.......... wwattuu..obbloco
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems,
36, 2024d.
Liu, Z., Tang, Z.. Shi, X., Zhang, A., Li, M., Shrivastava, A., and Wilson, A. G. Learning multimodal data augmen- tation in feature space, arXiv preprint arXiv:2212. 14453, 2022.
Vinyals, O., Blundell, C., Lillicrap. T., Wierstra, D., et al.
Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Wikidata. Wikidata: A free, collaborative, multilin- gual, secondary database, 2024. URL https:://wwtt /www. aakiidta.oo
Liu, Z., Liang, H.. Huang, X., Xiong, W., Yu, Q., Sun, L., Chen, C., He, C., Cui, B., and Zhang, W. Synthvim: High-efficiency and high-quality synthetic data for vi- sion language models. arXiv preprint arXiv:2407.20756 2024c.
Wu, D., Wan, Y., and Chang, K.-W. Visualized text-to- image retrieval. arXiv preprint arXiv:2505.2029: 2025.
Wu, Q., Zhao, M., He, Y., Huang, L., Ono, J., Wakaki, H., and Mitsufuji, Y. Towards reporting bias in visual-language datasets: bimodal augmentation by de- coupling object-atrribute association. arXiv preprint arXivv2310.01330, 2023a.
Nichol, A. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.0299) 2018.
OpenAL. Gpt-4v(ision) system card, 2023. URL https: org.oopuuS..//
//api.
263218031.
Wu, T.-L.., Zhou, Y., and Peng. N. Localizing active objects from egocentric vision with symbolic world knowledge. arXiv preprint arXiv:2310. 15066, 2023b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 3748-8763. PMLR, 2021.
Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., Xie, Z., Wu, Y., Hu, K., Wang, J., Sun, Y., Li, Y., Piao, Y.. Guan, K., Liu, A.. Xie, X., You, Y., Dong, K., Yu, X., Zhang, H., Zhao, L., Wang. Y., and Ruan, C. Deepseek-v12: Mixture-of- experts vasion-language models for advanced multimodal understanding, 2024. URL attps::/arxiiss /ttp//xxxivi..////// abs/ 2oo/2241.100
Recraft.Al. Recraft v3 model, 2024. URL https: / /ww. recraft.ai/
Sharifzadeh, S., Kaplanis, C., Pathak, S., Kumaran, D., Ilie, A., Mitrovic, J., Blundell, C., and Banino, A. Synth2 Boosting visual-language models with synthetic captions and image embeddings. arXiv preprint 24X 2024.
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Stability AL. 2024. URL. htttp::/staabilityyyt /nwws/ Antroddciing-staalll--dfdfuooooo
Introducing stable diffusion 3.5,
Xie, T., Cheng, X., Wang, X., Liu, M., Deng. J., Zhou, T., and Liu, M. Cut-thumbnail:: A novel data augmentation for convolutional neural network. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 1627-1635, 2021.
Sung, Y.LL, Cho, J., and Bansal, M. Vl-adapter: Parameter- efficient transfer learning for vision-and aibion-an-llnnguage tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5227-5237, 2022.

---

Contrastive Visual Data Augmentation
Yang, C.-F,, Yin, D., Hu, W., Peng, N., Zhou, B., and Chang, K.-W. Verbalized representation learning for interpretable few-shot generalization. arXiv preprint arXiv:2411.1865/ 2024.
Yang, K., Deng, J., An, X., Li, Feng, Z., Guo, J., Yang, J., and Liu, T. Alip: Adaptive language-image pre-training with synthetic caption. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2922- 2931, 2023.
Yang, S., Xiao, W., Zhang, M., Guo, S., Zhao, and Shen, F. Image data augmentation for deep learning: A survey. arXiv preprint #xXiv:204.086110 2022.
Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., and Zou, When and why rision-language models behave like bags-of words, and what to do about it? arXiv 2022.
preprint ar arXiv:2210.0/93
Zhang, Y., Unell, A., Wang, X., Ghosh, D., Su, Y., Schmidt, L., and Yeung-Levy, Why are goually-grounded lan- guage models bad image classification? arXiv preprint arXiv:2405. 18415, 2024.
Zheng. X., Chalasani, T., Ghosal, K., Lutz, S., and Smolic, A. Stada: Style transfer as data augmentation. arXiv preprint arv: 1909, 01056, 2019.
12

---

Contrastive Visual Data Augmentation
A. Appendix
A.1. Limitations and Future Work
Our work is not without limitations. First, in our experiments, we focus on the fine-tuning use case as it is the most general and intuitive way to utilize our augmented visual data. In the future, we plan to investigate other conceivable use cases for our augmented data, include model adaptation (Sung et al., 2022), test-time augmentation (Gidaris & Komodakis, 2018), visual information retrieval (Wu et al., 2025), and more. Third, the modularity of our method also invites other researchers to replace components of CoDA with superior models to achieve better performance. The NovelSpecies dataset, which can be updated with new species in future years, may also be used to evaluate future VLMs' novel concept recognition abilities. Finally, we also expect improved versions of T21 generation-based visual data augmentation techniques to eventually surpass CoDA in effectiveness and efficiency. Potential improvements may include more robust image feature filtering and more controllable text-conditioned image generation like multi-view synthesis. We hope our work can pave the way for future downstream advancements by demonstrating effective uses of our augmented visual data for enhancing model capabilities.
A.2. Data Selection Strategy
For each dataset, we focus on a randomly selected subset of concepts that the model is unable to recognize. The data selection strategy is as follows: In each iteration, we select a random subset of 15 species across different supercategories, including "Birds," "Mammals," and "Reptiles." This strategy allows us identify confusing pairs without overloading the system, progressively building collection of challenging cases from each subset. For each species within a subset, we create prompts in a multiple-choice format, incorporating the image and a randomized list of options from all species in the subset. Based on the response from the LMM, we are able to highlight specific species that are commonly mistaken for each other, guiding us in selecting pairs for further analysis. In particular, misclassification happens when an image of one species identified by the LLM to be an image of another species. A pair (A, B) is considered as a confusing pair if rate of misclassification on either direction is above the threshold 0.2. The process is repeated across new subsets, incrementally building ample dataset of concepts the model has difficulty recognizing.
A.3. Experiment Details
A.3.1. FEATURE EXTRACTION
For textual feature extraction, we use GPT-4o-mini with chain-of-thoughht reasoning, running with OpenAl API calls. Each API call processes up 2048 tokens, costing approximately 0.0025 per 1K input tokens and 0.005 per 1K output tokens. Given average of 500 tokens per query and 10 queries per concept, the estimated cost per concept is around $0.0375.
For visual feature extraction, we utilize GPT-4o-mini running with OpenAI API calls. Images are preprocessed to a resolution of 336x336 pixels and normalized before feature embedding extraction. Each image query incurs a cost similar to textual feature extraction. With an estimated 5 images processed per concept, the cost per concept amounts to approximately 0.1875.
With the rapid advancement of open-weights large language models and vision language models including DeepSeek V3 (Liu al., 2024a), DeepSeekVI.2 (Wu et al., 2024), Ilama 3.2 (Dubey al., 2024), and more; we expect that feature extraction LLMs and VLMs can be replaced with these models with none or minimal impact to performance. We plan to perform experiments on some of these models and provide comparison results in the next updated version of our work.
A.3.2. FEATURE FILTERING
We employ CLIP for automatic feature filtering, evaluating Discriminability and Generability scores. Discriminability is computed using cosine similarity between feature embeddings of target and misidentified concepts, with a threshold of 0.6. Generability is assessed by comparing feature presence in synthetic images using an ensemble of Stable Diffusion 3.5 Large and R3craftV3 models. The feature selection step is executed on an NVIDIA A100 GPU, processing features in approximately 2 hours. Top 5 ranked features are selected per concept.
3.3.3. IMAGE GENERATION AND VERIFICATION
For synthetic image generation, we employ Stable Diffusion 3.5 Large, running on a single A100 GPU. Additionally, we also integrate the R3 model through an API call. Image generation performed at a resolution 512x512 pixels
13

---

Contrastive Visual Data Augmentation
with guidance scale of 7.5. The pipeline generates 50 images per concept in approximately 1.2 seconds per image.
Post-generation, we perform automated verification using LLaVA V1.6-34b, running A6000 GPU. Each image would take approximately minutes to run for feature presence using a feature- matthing confidence threshold of 0.85. Images with satisfaction rate S (jyynthhetic ,F FM < 1.0 are discarded.
A.3.4. MODEL UPDATING
We train V1.6-34b with supervised fine-tuning (SFT) using LoRA with rank 128 and alpha 256, optimizing memory efficiency while maintaining model expressiveness. The training runs on two NVIDIA A6000 GPUs, leveraging DeepSpeed Zero-3 for distributed optimization and mixed precision (bf16) for efficiency. The vision encoder is CLIP-VYT-Large Patch14-336, with an MLP projector aligning visual and text features. We use cosine learning rate scheduler with 3 3% warmup ratio, training for 30 epochs with a batch size of 5 and learning rate of 2e-4. Images are padded for aspect ratio consistency, and gradient checkpointing is enabled to reduce memory usage. Checkpoints are saved every 50,000 steps, retaining only the most recent one.
A.3.5. EVALUATION
Automatic evaluation measures zero-shot classification accuracy on a held-out test set. Inference runs on single A6000 GPU with batch size of 20, taking approximately 1 hour to complete. The prompt templates for evaluation are attached to Appendix A.4
A.4. Prompt Construction
Prompt for Visual/Text Feature Extractions:
Contrastive Visual
GPT_Cont wat.oottraiiveeivvallaaaaaaaee
"You are an experienced and meticulously observant biological scientist who is asked to carefully assess the provided image. As labelled in the image, the left half of the image contains a picture of the animal (main_class) and the right half contains a picture of the animal (confusing_clasee is summarize the key distinctive visual attributes possessed by (main_class) (on the left of the image) that makes uniquely discernible from the (confusing_clas, (on the right half of the image). Reason step by step to produce the key visual attributes of a (main_class) (that make it distinet from a confusing_class)
Now, your task
answer. Finally, output
in a python list format containing short phrases of less than 8 words each. Do not output any features of the (con fcoiinnnngcl in your python list. Make sure not to name the (main_class) or the (confusing_class] in any of the attributes in your 1ist. Also, please try not to use negation the visual attributes you generate: for example, change features like "lack of facial markings' to 'plain brown face'. Additionally, do not use comparative form in any of the features you output, for example, change features like 'thinner body than the other class' to 'thin body"."
Visual
GPT GPT_Visual_Promp = "You are an experienced and meticulously observant biological scientist who is asked to carefully assess the provided image. The image contains picture of the animal (main_clasa). Now, your task is summarize the key distinctive visual attributes possessed by (main_class), Reason step by step to produce an answer. Finally, output the key visual attributes of a (main class] in a python list format containing short phrases of Less than 8 words each. Make sure not to name the main main_class) in any of the attributes in your list. Also, please try not to use negation the visual attributes you generate: for example, change features like lack of facial markings' to 'plain brown face'. Additionally, do not use comparative form in any of the features you output, for example, change features like 'thinner body than the other class' to rthin body'."
Contrastive Text wtt__ootrrattit___tt___etttt = "You are an exporienced and knowledgcable scene classification specialist who is tasked to summarize the key distinctive visual attributes possessed by :main_class) that makes uniquely discernible from the confusing_class) (just based on visual image) First retrieve your knowledge about the two different types of scenes, then reason step by step to produce an answer.
14

---

Contrastive Visual Data Augmentation
Finally, output the key visual attributes a (main_class) (distinct from a { confusing_olass) in a python 1ist format containing short phrases of less than 8 words each. Do not output any features of the {confusing_elass] in your python list. Make sure not to name the (main_class or the (confusing_class) in any of the attributes in your list. Also, please try not use negation the visual attributes you generate: for example, instead of saying 'no bright lights, use 'dark environment.' Additionally, do not use comparative forms in any of the features you provide. For instance, instead of saying 'smaller windows than the other place, use small windows.'
Text
GPT _er_Text_Prompt - "You are an experienced and knowl edgeable scene classi iistfiiation
specialist who is tasked to aummarize the key distinctive visual attributes possessed by classl. First retrieve your knowledge about the Imain class), then reason step by step to produce an answer. Finally, output the key visual attributes main main_class] class] in a python 1ist format containing short strings of less than 8 words each Make aure not to nane the (main_class) in any of the attributes in your list. Do not output any features of the (confusing_elass not to use negation
a
in your python list. Also, please try the visual attributes you generate: for example, instead of saying 'no bright lights,' use 'dark environment. Additionally, do not use comparative forms in any of the features you provide. For instance, instead of saying smaller windows than the other place,' use 'small windows.'
## Text to Image Generation Prompt Stable Diallleiittttttnnneeeee Ieag_to_Imaeeee_____eeeee_a Imagimmaaaeeeeeeeeeeeeaeeeeeeeee Geweretion mai main_class] that contains the fol lowing attributes: *+ ', i.join (attributes)
"Generate a 4K realistic image of
Feature Verification Prompt
lllallaaaa Verification Prompt "You are an image verification specialist. Your task is to llava
meticulously assess the image for specific attributes and confirm their presence. For each attribute in the List, carefully check the image, examine visual elements such color, shape, texture, position, and context clucs that might indicate whether the attribute is present. Provide a binary python output list, where each clement is either 1 (attribute is present) or 0 (attribute is absent), corresponding exactly to \nn AnExpectd
the order of attributes provided. Vnn AinlinAttributes to Output: A list of Os and is indicating the presence or absence of each attribute, in the same order as listed. Here an example output: [0, 11."
## Finetune and Evaluation Prompt
llava Finetune and Evaluation Prompt = "You are an image classification specialist with expertise in categorizimg images into specific groups. Given an image, identify its category from the following options: " ".join _pprorided_pption__appp
", or + htt.ddee_pttooo__capaaaaaaaeeeee category name for precise classification Please response with the category name only.
+ ". Provide your answer as only one
# Deduplication Prompt
GPT wtw_Deeuuplccatio__oooooe = "You are an experienced and knowledgeable blological scientist who is tasked to summarize the key distinctive visual attributes possessed by main_class) into coherent list. Given the following 1ist of attributes describing the animal species [main_class]: (attributes_list) duplicate features (which have the same or very similar meanings) into one, Then, you will order the remaining features in order of visual importance, the most visually significant / observable features will be at the front of the list while the least visually observable features will be at the back. Finally, output the key visual attributes of a [main class) in python list format containing short phrases of less than words each. Make sure not to name the (main_class) in any of the attributes in your 1ist. Also, please try not to use negation in the visual attributes you generate: for example, change features like "lack of facial markings' to 'plain brown face', Additionally, do not use comparative form in any of the features you output, for example, change features like 'thinner body than the other class' to 'thin body"."
You task is to combine the
# System Prompt GPT System Prompt "You helpful assistant."
15


Reviews:
[{"review_id": "OeAqodqcW6", "reviewer": "Reviewer_Xmi8", "summary": "The current submission addresses a known issue in LMMs (Large Multimodal Models) that of recognizing novel or even confusing visual concepts, due to their reliance on pre-trained knowledge and their limited ability to capture subtle visual details. To this en,d the authors introduce CoDA (Contrastive visual data augmentation) that extracts key contrastive both textual and visual features that differentiate target concepts from concepts they are commonly confused with. Afterward, text-to-image generative models are used to create synthetic training data that highlight these distinctive features, automatically filtering these images for quality assurance. The authors evaluate CoDA on three datasets: iNaturalist, SUN, and a newly introduced dataset called NovelSpecies, consisting of recently discovered animal species guaranteed to be unseen by the LMMs. CoDA is shown to generalize well across proprietary LMMs (GPT4o-mini) and traditional classifiers (ViT) and makes non-trivial improvements in accuracy over state-of-the-art augmentation methods in all the tested scenarios.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer pV1R for the insightful review. We are encouraged that you find:\n* **our claims well-supported by evidence**, \n* **our evaluation criteria and benchmarks appropriate**, \n* **our experiments sound and well-executed**. \n\nWe are also glad that you **enjoyed reading our submission**, while finding it **well-written, well-structured, with good discussion**. \n\nPlease find your suggestions carefully addressed below:\n\n \n---\n**Discussion on Failure Modes of Synthetic Image Generation**\n\nWe appreciate the reviewer for this constructive suggestion. **Given more space in the camera ready version, we will add the following section in the main paper to explain failure modes in synthetic image generation, along with image examples.**\n \nAs with any neural generative approach, our method is inherently constrained by the underlying T2I model’s capacity to represent the real world. While we expect this phenomenon to be gradually diminished with the advent of newer and more powerful T2I generative models, as of current publication we can still observe occasional generation results of nonsensical or biologically implausible artifacts (e.g., two-headed snakes) in rare cases. Additionally, while CoDA emphasizes class-discriminative features, it does not explicitly control for other attributes in the generated images. As a result, there can be unintended biases—for example, the backgrounds in generated images for some classes can be quite similar. While we did not observe any significant impact of such biases to our novel and confusing concept recognition tasks, users adapting our method to different use cases should be aware of potential unintentional impacts.\n\n---\n**Dedicated Future Works Section**\n \nWe fully agree with the reviewer that problems are never solved and would be useful to provide more insights on how this work could be improved upon or carried out further. **Given additional space in the camera ready version, we will add the following future works section to our main paper:**\n\nWhile a main contribution of our work is providing a state-of-the-art visual data augmentation strategy, we leave the downstream innovation on how best to use our augmented visual data to improve models for future work. In our experiments, we focus on the fine-tuning use case as it is the most general and intuitive way to utilize our augmented visual data. Besides this, other conceivable potential use cases for our augmented data include model adaptation, test-time augmentation, visual RAG or even pre-training. The modularity of our method also invites other researchers to replace components of CoDA with superior models to achieve better performance. The NovelSpecies dataset, which will continue to be updated with new species every year, may also be used to evaluate future VLMs' novel concept recognition abilities. Finally, we also expect improved versions of T2I generation-based visual data augmentation techniques to eventually surpass CoDA in effectiveness and efficiency, potential improvements may include more robust image / feature filtering and more controllable text-conditioned image generation like multi-view synthesis. \n\n---\n**Computational Overhead Comparison** \n\nWe thank the reviewer for bringing up this point: While we have thoroughly discussed the computational cost of CoDA in our appendix A.3.(L629-654), it would be very helpful to additionally compare this cost with that of existing visual data augmentation baselines such as ARMADA, so practitioners may reference it when considering adoption. **We plan to provide this information in the camera ready version via an additional figure.**\n\n\n---\n **Explanation of \"Confusable Concept\"**\n \nWe are glad that the reviewer brings up this point, which we would like to clarify: While CoDA chooses a confusable concept for each target concept, this process is very general and simply based on model misrecognition. \n\nFor example, when LMMs are tasked to classify a novel concept it has no previous knowledge of, the VLM will simply hallucinate and provide the closest confusable concept. CoDA then teaches the VLM the target concept by illustrating the visual differences between the target concept and its corresponding confusable concept. This effectively reduces the difficult and costly task of learning this novel concept from scratch to the much simpler task of learning the difference between the target concept and its confusable concept. \n\nThe effective learning cost reduction will depend on the level of similarity between the target concept and its closest confusable concept. However, we believe that as VLMs become more knowledgeable, they will also become better at finding higher quality similar confusable concepts, and thus dramatically reducing the cost of learning to recognize novel concepts.", "claims_and_evidence": "Claims are well-supported by evidence, such as:\n* CoDA outperforms existing visual data augmentation methods demonstrated through comparative evaluations on three datasets\n* CoDA is more effective in dealing with novel concepts, proved by the experiments conducted on a novel, dedicated and challenging benchmark for this task\n* broad applicability to different models such as ViTs and proprietary LMMs (GPT4o-mini)\n* solid ablation studies that prove the effectiveness of contrastive feature extraction and augmented image filtering (human evaluation is also used)", "methods_and_evaluation_criteria": "The evaluation criteria and benchmarks are appropriate. The authors evaluate on well-known benchmarks (iNaturalist, SUN) and a novel dataset (NovelSpecies) specifically designed for evaluating the recognition of novel concepts that are guaranteed to be outside the training data of any LMM (with a knowledge cutoff date).", "theoretical_claims": "No significant theoretical claims were made that required rigorous mathematical proofs. The paper is focused primarily on empirical and practical improvements to LMMs' visual recognition capabilities.", "experimental_designs_or_analyses": "The conducted experiments are sound and well-executed and include:\n1. CoDA comparison against multiple baselines on three datasets\n2. CoDA variations (textual-only, visual-only, both)\n3. Test on different model architectures\n4. Ablation studies to validate key components\n5. Human evaluations to verify feature and image quality", "relation_to_broader_scientific_literature": "The authors position their work well, addressing the existing literature on few-shot image recognition, visual data augmentation, and large multimodal models.", "essential_references_not_discussed": "The authors have cited a broad range of relevant literature, and I did not find any glaring omissions of essential works.", "other_strengths_and_weaknesses": "Overall, the strengths of this paper far outweigh the weaknesses.\n\nStrengths:\n* The paper addresses a practical and important problem for LMMs in a novel way\n* I fairly enjoyed reading the submission, well-written, well-structured, good discussion\n* The creation of NovelSpecies dataset is a valuable contribution to the field for benchmarking novel concept recognition\n* The method is model-agnostic and can work with any LMM or text-to-image generative model\n\nWeaknesses:\n* As I understood it, the method requires identifying a \"confusable concept\" for each target concept (one at a time), which could be challenging in some domains where the confusion patterns are not clear.\n* Limited discussion on potential failure modes or limitations of synthetic image generation.\n* Always valuable to have a dedicated section to future work or potential improvements, problems are never sold and would be useful to other scientists to have more valuable insights on how this work could be carried out further.", "other_comments_or_suggestions": "* A more detailed analysis of the computational overhead of CoDA compared to other methods would be useful for practitioners considering adoption.\n* Overall, I have very few critiques on writing or execution (I did not catch any obvious typos).", "questions_for_authors": "My questions related to CoDA's sensitivity to various factors, such as:\n1. How sensitive is CoDA to the quality of the identified \"confusable concept\", impact on performance?\n2. Performance changes to varying discriminability/generability thresholds?\n3. Increasing the number of synthetic images does not necessarily improve performance. Is there a way to predict in advance how many synthetic images would be optimal for a given concept?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hgaooyacEY", "reviewer": "Reviewer_pV1R", "summary": "The paper proposes a data augmentation technique for the tuning of large multimodal models on unseen concepts that are in a way 'very close' to known concepts included in the training. The method thus aims at expanding the knowledge of an existing model, including new concepts when they are encountered. \nThe augmentation strategy works on a contrastive learning basis, including text descriptions and generated images of the novel concepts. From this perspective, the paper has a novel approach to adapting (normally done via fine-tuning) models to new concepts that have not been seen during training. Next to the methodology, the authors present a new dataset of animal species, recently discovered, that can be used to effectively test novel concepts wrt the iNaturalistic dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer pV1R for their detailed review. We are delighted you recognize that:\n* our **method is novel and provides concrete contributions to the state of the art**, \n* our **claims are extensively demonstrated**,\n* our **datasets are solid choices to provide extensive validation**\n\nWe are also encouraged that you find our paper **well-placed in the current literature, addressing an important problem**. \n\nWe have carefully considered your valuable feedback and suggestions for improvement, which we address in detail below:\n\n---\n**Comparison to Fine-tuning Methods**\n\nWe very much appreciate the reviewer bringing up this important point and would like to take this opportunity to clarify: While a main contribution of our work is providing a state-of-the-art visual data augmentation strategy, we leave downstream users the freedom to decide how to use our augmented synthetic visual data, whether it is model adaptation, few-shot learning, fine-tuning, or even pre-training. **In our experiments, we focus extensively on the fine-tuning use case as it is the most general and intuitive way to utilize our augmented visual data and demonstrate its general usefulness for improving different models.**\n\nSpecifically, we compare **fine-tuning existing models with all real data, mixed real and synthetic data from baseline data augmentation strategies, and mixed real and synthetic data from our CoDA method**. Results in Table 2 and Table 3 show that basic fine-tuning with all real data or including synthetic data from baseline data augmentation strategies produces much lower performance compared to fine-tuning with mixed real and synthetic data from our CoDA method. (L330-420)\n\nFor better reference, we paste the following results subset from Table 2 of our main paper to illustrate the **performance disparity of basic fine-tuning with all real data vs including one synthetic image per concept from baseline data augmentation strategy ARMADA and our CoDA method**:\n\n|Dataset| Basic Fine-tuning Acc(%) | ARMADA Acc(%) | CoDA Acc(%) |\n|---|---|---|---|\n| NovelSpecies  | 61.2 | 60.7| 70.1|\n| SUN           | 73.4 | 75.9| 83.4|\n| INaturalist   | 49.2 | 60.1| 63.5|\n\n---\n**Relation to Test-time Augmentation Methods**\n\nWe thank the reviewer for pointing out the relations between our work and existing test-time augmentation techniques, a topic we extensively considered and explored during our project. As elaborated in our response to Issue #1 above, we position our work as providing a state-of-the-art visual data augmentation strategy, while allowing downstream users to decide whether to use our augmented data for pre-training, fine-tuning, or test-time augmentation.\n\n**To validate the usefulness of our augmented data, we currently focus on the fine-tuning scenario because:** \n- General applicability, as fine-tuning is generally applicable to LMMs as well as traditional classifiers.\n- Empirical effectiveness, as the difficulty of the visual novel/confusing concept recognition task limits the effectiveness of test-time augmentation methods.\n\nIn fact, during our exploration stage, we ran multiple test-time few-shot generalization experiments on VLMs. Here we attach our **earlier results for 1-shot and 3-shot generalization of LLaVA 1.6 34b, under the extremely simplified binary classification setting**:\n\n|Dataset|1-shot Acc(%)|3-shot Acc(%)|Random Acc(%)\n|-|-|-|-\n|NovelSpecies|48.6|50.7|50.0\n|INaturalist|49.3|47.9|50.0\n \nThe model achieves no better than random chance performance on the three datasets with few-shot learning. Further qualitative experiments on proprietary VLMs also showed little promise, which prompted us to move towards fine-tuning methods for better performance.\n\nHowever, we recognize that the reviewer's comment regarding test-time augmentation methods goes beyond basic few-shot learning. We have identified several **areas within multimodal test-time augmentation methods that may be promising given our CoDA method or relevant to the novel/confusing concept recognition task:**\n\n- Multimodal meta learning and fast adaptation methods such as Flamingo\n- Code-based multimodal reasoning methods including VisProg and ViperGPT\n- Multimodal reflection and self-critique methods including LLaVA-Critic, Critic-V, and VDebugger\n- RL-based inference time scaling methods such as Visual-RFT and VAGEN\n- Multimodal RAG methods including Wiki-LLaVA, UDKAG and UniRAG\n\nWe will include discussions relating our method to these test-time augmentation strategies in the camera ready version.\n\n---\n**Highlighting the Effectiveness/Efficiency Gains**\n\nWe thank the reviewer for recognizing the substantial effectiveness / efficiency gains of CoDA demonstrated in the experimental section. To highlight these gains, **we will include an additional bar chart comparing performances of CoDA against other baseline methods.** We believe this visual comparison will help readers more easily grasp the effectiveness / efficiency gains of CoDA.", "claims_and_evidence": "The authors claim that the proposed NovelSpecies dataset can be used to evaluate the ability to recognize novel concepts, which seems extensively demonstrated in the experimental section.\nFurthermore, a methodological claim that the method proposed can adapt existing models in an effficient and effective way, thus overcoming the ineffectiveness of methods based on fine-tuning due to the scarcity of data usable for model adaptation. However, comparisons to finetuning-based methods are not evident in the paper. This prevents from appreciating the efficiency and effectiveness contributions claimed.", "methods_and_evaluation_criteria": "The method is described in a detailed way. It makes sense to use text-guided generative models to expand the data available for augmentation at test-time. The filtering of features also seems to stabilize the adaptation phase, avoiding that the model learns degenerate representations. Overall the method seems fine and is evaluated on several datasets using reasonable metrics. In my opinion, it is novel and provides concrete contributions to the state of the art.", "theoretical_claims": "I did not find theoretical claims or proofs to validate. The paper makes hypotheses and verifies them with a mostly empirical approach.", "experimental_designs_or_analyses": "The chosen datasets and the proposed one are solid choices to provide an extensive validation. \nThe choice of the comparison methods could be extended with fine-tuning based approaches, as in the introduction they are claimed to be ineffcient and ineffective, two shortcomings that the proposed method addresses directly.\nThe ablation with different backbones provides extra evidence that the method can be plugged in different architectures.", "relation_to_broader_scientific_literature": "The paper seems well-placed in the current literature and addresses an important problem related to large multimodal models and their deployment in real scenarios. The paper relates, in my opinion, to the literature on test-time augmentation that is used to adapt models at test time. A coverage of this relation should be present in the paper, or the authors should motivate why not.", "essential_references_not_discussed": "I do not think there are particular missing references. I think that discussing the relation of this paper with test-time augmentation methods (and cite for instance seminal papers on the topic) would make the placement within the literature more solid.", "other_strengths_and_weaknesses": "The paper is clear, with potentially impactful contributions in the field.\nAs a weakness, I would say that the efficiency/effectiveness benefits should be better highlighted, as this is key to provide impact.", "other_comments_or_suggestions": "When use listings (e.g. 1)..., 2)...) there should be no dot '.' after the parenthesis ')' and the item should not start with a capital letter - that's really confusing while reading.\n\nto be explicit \n\"straightforwardly 1). Fine-tune text decoder on new textual\"\nshould be\n\"straightforwardly 1) fine-tune text decoder on new textual\"", "questions_for_authors": "- How does the proposed approach relate to test-time augmentation methods?\n- what is the effectiveness/efficiency gain wrt fine-tuning adaptation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Responsible Research Practice (e.g., IRB, documentation, research ethics, participant consent)"]}}, {"review_id": "0PRNVgFpgv", "reviewer": "Reviewer_bmru", "summary": "In this paper, authors proposes Contrastive Visual Data Augmentation (CoDA), a novel approach to improve LMMs' ability to recognize novel and easily confused visual concepts. CoDA extracts contrastive features between target concepts and the confusable counterparts, generating synthetic training data with test-to-image models to update LMM. This paper also contributes a novel dataset, NovelSpecies. The solid experimental results demonstrate the effectiveness of CoDA.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer bmru for their insightful and encouraging comments. We are pleased that you found:\n* our paper **well-written to follow**, \n* our proposed **data augmentation strategy effective**, \n* our **method's performances competitive**, \n* and our new **NovelSpecies dataset interesting and novel**. \n\nWe are also encouraged that you recognize that our **solid experimental results demonstrate the effectiveness of CoDA**.\n\nPlease find your suggesion and additional feedback carefully addressed below:\n\n---\n**Spacing Between Figure and Table Notation**\n\nThank you very much for catching this formatting issue, your detailed efforts are greatly appreciated. We fully agree that correcting this notation throughout our work would improve the readibility and clarity of our work. Therefore, we will make sure to correct all such usage instances in our camera ready version.\n\n---\n**More Visualizations like Figure 3 in Paper**\n\nWe fully agree with the reviewer that visualizations like Figure 3 can help illustrate our method and provide straightforward comparisons with other related methods. Moreover, we currently plan to include an additional figure in our camera ready version to provide example of success and (rare) failure cases in our visual data generation process. We believe this will help readers better understand the abilities and potential limitations of our work and facilitate better method adaptation to more domains.\n\n---\n**Whether Online Large Models can Recognize Novel Concepts**\n\nThis is a very interesting question raised by the reviewer. We believe that indeed online models with access to up-to-date internet information may be better suited for recognizing novel concepts compared to static models. However, when qualitatively testing current online VLMs such as GPT4o and Claude 3.7 Sonnet, we find that they still fail to recognize most of the concepts in our NovelSpecies Dataset. We believe this is likely due to their reliance on textual retrieval and inability to retrieve the relevant information based on novel images.", "claims_and_evidence": "Yes, they are.", "methods_and_evaluation_criteria": "Yes, the proposed method makes sense for the problem.", "theoretical_claims": "There are not theoretical claims in this paper.", "experimental_designs_or_analyses": "Yes, I did.", "relation_to_broader_scientific_literature": "1. This paper studies the data augmentation strategy, which is extensively applied for model training. \n2. A novel benchmark for current LMMs.", "essential_references_not_discussed": "Authors have cited and discussed related works.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-written to follow. \n2. The proposed data augmentation strategy is effective for novel concepts.\n3. The NovelSpecies dataset is interesting.\n4. Competitive performances. \n---\nFrom my perspective, there are no obvious weaknesses of this paper.", "other_comments_or_suggestions": "There are some typos in the paper. \n1. For cross reference in the paper, Fig.1, and Tab.1 should be shown as Fig. 1, and Tab. 1 or Figure 1 and Table 1, where there is a space between \".\" and the number.\n2. More visualizations like Figure 3 can be presented in the Appendix. \n3. I'm wondering whether up-to-date online large models (such as GPT-o3 mini) could recognize the novel concepts. (There is no requirement for this point during the rebuttal.)", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YdJVv64CJk", "reviewer": "Reviewer_DV9x", "summary": "This paper proposes a novel data augmentation technique aimed at improving the recognition capabilities of Large Multimodal Models (LMMs) on rare classes/concepts that are underrepresented in the training set. In particular, authors leverage text-to-image diffusion models to synthesize images of the rare concept, where the generation prompt is crafted via a contrastive technique focused on highlighting the differences between the rare concept and its similar, but common counterpart. Experiments demonstrate that if only a few (5-10) examples of the rare concept are available for fine-tuning, the proposed technique can significantly boost recognition performance over some simple baselines.\n\n## update after rebuttal\nI thank the authors for clarifying my questions and concerns. I raised my score accordingly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are pleased that reviewer DV9x finds:\n* our **idea original and sensible**\n* our **technique novel**\n* our **references sufficient**\n* and the **modularity of our approach beneficial**\n\nWe value your constructive comments and address them in detail:\n\n---\n**Related Data Augmentation Methods**\n\nWe want to highlight a key distinction between our **generation-based** and existing **editing-based** augmentation (ARMADA and all methods in [1]): While our method generates T2I augmented images from scratch, editing-based methods create augmented images by perturbing attributes of real images of other concepts (L94-105), leading to two drawbacks:\n- they require large amounts real images, which are rare for novel concepts. \n- augmented images often lack background and view variation. \n\nExperiments in Table 2 show that fine-tuning with such data yield very limited gains, on par with traditional editing-based augmentation methods like Flipping.\n\nFurthermore, while we aim to thoroughly compare with **published** related works such as ARMADA, at the time of submission, both TTIDA and [1] were **not published** in any known conference or journal. Nevertheless, we are happy to cite and thoroughly discuss both works in our camera ready version.\n\n---\n**Can VLMs Describe Features of Truly Novel Concepts**\n\nWhile editing-based methods can only create augmented images by perturbing known attributes present in existing concepts, our generation-based approach can design any novel feature to describe novel concepts. Therefore, **this limitation only exists for editing-based methods**, while CoDA is able to create any novel feature that can be described with human language. \n\nFurthermore, Chiquier et al. [2] showed modern VLMs can generate high-quality novel descriptive features for rare species and even otherworldly concepts such as those in the KikiBouba Dataset. \n\n---\n**Clarification on Compute Efficiency**\n\nWe'd like to clarify: By \"compute efficiency\", we refer to **efficiency during fine-tuning** with augmented / real images. As described in L631-654, CoDA is inference-only and does not involve any expensive back-propagation. Thus, CoDA's compute cost is an order of magnitude less than the fine-tuning process. \n\nOur efficiency claim refers to: when comparing models fine-tuned with CoDA augmented images vs models fine-tuned with all real images or ARMADA images, models fine-tuned with CoDA images can achieve higher performance with lower fine-tuning cost. Table 2 results justify this: training on 6 images per concept of mixed Real+CoDA data outperforms training with 20 real images per concept (using over 2x the compute).\n\nWe acknowledge the word 'compute efficiency' was ambiguous in the paper and will clarify it in the camera-ready version.\n\n---\n**Can T2I Models Generate Multiple Attributes**\n\nThis is an issue we encountered many times. Consistent with Conceptmix, we found older T2I models like SD-2.1 tend to fail at generating >3 attributes in a single image. However, newer T2I models not included in the benchmark, like SD-3.5, can reliably generate >5 concepts. \n\nFurthermore, our two-layer feature filtering (L171-209) and image filtering (L236-251) strategy is specifically designed to filter out any remaining generation failures. Our human evaluation in (Table 1) demonstrates that the vast majority of features (83.97%) were successfully generated:\n\n|Image Type|Feature Presense(%)|IAA(κ)\n|-|-|-\n|Real|92.51|0.87\n|Synthetic|83.97|0.82\n\n---\n**CLIP Discriminability/Generability scores**\n\nOur intentional math formulation of Discriminability/Generability scores ensure that the proposed association bias of CLIP would actually **aid in filtering out non-discriminative features**, while having no impact on the relative ranking of generability scores. Due to word limits, please respond for a detailed breakdown.\n\n---\n**Dataset Size and Generality of Results**\n\nWe want to clarify: while the NovelSpecies dataset has **64 different species**, it actually contains **2240 annotated images**. Results on NovelSpecies is reported on val and test sets each containing **960 unique images** (L275-277). We tested **three models** on this dataset to show CoDA's generality (Table 3).\n\nTo further prove the statistical significance of our NovelSpecies results, we ran 5 independent sets of experiments on 10 different settings of NovelSpecies with different random seeds:\n\nResults show remarkable consistency: **average standard deviation of scores is 0.0133** across 10 settings, with a max of 0.017. The **average coefficient of variation is 1.82%** across all settings, **far below the general 5% threshold** for statistical stability.\n\n|Statistic|Min|Max|Avg\n|-|-|-|-\n|Standard Deviation|0.0069|0.0170|0.0133\n|Coefficient of Variation|0.0087|0.0249|0.0182\n\n(Respond to see  full score table)\n\n[1] Advances in diffusion models for image data augmentation. AI Review (2025)\n\n[2] Evolving Interpretable Visual Classifiers with Large Language Models. ECCV (2024)", "claims_and_evidence": "- The claim in the contributions list \"CoDA is also the first widely successful method using text-to-image generation for visual data augmentation\" in my opinion is too strong. T2I diffusion models have been used for data augmentation before (see ARMADA [1], a baseline from the paper, TTIDA [2] and a survey paper [3]) and \"widely successful\" is subjective.\n\n- Authors claim that CoDA \"significantly improve data and compute efficiency compared to existing methods\", and while I can get behind the data efficiency aspect (with the same number of real samples, CoDA achieves higher accuracy), compute efficiency is not demonstrated in any way in the paper. In fact, it appears that the technique requires significant effort, involving several steps of feature extraction using LLMs and frontier diffusion model-based generation. Comparative study on the cost of the technique is missing.\n\n[1] Jin, Xiaomeng, et al. \"ARMADA: Attribute-Based Multimodal Data Augmentation.\" Proceedings of the First Workshop on Advancing Natural Language Processing for Wikipedia. 2024.\n\n[2] Yin, Yuwei, et al. \"Ttida: Controllable generative data augmentation via text-to-text and text-to-image models.\" arXiv preprint arXiv:2304.08821 (2023).\n\n[3] Alimisis, Panagiotis, et al. \"Advances in diffusion models for image data augmentation: A review of methods, models, evaluation metrics and future research directions.\" Artificial Intelligence Review 58.4 (2025): 1-55.", "methods_and_evaluation_criteria": "The proposed approach is sensible. Explicitly highlighting the differences between known concepts and a novel concept can be an efficient way to learn the characteristic attributes of the novel concept in a data-efficient way. \n\nOne caveat I have with the methodology is that it is bottlenecked by the VLMs/LLMs used for extracting visual/textual features. If the concept is truly novel, it is possible that the defining attributes cannot be extracted this way (as the feature extractors haven't seen the concept during training either). In other words, the proposed technique only works in cases where the novel concept is defined by a unique/novel combination of already known attributes. However, for recognizing truly novel concepts, external knowledge may be required.\n\nAnother concern I have is the current limitations of text-to-image models in correctly generating multiple attributes based on the input prompt, which is crucial for the technique. In particular, [1] shows that even SOTA proprietary T2I models fail to generate as low as 3 distinct attributes correctly in 50% of the cases, and the success probability approaches 0 for 7 attributes. Thus, I believe the granularity of how much contrastive information can be represented in the synthetic images is seriously limited by the current performance of T2I models.\n\nFinally, I am unsure about using CLIP for calculating Discriminability/Generability scores. In particular, CLIP may have the exact same bias as the LMM towards the well-represented concept and thus  may more frequently associate attributes with known concepts simply due to training set bias. \n\n[1] Wu, Xindi, et al. \"Conceptmix: A compositional image generation benchmark with controllable difficulty.\" arXiv preprint arXiv:2408.14339 (2024).", "theoretical_claims": "No proofs/theoretical claims.", "experimental_designs_or_analyses": "I have some concerns with respect to the experimental evaluation. \n\nFirst, the primary dataset, NovelSpecies, is very small. The results are reported over 64 datapoints, which casts some doubt on the statistical significance of the reported results. \n\nSecond, the improvements are very inconsistent: in some experiments textual features alone are the best, in other cases visual, and in some cases the combination of both. In some cases the proposed contrastive feature extraction helps, in other cases it doesn't. I believe there has to be more thorough study on the role of the different components of the pipeline, and some guidelines how to select which arrangement to use. The large variance in results may also be a result of the small size of the dataset.\n\nThird, it would be necessary to add some naive baselines to gauge the effectiveness of the technique. What happens if we use few-shot prompting with or without augmented samples? What happens if we use few-shot prompting highlighting the differences in text format? \n\nLastly, experiments on larger scale datasets are only performed using a single model (LLaVA-1.6), casting some doubt on the generality of the results.", "relation_to_broader_scientific_literature": "The paper is closely related to and advances the work proposed in [1] by performing the editing in a more targeted way: the edit highlights the features that are different in the novel class from a common class the model confuses it with.\n\n[1] Jin, Xiaomeng, et al. \"ARMADA: Attribute-Based Multimodal Data Augmentation.\" Proceedings of the First Workshop on Advancing Natural Language Processing for Wikipedia. 2024.", "essential_references_not_discussed": "I believe the references are sufficient to understand the context and contributions.", "other_strengths_and_weaknesses": "I believe that the core idea of the paper is original and sensible. The significance of the findings is somewhat diminished due to (1) limited dataset size, limited number of models, (2) lack of discussion on cost/compute and (3) lack of ablation studies on the different components of the method. The modularity of the approach and the potential to improve with the emergence of better models is a nice benefit. Moreover, adapting LMMs to novel concepts efficiently is an important direction giving strong motivation to the paper. I find the writing to be more or less clear.", "other_comments_or_suggestions": "I found the term \"hyper-domain-specific\" verbose and vague at the same time. How is it defined and how is it different from domain-specific?", "questions_for_authors": "See my points under Other Strengths and Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers agree this paper addresses a practical and timely challenge in LMMs through a novel contrastive data augmentation framework, CoDA. Reviewers agree that the method is well-motivated, well-executed, and broadly applicable. The authors provide strong empirical evidence across diverse datasets and models, including a valuable new benchmark (NovelSpecies). Rebuttals are detailed and constructive, with thoughtful clarifications and additional experiments addressing reviewers’ concerns, including naive baselines and TTA. While some limitations in scope and compute claims exist, they are revised in the response. Overall, the work is a solid extension of data-efficient adaptation for multimodal models.
Decision: Accept (poster)