What Makes ImageNet Look Unlike LAION
Authors: Ali Shirali
Moritz Hardt
Venue: Accepted by TMLR ()

Abstract: ImageNet was famously created by querying several image search engines such as Flickr. What if we recreated ImageNet instead by searching the massive LAION dataset based on image captions alone? In this work, we carry out this counterfactual investigation. We find that the resulting ImageNet recreation, which we call LAIONet, looks distinctly unlike the original. Specifically, the intra-class similarity of images in the original ImageNet is dramatically higher than it is for LAIONet. Consequently, models trained on ImageNet perform significantly worse on LAIONet. We propose a rigorous explanation for the discrepancy in terms of a subtle, yet important, difference in two plausible causal data-generating processes for the respective datasets, that we support with systematic experimentation. In a nutshell, searching based on an image caption alone creates an information bottleneck that mitigates the selection bias otherwise present in image-based filtering. Our explanation formalizes a long-held intuition in the community that ImageNet images are stereotypical, unnatural, and overly simple representations of the class category. At the same time, it provides a simple and actionable takeaway for future dataset creation efforts.

Published in Transactions on Machine Learning Research (02/2025)
What Makes ImageNet Look Unlike LAION
Ali Shirali
shirali ali@berkeeey.....ooo edu
University of California, Berkeley
Moritz Hardt
Max Planck Institute for Intelligent Systems, Tübingen and Tübingen Al Center
Reviewed on OpenReview: https: // openreview net/ forum? id= IrBYuh9W37
Abstract
ImageNet was famously created by querying several image search engines such Flickr. What if we recreated ImageNet instead by scarching the massive LAION dataset based on image captions alone? In this work, we carry out this counterfactual investigation. We find that the resulting ImageNet recreation, which we call LAIÖNet, looks distinctly unlike the original. Specifically, the intra-class similarity of images the original ImageNet is dramat- ically higher than it is for LAIONet. Consequently, models trained on ImageNet perform significantly worse on LAIONet. We propose rigorous explanation for the discrepancy in terms of subtle, yet important, difference in two plausible causal data-generating processes for the respective datasets, that we support with systematic experimentation. In mutshell, searching based on an image caption alone creates an information bottleneck that mitigates the selection bias otherwise present in image-based filtering. Our explanation formalizes a long-held intuition in the community that ImageNett images are stereolypical, unnatural, and overly simple representations of the class category. At the same time, it provides simple and actionable takcaway for future dataset creation efforts.
1 Introduction
For nearly a decade, ImageNet (Deng ct al. 2009 was the focal benchmark for much of computer vision and deep learning. Created from image web search results and human filtering, ImageNet contributed curated images suitable for supervised learning at the time. In recent years, however, the community has seen a new generation of models trained on massive amounts of noisy image-text data gathered from the web with minimal curation. Available to the academic public is the massive scale LAION dataset, in two versions, featuring 400 million (Schuhmann et. al. 2021) and 5 billion (Schuhmann et al. 2022 crawled image-text pairs, filtered by the OpenAl CLIP model (Radford et al. 2021 for sufficient image-text relevance rather than human annotators.
At the outset, LAION works much like text-based web image search. We can specify a query and retrieve images with high similarity between the query and the text surrounding the image on the website from which was crawled. We can therefore search LAION for each of the 1000 categories in the ImageNet ILSVRC-2012 dataset] and retrieve images corresponding the first step of creating ImageNet from Flickr search results, except that LATON replaces Flickr, but either way, both are based on web crawls. Where the creators of ImageNet hired human annotators to filter images, we analyze image captions to ensure that the resulting images have high fidelity to the class category.
each of the classes. This process is much like
We might expect that for a suitably chosen textual similarity threshold, the resulting datasct would bear resemblance to the original ImageNet. However, we demonstrate that this is anything but the case. The dataset, so created from LAION, very much looks unlike ImageNet. And we explain why, supported by
Unless otherwise stated, by ImageNet mean the ImageNet I.SVRC-2002 dataset.

---

Published in Transactions on Machine Learning Research (02/2025)
independent evidence from other well-curated datasets. This explanation, although subtle, reveals a funda- mental fact about the difference between ImageNet and LATON that has consequences for understanding dataset creation at large.
1.1 Our Contributions
We introduce a new research artifact, called LAIONet, that aims at a recreation of ImageNet on the basis of LAION. We start. from LATOON-00M,, collection of 400M image-text pairs extracted from web pages in Common Crawl (comoncraww. org) between 2014 and 2021. The relevance of images and their corresponding texts was quality-controlled with OpenAl CLIP model, excluding instances with a cosine similarity of image and text embeddings less than 0.3.
Creation of LAIONet. We create LAIONet solely on the basis of text-based selection. We require the exact "lemmas" (terms) in so-called "synset" of an ImageNet category to appear in the text corresponding to an image. Morcover, we require a high similarity between the text and the synset name and definition. We use the cosine similarity of CLIP text embeddings to calculate this similarity, however, we make consistent observations using MPNet (Song et al., (2200) as the text encoder. LAIONet selection criteria are conservative in that they tend toward images that are easy classify; least from the CLIP point of view, there no evidence that LAIONet images are harder to classify than ImageNet.
Contrasting LAIONet and ImageNet. To begin understand the differences between LAIONet and ImageNet, we evaluate slew of Imagenet models on LAIONet. As show, the accuracy of models trained ImageNet drops by 5 to 12 percentage points when evaluated on LATONet (Figure 1. In calculating accuracy, we weight classes uniformly as is done in ImageNet. When classes are weighted based on the frequency of each class in LAIONet, accuracy drops by another 5 to 10 percentage points.
Top 1 accuracy
Top 5 accuracy
0.85
0.95
Imagetiet Set)
0.80
(valld (validation set)
ImageNet
0.85
0.70
1k models models ptz2k models
0.80
Ik models
0.65
pk2 models pt2zk models
0.75
0.60
0.70
0.6
0.7
0.8
0.7
0.8
0.9
LAIONet
LAIONet
(equi-weighted classes)
(equi-weighted classes)
Figure 1: Accuracy of TmaggNNet-trained models when evaluated on ImageNet. validation set versus eAT Three types of models are distinguished based on whether they are pre-trained on ImageNet-222k and whether they are fino-tuned on ImagoNet-Ik. Accuracy is defined as the average of the recalls calculated for cach class that present in LAIONet.
Drops in accuracy, such as these, are well-documented phenomenon in machine learning at this point. In this work, we go a step further by providing a substantive explanation for the difference between LAIONet and ImageNet.
Diagnosing the Difference. In the first step, we observe that the intra-class similarity, mcasured as the pairwise similarity of the images within a class, is lower for LAIONet than for ImageNet. In other words, LAIONet images are more diverse in each class. The recall of the models is also lower in the classes with lower intra-class similarity. Honce, lower intra-class similarity gives concrete reason for why the accuracy of ImageNet models drops on LAIONet. But why does LAIONet have lower intra-class similarity in the first place?
2

---

Published in Transactions on Machine Learning Research (02/2025)
We answer this question in terms of two plausible causal graphs for the respective data-generating pro- cesses (Figure 2 Both graphs are based on the standard anti-causal representation of classification prob-
lems (Schölkopf et al. 2012], whereby for each category Y there is a mechanism to generate data (here, image and text 7) given Y. But, the graphs differ in one important aspect.
Y
T
X
T
X
S
S
(a) LAIONet data collection
(b) ImageNet data collection
Figure 2:
The suggested underlying mechanism of data generation and selection in LAIONet and ImageNet.
Class Y, text description T, image X, selection S or S" The dashed line between X and T means that such links can exist in the graph
In the case of LATONet (Figure 2al, selection is based on text alone. The causal graph has the important property that the distribution of the image is independent of the selection decision conditional on the text. In other words the text serves an information bottleneck between the selection mechanism and the image. Choosing an image reveals nothing more about the image than what can be learned from its textual representation. This powerful conditional independence property limits how much selection can bias the distribution of the image. In contrast, in the case of ImageNet (Figure 2b there is a link from the image to the selection decision. For example, this link exists when human annotators see the full image and decide to select or discard an image. The existence of this link is what can strongly bias the distribution of the image conditional on selection. It is this selection bias that is visible the higher intra-class similarity.
Our case hinges on the existence and strength of the image-to-selection link in the causal graph for Image We then go beyond LAtONet and provide three complementary arguments as evidence:
We can weaken the image-to-selle soseleconnn link by considering ImageNet instances of different selection frequencics. The selection frequency describes the rate which Amazon MTurk workers selected candidate image into the dalasel within a larget class. This allows us to modulate the strength of the image-to-selection link. Looking at three versions of TmageNet (Recht et al. 2019), we find that for a lower selection frequency, the resulting images come closer to LAIONet.
We show that text alone cannot explain why an image selected into ImageNet. The ImageNet- Captions dataset Fang et al., 2022 has restored the captions for one-third of the original ImageNet images. If the text was the only factor in determining the relevance to a synset, it should explain are selected. Looking at the similarity between texts and
why the images their synsets, a majority of text-synset pairs exhibit high similarity, but the distribution has a heavy tail and there are instances with low similarity. For pairs with low similarity, there are often many other synsets more similar to the text. This makes these instances unlikely to have been selected solely based on their text.
We search LAION for the texts most similar to the texts from the ImageNet-CCationn dataset. The resulting images show significantly higher variability (in other words, lower intra-class similarity) Ihan ImageNet. This suggests that another mechanism must have been at play.
In conclusion, we argue that the TmageNet. It is this mechanism that makes ImageNet look unlike LAION. This insight has direct prescriptive
mechanism was significantly play in the creation of
3

---

Published in Transactions on Machine Learning Research (02/2025)
value for dataset creation efforts in general. When creating dataset and diversity is desired, we should select candidates on the basis of an information bottleneck. A succinct text caption, for example, generally carries much less information than the entire image. Selecting on the basis of the text caption, therefore, retains much of the entropy present in the image distribution.
All code available at: httppe//ggtaabb.ccm/aaaaaaaaarar
1.2 Related Work
Torralba & Efros (2011) introduced cross-dataset evaluation in computer vision and specifically illustrated the uniqueness that each common dataset at the time, including ImageNet, possesses. Recreating an ImageNet test set, called ImageNetV2, although with different motivation, was the subject of the seminal paper by RRecht, Roelofs, Schmidt, and Shankar (2019). Engstrom et al. (2020) argue that there is subtlety in thresholding empirical estimates of the true underlying selection frequency of an image in ImageNetV2. Our argument, however, does not rely on any specific threshold of the selection frequency. We only need to observe what happens as we vary it from small to large. In contrast to ImageNet our goal is not to recreate ImageNet as closely as possible. Rather it is the differences between ImageNet and LAION that are the focus of our investigation.
Many other works have modified ImageNet for variety of reasons. Geirhos et al. (2019) created stylized version of ImageNet to reduce the reliance of the trained model on texture. Xiao et al. (2021) disentangled the foreground and background of ImageNet images to show the tendency of the models to rely on the background. Li et al. (2023b) proposed ImageNet-W test set by inserting a transparent watermark into the images of ImageNet validation set, revealing the reliance of the models on watermarks. ImageNet undergoes ongoing augmentation over lime. For example, the ImageNet-Captions (Fang el al. (0022) project has restored the captions of about one-third of original ImageNet images from Flickr. ImageNet-X (Idrissi et al.) 2023) provides set of human annotations pinpointing 16 failure types for ImageNet such posc; background, or lighting. The peculiarities of ImageNet have been the subject of multiple studies. For example, Huh et al. 2016 found the large size and many classes, including very similar classes, do not affect the successful transfer performance of -atained-t features.
On the side of LAION, researchers are keenly interested in understanding the strong zero-shot accuracy of contrastive language image models using L.AION (Vogel al. 2022). Fang et al. (2022) found none of the large training set size, language supervision, and contrastive loss function determines this robustness and more diverse training distribution should be the main cause. Our work demystifies this distributional
advantage by contrasting ImageNet and LAION. Nguyen et al. (2022) compared various large image-text datasets differing in the crcation process and found the robustness induced by each varies widely in different aspects, suggesting further studies of the role of dataset design. Our work highlights an important mechanism at play in dataset design that can move the dataset further away from a natural distribution.
2 LAIONet: An ImageNet Out of LAION
Our starting point is to create an ImageNet-like dataset from LAION. This dataset is a research artifact intended to highlight the differences between LAION and ImageNet. Our goal is not to provide a new benchmark or a new training set. However, LATONet might be of interest to obtain diverse samples, or variants of LAIONet may be created to improve our understanding of benchmarks.
To start, recall that every ImageNet class corresponds to WordNet Miiller 1998) synset which consists of so-called lemmas. Synsets also come with short definition known gloss. We label LAION instance with a WordNet synset if 1) at least one lemma from the synset exists in the text of the instance, and 2) this text is sufficiently similar to the name and definition of the synset. Out of LAION 400M samples, 21M of them passed the first condition. The second condition ensures the lemma as found in the LAIÓN sample has the intended meaning. To quantify the similarity of the LAION text and a synset, we first crcate a textual representation for the synset by concatenating its name and definition (to be called the synset text). We then calculate the embedding vectors for both the synset text and LAION text using CLII and compute
4

---

Published in Transactions on Machine Learning Research (02/2025)
their cosine similarity. Alternatively, one may use any sufficiently powerful text encoder for this purpose. For instance, we repeat this process using MPNet. (Song et al. 2020) in Appendix A
Figure 3a illustrates the distribution of LAION text to synset text similarities. In general, & high value for textual similarity ensures the LAION text is describing the same object as the synset. But as Figure 3b shows, we cannot set a very high similarity threshold since the extracted dataset will lose its coverage over the ImageNet's 1k classes. We found the threshold of 0.82 the highest reasonable choice it allows for covering most classes while going beyond it sharply reduces the number of covered classes (Figure 3b) with no significant reduction in the dataset size (Figure Bch. To further support this choice, in Section (Figure 11bj, we demonstrate that using the restored captions of ImageNet, textual similarity of above 0.7 is sufficient to ensure that sample belongs uniquely to the synset. Also refer to Appendix C for an example of when the second step of filtering is necessary and why the chosen threshold is conservative.
2.5
included
1c7
1.25
2.0
900
1.00
1.5
0.75
800
1.0
0.5
0.25
700
0.5
0.6
0.7
0.8
0.5
0.6
0.7
0.8 0.9
0.5
0.7
0.8 0.9
text-to-[namer def] similarity
text-to-[namer def) similarity
text-to-[name: def] similarity
(a)
(b)
(c)
Figure 3: Filtering LAION samples based on their textual similarity to the candidate synsets. The dashed line shows the chosen threshold. (a) The overall probability density function (pdf) of the similarities prior to the second step of filtering. in the dataset and the size of the dataset for different levels of similarity threshold.
(b and c) The number of ImageNet classes covered by at least one example
relative frequency
10-2
10-1
basketball
laien mowar
brown bear
forkilit
projectoo
swing
barball
laaffish
tree, frog
galditann
hay
altar
kmonno
binder
giant panda
digital watch
hernit crab lnttle blue heron bassinet
Site
Figure 4: Relative frequencies of different classes in LAIONet sorted in descending order for the 500 most frequent classes. Some class names are shown. The red line shows uniform weight.
We take few additional measures to guarantee the safety and quality of the chosen instances. First, we drop samples with more than one label to simplify the evaluation on the dataset. Second, we drop images tagged as not-safe-for-wort in LAION. Finally, we exclude images that contain text matching the name of their synset. This will ensure the captions are describing an object in the image and not just reflecting on another text. To achieve this, we employ EAST for text detection (Zhou al. 2017) and TrOCR for text recognition (Li et al. 2023a). This step climinates 1.1% of the samples.
The final dataset, which we call LAIONet, consists of 822k samples from 915 ImageNet classes, sufficiently large for fine-gras grained evaluation purposes statistical significance. Unlike ImageNet which provides about the same number of images per class, the large variation in the relative frequency of the classes in LAIONet reflects the natural distribution of each class (Figure 4p. In particular, 95% of LAIONet classes have at least samples, 90% have at least 13 samples, 80% have at least 36 samples, and 70% have at least 75 samples. We will later use the relative frequency classos to compare the performance of models in frequent and infrequent classes. Note that we can also create a more conservative version of LAIONet mimicking the ImageNet validation set by retaining only the top 50 most similar instances for each class. This version
5

---

Published in Transactions on Machine Learning Research (02/2025)
of LAIONet yields consistent observations in general (Appendix Appendix 0
B Find sample images of LAIONet in
Are LAIONet images harder to classify? To find out, we compare CLIP zero-shot accuracy on LAIONet and ImageNet. For every image, we predict the label of the image based on what synset has the highest cosine similarity between the image embedding and the synset text embedding. To make accuracy estimates on LAIONet comparable with ImageNet, we calculate accuracy as the average recall across the classes present in LAIONet. This uniform weighting is consistent with the setup of ImageNet validation with 50 images per class. We found CLIP zero-shot top 1 accuracy to only differ by 2% across datasets. Hence, at least from the CLIP view, LAIONet images are not harder to classify in terms of average accuracy. We acknowledge a limitation in that the CLIP text embeddings used for LAIONet creation are jointly trained with the image embeddings used for zero-shot accuracy calculations. This may give CLIP an advantage on LAIONet, and CLIP zero-shot accuracy should be interpreted with caution. Appendix D offers a more direct assossment of the level of difficulty involved in identifying the intended object in LAIONet. This is achieved by directly computing the cross-modality similarity between an image and its associated synset. Overall, LATONet images do not exhibit significant difficulty compared to ImageNet.
3 LAIONet Versus ImageNet
We begin to understand the differences between the two datasets by looking the accuracy of various ImageNet classifiers LAIONet. After observing significant accuracy drop, consider the disparity in intra-class similarity as possible explanation.
3.1 Comparing Accuracy
2016), Vision Transformers (ViT) (Dosovitskiy et al, 2021), modernized ConvNet (ConvNeXt) (Liu et al. 2022), and Bidiroctional Encoder reprosentation from Image Transformers (BEiT) (Bao el al. 2022). All models are trained on ImageNet without extra training data. We use various versions of each model in terms of the size (small, base, large, etc.), image resolution (224x224 or 384x384), patch resolution (16x16 or 32x32), and whether models are pre-trained on the complete ImageNet with 22k classes or not. All models come from HuggingFace (huggingface co) checkpoints.
We consider four model families: ResNet
(He et al.
We first compare the (equally weighted) accuracy defined by the average of recalls across the classes covered by LAIONet. Figure | compares the top and top 5 accuracy on ImageNet and LAIONet. In most of the highly accurate models, accuracy drops by at least 10 percentage points when estimated LAIONet with models pre-trained on ImageNet-22k showing slightly more robustness.
Next, we use the relative frequency of each class in LAIONet to weight its recall and obtain a LAION- weighted accuracy. Figure compares -eigteweighted and equally-weighted accuracy on LAIONet. The weighted accuracy is consistently lower by to 10 percentage points. This can partially be explained by the observation that ImageNet-traiie trained models are performing worse when the class is describing a more common object (Appendix G.1 We also compared LAION-weighted and equally- accuracy on ImageNet and found consistent results in Appendix
3.2 Comparing Intra-Class Similarity
While LAIONet images are in a precise sense not more difficult than ImageNet, there is another factor that can explain the accuracy drop: the intra-class similarity of images. We deline this similarity as the pairwise similarity of the images from the same class, measured by the cosine similarity of their CLIP image enbeddings. The lower these similarity values, the more diverse the images from that class.
Figure ba shows the distribution of intra-class similarities aggregated over all the classes that have al least samples in LAIONet. To make the distributions comparable, we sampled (with replacement) the similarities from LATONet to match ImageNet. The left tail of the L.AONet intra-class similarity distribution makes it clear that LAIONet overall provides a more diverse set of images. To observe the effect in greater detail, for each class, Figure 0b shows the average intra-class similarity of LATONet images subtracted by the average
6

---

Published in Transactions on Machine Learning Research (02/2025)
Top accuracy
Top accuracy
(LAION-WeI wew-weigttedde classes)
Ik models
LAIONet (LAION- won--weighedd classes)
1k models
pt22k-ft1k models pt22k models,
pt22k models
0.7
pt22k models
yex
y-x
LAIONe!
0.7
0.5
0.6
0.5
0.7
0.7
0.8
0.9
LAIONet
LAIONet
(equi-weighted classes)
(equi-weighted classes)
Figure A weigN-weighted accuracy is calculated according to the relative frequency of the classes in LAIONet and compared to the accuracy with equally weighted classes.
intra-class similarity of ImtgeNNet images from the samn class. In almost two-thirds of the classes, LAIONet has significantly lower intra-class similarity. This provides further evidence that LAIONet images exhibit greater variability within each class.
Intra-Class image-lmage CLIP similarity
LAIONet
0.2
ImageNet
0.0
LAIONet ImageNet
1
-0.2
UCB
0.0
0.2
0.4
0.6
0.8
1.0
200
400
600
intra-class image similarity
(a) Dist. aggregated similarities
(b) Comparison across classes
Figure Comparing the intra-class similarity of LAIONet and ImageNet. (a) each class, pairwise similarities of LAIONet images are sampled to match ImageNet in number. All the classes combined, the distribution of intra-class similarity is depicted. (b) For each class, the average intra-class similarity of ImageNet images is subtracted from the same value in LAIONet. The blue and red curves show upper and lower 95% confidence intervals. All values are sorted ascendingly.
we show that models struggle more with classes where LAIONet and ImageNet have significantly different intra-class similarity. This, combined with our observation LAIONet having lower intra-class similarity, supports our argument that intra-class similarity plays crucial role in reducing accu- racy.
In Appendix 2
4 Diagnosing ImageNet
As is standard modeling practice, think of data-generating process that for given class Y generates pair of image X and text T. Idcally, when we scarch for images of particular class y, we would like to draw samples from distribution p(X = y) Unless we have access to the generative process or we have a completely random set of images all correctly labeled, drawing samples directly from p(X = y) will not possible. In particular, none of these options are available when rescarchers collect new datasct. Instead, researchers have to define selection mechanism S for choosing images. What observe is the conditional distribution of X given S.

---

Published in Transactions on Machine Learning Research (02/2025)
In creating LAIONet, we relied on texts to select the samples (Figure 2ah. LAIONet images follow p(X]S 1) where S = 1 if T is sufficiently similar to Y. With our conservative selection criteria, we can assume every T' passed our similarity threshold is generated from the intended Y = y. Therefore, p(X = 1) = p = 1, = y). Generally, an image carries much more information than the text. So, for the images of certain class, conditioning on the lext alone should not alter the distribution significantly. Intuitively speaking, p(X = y, = f) = p(X = y)- In our setting, a weaker independence is sufficient to show LAIONet images follow the desired distribution. Even if information from X beyond Y is present in T, since we deliberately refrained from searching for visual descriptions in the text, we expect 5 to be independent from X for a given Y = y- Hence, we have reason to hope p(X[S= - ~ pl [ = 1, = y) ~ p(X = y)-
In general, a selection S" can rely on both text and image directly (Figure 2bp. In this case, the distribution of observed images p(X]S' = 1) can be far from the desired distribution p(X = y). We believe this has happened in the collection of ImageNet, primarily through human annotators examining and acting on images. Incorporation of visual features at the side of the scarch engine provider is another plausible mechanism. While we may not be able to pinpoint the exact mechanism at play, we will now move beyond LAIONet and demonstrate, through three independent experiments, strong link between the image X and the selection criterion S' in the creation of ImageNet.
4.1 A Weaker Image- Te-Selection Link Makes ImageNet More Like LAIONet
Image annotation is one clear mechanism by which the image X inffuences selection S'. Changing the strictness of annotation allows us to modulate the strength of this mechanism and measure its effect. This experiment is possible due to the availability of ImageNet (Recht et al.) 2019) that comes with three different versions. The three versions of ImageNet V2, called a, b, and c, differ in the level of agreement among annotators. More precisely, each image comes with an MTurk selection frequency which is what fraction of MTurk workers selected the image to be from the target class. ImageNet versions a, b, and e have an average MTurk selection frequency of 0.85, 0.73, and 0.93, respectively. Note that version b has the lowest and version has the highest selection frequency.
We first observe that allowing for more disagreement among annotators results in the inclusion of more diverse images. Figure 7a shows the distribution of intra-class similarity for V2 versions b and e. One can see that in version b, with the lowest average MTurk selection frequency, the intra-class similarity is shifted toward lower values.
We next further show as the average MTurk selection frequency increases, ImageNet becomes more similar to ImageNet and less similar to LATONet. In this regard, to compare two datasets, we count the number of classes in which the first dataset has significantly lower intra-class similarity than the second dataset, and vice versa. Figure 7b compares LAIONet and three versions of ImageNetV2. As the figure suggests, LAIONet and ImageNetV2 are similarly diverse when the average MTurr selection frequency is low (corresponding to ImageNet V2 version b). However, as the MTurk selection frequency increases, ImageNet V2 shows higher intra-class similarity than LAIONet. At the same time, Figure 7c shows ImageNet V2 becomes more similar to ImageNet as we increase the MTurk selection frequency.
We also contrast the accuracy of TmageNet-trainedd models LAIONet and ImageNetV2 in Figure Reaffirming our previous observation, as the MTur selection frequency declines from version to a to b, the accuracy on ImageNet and LAIONet increasingly aligns. Notably, most models experience similar accuracy drops on ImageNet V2-b and LAIONet.
Together, these observations show the impact the image has on the selection, particularly during annotation, is significant and can partially explain the divergence between LATONet and ImageNet. Further, the extra intra-class diversity of LAIONet achievable from less stringent human annotation and can explain the consistent accuracy drop on LAIONet and ImageNetV2.
4.2 Introducing an Image- To-Selection Link Makes LAIONet More Like ImageNet
We deliberately refused to use visual information in selecting LAIONet samples. Complementing Section 4.1 natural question arises: What would happen if we used multimodal LATON image-to-synset text similarity,
8

---

Published in Transactions on Machine Learning Research (02/2025)
1.0
1.0
3
ImageNetV2-b ImageNetV2-c
proportion classes
LAIONet < ImageNetV2
propertion classess
ImageNet < ImageNetV2
0.8
LAIONet ImageNetV2
0.8
ImageNet ImageNetV2
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0 0.2 0.4
0.6
0.8
1.0
0.75
0.80
0.85
0.90
0.75
0.80
0.85
0.90
intra-class image-image CLIP similarity
MTuu selection frequency
MTurk selection frequency
(a) Dist. of aggregated similarities
(b) LAIONel vs. ImageNet
ImageNee vs. InageNet
Figure 7: The effect of MTurk selection frequency on intra-class similarity. (a) The distribution of intra-class similarity aggregated over all classes for ImageNet versions b and c. (b) LATONet versus three versions of ImageNet The vertical axis shows the proportion of classes in which one dataset has significantly lower intra-class similarity than the other. To determine whether intra-class similarity is significantly lower in a class, we compute the average intra-class similarity of images within that class for both datasets and compare their difference using a 95% bootstrapped confidence interval. Blue curve: the proportion of classes where LAIONet has lower intra-class similarity than a specific version of ImageNet Green curve: ImageNet V2 has lower intra-class similarity. (c) ImageNet versus ImageNet Red curve: ImageNet has lower intra-class similarity. Green curve: ImageNet has lower intra-class similarity.
Top accuracy
Top accuracy
0.90
1.00
0.85
0.95
ImageNeetV22
0.80
ImageNet-V2
0.90
0.75
0.85
0.70
0.80
0.65
0.60
0.75
y-x
y-x
0.70
0.6
0.7
0.8
0.8
0.9
LAIONet
LAIONet
(equi-weighted classes)
(equi-weighted classes)
Figure 8: Accuracy of TmageNet-trained models evaluated on three versions of ImageNet versus L.TOOett.
either in place of or alongside LAION text-lo-synsel text similarity, to select samples? Answering this question, we construct new datasets out of LAION where we require an included sample to have image-to- Iname: def] similarity greater than threshold and text-to-[namer [name: def] similarity also greater than another threshold. To choose these two thresholds, we control the total number of selected samples to be similar to our original version of LATONet. Using CLIP embeddings to calculate similarities, we derive a multimodal similarity threshold for each value of the textual similarity threshold as shown in Figure g From this curve, we select four distinct threshold values and create a dataset for each. We call these datasets A, B, C, and D as depicted on Figure D.
For all four datasets, we calculate the intra-class similarities of all the common classes among them. We show the distribution of intra-class similarities of datasets A, B, and D in Figure [0a] As the figure suggests, dataset A, which is the most similar to LAIONet, overall has the lowest intra-class similarities. As we strengthen the image-to-sclection link by using a lower textual similarity and higher multimodal similarity, datasets show higher intra-class similarity. This is also evident in Figure 10b where we plot the average of intra-class similarities across classes. In Figure [0c] we also look into the number of classes where a dataset has lower intra-class similarity compared to ImageNet. Figures 10b and 10c together show that datasct A, the most similar to LAIONet, exhibits greater diversity than ImageNet, both in terms of average intra-class similarity and the number of classes where A is more diverse. However, we increase the multimodal
9

---

Published in Transactions on Machine Learning Research (02/2025)
Image-to-[name: similarity
0.30
0.29
0.27
0.26
0.725 0.750 0.775 0.800 text-to-[name: def) similarity
Figure 9: For each textual similarity threshold, we find the corresponding multimodal similarity threshold such that the resulting dataset, satisfying both similarity criteria, contains the same number of samples as LAIONet. We then choose four points on this curve, A, B, C, and D, and create four such datasots.
threshold, the datasets consistently become less diverse, eventually even less diverse than ImageNet. To make sure our measurement of intra-class similarities is not biased by the CLIP visual encoder, we have repeated this experiment using a variety of encoders in Appendix and found even stronger results.
In summary, in this experiment, used CLIP multimodal similarity to introduce image-to-sclection link into our dataset creation process. Our observations strongly suggest that this link can reduce the diversity of the included images when we control the total number of selected examples. While CLIP multimodal similarity, which is known to suffer from specific biases (Tong et al.) 2024), may not accurately reflect how link worked in ImageNet, it demonstrates how relying on a richer modality
the original can make the selection process more prone bias.
avg. similarity
1.0
proportion of classees
3
w/ intra-daa dlrs-
0.72
ImageNet avg. sim.
0.71
1
0.2
0
0.0
0.0
0.6
C
intra class image-image CLI similarity
(a) Dist. of aggregated similarities
(b) Avg. intra-class sim. in A-D
(c) A-D vs. ImageNet
Figure 10: (a) The distribution of intra-class similarity aggregated over all classes for datasets A, B, and D. (b) The average of intra-class similarities compared across datasets A, B, C, and D. To calculate this average, we first find the average of intra-class similarity in each class and then take their average across all common classes A, B, C, D. (c) The proportion of classes where the new datasets have lower-intra class similarities than ImageNet.
4.3 Text Alone Cannot Explain Why Image Is Selected Into ImageNet
ImageNet-Captions Fang al. 2022 is a subset of Image training data with restored title, descrip- tion, and tags from Flickr. We assume the samples in ImageNet-Captions are random subset of the original ImageNet and the captions are accurately restored. If there was no link X - S', the accompanying caption of an image in ImageNet-Caations should be able to explain why this image is selected.
We follow Fang et al. 20222 and define the text as the title, description, and tags concatenated. Figure Ila illustrates the similarity between the texts and their respective synsets using CLIP text emboddings. Al- though most of the texts have high similarity of 0.6 or above their synsets, the distribution has a heavy left tail. The fact that text has low similarity to the intended synset does not necessarily mean it could
10

---

Published in Transactions on Machine Learning Research (02/2025)
not be chosen by the search engine. However, we show many of the texts that have low similarity to the intended synsets actually have high similarity to numerous other synsets, making them less likely to have appeared for the intended meaning. For every text, we find the similarity to all synsets, i.e. the similarity to their names and definitions, and count the proportion of unintended synsets (false classes) that are more similar to the lext than the intended synset. A low value for this proportion shows the lext well represents intended synset whereas a significant non-zero value indicates that there are considerable other synsets that are more strongly present in the text. As Figure 11b demonstrates, for text with low similarity to its synset there observations show that at least. based on the restored texts in ImageNet-Captions the text alone cannot fully explain why an image is selected and another mechanism should have been at play.
average 5% (equivalently, 200) or more other synsets more similar to the text. These
praportion faise classes
0.25
LAIONet
60000
ImageNet- Captions
0.20
count
40000
more similar to
0.15
0.10
20000
0.05
0
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.8 text:(name: def] similarity
0.2
0.4
0.6
1.0
text: (name: def] similarity
(a)
(b)
Figure 11: (a) The distribution of the text-to-synse similarity. (b) For every bin of text-to- sott-to-yynet similarity, the average proportion of unintended classes which are more similar to the text than the intended class is depicted in black.
4.4 ImageNet, Had It Been Created Solely Searching Texts, Does Not Resemble Current ImageNet
If the link from X to S" did not exist, regardless of how the selection algorithms works, p(X = =() would look similar in both graphs of Figure 2 To test this hypothesis, we extract a new dataset from LAION. For every image in ImageNet with corresponding text T = in ImageNet-Captions we find the LAION sample with the most similar text to We only keep a LAION sample the similarity above 0.7. This choice ensures the two lexts are sufficiently similar as we can consider them roughly the same while the dataset covers more than 95% of the ImageNet classes (Appendix E
A Intra-class image-image CLIP similarity
ImageNet
0.2
ImageNet
0.0
2
LAIONet
ImageNet
1
UCB
-0.4
LCB
0
0.0
0.2
0.4
0.6
0.8
250
500
750
intra-class image-image CLIP similarity
(a) Dist. of aggregated similarities
(b) Comparison across classes
Figure 12: Comparing the intra-class similarity of the new datasct and ImageNet. The new datasct. is obtained by selecting LAION examples with the most similar texts to the lexts in ImageNel-Captions. (a) Distribution intra-class similarity aggregated across all classes. In each class, pairwise similarities of the images the new datasct are sampled to match ImageNet in number make the distributions comparable. (b) For each class, the average of the intra-class similarity of the images in the new dataset minus the corresponding value in ImageNet is plotted in black. The upper and lower 95% confidence bounds are depicted in blue and red. All values sorted ascendingly.
11

---

Published in Transactions on Machine Learning Research (02/2025)
suggests, images in the new dataset have significantly lower intra-class similarity. Looking at each class separately, Figure 12b shows in almost 70% of the classes, the images from the new dataset are significantly more diverse (have lower intra-class similarity). These observations reject the hypothesis that the graphs of Figure 2 have the same structure and show a potential Icak from the image to the selection. We note the limitation that lexts the ImageNel-CCaptions dataset may not completely include the text available the time of ImageNet creation. Second, for many cases, we were unable to find great matches for the ImageNet texts in LAION-400M and scaling our analysis to LAION-5B might help here.
As Figure 12a
5 Conclusion
In conclusion, argue that the image-to-selection mechanism played significant role in the creation of ImageNet, distinguishing it from LAION. We demonstrated this through three experiments. First, we modulated the speculated link from image to selection, showing the significant contribution this mechanism has in reducing the diversity of the selected images. The next two experiments rejected the hypothesis that image plays no or negligible role in the selection by showing ImageNet captions cannot solely explain the selection.
This insight carries valuable implications for dataset creation efforts in general. When developing dataset and diversity is desired, we advise selecting candidate instances based on information bottleneck, like a succinct textual description of the instance, rather than the full instance. This will mitigate the selection bias that may otherwise distort the distribution of data conditional on selection.
References
Hangbo Bao, Li Dong, Songhao Piao, and Furn Wei. BEit: BERT pro-training of image transformers. International Conference on Learning Represenlations, 2022. URL httpps//operrerree.ee/ffoooo www.:nernfoommmieoeo
p BhZSz5990
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 TEEE conference on computer vision and pollern recognition, pp. 248 255. Teee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image worth 16x16 words: Transformers for image recognition scale. In International Conference Learning Representations, 2021. URL
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Jacob Steinhardt, and Aleksander Madry. Identifying statistical bias in dataset replication. In International Conference on Machine Learning, pp. 2922 2932. PMLR, 2020.
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distribbutinal robustness in contrastive language image pre-training (clip). Tn International Conference on Machine Learning, pp.
PMLR, 2022.
Robert Geirhos, Patricia Rubisch, Claudio Michaclis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019. URL https://openrevier: net:/forummidd--yyygggooo0
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770 778, 2016.
Minyoung Huh, Pulkit Agrawal, and Alexci A Efros. What makes imagenet good for transfer learning? arXiv preprint or oix:::6000 08614, 2016.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero, Ivan Evtimov, Caner Hazirbas, Nicolas Ballas, Pascal Vincent, Michal Drozdzal, David Lopez-Paz, and Mark Tbrahim. Imagenet-x: Understanding model mistakes with factor of variation annotations. In The Eleventh International Conference on Learning wttp//orrniiii/////a
Representations, 2023. URL https:://penrevveer
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. Trocr: Transformer-baser optical character recognition with pre-trained models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 13094 13102, 2023a.
Zhiheng Li, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim. A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one am- plifies others. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 20071-20082, 2023b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEFE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 11976 11966, 2022.
George A Miller. WordNet: An electronic lerical database. MIT press, 1998.
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of CLIP. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
https:///openreview
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 2021.
PMLR,
Benjamin Recht,, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International conference on machine learning, pp. 5389 5400. PMLR, 2019.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal lcarning. In International Coference on International Conference Machine Learning, pp. 2012.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint 3xi::2111.02111 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Sri- vatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-texi models. In Thirly- siath Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openrevier http:///ww...oo.eeeeeeo//aaaaaaa
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tic-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in Newral Information Processing Systems, 13:16877-16867, 2020.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognétion, pp. 9568-9578, 2024.
Antonio Torralba and Alexci A Efros. Unbiased look at dataset bias. In CVPR 2011, pp. 1521-1528. IEEE, 2011.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Felix Vogel, Nina Shvetsova, Leonid Karlinsky, and Hilde Kuehne. Vl-taboo: An analysis of attribute-based zero-shot capabilities of vision-language models. CoRR, abs/2209.061033, 2022. URL https://dii.orrr org/
10.48550/arXiv. 220.9xiii...0....mmmmmm 06103
Kai Yuanqing Xiao, Logan Engstrom, Andrew Tyyss and Aleksander Madry. Noise signal: The role of image backgrounds in object rocognition. In International Conference Learning Representations, 2021. URL
Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. East: an efficient and accurate scene text detector. In Proceedings of the IEEE conference Compuler Vision and Pattern Recognition, pp. 5551-5560, 2017.
14

---

Published in Transactions on Machine Learning Research (02/2025)
A An MPNet-Filtered LAIONet
The creation of LAIONet relies on textual similarity of the LAION text and synset text. Section we used the cosine similarity of CLIP text embeddings to calculate this similarity, however, any sufficiently strong text encoder can be used for this purpose. In particular, we use MPNct {Song et al. 2020) finc-tuned on 1B sentence pairs with contrastive objective by HuggingFace [ We follow a similar procedure to Section 2 and choose the maximum similarity threshold that the resulting dataset does not lose its coverage over classes. We select the similarity threshold of 0.58. As Figure 13 suggests, threshold larger than 0.58 may exclude many classes without reducing the size of the resulting dataset. Refer to Appendix for additional evidence that this threshold works.
1e7
2.5
of included
2.0
900
dataset size
1.0
1.5
1.0
0.5
0.5
700
0.0
0.0
0.4
text-to-[name: def) similarity
text-to-[name: def) similarity
text-to-[name: def] similarity
(a)
(b)
(c)
Figure 13: Filtering LAION samples based their MPNet. textual similarity to the candidate synsets. The dashed line shows the chosen threshold. (a) The overall distribution of the similarities prior the second step of filtering. (b and e) The number of ImageNet classes covered by the dataset and the size of the dataset for different levels of similarity threshold.
Proceeding with the similarity threshold of 0.58, and after dropping samples labeled as not-safe- foroo-fo-woorrr samples with multiple labels, and images containing text of their associated synsels, this version of LATONet will have 831k samples covering 938 classes. As Figure 14 shows, consistent with our observation from CLIP- filtered LAIONet, models trained on ImageNet experience 10 to 15 percentage points of accuracy drop on IiPNe--fiteree LAIONet.
Top 1 accuracy
Top accuracy
ImageNet (validation set) set)
(valld (validation set) )
0.90
ImageNet
0.85
0.80
1k models
p127k-Rtkk models
ptl models pt22k models
ptzzk
yEx
0.70
0.5
0.6
0.7
0.8
0.7
0.9
LAIONet
LAIONet (equi-weighted classes)
(equi-weighted classes)
Figure 14: Accuracy of Teagennet-rr models when evaluated ImageNet validation set MPNet- filtered LAIONet. Three types of models are distinguished based whether they are pre-trained ImageNet-2D1 22k and whether they are fine-tuned on ImageNet-lk. Accuracy is defined as the average of the recalls calculated for each class that is present in LAIONet.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Last but not least, Figure 15 suggests that MPNet-filtered LAIONet also exhibits lower intra-class similarity compared to ImageNet. In particular, in more than 70% of the classes, LATONet has significantly lower intra-class similarity than ImageNet.
4 Intra-cCass image-image
0.2
LAIONet
3
ImageNet
CUP similarity
0.1
0.0
-0.1
LAIONet ImageNet
-0.2
UCB
LCB
-0.3
250
500
750
intra-class imago-image CLIP similarity
(a) Dist. of aggregated similarities
(b) Comparison across classes
Figure 15: Comparing the intra-class similarity of LAIONet and ImageNet. (a) In each class, pairwise similarities of LAIONet images are sampled to match ImageNet in number. All the classes combined, the distribution of intra-class similarity depicted. (b) For each class, the average intra-class similarity of ImageNet images subtracted from the same value in LAIONet. The blue and red curves show upper and lower 95% confidence intervals. All values are sorted ascendingly.
B A LAIONet From Most Similars
We created LAIONet by ensuring the presence of at least one lemma from the associated synset in the LAION text and by ensuring sufficient similarity between the synset lext and LAION lext. The frequency each class in LAIONet reflects the natural distribution of that class on the web and likely worldwide. However, we can create a more conservative version of LAIONet by retaining only the top 50 most similar instances for each class. This will make LAIONet more similar to the ImageNet validation set. Such a version of LATONet will have 39k samples covering 915 classes if initially filtered by CLIP similarity threshold of 0.82, and 41k samples covering 938 classes if initially filtered by MPNet similarity of 0.58.
Figure I6 illustrates that models performing well on ImageNet consistently experience to 10 reduction accuracy on this version of LATONet.. Hence, the reduction in accuracy consistent across all versions of LAIÖNets, including the most conservatively created ones. Figure [7 also confirms that this version of LAIONet. stiill exhibits longer tail of small intra-class similarity compared to ImageNet, potentially explaining the accuracy drop.
C On the Choice of LAION Text to Synset Text Similarity Threshold
In Section 2 we described how LAIONet is generated through substring matching LAION texts with Ima- geNet synset lemmas, followed by filtering out the cases where the LAION text is not sufficiently similar to the synset name and definition. critical choice in the second filtering step the choice of the minimum required textual similarity. We conservatively chose this threshold to be the largest value such that the remaining examples cover & large number of ImageNet's classes. To show this filtering necessary and our threshold of 0.82 for CLIP-based filtering and threshold of 0.58 for MPNet-based filtering is conservative, have provided example in Figure I8 Here the synset "cougar" has lemma. "puma". From WordNet definition, "cougar" is "large American feline resembling lion". But the common usage of "puma" on the web about a brand. As Figure (8)ssows for small similarity to the synset, data most likely will represent the brand instead of the animal. As we increase the similarity threshold, the examples become more and more likely be from the intended mcaning. Our manual inspections show similar this example, the chosen thresholds most likely result in high-quality matching the intended meaning of the synset even if the web is dominated by other meanings.
16

---

Published in Transactions on Machine Learning Research (02/2025)
Top accuracy
Top accuracy
0.85
0.95
ImageNet (validation set)
ImageNet (validation set)
0.80
0.90
0.75
0.85
0.70
models
0.80
mode el pt22k-ft:kk models pt22k models y=x
pt22k-ftik models ptz2k models
0.75
yTx
0.6
0.9
LAIONet (equl-weighted classes)
LAIONet (equi-welghted classes)
(a)
top 50
(b) CLIP-filterd top 50
Top 1 accuracy
Top accuracy
0.85
0.95
ImageNet (validation set)
ImageNet
0.80
0.90
0.85
0.70
models pt22k- models pt22k
mod els pt22k ttk models pt22k models
0.65
y-x
0.75
y-x
0.60
0.6
0.7
0.8
0.7
0.8
0.9
LAIONet
LAIO classess
(equi-weighted classes)
(equi-weighted classes)
(c) MPNet-filtered top 50
(d) MPNet-filtered top 50
Figure 16: Accuracy of ImageNet-trained models when evaluated on ImageNet validation set versus LAIONet created by retaining top 50 most similar instances for each class.
LAIONet
LAIONet
ImageNet
3
ImageNet
1-
1
0
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
intra-class image-image CLIP similarity
Intra-class image-image CLIP similarity
(a) CLIP-filtred lop 50
(b) MPNet-filtreed top 50
Figure 17: Comparing the intra-class similarity of LAIONet and ImageNet. In each class, pairwise similarities of LATONet images are sampled to match ImageNet number. All the classes combined, the distribution of intra-class similarity is depicted. LAIONet is created by retaining top 50 most similar instances to the synset text in each class after textual similarity filtering with CLIP or MPNet.
D On the (Non) of LAIONet Image Classification
To obtain a better idea how hard is to recognize an object in LAIONet, we calculate the cross-modal similarity of the images the texts of their associated synsets using CLIP embeddings. A high value of image-to-synset similarity indicates CLIP is able identify an object from the synset in the image. On the other hand, low value could indicate that the intended object is either absent from the image difficult
17

---

Published in Transactions on Machine Learning Research (02/2025)
5.0
2
0.0
(a) CLIP-based lextual similarity
(b) MPNel-based textual similarity
Figure 18: Sample images from five intervals of LAION text synset text similarity.
to recognize. We compare the image-to-synset similarities obtained from the ImageNet validation set and LAIONet.
Figure 19a illustrates the distribution of image-to-synset similarity for LAIONet and ImageNet. To ensure these distributions are comparable, we sampled LAIONel with replacement match the number of images per class in the ImageNet validation set. As the figure suggests, the two datasets are not significantly different. In more fine-grained test, compared the image-to-synset similarity of the LAIONet and ImageNet for each class. Figure 19b shows the average similarity in each class for LAIONet subtracted by the average similarity in the same class for TmageNet along 95% upper and lower confidence bounds. Overall, there is no strong signal that LAIONet images are harder in particular.
LAIONet
Image-to-Iname def) CLIP similarity
0.10
LAIONet ImageNet
ImageNet
UCB
10.0
LCB
edr
7.5
0.05
2.5
0.0
0.0
0.2
0.4
0.6
0.8
1.0
200
400
600
800
image-to((name: def] CLIP similarity
(a) Dist. of aggregated similarities
(b) Comparison across classes
Figure 19: Comparing image-to-synsed similarities of LAIONet and ImageNet. (a) For each class, LAIONet is sampled with replacement to have the same number of images as ImageNet, and all samples are aggregated to obtain the distribution.
(b) For every class, the average similarity the images to synset text is calculated for LAIONet and ImageNet and the difference is plotted. The upper and lower 0.95% confidence bound for this difference is plotted in red and blue. All values sorted ascendingly.
18

---

Published in Transactions on Machine Learning Research (02/2025)
E
.On the Choice of Textual Similarity Threshold in Extracting Most Similar LAION Instances to CrageNet-Captions
In Section 4.4 we selected a similarity threshold of 0.7 as the minimum requirement for similarity between LAION text and ImageNet. text. in order to include a sample from LAION. Ideally, we look for LAION examples with identical text the ImageNet but due to the limited number of samples available in LAION, this is not possible. As Figure 3b shows, increasing the similarity threshold beyond the chosen level of 0.7 significantly decreases the number of covered classes. Meanwhile, for larger thresholds, the new dataset looks more like ImageNet but still distinguishable. As Figure [0b shows, the proportion of classes with significantly lower intra-class similarity in ImageNet increases as the threshold increases, while the proportion of classes with significantly lower intra-class similarity in the new dataset decreases. The gap still persists but can potentially become smaller in the region our data cannot cover. In sum, the dataset extracted based on ImageNet looks unlike ImageNet but to the extent it is possible to find similar texts in LAION.
num. of classes included
1000
proportion of classes
1.00
0.75
950
LAION ImageNet
0.50
LAION ImageNet
900
0.25
850
0.00
0.5
0.6
0.7
0.5
0.6
0.7
LAION text to ImageNet text CLIP similarity
LAION text to ImageNet text
CLIP similarity
(a)
(b)
Figure 20: The effect of similarity threshold on the dataset extracted from LAION samples with most similar texts to the ImageNet texts. (a) Number of the classes covered in the new dataset versus the similarity threshold. (b) Proportion of classes with significantly lower intra-class similarity in the new dataset (blue) and proportion of classes with significantly lower intra-class similarity ImageNet (red) versus the similarity threshold.
F LAiON-Weighted Versus Equally-Weighted Accuracy Evaluated on ImageNet
In Section 3.1 we introduced weigNe accuracy where we use the relative frequency of each class in LAIONet to weight recall. As we presented in Figure 5 the LATON-weighte accuracy is consistently lower than the equally-weighted accuracy when models are evaluated on LAIONet. This observation is not limited to evaluation on T.AIONet. In fact, Figure 21 shows when weight the classes according to their relative frequency LAIONet, ImageNet accuracy also decreases. This can be attributed to the challenge of recognizing more frequent objects, given their potentially diverse types.
G The he Relation of Recall, Relative Frequency, and Intra-Clas Similarity
G.1 Recall Versus Relative Frequency
we observed accuracy drops when we weight different classes according to their frequency in LAIONet. This can be partially explained as models perform worse in more frequent classes. To directly observe this, Figure shows the rocall in each class versus the relative frequency of the class LATONet. Regardless of whether LAIONet created by filtering based on CLIP textual similarity or MPNet similarity, there exists weak but consistent trend that more frequent classes more likely to be misclassifidd.
In Section 3.1
19

---

Published in Transactions on Machine Learning Research (02/2025)
Top accuracy
Top accuracy
0.9
1k models
ImageNet (LAION- MiON-wwighted classes)
1k models
pt22k-ttlk models
pt22k-ftik models
ImageNes (LAION- weighted
pi2 models
y-x
0.9
yex
pt22k models
0.7
0.8
0.6
0.7
0.6
0.7
0.8
0.9
0.7
ImageNet Net
ImageNet
(equi-weighted classes)
(equi-weighted classes)
Figure 21: On ImageNet, a weigheee accuracy is calculated according to the relative frequency of the classes in LAIONet and compared to the accuracy with equally weighted classes.
G.2 Recall Versus Intra- Class Similarity
Section 3.2 introduced the hypothesis that higher intra-class similarity may account for the lower-than- expected performance of ImageNet models LAIONet. To observe that intra-class similarity can be re- sponsible for accuracy drop, Figure 23 demonstrates that models struggle on classes where LA1ONet is more diverse than ImageNet, as shown by the recall rates plotted against the difference in average intra-class similarity. This is true regardless of what notion of accuracy and what version of LATONet, CLIP-filtere or JLPNet-fittered is used.
20

---

Published in Transactions on Machine Learning Research (02/2025)
1.0
1.0
1.0
1.0
recal @1
recalle)
0.5
0.5
0.5
0.5
0.0
renIee-50 20
0.0
0.0
convnecc-asn- 221-221--1
0.0
ve-cmn--attti---- 224
10-3
10-3
10-3
10-3
relative frequency
relative frequency
frequency
relative frequency
(a) Recall&01 (CLIP-filtered LAIONet)
1.0
1.0
1.0
1.0
recall
recall
recall
0.0
mmne:: 20
0.0
0.0
convrecc 21-2
0.0
10
10-2
10
10
10"4
10"2
10
10-2
relative frequency
relative frequency
relative frequency
relative frequency
(b) Recallud (MPNet-filtered LATONet)
1.0
1.0
1.0
1.0
recalies
recalles
recalles
0.5
0.5
D.5
0.5
226-3k
10-3
10"3
10
10-3
relative frequency
relative frequency
relative frequency
relative frequency
(c) Recallã5 (CI.IP-flttred LATONet)
recallas
recall@s
recalles
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
10-2
10
relative frequency
relative frequency
relative frequency
relative frequency
(d) Recallãs (MPNet-filtered LATONet)
Figure 22: Recall per class evaluated on LAIONet versus how frequent the class LAIONet. Four different models are used, where two of them are pretrained on ImageNet-21k and of them are not, Two versions of LAIONet, CLIP-filterd and MPNet-filtered are included. Trends consistent.
21

---

Published in Transactions on Machine Learning Research (02/2025)
1.0
1.0
1.0
1.0
recalled
recalled
recalli01
0.0
POSNS 50
0.0
Commen 2260-26
0.0
cantereon bass 234 22
0.0
ME 9U50 PUTLLES 224
-0.25
0.00
0.25
-0.25 0.00
0.25
-0.25
0.25 intra similarity
0.00
-0.25
0.00
0.25
intra similarity
class similarity
similarity
(a) Recalla01 (CIIP-filtered LATONet)
1.0
1.0
1.0
1.0
recal
recalig1
recale1
recaligg
0.5
0.5
0.5
CATATIOn: base 228
corment 22-22
vi sathhe-222
-0.25
0.00
-0.25
0.00
-0.25
0.00
-0.25
0.00
& intra class similarity
à intra class similarity
intra class similarity
à intra class similarity
(b) Recall@1 (MPNet-fitereed LAIONet)
recalle5
recailes
0.5
0.5
0.5
0.5
0.0
0.0
0.0 0.25
0.25
-..55
0,00
0.25
-0.25
0.00
0.25
0.25
0.00
A Intra-class similanity
Intra-class similarity
& intra-class similarity
Intra-class simllarity
(c) Recall@5 (CLIP-fitered LAIONet)
1.0
recallas
resalles
recaligs
0.5
0.5
0.5
0.5
0.0 -0
0.0
0.0
conomest 26-2--1------
0.0
0.25
0.00
-0.55
0.00
-0.55
0.00
0.25
0.00
Intra-class similarity
intra-class similarity
Intra-class similarity
similarity
(d) Recall@5 (MPNet-fittreed LAIONet)
Figure 23: Recall on LAIONet for each class versus the disparity in intra-class similarity between LAIONet and ImageNet. This disparity (horizontal axis) measured by subtracting the class-average intra-class similarity in ImageNet from that in LAIONet. Four exemplary models shown, where two of them are pretrained Tel-21l (yellow) and two of them are not (red). Two versions of LATONet. considered. Trends are consistent.
22

---

Published in Transactions on Machine Learning Research (02/2025)
H Using Variety of Image Encoders to Calculate Intra-Clas Similarities
In Section 3.2 we introduced intra-class similarity and used CLIP image embeddings to calculate that. Since CLIP has well-known visual biases [Tong et al. 2024), we repeat our calculations using a variety of other encoders. This includes using the last hidden layer of the base ViT, BEiT, and ConvNeXT models, all pre-trained on the larger ImageNet-227 datasets.
Figure 24 illustrates the proportion of classes where LAIONet has significantly lower (blue) higher (red) intra-class similarity compared to ImageNet. As the figure suggests, using CL.IP and ViT-based image embeddings yields highly consistent results. Furthermore, using other embeddings only strengthens argument about the additional diversity in LAIONet.
1.0
classes
LAIONet ImageNet
LAIONet ImageNet
0.4
0.2
0.0
CLIP
ViT
ConvNeXt
BEIT
Image encoder
Figure 24: Comparing the intra-class similarity LATONet and ImageNet for various encoders. The vertical axis shows the proportion of classes in which dataset has significantly lower intra-class similarity than the other, where we define significance at 95% confidence levels.
We also repeal our experiment in Section 4.2 using various image encoders. Remember that in Section 4.2 generated new datasets A, B, C, and D by utilizing both the multimodal similarity between LAION images and synset text, as well as the textual similarity between LAION text and synset text. We then dete in Figure 10cl the fraction of classes where these datasets show lower intra-class similarity than TmageNet. In Figure 25 we show have repeated this experiment using other image embeddings. One can see that using any image encoder, going from dataset A to D, by increasing the role of multimodal similarity dataset creation, the datasets become less and less diverse compared ImageNet. This once again reaffirms conclusion regarding the potential bias that the image-to-selection link can introduce in dataset creation.
1.0
proportion of classes
0.8
0.6
ConvNeXT
0.4
Figure 25: The proportion classes whore the new datascts A, B, C, and D in Scction 4.2 have lower-intra class similarities than ImageNet, where we use various image encoders calculate intra-class similarity.
23

---

Published in Transactions on Machine Learning Research (02/2025)
Sample Images From LAIONet
We provide randomly picked images from both CLIP-filtered and MPNet-filtreed LATONet (Appendix B in this section. These images have been chosen based on various levels of difficully. Figure 26 illustrates the distribution of the recall@5 difference of the ViT-base model for each common class between LATONet and ImageNet. We choose recall@5 as a more reliable metric where the multiplicity of labels is less concern. One can see that there exist classes for which the recall on LAIONet is less than ImageNet for 0.5 more. These lypically the classes for which LATONet may have used broader meaning for the synset the images have appeared in different context than ImageNet. It worth noting that these classos make up very small portion of all classes and have minimal impact evaluations, whether not including such images is desired.
For the classes labeled on the graphs Figure 26 we have provided 10 random images from all datasets in the following. Each figure comes with potential explanation for the failure of ImageNet models in the caption.
papppillo thimble egyptian cat
200
egyptiaa cat
count
count
150
100
bress
grile
grille
pillow
kalbee
-0.5
-0.5
&recali@5 (LMONet ImageNet)
Arecall@s MuINNetImaggNNel
(a) MPNet-filtered
(b) CLIP-filtered
Figure 26: Distribution of recallã5 on LATONet subtracted by recall@5 on ImageNet. Only common classes are considered. The texts show the chosen classes for which example images are provided. The position of each text on the horizontal axis is the difference in recalls for that class.
00
date OCEaM
aami bock.n00
(a) MPNet-filtered
milk 0
Lable Imman.i
codestain.g
(b) CLIP-filtered
(c) ImageNet validation
Figure 27: Egyption cat. ImageNet models primarily struggle with Egypt cat statues or painted graphics, which are not well-represented or are rare in the ImageNet dataset.
24

---

Published in Transactions on Machine Learning Research (02/2025)
stade e
(a) MPNet-filteree
Redi coudh Okalest
shudin couchc
(b) CLIP-filtered
(c) ImageNet validation
Figure 28: Pillow. ImageNet models struggle to identify pillows when they deviate from the predominantly rectangular shape that is common in ImageNet.
coo boll
(a) MPNeet-iltered
bal40
(b) CLIP-filterd
(c) ImageNet validation
Figure 29: Bolete. ImageNet models are challenged when a bolete appears in contexts outside of nature, such as being picked by a girl or found in a pan.
ttber owwernnrrreeee kenat
(a) MPNet-filteed
(b) CLIP CLIP-fitered
(c) ImageNet: validation
Figure 30: Thimble, ImageNet models are challengod when the thimble among many other items.
25

---

Published in Transactions on Machine Learning Research (02/2025)
(a) MPNet-filtered
anotee 022
(b) CLIP-filtered
(c) ImageNet validation
Figure 31: Torch. ImageNet models have difficulty with recognizing graphical depictions of torches and identifying variations in torch orientation.
mergamern.
(a) MPNet-filtered
(b) CLIP-filtered
(c) ImageNet validation
Figure 32: Hen. Graphical hens pose a challenge for Image models. MPNet-filtered images also include blue and green-winged teal hens, which not present in the ImageNet dataset.
bestzra
(a) MPNet-filtere
1re 10100000
space haster eh sow s00sn
asenero modin 19n0-01
(b) CLIP-filtrrd
(c) ImageNet. validation
Figure 33: Grille. ImageNet models only recognize grille when installed include various kinds of grille which are not meant by ImageNet class.
LAIONet images also
26

---

Published in Transactions on Machine Learning Research (02/2025)
TOBEEA
SEaeer DMS
(a) MPNet-filtered
(b) CLIP-filterd
(c) ImageNet validation
Figure 34: Brass. The intended concept of this class ImageNet memorial made of brass. However, LA1ONet images correspond the broader meaning and the model not expected predict that.
27


Reviews:
[{"review_id": "OvxDBd9W1x", "reviewer": "Reviewer_2Yxa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. This paper is well-written, well-organized, and well-motivated, tackling a fundamental issue in vision dataset creation. \n2. The authors provide comprehensive experiments comparing LAIONet and ImageNet, evaluating accuracy, intra-class similarity, and the causal data-generation processes. \n3. The study aims to understand the role of human annotation through ImageNetV2 analysis, and the relationship between text captions and image selection. They compare ImageNetV2 and ImageNet-Captions to support their claims about dataset biases and diversity. \n4. Additionally, they offer insightful analysis of the selection approaches on dataset structure. These strengths make the work a valuable contribution to understanding and improving dataset curation. \n\nWeakness:\n1. Using textual similarity as the sole selection criterion may oversimplify the multimodal factors in dataset creation and fail to account for contextual biases in text captions. \n2. While the paper shows LAIONet has greater diversity, it doesn't show whether this diversity always represents meaningful variations or might include some noisy/irrelevant variations.\n3. Although the authors propose using text as an information bottleneck for future dataset creation, they don't provide concrete guidelines about optimal text similarity thresholds or how to balance coverage versus precision in text-based filtering. Some hyperparameter settings seem very arbitrary, for example “excluding instances with a cosine similarity of image and text embeddings less than 0.3.” Why 0.3? Why not higher/lower? Also “we require a high similarity between the text and the synset name and definition” but what if the original LAION caption doesn’t include the relevant information at all? How to handle this situation?\n4. The reported 5-12 percentage point accuracy drop when evaluating ImageNet models on LAIONet could be attributed to dataset shift rather than intra-class similarity differences. Is there any experiment on the other way around? For example testing out the models trained with LAIONet on ImageNet.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces LAIONet, a dataset from LAION-400M based on textual similarity to ImageNet, supporting a counterfactual analysis of ImageNet's structure. It reveals differences between LAIONet and ImageNet, such as LAIONet's lower intra-class similarity, which causes ImageNet-trained models to perform worse on it. They further provide causal explanations for these differences, attributing ImageNet's higher intra-class similarity to human annotation, which introduces selection bias, whereas LAIONet reflects more natural data distributions due to its text-based selection. This study offers actionable insights for dataset creation, advocating for the use of information bottlenecks like text captions to reduce bias and improve diversity in datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Potentially including some small-scale ablation studies across different similarity thresholds and providing empirical justification for the chosen values would be helpful to understand why 0.3 is a good threshold\n- To properly analyze the performance gap, I suggest that the authors conduct bidirectional experiments by training models on LAIONet and testing them on ImageNet.", "broader_impact_concerns": "It might be helpful to elaborate further on the process used to filter out NSFW content from LAION-400M and discuss the measures taken to make sure those content is excluded from LAIONet.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9omgr6wloq", "reviewer": "Reviewer_DWnV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The paper presents a new dataset that can be used to evaluate ImageNet models, with the same classes but significantly more varied images.\n\n1. The authors make use of pre-existing ImageNet datasets in their analysis, like ImageNet-Captions, and ImageNetV2.\n\n1. The authors come up with a hypothesis about the data generating process that could account for the observed difference, and the submission describes multiple experiments that try to give support to that hypothesis.\n\nWeaknesses\n\n1. The paper compares two datasets that were generated in completely different ways: ImageNet was using various image search engines to find images matching the textual descriptions, LAIONet sources images exclusively from a dataset that is already prefiltered in various ways, one of them being a CLIP image/text similarity threshold; ImageNet being collected from the web before the year 2009, LAIONet containing images up to 2021; ImageNet being selected by human rater agreement, LAIONet completely relying on a CLIP-based selection process to remove noise; (and the list goes on...) But the author's central claim is that the observed difference of the two datasets (namely its lower intra-class similarity) is due to information bottleneck in the selection criteria, and the higher intra-class variance of LAIONet being due to the fact that only the text was used for the selection process. I agree that this is a possible source of the observed difference, but the paper is lacking a quantification of **how much of the observed difference is due to this difference vs. all the other differences** in the dataset generation process.\n\n1. The introduced dataset LAIONet lacks some detail. For example, I only found Figure 4 about the class frequencies, and that figure only has approximate information about the 500 most frequent classes (for the least frequent classes, how often do they actually appear in the dataset?).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission presents a new dataset LAIONet that is sourced from LAION 400M by searching for lemmas from the imagenet net classes' synsets in the LAION text attributes, and then further filtering these candidates with a fixed threshold by their CLIP text similarity to the synset text.\n\nModels trained on ImageNet-1k/22k perform worse on LAIONet, and LAIONet has a lower intra-class similarity than ImageNet.\n\nThe authors propose that this difference can be explained by the data generation process, namely that the selection criteria of LIONet is limited to the text vs a selection criteria that involves both text and image in the case of ImageNet.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. `[critical]` I would want to see a **controlled setup**, where only specific parts of the dataset generation process are changed, and then the resulting datasets are compared against each other. If the authors want to show that higher intra-class similarity is due to a selection condition based on both the image and the text, vs. only the text, then they should define an additional dataset generation process that shares all characteristics with the dataset generation process of LAIONet, but changes the selection criteria, and then compare the resulting datasets.\n\n1. Provide **additional LAIONet characterization**, such as the number of examples per class.\n\n1. The authors mention multiple times that LAIONet contains \"images that are easy to classify\" (Section 1.1), and that \"at least from the CLIP view, LAIONet images are not harder to classify.\" (Section 2). But images could actually be quite hard to classify, for example because the text does not describe the image well in some cases. It would be interesting to have a **more detailed analysis of the mis-classifications** from different models. I found some examples in Figures 21-29, but these classes are not representative for the majority of the images in LAIONet.\n\n1. The **filtering** is crucial for the generation of LAIONet. The paper should explain why examples were first filtered by presence of a lemma in the text, and then by applying a threshold to the CLIP text-similarity of the example text and name and definition of the synset. Why is the first step necessary (the text only argues why the second step is necessary)? What happens to the intra-class similarity when the threshold is changed?\n\nAdditional Questions\n\n1. The first phrase of the abstract states that \"ImageNet was famously created from Flickr image search results.\" Where exactly is this described? Looking at [(Deng et al, 2009)] I read \"We collect candidate images from the Internet by querying several image search engines\" in Section 3.1\n1. Why was LAION 400M used and not LAION 5B? Having a larger dataset would allow for better class coverage and more freedom in the selection procedure.\n1. Figure 2: What is the meaning of some lines being dashed?\n1. Figure 3b: \"ImageNet classes covered\" – covered with how many examples?\n\n[(Deng et al, 2009)]: https://www.image-net.org/static_files/papers/imagenet_cvpr09.pdf", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QuBDzugufm", "reviewer": "Reviewer_Zheg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**  \n1. This paper investigates the differences between the widely-used LAION and ImageNet, which is interesting.  \n2. Analyzing image-to-selection link is a novel perspective to understand dataset differences. \n3. The experiment and analysis may inspire future studies in related areas. \n\n**Weaknesses:**  \n1. ImageNet V2 also leads to significant performance drop than ImageNet.  How is the trend in Figure 1 when comparing ImageNet V2 vs LAIONet? \n2. LAIONet is selected according to text similarities between LAION text and a ImageNet synset using text encoders. Do CLIP and MPNet show biases in long-tail categories that has negative influence in data selection?  \n3. How many objects are typically in each image? Are there cases where an image includes multiple ImageNet categories?  \n4. Section 2 claims LAIONet images are not harder to classify from a CLIP perspective. Since LAIONet was curated using CLIP embeddings, does CLIP select easy images for itself to LAIONet?\n5. Section 3.2 uses CLIP to evaluate visual similarity, but CLIP's vision encoder has known limitations [1]. How about using other vision models?  \n6. More details of how Figure 7(b)(c) is ploted are suggested.\n\n[1] Tong et al. *Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs.* CVPR 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces LAIONet, a new dataset constructed by selecting images from the LAION dataset according  to ImageNet categories. Experimental results show the differences of model accuracy between LAIONet and ImageNet. This paper also an.alyzes the role of image-selection strategies in the dataset generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is of clear interest to the community. All three reviewers are convinced by the revised version.
Decision: Accept (poster)