Inferring Offensiveness In Images From Natural Language Supervision
Authors: Patrick Schramowski
Kristian Kersting
Venue: ICLR 2022 Submitted (2022)

Abstract: Probing or fine-tuning (large-scale) pre-trained models results in state-of-the-art performance for many NLP tasks and, more recently, even for computer vision tasks when combined with image data. Unfortunately, these approaches also entail severe risks. In particular, large image datasets automatically scraped from the web may contain derogatory terms as categories and offensive images, and may also underrepresent specific classes. Consequently, there is an urgent need to carefully document datasets and curate their content. Unfortunately, this process is tedious and error-prone. We show that pre-trained transformers themselves provide a methodology for the automated curation of large-scale vision datasets. Based on human-annotated examples and the implicit knowledge of a CLIP based model, we demonstrate that one can select relevant prompts for rating the offensiveness of an image. 
In addition to e.g. privacy violation and pornographic content previously identified in ImageNet, we demonstrate that our approach identifies further inappropriate and potentially offensive content.

Under review as a conference paper at ICLR 2022
ANTICIPATING ETHICAL ISSUES IN IMAGE TRAINING- DATA USING LARGE-SCALE TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
A This paper contains images and descriptions that offensive in nature. Probing fine-tuning (large-scale) pre-trained models results in sfate-of-the-art performance for many NLP tasks and, more recently, even for computer vision tasks when combined with image data. Unfortunately, these approaches also entail severe risks. In particular, large image datasets automatically scraped from the web may contain derogatory terms as categories and offensive images, and may also underrepresent specific classes. Consequently, there is an urgent need to carefully document datasets and curate their content. Unfortunately, this process is tedious and error-prone. We show that pre-trained transformers themselves provide a methodology for the automated curation of large-scale vision datasets. Based on examples and the implicit knowledge of a CLIP based model, we demonstrate that one can select relevant prompts for rating the offensiveness of an image. In addition to e.g. privacy violation and pomographic content previ- ously identified in ImageNet, we demonstrate that our approach identifies further inappropriate and potentially offensive content.
INTRODUCTION
Deep learning models yielded many improvements in several fields. Particularly, transfer learning from models pre-trained on large-scale supervised data has become common practice in many tasks both with and without sufficient data to train deep learning models. Whereas approaches like semi- supervised sequence learning (Dai & Le 2015) and datasets such as ImageNet (Deng et al. 2009). especially the MaangeNet-LLSRRC-222
dataset with 1.2 million images. established pre-training approaches, in the following years, the training data size increased rapidly to billions of training examples (Brown et al. 2020 et al. 2021), steadily improving the capabilities of deep models. Recently, autoregressive (Radford a.. 2019), masked language modeling (Devlin et al., 2019) as well as natural language guided vision models (Radford et al., 2021) have enabled zero-shot transfer to downstream datasets removing the need for dataset-specific customization. Besides the parameter size of these models, the immense size of training data has enabled deep learning models to achieve high accuracy on specific benchmarks in natural language processing (NLP) and computer vision (CV) applications. However, in both application areas, the training data has been shown to have problematic characteristics resulting in models that encode e.g. stereotypical and derogatory associations (Gebru et al. 2018 Bender et al., (02II. Unfortunately, the curation of these large datasets is tedious and error-prone. Pre-trained models (PM) used for downstream tasks such as face detection propagate retained knowledge to the downstream module e.g. the classifier.
To raise the awareness of such issues, Gebru et al. (2018) described how large, uncurated, Internet- issues,[Gebru based datasets encode e.g. dominant and hegemonic views, which further harms people the margins, The authors urge researchers and dataset creators to invest significant resource allocation towards dataset curation and documentation practices. As a result, Birhane & Prabhu (2021) provided modules to detect faces and post-process them to provide privacy, as well as pornographic content classifier to remove inappropriate images. Furthermore, Birhane & Prabhu (202 conducted a hand surveyed image selection to identify misogynistic images in the - Unfortunately, such a curation process is tedious and does not scale to current dataset sizes. Moreover, misogynistic images, well as pornggrppii content, only two subsets of offensive images. It remains an open question how to infer general offensiveness in images, including abusive, indecent, obscene, or menacing content, and how to identify them in automated dataset curation process.
dataset.

---

Under review as a conference paper at ICLR 2022
Soreedflerr
twicher shop
metchec
TEVONNIT
bo0k Jncket
hammerhead
SPONAAL ATI
Ioe boor
sbaya
The single image identified by the hand surveyed image selection of Birhane & Prabhu (202I). Right: Range of samples from our CLIP pre-selection. In summary. CLIP is detecting over 1,5% out of 50,000 images from ImageNet's validation set as possible offending and over 304 out of 2,28 167 from the training
Figure 1: Results from the ImageNat-ILSVRC-200 dataset (validation set). Left:
set. Like our classifier, the pornographic classifier used in (Birhane & Prabhu 2021) identifies the 5th image in the first row as inappropriate. However, our classifier finds additional images along other dimensions of inappropriate content. This provides an extension to private obfuscation of the faces and pornographic content classifiers provided by Birhane & Prabhu (022)) We blurred the images to not offend the reader and to not violate privacy.
Whereas large image datasets automatically scraped from the web may contain derogatory terms as categories and offensive images, which results in models with undesirable behavior, pre-trained models may also reflect desirable implicit knowledge and biases such as our social, ethical, and moral choices (Jentzsch et al. 2019 Schramowski et al. 2020) reflected within the training data.
In study, we investigate modern vision PMs trained on large-scale datasets, in particular, the Contrastive Language-Image Pre-trained model (CLIP) (Radford et al.] (02T) and argue that they show that they encode
themselves pave a way mitigate the associated risks. Specifically, implicit knowledge to infer offensiveness in images overcoming previous issues, namely the lack of adequate and sufficient training data. Furthermore, we demonstrate that our approach can be utilized to annotate offensive images in vision datasets and, therefore, reliably assist the curation process of such datasets. We illustrate our approach on the popular SmageNet-ILSVRC-220 show that large computer vision datasets contain additional inappropriate content, which previous documentations had not detected. With our proposed method this content can be automatically and reliably pre-selected.
dataset and
As an example, Fig. lleft) shows an exemplary image from the ImageNe==ILSVRC-201 validation set identified as misogynistic content categorized as "beach voyeur" by hand-surveyed image selection in (Birhane & Prabhu 2021). Next to this human-selected image, [Birhane & Prabhu (2021) applied different models to detect visible faces (thus violating privacy rights) and pornographic content. However, as we will show with our study, further inappropriate images, which we refer to as offensive, can be identified within the dataset. For instance, Fig. rightt shows sixteen hand- picked images from a set of automatically detected possibly offensive images, utilizing our proposed approach. Depending on the task and stakeholders, this ranges from offensive objects such weapons (second row, first and fifth image) and dead animals (first row, sixth image) to immoral actions such as harming or even killing animals (second row, second image) and humans (second row, seventh image), as well as offensive text and symbols (first row, third image).
With our study we therefore strongly advocate for curating and documenting dataset by the categories and models provided by Birhane & Prabhu (202T) but also by taking the possible general offensiveness in images into account. To this end. we provide our models and the necessary data to reproduce our experiments and utilize our proposed methoc
We proceed as follows. We start with a brief overview of related work and required background introducing pre-trained models and their successes as well as concerns raised. Next, we describe the term offensiveness and show that common deep models can not reliably detect offensive image content due to the lack of sufficient data. We then continue by demonstrating that recent models, guided by natural language during the pre- pre-training phase, can infer offensiveness in images based on their implicit knowledge. Before concluding, we present automated dataset curation exemplary on the MengeNet--LSSRR-2000 dataset.
Anonymous link placeholder. Please see supplement.
2

---

Under review as a conference paper at ICLR 2022
2 BACKGROUND AND RELATED WORK
Concerns about large-scale data sets. Pre-training has become an essential approach in many vision and language tasks. In the vision domain, pre-training on large-scale supervised data such as ImageNet (Deng et al. 2009) has shown to be crucial for enhancing performance on downstream tasks via transfer learning. Since these datasets contain millions of data samples, curating such pre-training datasets requires heavy work on data gathering, sampling, and human annotation, making error-prone and difficult to scale. Moreover, in the language domain, task-agnostic objectives such as autoregressive (Radford et al. (2019) and masked language modeling (Devlin et al. 2019) have scaled across many orders of magnitude, especially in model capacity and data, steadily increasing performance but also the capabilities of deep models. With their standardized input-output (text-to- text) interface Radford et al. (2019) have enabled zero-shot transfer to downstream datasets. Recent systems like GPT-3 (Brown et al.) 2020) are now competitive across many tasks with specialized models whereas requiring only a small amount to no task task-specific training data. Based on these advances, more recently, Radford et al. (202T) and Jia et al. (202I) introduced models with similar capabilittiss in the vision domain. However, pre-training such models requires particularly large-scale training data, and the datasets' curation process is tedious and error-prone.
To tackle this issue Gebru et al. (2018) suggest to provide dataset audit cards to document datasets. This provides stakeholders the ability understand training data characteristics in order alleviate known as well as unknown issues. The authors argue that whereas documentation allows for potential accountability, undocumented training data perpetuates harm without recourse.
Birhane & Prabhu (2021) provided such dataset card for the popular computer-vision ImageNet- ILSVRC-2012 dataset, including several metrics and the hand surveyed identification of images with misogynistic content. More importantly, the authors raised the awareness of polluted image datasets by the example of poroggrppicc content inside several popular computer vision benchmark datasets. Although the authors raised criticism against ImageNet and identified several inappropriate images, the Se-eeeee--ILSVCC----
dataset- and the pre-trained models- are still under the most popular datasets in the MI. community. In line with Gebru et al. (2018), Birhane & Prabhu (202T) urge that ethics checks for future dataset curation endeavors become an integral part of the cuman-in-the-lowe validation phase.
ImageNet. The ImageNet (Deng et al. 2009) data collection is one of the most popular datasets in the computer vision domain and mostly refers to the subset ImageNet k dataset with 1.2 million images across 1000 classes. This was introduced in 2012 for the classification challenge in the ImageNet Large Scale Visual Recogmition Challenge (ILSVRC). However, in total the collection (ImageNet21k) covers over 14 million images spread across 1,841 classes.
As Birhane & Prabhu (021) state, the ImageNet dataset remains one of the most influential and powerful image databases available today, although was created over decade ago. To apply transfer learning, the most popular deep learning frameworks provide downloadable pre-trained models for ImageNetlk. Recently, [Ridiik et al. (202T) provided a novel scheme for high-quality, efficient pre-training on ImageNet2 I and, along with it, the resulting pre-trained models.
Pre-training vision models with natural language supervision. Pre-training methods that learn directly from raw data have revolutionized many tasks in natural language processing and computer vision over the last few years. RRafford et al. (2021) propose visual representation learning via natural language supervision in a contrastive learning setting. The authors collected over 400M image-text pairs (WebImageTeex dataset) to show that the improvement with large-scale transformer models in NLP can be transferred to vision. More precisely. whereas typical vision models jointly train an image feature extractor and linear classifier, CLIPP jointly trains an image encoder and a text encoder to predict the correct pairings of a batch of (image, text) training examples. At test time the authors propose to synthesize the learned text encoder with a (zero-shot) linear classifier by embedding the names or descriptions of the target dataset's 's classes, e.g. "The image shows <label>.". For simplicity, we refer to a model trained in a contrastive language- fine-tuned probed for downstream task CLIP model. Closely related to CLIP, the ALIGN (Jia jet al.) 202I) model is a family of multimodal dual encoders that learn to represent images and text in shared embedding space. Instead of Vision-Tran Transformers (ViT) or ResNet models ALIGN uses the
pre-training setting and
3

---

Under review as a conference paper at ICLR 2022
EfficientNet (Tan & Le 2019) and BERT (Devlin al.) 2019) models as vision and text encoders. These encoders are trained from scratch on image-text pairs (1.8B pairs) via contrastive learning.
These models and their zero-shot capabilities display significant promise for widely-applicable tasks like image retrieval or search (Radford et al. 202T). For instance, since image and text are encoded in the same representational space, these models can find relevant images in a database given text relevant text given an image. More importantly, the relative ease of steering CLIP toward various applications with little or no additional data or training unlocks novel applications that were difficult to solve with previous methods, e.g. as we show, inferring the offensiveness in images.
Carried Knowledge of Pre-trained Large-Scale Models. As already described, with training on raw text, large-scale transformer-baat -based language models revolutionized many NLP tasks. Recently, Radford et al. (2021). Ramesh et al. (2021) and Jia et al. 2222)) presented encouraging results that a similar breakthrough in computer vision will be possible. Besides the performance improvements in generation, regression, and classification tasks, these large-scale language models show surprisingly strong abilities recall factual knowledge present in the training data (Petroni et al. 2019). Further, Roberts et al. (2020) showed that large-scale pre-trained language models' capability to store and retrieve knowledge scales with model size.
Since such models are often trained on unfiltered data, the kind of knowledge acquired is not controlled, leading to possibly undesirable behavior such as stereotypical and derogatory associations. However, Sooweer,,Schick et al. (202T) demonstrated that these models can recognize, to considerable degree, their undesirable retained knowledge and the toxicity of the content they produce. The authors further showed that a language model with this ability can perform self-debiasing to reduce its probability of generating offensive text. Furthermore, Jentzsch et al. (2019) and |Schrmmoskii et al. (2020) even showed that the retained knowledge of such models carries information about moral norms aligning with the human sense of "right" and "wrong" " expressed in language. Similar to (Schick et al. 202I), Schramowski al. (202T) demonstrated how utilize this knowledge guide autoregressive language models' text generation to prevent their toxic degeneration.
In this work, we investigate if able utilize the carried knowledge of large-scale vision models in similar way, i.e. detecting possible offensive images in large-scale vision datasets.
3 PRE-TRAINED MODELS ARE ABLE TO INFER OFFENSIVENES, IN IMAGES.
Inspired by these previous results, utilize pre-trained multi-modal (language-visse -visio)) model to investigate its carried visual knowledge. We make use of the multimodality by prompting the model with natural language to analyze if and to which extent it is able to infer the offensiveness in images.
Offensive images. Let us start by defining the term "offending" and describing it in the context of images. According to the Cambridge dictionary "offending" can be phrased as "unwanted, often because unpleasant and causing problems". Additionally, in the context of images and text, according to Law Insider" Offending Materials means any material, data, images, or information which is (a) in breach of any law, regulation, code of practice or acceptable use policy; or (b) defamatory, false, inaccurate, abusive, indecent, obscene or menacing or otherwise offensive; or (c) in breach of confidence, copyright or other intellectual property rights, privacy or any other right of any third party. In this work, we focus on images following the definition (b). This definition aligns with definitions of previous work detecting hate speech (Gomez et al. 2020) and offensive product images (Gandhi et al. 2020).
As Gandhi et al. (2220) have described, technical challenges of building such a system are, among others, the lack of adequate training data, an extreme class imbalance, and changing test distribution. However, we will showcase that recent pre-trained models trained on large-scale data guided by natural language supervision are able to distinguish between inappropriate, possible offending image content and other images based on their carried knowledge acquired during the pre-training phase. This is due to the self-supervised learning and the implied task-independence of the involved transformer.
accessed on 3rd October 2021 http:://wwwwlliieieccccooccooiin accessed on 3rd October 2021
4

---

Under review as a conference paper at ICLR 2022
3.0
Offensive- ness rating
Offensive- ness rating
200
PC2
4.35
5:00
PC1
PCI
(a) SMID data distribution.
(b) ResNet50 pre-trained ImageNetlk.
(c) ViT-B/16 pre-trained WebImageTess via CLIP.
Figure 2: The SMID dataset. a) rating < 2.5 are samples with possible offensive content and >3.5 images with positive content. b-c) PCA visualization of SMID feature space using different pre-trained models. Coloring of data samples indicates the moral rating of the image's content. A rating of four and five are immoral content and one and two moral content.
Note that offensiveness is concept that is function of social norms and people have diverse sentiments. In the present study, we define offensive content based on a data-centric approach. Therefore, the investigated "offensiveness" may primarily surfaces from the group of people that have generated the selected data and the annotators but also the pre- pre-trained model's s implicit knowledge. Both, the model and the data could contain biases which can result in ethical issues. We refer the ethics statement (Section (6 for details.
To steer the model towards detecting (morally) offensive image content, we used the images contained in the Socio-Moral Image Database (SMID) Crone et al. 2018). The dataset contains 2,941 images. covers both the morally good and bad poles of range of content dimensions including objects, symbols as well as actions portrayed. In large-scale survey, these images were annotated by 2.716 participants. Fig. shows the density distribution of the annotated labels, Based on their findings, Crone et al. (2018) divided the data into good (green; mean rating > 3.5), bad (red; mean rating 2.5)), and neutral (grey; rest) images. As the creators suggest, we discretised rating 2.5 as (morally) offending, and rating > 3.5 as not offending, cf. Fig. 2 We split the dataset (962 negative images and 712 positive images) into two random splits for our experiments. The training set contains 90%, and the test set the remaining 10% images. In the following experiments, 10-fold cross-validated results are reported.
Deep Learning to classify offensive image content. In the context of the Christchurch mosque shooting video streamed on Facebook, officials at Facebook replied that one reason the video was not detected and removed automatically is that artificial intelligence systems are trained with large volumes of similar content. However, in this case, there was not enough comparable content because such attacks are rarg") Also Gandhi et al. (200)) have deseribed the lack of adequate training data to train a classifier to, in their case, detect offensive product content. We further investigate this issue with our next experiment by fine-tuning common pre-trained model on few offensive labeled images.
To measure how well model can encode what human could consider to be offending, considered the above-mentioned Database (in total 1,674 images). We start by training a deep vision model on this dataset. Similar to Gandhi et al. (2020) we chose the ResNet50 architecture (He et al. 2016) pre-trained on ImageNet datasets (Deng et al. 2009). Fig. 2 shows dimension reduction via PCA of the embedded representations of the traiteained model, i.e. before trained on the SMID dataset. Based on this dimension reduction, it is unclear if the ImageNetlk pre-trained ResNet50 variant is able to infer offensiveness in images reliably. Also, after training the network, the performance of the fine-tuned (training all model parameters), as well as linear probed model (cf. Tab. DT. shows inconclusive results; even if the performance increases when larger dataset (ImageNet2 IK) is used. Specifically, the resulting weak precision and weak recall of probing the ImagNetlk k based models shows us that it has issues classifying truly offensive images as well as distinguishing between truly
are- ww..finggeeaffee-ccsstchhhhhhhhe accessed 4th October
5

---

Under review as a conference paper at ICLR 2022
Arch.
Dataset
Accuracy (%)
Precision
Recall
F1-Score
78.36 1.76
0.75 0.05
0.74 + 0.09
0.76 + 0.02
ImageNetIk
80.81 + 2.95
0.75 + 0.02
0.81 + 0.02
0.80 + 0.03
ResNe150
82,11 + 1.94
0.02
0.80 1 0.05
0.78 1 0.04
ImageNet21k
84.99 +: 1.95
0.82 1 0.01
0.85 + 0.06
0.82 + 0.04
Weblmage'Text
690.57 1:111:1 1.82
e0.91
c0.89
o0.88 0.03
VIT-B/32
Weblmage' Text
94.52 + 2.10
0.94 0.04
0.91 + 0.02
0.92 - 0.01
VIT-B/16
WebImageTText
96.30 1 1.09
0.95 1 0.02
0.97 1 0.01
0.97 0.02
Table Performances of pre-trained models ResNet50 and VīT-B. The ResNet50 is pre-trained on ImageNetIk, ImageNet2 (Deng et al. 2009) and the WebTextImage dataset (Radford et al., (2021). The ViT is pre-trained on the Weblextlmage dataset. On the ImageNet datasets, we applied linear probing (top) and fine-tuning (bottom), and the Weblmage'Text Text-baedd models, soft-prompt tuning. The overall best results are highlighted bold with the * marker and best on the ResNet50 architecture with markers. Mean values and standard deviations are reported.
100
Test Accuracy
80
Prompt (VIT-B/16 CLIP-Weblmage Text)
Prompt-tuning [ViT-8/22 CLIP-Weblmage Text)
Promps-uning [ReNN150 CLIP-WildmaggTeel
Linear Probing [ReNie50 CLIP-WeblmigeTets
Probing (ResNett0 Imagerer21k]
60
Linear Probing (ResNetti Imageretik)
40
1%
2%
4%
6%
8%
10%
20%
40%
60%
80%
1009
0%
# of labeled training examples used, shown in % of training set.
Figure 3: Performances of pre-trained models ResNer50 and ViT-B. The ResNet50 is pre-trained on ImageNetlk, ImageNet2 (Deng et al. 2009) and the WebTextImage dataset (Radford et al., (02T). The VIT is pre-trained on the WebTextlmage dataset. On the ImageNet datasets, applied linear probing (top), and on the seblmageeText-baan on different sizes of the SMID training set where 100% corresponds to 1506 images. One that steering CLIP towards inferring the offensiveness in images requires only little additional data. In contrast other pre-trained models, therefore provides reliable method to detect offending images.
models soft-prompt tuning. Tuning was performed
non-offensive and offensive images. After fine-tuning the whole model, it can increase its recall; however, the precision stays comparable low.
This supports the previous findings mentioned above. Next, we will consider these models as baselines investigate if more advanced PMs trained on larger unfiltered datasets carry knowledge about offensiveness.
Pre-trained, natural language guided models carry knowledge about offensiveness. By training on over 400M data samples and with natural language supervision, CLIP (Radford et al.] 2021). and other similar models (Jia et al. 2021), acquire (zero-)few-sher sfew-shot capabilities displaying significant promise for applications with little or no additional data. Next, we will investigate if this includes the detection of offensive image content.
Fig. 2 () shows the embedded representations the ViT-B/I6 model pre-trained on WebImageTess via Contrastive Language-Iman Pre-training (Radford et al. 2021). One can observe that the VIT's learned representation encodes knowledge of the underlying task, i.e. distinguish offensive and not offensive images without being explicitly trained do These results confirm assumption that due to the natural language supervision, CLIP implicitly acquired knowledge about what a human could depending the context- perceive as offending.

---

Under review as a conference paper at ICLR 2022
Offensive-
ness rating
4
Type
Sample
Prompt
Prompt Init
Figure 4: Soft-prompt tuning on lision-language representation space. The squared data samples visualize the locations of the initial prompt and the crosses the final prompts. On the right, the nearest image samples for each prompt are displayed.
Furthermore, the natural language supervision of CLIP allows us probe the model without training it (zero-shot). More precisely, as with the previous model (ResNet50 pre-trained on ImageNet), the images are encoded via the pre-trained visual encoder. Instead of training a linear classifier, we operate on the similarity of samples, in this case, the cosine similarity, in the representational space:
Evienal (I) * Etent (2)
Sim(x, 2) Brrisnaa (z) ||2 * Eteze (=)ll2
(I)
where Evienal and Etezt the visual and text encoders, and image sample and a prompt. We embedded the classes, as suggested by RRadford et al. (202T), into natural language prompts such "This image is about something <label>.", which has shown be good default helping to specify that the text is about the content of the image. Following the collection of human annotations contained in the SMID dataset (Crone al., 2018), we applied various prompt classes: bad/good behavior, whwpeewwworr/rrrsewwrllyy positive/negatte
and moral/immoral. Whereby, the labels positive and negative resulted the best zero-shi shot performance.
Fig. 3 (0%) shows that this zero-shot approach utilizing the implicit knowledge of the CLIP models is already performing on par with the ImageNet-basee based PMs which were fine-tuned on SMID. However, we noticed that the zero-shot approach is able to classify true-negativ samples well but performs less well on classifying positives. This suggests that both, or at least the prompt corresponding the positive class label, are not chosen optimally. The nearest image neighbors extracted from the SMID dataset (ef. Fig. top-right) confirm this observation.
No need to learn new features: Learning how to ask the model. The previously defined prompts may not be the optimal way to query the model's implicit knowledge to infer the offensiveness in images. To further steer the model, we, therefore, searched for optimal text embeddings, i.e. optimize the prompts, which is also called (soft-) prompt-tuning (Zhong et al.) [021] Qin & Eisner] (021). As an optimization task we defined the distinction of offensive and not offensive images and optimized the prompts by gradient descent as follows:
â arg max [(z))
(2)
where
[(z)
Σ y log(}), with ý softmax(Simm 2)) xEX
X
Note, that we do not updated the parameters, 0, of Fepsunt and Frest- The term y is the ground truth class label and X batch during the stochastic gradient descent optimization. The resulting prompts are shown in Fig. 4 and clearly portray on the one side possible offending image content and the other side positive content.
Next, we will evaluate the resulting CLIP model equipped with the newly designed prompts. Fig. shows that even a small portion of the training data (e.g. 4%, 60 images) increases the vision transformer's 's (ViT-B) performance to over 90%. In general, the vision transformer outperforms the pre-trained ResNet50 models. Furthermore, the vision transformer with higher model capacity
7

---

Under review as a conference paper at ICLR 2022
outperforms the smaller variant, indicating that not only the dataset's size is important, but also the capacity of the model. Training with the full training set reaches a final test accuracy of 96.30%= 1.09 (cf. Tab. DT. These results clearly show that large-scale pre- pre-trained transformer models are able to infer the offensiveness in images and that they already acquire this required knowledge during their pre-training phase guided by natural language supervision.
4 MACHINES ASSIST TO DETECT OFFENDING IMAGES IN CV BENCHMARKS
Next, we utilize the pre-trained CLIP model, and the SMID- selected prompts to identify possible offending images from popular computer vision benchmark datasets. As Birhane & Prabhu (202T) we focus on ImageNet and use its most-popular subset the ImageNet-ILSVRC-200 dataset as an example.
Using our previously described approach the pre-selection by CLIP extracts possible offensive images. However, offensiveness is subjective to the user and, importantly, the task at hand. In our case, the steered model may primarily mirror the moral compass and social expectations of the 2,716 annotators. Therefore, it is required that humans and machines interact with each other, and the human user can select the images based on a given setting and requirements. Hence, we do not advise removing specific images but investigate the range of examples and offensiveness selected by the system and thereby document the dataset. We here provide an exemplary list of contents and disguised images (Fig. 5). Additionally, we provide Python notebooks with the corresponding images along with the classifier in the supplemental material. Moreover, to enforce targeting possible strongly offensive images, we determined the prompts by shifting the negative threshold rating of 1.5 instead of 2.5.
Due to the complexity of offensive context, we separate the identified offensiveness into offending objects, symbols, and actions in images.
Objects. The ImageNetlk dataset, also known as ImageNet-ILSVRC-201 formed the basis of task-1 of the ImageNet Large Scale Visual Recognition Challenge. Hence, all images display animals or objects. Therefore it is not surprising that the largest portion of offensive content concerns negative associated objects and animals. In total, 40,501 images were identified by the offensiveness classifier, where the objects "gasmask" (797 images), "guillotine" (783), and "revolver" (725) are the top-3 classes. However, whereas most people would assign these objects morally questionable and offensive, they can not be treated as offensive when training a general object classifier. The same applies to the animal-classe tick (554) and spider (397).
To infer the offensiveness of images contained in ImageNet, may be more applicable to investigate classes with only small portion of possible offensive images. Next to injured (e.g. "koala", "king penguin") and aggressive animals (e.g. "pembroke", "redbone")), our proposed classifier detects caged (e.g. "great pyrenees", "cock") and dead animals (e.g. "squirrel monkey". "african elephant"). Additionally, objects in inappropriate, possible offensive scenes, like a bathtub tainted with blood ("tub") are extracted.
Symbols. Furthermore, one is able to identify offensive symbols and text on objects: several National Socialist symbols especially swastika (e.g. "mailbag", "military uniform""). persons in Ku-Klux- Klan uniform (e.g. "drum")), insults by e.g. showing the middle finger (e.g. "miniature pinscher", "gorilla", "lotion"), "), and inappropriate text such as "child porn" ("file") and "bush=i*** pa** off USA" ("pay-phone")..
Actions. In addition to objects and symbols, our proposed classifier is able to interpret scenes in images and hence identify offensive actions shown in images. Scenes such as burning buildings (e.g. "church") and catastrophic events (e.g. "airliner", "trailer truck") are identified. More impor- tantly, offensive actions with humans involved are extracted such as comatose persons (e.g. "apple", "brassiere", "tub"), persons involved in an accident (e.g. "mountain bike"), the act of hunting ani- mals (e.g. "African elephant", "impala"), a terrifying person hiding under a children's crib ("crib"), scenes showing weapons or tools used to harm, torture and kill animals (e.g. "hamster") and people (e.g. "hatchet". "screwdriver", "ballpoint", "tub").
Furthermore, derogative scenes portraying men and women wearing muzzles ("muzzle"), clearly misogynistic images egg. harmed women wearing an abaya, but also general nudity with exposed
8

---

Under review as a conference paper at ICLR 2022
king penguin
mailbaa
miltary
pin.
church
Granny Smith
Impala
ballpoint
plastic bag
pay-ahone
crib
muZ3le
scrowbriver
hatchot
IOEET
abaya
Figure 5: Exemplary hand-picked images with offensive content from the pre-selection of our proposed method. The images visualize the range of offensiveness (objects, symbols, actions) detected. Due to their apparent offensive content, we blurred the images. Their content can be inferred from the main text.
genitals (e.g. "bookshop", "bikini", "swimming trunks") and clearly derogative nudity (e.g. "plastic bag") are automatically selected by our proposed method. Note that e.g. the misogynistic image showing harmed woman wearing an abaya not identified by the human hand surveyed image selection of BBirhane & Prabhu (202T). Therefore, we strongly advocate utilizing the implicit knowledge of large-scale state-of-the-ee models in human-in-the-loop curation process to not only partly automatize the process but also to reduce the susceptibility to errors.
5 CONCLUSION
In recent years, deep learning approaches, especially transfer learning from models pre-trained on large-scale supervised data, have become standard practice for many applications. To train such models, a tremendous amount of data is required. As a result, these datasets are insufficiently filtered collections crawled from the web. Recent studies (Gebru et al.) 2018 8 Birhane & Prabhu 2021 Bender al. 202T have revealed that models trained on such datasets, and the resulting models for downstream tasks benefiting from these pre-trained models, implicitly learn undesirable behavior, e.g., stereotypical associations or negative sentiment towards certain groups. Consequently, there an urgent need to document datasets and curate their content carefully. Unfortunately, current processes are tedious, eтror- proon, and do not scale well to large datasets.
To assist humans in the dataset curation process, we, therefore. introduced novel approach utilizing the implicit knowledge of large-scale pre- pre-trained models and illustrated its benefits. We showed that CLIP (Radford al.) 2021) retains the required knowledge about what human would consider to be offending during its pre-training phase. As a result, offers a solution to overcome previous issues, namely the lack of sufficient training data identify offensive material automatically. In this regard, we have outlined a new solution to assist the curation process on large-scale datasets. dataset, showcased that our proposed approach
On the example of the MenggeNe-ILLSRRC200 can identify additional inappropriate content compared to previous studies. Our approach can be transferred any other vision dataset.
In future work, we thus plan to extend our analysis to other datasets such as the OpenImage dataset (Kuznetsova et al. 2020) and multi-modal datasets (Jia et al. (021) Further possible avenues for future work are the extensions of the proposed method to labti-label classification to directly separate offensive objects, symbols, and actions or derive other categories offensive content. Moreover, classifying different levels of offensiveness could further provide details to document datasets; however, this could require additional data. Since the underlying model of our proposed classifier is a deep learning model, it inherits its black-box properties. This makes it hard to understand why the model is identifying specific images. Applying explainable Al methods such as (Chefer et al. 2021 explain the reasoning process could lead to further improvement of the curation process.

---

Under review as a conference paper at ICLR 2022
6 ETHICS STATEMENT
Our proposed method provides a solution to automatically infer offensiveness in images with the intention to assist the curation process and documentation of datasets. However, we strongly advise applying such methods in human-in -aeuu----tte-looet setting. Since CLIP models themselves trained with weak supervision on not freely accessible data sources and only the pre- pre-trained models are provided by its' creators, it is unclear if e-g. social biases are inherent to the model. Details about possible biases and other potential misuses (e.g. surveillance) of the CLIP models can be found the original work of Radford al. (202I).
Furthermore, in the present study, offensive content is detected by a data-centric Al system, where the model is steered towards the sentiment reflected in the data. Therefore, next the model's contained biases, the investigated "offensiveness" may primarily surfaces from the group of people that have generated the selected data and the annotators but also the pre-trained model's implicit knowledge. An interesting and important avenue for future work is addressing what different groups of society, e.g. different cultures, would consider as offensive by the creation of datasets representing their corresponding sentiments. As other social norms, offensiveness is constantly evolving. This makes it necessary to update the data, system and documentations.
7 REPRODUCIBILITY STATEMENT
The code to reproduce the figures and results of this article, including pre-trained models, can be found in the publicly available repository. Furthermore, we provide the source code needed in order to determine prompts to steer the CLIP model towards inferring offensiveness images and apply to detect possible offending images in vision datasets. The figures with disguised images are provided in original form in the supplement material.
REFERENCES
Emily M. Bender, Timnit Gebru, Angelina ScMillan-Majjo and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Madeleine Clare Elish, William Isaac, and Richard Zemel (eds.), Proceedings of ACM Conference on Fairness, Accountability, and Transparency (FAceT), pp. 610-623, ACM, 2021. doi: 10.1 D0.1145/34421 01.4442288.344922
Abeba Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1536-1546 IEEE, 2021. doi: 00.1099/ACV44833.022000000
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Lin (eds.), Proceed- ings of Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
Hila Chefer, Shir Gur. and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp- 782-791. Computer Vision Foundation / IEFF, 2021.
Damien L. Crone, Stefan Bode, Carsten Murawski, and Simon M. Laham. The socio-moral image database (smid): A novel stimulus set for the study of social, moral and affective processes. PLOS ONE, 13(1):1-34 01 2018. doi: 10. /.joun//ju won...19099.....
Andrew M. Dai and Quoc V. Le. Supersupe sequence learning. In Corinna Cortes. Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Proceedings of Annual Conference Neural Information Processing Systems (NeurIPS), pp. 3079-3087, 2015.
10

---

Under review as a conference paper at ICLR 2022
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255. IEEE Computer Society, 2009. doi: 10. (009/CVPR.0009.5200
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pp. 4171-4186, 2019.
Shreyansh Gandhi, Samrat Kokkula, Abon Chaudhuri, Alessandro Magnani, Theban Stanley, Behzad Ahmadi, Venkatesh Kandaswamy, Omer Ovenc, and Shie Mannor. Scalable detection of offensive and non-compliant content logo product images. In Proceedings of IEEE Winter Conference on Applications of Computer Vision ( (WACV), pp. 2020. 2990.90.3454.
IEEE, 2020. doi: 10.1 10.1109/WACV4557.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. CoRR, abs/ 190,09010, 2018.
Raul Gomez, Jaume Gibert, Lluís Gómez, and Dimosthenis Karatzas. Exploring hate speech detection in multimodal publications. In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1459-1461 IEEE, 2020. doi: N0.1099WACV4557202200000...
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.20166
Sophie Jentzsch, Patrick Schramowski, Constantin A. Rothkopf, and Kristian Kersting. Semantics derived automatically from language corpora contain human-like moral choices. Proceedings of the AAAT/ACM Conference on AI, Ethics, and Society (ATES), pp. 37-44, 2019.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and uisionlanguaag representation learning with noisy text supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the International Conference on Machine Learning, (ICML), pp. 1904-4916 PMLR, 2021.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont- Tust-, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset V4. Int. J. Comput. Vis., 28(7):1956-1981 2020. doi: 10.1007/s1 1263-020-00116--2
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pp. 2463-2473. Association for Computational Linguistics, 2019. doi: 10.186533w1/D11-11
Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tüt Beltagy. Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL-HLT), pp. 5203-5212. Association for Computational Linguistics, 2021. doi: 10. 10.18653/v1/2221.11. www.://22...accmmii..4..4
A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. CoRR, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.). Proceedings of the International Conference on Machine Learning (ICML), pp. PMLR, 2021.
11

---

Under review as a conference paper at ICLR 2022
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the International Conference Machine Learning (ICML), pp. 8821-8831. PMLR, 2021.
Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. CoRR, abs/2104. 10972, 2021.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418-5420 Association for Computational Linguistics, 2020. doi: 10.18653/v1/ 2020.emnlp-mainn main.437.
Timo Schick, Sahana Udupa, and Hinrich Schüize. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. CoRR, 0t5/2103.004/0 2021.
Patrick Schramowski, Cigdem Turan, Sophie Jentzsch, Constantin A. Rothkopf, and Kristian Kersting. The moral choice machine. Frontiers Artif. Intell., 3:36, 2020. doi: 1..3389/frai.2222.0000
Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A. Rothkopf, and Kristian Kersting. Language models have a moral dimension. CoRR, abs/2103.11790, 2021.
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the International Conference on Machine Learning (ICML), pp. 6105-6114, PMLR, 2019.
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [MASK]: learning vs. learning to recall. In Kristina Toutanova, Anna Rumshisky, Luke Zeulleooee, Dilek Hakkani-Tür, Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL-HLT), pp. 5017-5033. Association for Computational Linguistics, 2021. doi: 10.18653/v11200 mai.I-main.398
12


Reviews:
[{"review_id": "xnELzK7WlaQ", "reviewer": "Reviewer_yFoK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To begin, I don’t think the work adequately defines what they are scoping/defining by \"offensiveness\". In Section 3, the work draws from some other works and focus in on “defamatory, false, inaccurate, abusive, indecent, obscene or menacing or otherwise offensive” as their scope. In what follows in the rest of the paper, there is an assumption that a large model trained on large data solves for this definition, when in fact, it’s like using an answer to “how” to a question about “what”. As a result, the problem statement of the paper is rather muddled since there’s essentially a “you know it when you see it” argument to the question of \"what is offensive content?\". Just to name a few other quandaries with problem scoping: to “whom” are we considering the content to be “offensive”? Since “offensiveness” is a concept that is a function of social norms, over what period of time are we considering or is the modeling a function somehow of time?\n\nThese questions have important design implications, for example, in the empirical experiments of this work, they discretize ratings in their Socio-Moral Image Database and threshold on <2.5 and >3.5 as offending and non-offending, respectively, but since it’s not clear how we are defining the “whom” of offensiveness, it’s unclear if these choices make sense [N.B. authors followed prior art with those thresholds, but that in itself is a choice by the authors]. To what extreme do we take “injustice anywhere is a threat to justice everywhere” (Martin Luther King Jr)? If it was found to be offensive to one rater? Do we define it at the level of a locale or people group sharing similar socio-cultural values? These are challenging philosophical questions even that I do not expect an ICLR paper to fully address, but a position on the scope needs to be made explicit, otherwise, the objective is unclear and it's difficult to even assess if this paper is making progress on the problem.\n\nFurther, I was confused by some of the reasoning in the presentation, which summarizing using quotes follows as:\n\t\n> \"...training data has been shown to have problematic characteristics resulting in models that encode e.g. stereotypical and derogatory\nassociations...  large, uncurated, Internet-based datasets encode e.g. dominant and hegemonic views...\"\n\t\n> \"...we investigate modern vision PMs trained on large-scale datasets, in particular, the Contrastive Language-Image Pre-trained model (CLIP) (Radford et al., 2021) and argue that they themselves pave a way to mitigate the associated risks... they encode implicit knowledge to infer offensiveness in images overcoming previous issues, namely the lack of adequate and sufficient training data.\"\n\t\n> \"...we demonstrate that our approach can be utilized to annotate offensive images in vision datasets and, therefore, reliably assist the curation process of such datasets.\"\n\t\n> \"We illustrate our approach on the popular ImageNet-ILSVRC-2012 dataset and show that large computer vision datasets contain additional inappropriate content, which previous documentations had not detected.\"\n\nThe logic seems be that large models trained on large data have undesirable biases and to address this, they propose to use another large model trained on large data to filter out the undesirable biases because they believe a priori that it, like other such models is attuned to the undesirable biases and can be used to pre-train a supervised task to identify one dimension of these biases: “offensiveness”. There’s a weird “who watches the watchmen” problem here. It’s unclear if there is actual a reinforcing loop where model used for filtering (CLIP) has biases and as a result, any downstream tasks will just narrowly focus on its representing of those biases. I will grant that the authors acknowledge this in the Ethics Statement, but I actually think it’s far more central the methods proposed and there's a need to address it head-on versus just having it as an Ethics addendum.\n\nUltimately, perhaps most puzzling is that despite the argument that such a filtering technique could reduce harmful biases in large datasets like ImageNet1k, the impact of the qualitative examples provided in Section 4 is opaque. For example, are the examples given coming from a high-recall setting where we want to trigger on even very low softmax scores? Even if these are identified in ImageNet1k, do we have reason to believe the base rates are sufficiently high that a naively trained model would encode these biases? If so, why was this then not validated with experiments?", "summary_of_the_paper": "This work investigates using CLIP for pre-training a model on the Socio-Moral Image Database (prior art) for the task of “offensiveness” detection for which they argue will assist in reducing bias in large models trained from large corpora like ImageNet.", "summary_of_the_review": "Overall, the paper puts forward a number of interesting empirical experiments, but because the problem statement (“offensiveness”) to begin with is not well-scoped, it then becomes difficult to assess the contributions of the proposed methodology. In addition, the ultimate impact of the work is only shown qualitatively with example concepts (and occurrences) that it’s identified in ImageNet1k, but the results of these are not validated and its impact on say models trained on ImageNet1k without these “offensive” examples is not quantified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "v4cP5z6isfN", "reviewer": "Reviewer_dcH1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The contribution of the paper is not wholely clear to this reviewer. \n\nFirst a short summary of the paper: (1) the paper picks a fairly obscure dataset for prediction (2) constructs a good classifier using prompt fine tuning and CLIP (cool!) (3) applies it to ImageNet and provides a few categories with images deemed immoral. \n\nOverall, the paper is pitched as a dataset analysis tool for problematic images. But little to no effort is put into presenting results of analysis, beyond listing a few categories in ImageNet that had a lot of 'immoral' judgements predicted. I am left essentially knowing that a classifier can be built to predict morality based on an obscure dataset.  \n\nOn a technical level, while the results of prompt tuning of CLIP in particular could be new, the application is a fairly obvious combination of existing ideas, although I am excited to know its possible. I would rather have seen a full exploration of prompt fine tuning in CLIP rather than a narrow application to this morality dataset.  ", "summary_of_the_paper": "The paper constructs a classifier of whether or not an image is offensive. This is operationalized by finding a dataset from the psychology community of a few thousand images and ordinal judgements of 'morality' from a study. Prediction of these judgements is predictably hard, in no small part because the data is small. To combat these issues, the paper uses CLIP and soft-prompt tuning. Soft-prompt tuning appears to be very effective for CLIP in this context reaching accuracies of over 95%, where baseline fine tuning only gets about 85%. ", "summary_of_the_review": "The paper does not provide deep analysis of offensive content in any visual dataset. Technically, the paper combines currently trendy and exciting techniques in fairly straightforward ways to create a classifier from little data for morality judgements. This technical contribution is not fully explored, and in itself does not appear novel. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nE3eVpWXQ5l", "reviewer": "Reviewer_zJbR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. This paper focused on an important aspect of fairness/dataset bias in modern deep learning models, and showed an interesting direction to reveal questionable instances from the dataset. As the authors advocate this approach if combined to the human-in-the-loop protocol for dataset collection, may help creating more responsible datasets while saving some tedious human efforts.\n2. The authors give very detailed introduction, background and related work sections, highlighting the importance of the task.\n3. The code and pre-trained checkpoint provided in the supplementary might be useful for future work.\n\nWeakness:\n1. The technical contribution of this paper is very limited. Although it's arguably as important to identify new question and explore new application of existing techniques, essentially what this paper did is just an extension of CLIP-based models. The baseline experiments are simply finetuning/linearly probing existing models (ResNet-50 and ViT) on a public dataset (SMID), where the offensiveness is more of a synonym of \"immoral/blameworthy\". The proposed work regarding CLIP is more like a specific zero-shot learning application (i.e. offensiveness), and the prompt tuning is also highly related to previous work. Maybe this work is more suitable for a dedicated conference/workshop rather than a generic top-tier AI conference like ICLR.\n2. One important section that is missing from this paper is the analysis of precision/recall of the proposed work. In Table 1, the authors show precision/recall/F1 but this is on SMID dataset (if I'm understanding correctly), where the \"ground truth\" labels are inferred based on discretization of rating (<2.5 or >3.5). This is different from the true offensiveness detection. Specifically, since the authors already compared with Birhane & Prabhu (2021) with their hand surveyed image selection, how accurate does the proposed model match the performance of the human selection? In Page 9 the author mention that one image of Figure 5 (misogynistic image) is missed by Birhane & Prabhu, but I'm curious in a statistical level how well do they agree. Alternatively a user study might be useful, that within the 40,501 detected images how many are truly \"offensive\"? Right now it seems to me that the authors only show several methods effective on SMID dataset, but the key contribution of \"inferring offensiveness\" is built upon the assumption that SMID rating is a proxy for offensiveness.\n3. The concerns about ethical issues are not fully addressed. See Ethics Concerns section for more comments.\n4. Some minor issues need to be resolved: For example, Figure 4 caption is it \"On the right\" (not \"On the left\")? Footnote 4 on Page 5, \"Oktober\" -> \"October\". The format of citation is mistaken in multiple occurrences, such as Figure 3 caption \"(Deng et al. 2009)\", and second paragraph of page 6, in \"CLIP (Radford et al. 2021)\", \"other similar models (Jia et al 2021)\".", "summary_of_the_paper": "This paper described an interesting idea to leverage massively pre-trained models (specifically CLIP-based models) to infer offensiveness in images. The authors first gave detailed literature review regarding a recently raising concern about inappropriate images in computer vision datasets, and also performed several finetuning/probing baseline experiments to illustrate the feasibility to detect such inappropriate contents. Next the authors explored the possibility to mitigate the potential risk by utilizing the implicit knowledge learned by CLIP-based pre-\ntrained models. Lastly the authors choose ImageNet for a proof-of-concept validation and show that the proposed approach can discover previously neglected offensive images.", "summary_of_the_review": "In general I think it's a solid paper with proper experiments as proof-of-concept, but the lack of technical contribution and missing the actual analysis on the detection of offensive contents are major weakness of the submission. For top-tier AI conference like ICLR this paper is not good enough.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors set out on an important question of whether abstract and culturally specific concepts like offensiveness can be detected in images. The novelty of this work comes in part from tackling this question and attempting to create a technology which can operationalize it.  However, despite the authors' insistence that offensiveness is not "just another label", in practice the work treats it very much that way and therefore does not present a compelling innovation either in modeling or in juxtaposition to other labeling tasks.  Known training and inspection techniques are used on existing representations and more powerful models with more training data generalize better. It is unclear what is novel in the approach or unique to offensiveness over other labels (including abstract ones).
Decision: Reject