Inferring Offensiveness In Images From Natural Language Supervision

1 INTRODUCTION . Deep learning models yielded many improvements in several fields . Particularly , transfer learning from models pre-trained on large-scale supervised data has become common practice in many tasks both with and without sufficient data to train deep learning models . Whereas approaches like semisupervised sequence learning ( Dai & Le , 2015 ) and datasets such as ImageNet ( Deng et al. , 2009 ) , especially the ImageNet-ILSVRC-2012 dataset with 1.2 million images , established pre-training approaches , in the following years , the training data size increased rapidly to billions of training examples ( Brown et al. , 2020 ; Jia et al. , 2021 ) , steadily improving the capabilities of deep models . Recently , autoregressive ( Radford et al. , 2019 ) , masked language modeling ( Devlin et al. , 2019 ) as well as natural language guided vision models ( Radford et al. , 2021 ) have enabled zero-shot transfer to downstream datasets removing the need for dataset-specific customization . Besides the parameter size of these models , the immense size of training data has enabled deep learning models to achieve high accuracy on specific benchmarks in natural language processing ( NLP ) and computer vision ( CV ) applications . However , in both application areas , the training data has been shown to have problematic characteristics resulting in models that encode e.g . stereotypical and derogatory associations ( Gebru et al. , 2018 ; Bender et al. , 2021 ) . Unfortunately , the curation of these large datasets is tedious and error-prone . Pre-trained models ( PM ) used for downstream tasks such as face detection propagate retained knowledge to the downstream module e.g . the classifier . To raise the awareness of such issues , Gebru et al . ( 2018 ) described how large , uncurated , Internetbased datasets encode e.g . dominant and hegemonic views , which further harms people at the margins . The authors urge researchers and dataset creators to invest significant resource allocation towards dataset curation and documentation practices . As a result , Birhane & Prabhu ( 2021 ) provided modules to detect faces and post-process them to provide privacy , as well as a pornographic content classifier to remove inappropriate images . Furthermore , Birhane & Prabhu ( 2021 ) conducted a hand surveyed image selection to identify misogynistic images in the ImageNet-ILSVRC-2012 dataset . Unfortunately , such a curation process is tedious and does not scale to current dataset sizes . Moreover , misogynistic images , as well as pornographic content , are only two subsets of offensive images . It remains an open question how to infer general offensiveness in images , including abusive , indecent , obscene , or menacing content , and how to identify them in an automated dataset curation process . Whereas large image datasets automatically scraped from the web may contain derogatory terms as categories and offensive images , which results in models with undesirable behavior , pre-trained models may also reflect desirable implicit knowledge and biases such as our social , ethical , and moral choices ( Jentzsch et al. , 2019 ; Schramowski et al. , 2020 ) reflected within the training data . In our study , we investigate modern vision PMs trained on large-scale datasets , in particular , the Contrastive Language-Image Pre-trained model ( CLIP ) ( Radford et al. , 2021 ) and argue that they themselves pave a way to mitigate the associated risks . Specifically , we show that they encode implicit knowledge to infer offensiveness in images overcoming previous issues , namely the lack of adequate and sufficient training data . Furthermore , we demonstrate that our approach can be utilized to annotate offensive images in vision datasets and , therefore , reliably assist the curation process of such datasets . We illustrate our approach on the popular ImageNet-ILSVRC-2012 dataset and show that large computer vision datasets contain additional inappropriate content , which previous documentations had not detected . With our proposed method this content can be automatically and reliably pre-selected . As an example , Fig . 1 ( left ) shows an exemplary image from the ImageNet-ILSVRC-2012 validation set identified as misogynistic content —categorized as “ beach voyeur ” — by a hand-surveyed image selection in ( Birhane & Prabhu , 2021 ) . Next to this human-selected image , Birhane & Prabhu ( 2021 ) applied different models to detect visible faces ( thus violating privacy rights ) and pornographic content . However , as we will show with our study , further inappropriate images , which we refer to as offensive , can be identified within the dataset . For instance , Fig . 1 ( right ) shows sixteen handpicked images from a set of automatically detected possibly offensive images , utilizing our proposed approach . Depending on the task and stakeholders , this ranges from offensive objects such as weapons ( second row , first and fifth image ) and dead animals ( first row , sixth image ) to immoral actions such as harming or even killing animals ( second row , second image ) and humans ( second row , seventh image ) , as well as offensive text and symbols ( first row , third image ) . With our study we therefore strongly advocate for curating and documenting a dataset by the categories and models provided by Birhane & Prabhu ( 2021 ) but also by taking the possible general offensiveness in images into account . To this end , we provide our models and the necessary data to reproduce our experiments and utilize our proposed method1 . We proceed as follows . We start with a brief overview of related work and required background introducing pre-trained models and their successes as well as concerns raised . Next , we describe the term offensiveness and show that common deep models can not reliably detect offensive image content due to the lack of sufficient data . We then continue by demonstrating that recent models , guided by natural language during the pre-training phase , can infer offensiveness in images based on their implicit knowledge . Before concluding , we present our automated dataset curation exemplary on the ImageNet-ILSVRC-2012 dataset . 1Anonymous link placeholder . Please see supplement . 2 BACKGROUND AND RELATED WORK . Concerns about large-scale data sets . Pre-training has become an essential approach in many vision and language tasks . In the vision domain , pre-training on large-scale supervised data such as ImageNet ( Deng et al. , 2009 ) has shown to be crucial for enhancing performance on downstream tasks via transfer learning . Since these datasets contain millions of data samples , curating such pre-training datasets requires heavy work on data gathering , sampling , and human annotation , making it error-prone and difficult to scale . Moreover , in the language domain , task-agnostic objectives such as autoregressive ( Radford et al. , 2019 ) and masked language modeling ( Devlin et al. , 2019 ) have scaled across many orders of magnitude , especially in model capacity and data , steadily increasing performance but also the capabilities of deep models . With their standardized input-output ( text-totext ) interface Radford et al . ( 2019 ) have enabled zero-shot transfer to downstream datasets . Recent systems like GPT-3 ( Brown et al. , 2020 ) are now competitive across many tasks with specialized models whereas requiring only a small amount to no task-specific training data . Based on these advances , more recently , Radford et al . ( 2021 ) and Jia et al . ( 2021 ) introduced models with similar capabilities in the vision domain . However , pre-training such models requires particularly large-scale training data , and the datasets ’ curation process is tedious and error-prone . To tackle this issue Gebru et al . ( 2018 ) suggest to provide dataset audit cards to document datasets . This provides stakeholders the ability to understand training data characteristics in order to alleviate known as well as unknown issues . The authors argue that whereas documentation allows for potential accountability , undocumented training data perpetuates harm without recourse . Birhane & Prabhu ( 2021 ) provided such a dataset card for the popular computer-vision ImageNetILSVRC-2012 dataset , including several metrics and the hand surveyed identification of images with misogynistic content . More importantly , the authors raised the awareness of polluted image datasets by the example of pornographic content inside several popular computer vision benchmark datasets . Although the authors raised criticism against ImageNet and identified several inappropriate images , the ImageNet-ILSVRC-2012 dataset —and the pre-trained models— are still under the most popular datasets in the ML community . In line with Gebru et al . ( 2018 ) , Birhane & Prabhu ( 2021 ) urge that ethics checks for future dataset curation endeavors become an integral part of the human-in-the-loop validation phase . ImageNet . The ImageNet ( Deng et al. , 2009 ) data collection is one of the most popular datasets in the computer vision domain and mostly refers to the subset ImageNet1k dataset with 1.2 million images across 1000 classes . This was introduced in 2012 for the classification challenge in the ImageNet Large Scale Visual Recognition Challenge ( ILSVRC ) . However , in total the collection ( ImageNet21k ) covers over 14 million images spread across 21,841 classes . As Birhane & Prabhu ( 2021 ) state , the ImageNet dataset remains one of the most influential and powerful image databases available today , although it was created over a decade ago . To apply transfer learning , the most popular deep learning frameworks provide downloadable pre-trained models for ImageNet1k . Recently , Ridnik et al . ( 2021 ) provided a novel scheme for high-quality , efficient pre-training on ImageNet21k and , along with it , the resulting pre-trained models . Pre-training vision models with natural language supervision . Pre-training methods that learn directly from raw data have revolutionized many tasks in natural language processing and computer vision over the last few years . Radford et al . ( 2021 ) propose visual representation learning via natural language supervision in a contrastive learning setting . The authors collected over 400M image-text pairs ( WebImageText dataset ) to show that the improvement with large-scale transformer models in NLP can be transferred to vision . More precisely , whereas typical vision models jointly train an image feature extractor and a linear classifier , CLIP jointly trains an image encoder and a text encoder to predict the correct pairings of a batch of ( image , text ) training examples . At test time the authors propose to synthesize the learned text encoder with a ( zero-shot ) linear classifier by embedding the names or descriptions of the target dataset ’ s classes , e.g . “ The image shows < label > . ” . For simplicity , we refer to a model trained in a contrastive language-image pre-training setting and fine-tuned or probed for a downstream task as CLIP model . Closely related to CLIP , the ALIGN ( Jia et al. , 2021 ) model is a family of multimodal dual encoders that learn to represent images and text in a shared embedding space . Instead of Vision-Transformers ( ViT ) or ResNet models ALIGN uses the EfficientNet ( Tan & Le , 2019 ) and BERT ( Devlin et al. , 2019 ) models as vision and text encoders . These encoders are trained from scratch on image-text pairs ( 1.8B pairs ) via contrastive learning . These models and their zero-shot capabilities display significant promise for widely-applicable tasks like image retrieval or search ( Radford et al. , 2021 ) . For instance , since image and text are encoded in the same representational space , these models can find relevant images in a database given text or relevant text given an image . More importantly , the relative ease of steering CLIP toward various applications with little or no additional data or training unlocks novel applications that were difficult to solve with previous methods , e.g. , as we show , inferring the offensiveness in images . Carried Knowledge of Pre-trained Large-Scale Models . As already described , with training on raw text , large-scale transformer-based language models revolutionized many NLP tasks . Recently , Radford et al . ( 2021 ) , Ramesh et al . ( 2021 ) and Jia et al . ( 2021 ) presented encouraging results that a similar breakthrough in computer vision will be possible . Besides the performance improvements in generation , regression , and classification tasks , these large-scale language models show surprisingly strong abilities to recall factual knowledge present in the training data ( Petroni et al. , 2019 ) . Further , Roberts et al . ( 2020 ) showed that large-scale pre-trained language models ’ capability to store and retrieve knowledge scales with model size . Since such models are often trained on unfiltered data , the kind of knowledge acquired is not controlled , leading to possibly undesirable behavior such as stereotypical and derogatory associations . However , Schick et al . ( 2021 ) demonstrated that these models can recognize , to a considerable degree , their undesirable retained knowledge and the toxicity of the content they produce . The authors further showed that a language model with this ability can perform self-debiasing to reduce its probability of generating offensive text . Furthermore , Jentzsch et al . ( 2019 ) and Schramowski et al . ( 2020 ) even showed that the retained knowledge of such models carries information about moral norms aligning with the human sense of “ right ” and “ wrong ” expressed in language . Similar to ( Schick et al. , 2021 ) , Schramowski et al . ( 2021 ) demonstrated how to utilize this knowledge to guide autoregressive language models ’ text generation to prevent their toxic degeneration . In this work , we investigate if we are able to utilize the carried knowledge of large-scale vision models in a similar way , i.e . detecting possible offensive images in large-scale vision datasets .

Summary: The paper constructs a classifier of whether or not an image is offensive. This is operationalized by finding a dataset from the psychology community of a few thousand images and ordinal judgements of 'morality' from a study. Prediction of these judgements is predictably hard, in no small part because the data is small. To combat these issues, the paper uses CLIP and soft-prompt tuning. Soft-prompt tuning appears to be very effective for CLIP in this context reaching accuracies of over 95%, where baseline fine tuning only gets about 85%.