Inferring Offensiveness In Images From Natural Language Supervision

1 INTRODUCTION . Deep learning models yielded many improvements in several fields . Particularly , transfer learning from models pre-trained on large-scale supervised data has become common practice in many tasks both with and without sufficient data to train deep learning models . Whereas approaches like semisupervised sequence learning ( Dai & Le , 2015 ) and datasets such as ImageNet ( Deng et al. , 2009 ) , especially the ImageNet-ILSVRC-2012 dataset with 1.2 million images , established pre-training approaches , in the following years , the training data size increased rapidly to billions of training examples ( Brown et al. , 2020 ; Jia et al. , 2021 ) , steadily improving the capabilities of deep models . Recently , autoregressive ( Radford et al. , 2019 ) , masked language modeling ( Devlin et al. , 2019 ) as well as natural language guided vision models ( Radford et al. , 2021 ) have enabled zero-shot transfer to downstream datasets removing the need for dataset-specific customization . Besides the parameter size of these models , the immense size of training data has enabled deep learning models to achieve high accuracy on specific benchmarks in natural language processing ( NLP ) and computer vision ( CV ) applications . However , in both application areas , the training data has been shown to have problematic characteristics resulting in models that encode e.g . stereotypical and derogatory associations ( Gebru et al. , 2018 ; Bender et al. , 2021 ) . Unfortunately , the curation of these large datasets is tedious and error-prone . Pre-trained models ( PM ) used for downstream tasks such as face detection propagate retained knowledge to the downstream module e.g . the classifier . To raise the awareness of such issues , Gebru et al . ( 2018 ) described how large , uncurated , Internetbased datasets encode e.g . dominant and hegemonic views , which further harms people at the margins . The authors urge researchers and dataset creators to invest significant resource allocation towards dataset curation and documentation practices . As a result , Birhane & Prabhu ( 2021 ) provided modules to detect faces and post-process them to provide privacy , as well as a pornographic content classifier to remove inappropriate images . Furthermore , Birhane & Prabhu ( 2021 ) conducted a hand surveyed image selection to identify misogynistic images in the ImageNet-ILSVRC-2012 dataset . Unfortunately , such a curation process is tedious and does not scale to current dataset sizes . Moreover , misogynistic images , as well as pornographic content , are only two subsets of offensive images . It remains an open question how to infer general offensiveness in images , including abusive , indecent , obscene , or menacing content , and how to identify them in an automated dataset curation process . Whereas large image datasets automatically scraped from the web may contain derogatory terms as categories and offensive images , which results in models with undesirable behavior , pre-trained models may also reflect desirable implicit knowledge and biases such as our social , ethical , and moral choices ( Jentzsch et al. , 2019 ; Schramowski et al. , 2020 ) reflected within the training data . In our study , we investigate modern vision PMs trained on large-scale datasets , in particular , the Contrastive Language-Image Pre-trained model ( CLIP ) ( Radford et al. , 2021 ) and argue that they themselves pave a way to mitigate the associated risks . Specifically , we show that they encode implicit knowledge to infer offensiveness in images overcoming previous issues , namely the lack of adequate and sufficient training data . Furthermore , we demonstrate that our approach can be utilized to annotate offensive images in vision datasets and , therefore , reliably assist the curation process of such datasets . We illustrate our approach on the popular ImageNet-ILSVRC-2012 dataset and show that large computer vision datasets contain additional inappropriate content , which previous documentations had not detected . With our proposed method this content can be automatically and reliably pre-selected . As an example , Fig . 1 ( left ) shows an exemplary image from the ImageNet-ILSVRC-2012 validation set identified as misogynistic content —categorized as “ beach voyeur ” — by a hand-surveyed image selection in ( Birhane & Prabhu , 2021 ) . Next to this human-selected image , Birhane & Prabhu ( 2021 ) applied different models to detect visible faces ( thus violating privacy rights ) and pornographic content . However , as we will show with our study , further inappropriate images , which we refer to as offensive , can be identified within the dataset . For instance , Fig . 1 ( right ) shows sixteen handpicked images from a set of automatically detected possibly offensive images , utilizing our proposed approach . Depending on the task and stakeholders , this ranges from offensive objects such as weapons ( second row , first and fifth image ) and dead animals ( first row , sixth image ) to immoral actions such as harming or even killing animals ( second row , second image ) and humans ( second row , seventh image ) , as well as offensive text and symbols ( first row , third image ) . With our study we therefore strongly advocate for curating and documenting a dataset by the categories and models provided by Birhane & Prabhu ( 2021 ) but also by taking the possible general offensiveness in images into account . To this end , we provide our models and the necessary data to reproduce our experiments and utilize our proposed method1 . We proceed as follows . We start with a brief overview of related work and required background introducing pre-trained models and their successes as well as concerns raised . Next , we describe the term offensiveness and show that common deep models can not reliably detect offensive image content due to the lack of sufficient data . We then continue by demonstrating that recent models , guided by natural language during the pre-training phase , can infer offensiveness in images based on their implicit knowledge . Before concluding , we present our automated dataset curation exemplary on the ImageNet-ILSVRC-2012 dataset . 1Anonymous link placeholder . Please see supplement . 2 BACKGROUND AND RELATED WORK . Concerns about large-scale data sets . Pre-training has become an essential approach in many vision and language tasks . In the vision domain , pre-training on large-scale supervised data such as ImageNet ( Deng et al. , 2009 ) has shown to be crucial for enhancing performance on downstream tasks via transfer learning . Since these datasets contain millions of data samples , curating such pre-training datasets requires heavy work on data gathering , sampling , and human annotation , making it error-prone and difficult to scale . Moreover , in the language domain , task-agnostic objectives such as autoregressive ( Radford et al. , 2019 ) and masked language modeling ( Devlin et al. , 2019 ) have scaled across many orders of magnitude , especially in model capacity and data , steadily increasing performance but also the capabilities of deep models . With their standardized input-output ( text-totext ) interface Radford et al . ( 2019 ) have enabled zero-shot transfer to downstream datasets . Recent systems like GPT-3 ( Brown et al. , 2020 ) are now competitive across many tasks with specialized models whereas requiring only a small amount to no task-specific training data . Based on these advances , more recently , Radford et al . ( 2021 ) and Jia et al . ( 2021 ) introduced models with similar capabilities in the vision domain . However , pre-training such models requires particularly large-scale training data , and the datasets ’ curation process is tedious and error-prone . To tackle this issue Gebru et al . ( 2018 ) suggest to provide dataset audit cards to document datasets . This provides stakeholders the ability to understand training data characteristics in order to alleviate known as well as unknown issues . The authors argue that whereas documentation allows for potential accountability , undocumented training data perpetuates harm without recourse . Birhane & Prabhu ( 2021 ) provided such a dataset card for the popular computer-vision ImageNetILSVRC-2012 dataset , including several metrics and the hand surveyed identification of images with misogynistic content . More importantly , the authors raised the awareness of polluted image datasets by the example of pornographic content inside several popular computer vision benchmark datasets . Although the authors raised criticism against ImageNet and identified several inappropriate images , the ImageNet-ILSVRC-2012 dataset —and the pre-trained models— are still under the most popular datasets in the ML community . In line with Gebru et al . ( 2018 ) , Birhane & Prabhu ( 2021 ) urge that ethics checks for future dataset curation endeavors become an integral part of the human-in-the-loop validation phase . ImageNet . The ImageNet ( Deng et al. , 2009 ) data collection is one of the most popular datasets in the computer vision domain and mostly refers to the subset ImageNet1k dataset with 1.2 million images across 1000 classes . This was introduced in 2012 for the classification challenge in the ImageNet Large Scale Visual Recognition Challenge ( ILSVRC ) . However , in total the collection ( ImageNet21k ) covers over 14 million images spread across 21,841 classes . As Birhane & Prabhu ( 2021 ) state , the ImageNet dataset remains one of the most influential and powerful image databases available today , although it was created over a decade ago . To apply transfer learning , the most popular deep learning frameworks provide downloadable pre-trained models for ImageNet1k . Recently , Ridnik et al . ( 2021 ) provided a novel scheme for high-quality , efficient pre-training on ImageNet21k and , along with it , the resulting pre-trained models . Pre-training vision models with natural language supervision . Pre-training methods that learn directly from raw data have revolutionized many tasks in natural language processing and computer vision over the last few years . Radford et al . ( 2021 ) propose visual representation learning via natural language supervision in a contrastive learning setting . The authors collected over 400M image-text pairs ( WebImageText dataset ) to show that the improvement with large-scale transformer models in NLP can be transferred to vision . More precisely , whereas typical vision models jointly train an image feature extractor and a linear classifier , CLIP jointly trains an image encoder and a text encoder to predict the correct pairings of a batch of ( image , text ) training examples . At test time the authors propose to synthesize the learned text encoder with a ( zero-shot ) linear classifier by embedding the names or descriptions of the target dataset ’ s classes , e.g . “ The image shows < label > . ” . For simplicity , we refer to a model trained in a contrastive language-image pre-training setting and fine-tuned or probed for a downstream task as CLIP model . Closely related to CLIP , the ALIGN ( Jia et al. , 2021 ) model is a family of multimodal dual encoders that learn to represent images and text in a shared embedding space . Instead of Vision-Transformers ( ViT ) or ResNet models ALIGN uses the EfficientNet ( Tan & Le , 2019 ) and BERT ( Devlin et al. , 2019 ) models as vision and text encoders . These encoders are trained from scratch on image-text pairs ( 1.8B pairs ) via contrastive learning . These models and their zero-shot capabilities display significant promise for widely-applicable tasks like image retrieval or search ( Radford et al. , 2021 ) . For instance , since image and text are encoded in the same representational space , these models can find relevant images in a database given text or relevant text given an image . More importantly , the relative ease of steering CLIP toward various applications with little or no additional data or training unlocks novel applications that were difficult to solve with previous methods , e.g. , as we show , inferring the offensiveness in images . Carried Knowledge of Pre-trained Large-Scale Models . As already described , with training on raw text , large-scale transformer-based language models revolutionized many NLP tasks . Recently , Radford et al . ( 2021 ) , Ramesh et al . ( 2021 ) and Jia et al . ( 2021 ) presented encouraging results that a similar breakthrough in computer vision will be possible . Besides the performance improvements in generation , regression , and classification tasks , these large-scale language models show surprisingly strong abilities to recall factual knowledge present in the training data ( Petroni et al. , 2019 ) . Further , Roberts et al . ( 2020 ) showed that large-scale pre-trained language models ’ capability to store and retrieve knowledge scales with model size . Since such models are often trained on unfiltered data , the kind of knowledge acquired is not controlled , leading to possibly undesirable behavior such as stereotypical and derogatory associations . However , Schick et al . ( 2021 ) demonstrated that these models can recognize , to a considerable degree , their undesirable retained knowledge and the toxicity of the content they produce . The authors further showed that a language model with this ability can perform self-debiasing to reduce its probability of generating offensive text . Furthermore , Jentzsch et al . ( 2019 ) and Schramowski et al . ( 2020 ) even showed that the retained knowledge of such models carries information about moral norms aligning with the human sense of “ right ” and “ wrong ” expressed in language . Similar to ( Schick et al. , 2021 ) , Schramowski et al . ( 2021 ) demonstrated how to utilize this knowledge to guide autoregressive language models ’ text generation to prevent their toxic degeneration . In this work , we investigate if we are able to utilize the carried knowledge of large-scale vision models in a similar way , i.e . detecting possible offensive images in large-scale vision datasets .

Summary: This paper described an interesting idea to leverage massively pre-trained models (specifically CLIP-based models) to infer offensiveness in images. The authors first gave detailed literature review regarding a recently raising concern about inappropriate images in computer vision datasets, and also performed several finetuning/probing baseline experiments to illustrate the feasibility to detect such inappropriate contents. Next the authors explored the possibility to mitigate the potential risk by utilizing the implicit knowledge learned by CLIP-based pre- trained models. Lastly the authors choose ImageNet for a proof-of-concept validation and show that the proposed approach can discover previously neglected offensive images.