Self-supervised Contrastive Zero to Few-shot Learning from Small, Long-tailed Text data

1 INTRODUCTION . The current prevailing approach to supervised and few-shot learning is to use self-supervised pretraining on large-scale ‘ task-external ’ data and then fine-tune on end-task labels . Recent studies have found that , thus far , this way of pretraining fails in low-resource settings ( Yogatama et al. , 2019 ; Şerbetci et al. , 2020 ) and that reported performance improvements are caused in part by evaluation setups that are designed in line with the paradigm that “ massive resources are pivotal ” to improving language understanding ( Linzen , 2020 ; Schick & Schütze , 2020a ; Dodge et al. , 2020 ; Brown et al. , 2020 ) or computer vision ( Chen et al. , 2020 ) . Despite these critiques , the underlying goal of better initialisation of layer weights is a core requirement of successful learning with neural networks , where self-supervised layer-wise pretraining ( Bengio et al. , 2006 ) was replaced by better layer initialisation ( Glorot & Bengio , 2010 ) , which was in turn replaced by pretraining on growing amounts of external data ( Bojanowski et al. , 2017 ; Devlin et al. , 2019 ; Chen et al. , 2020 ; Brown et al. , 2020 ) – i.e . FastText , BERT , SIMCLR and GPT-3 . The latter three approaches require massive compute and data resources , but enable marked learning improvements in few-shot ( SIMCLR , GPT3 ) or zero-shot ( GPT-3 ) scenarios compared to models that have several orders of magnitude fewer parameters . There are efforts to reduce model size requirements for few and zero-shot adaptation by orders of magnitude ( Schick & Schütze , 2020a ; b ; Plank & Rethmeier , 2019 ) , with some being increasingly beneficial in scenarios with low input data ( X ) , label resources ( Y ) , and rare events in X , Y . Crucially , such approaches do not simply rely on more data , but on creating better initialised input features X . In contrast , approaches like SIMCLR or BERT ( Chen et al. , 2020 ; Devlin et al. , 2019 ) use self-supervision via contrastive learning and input masking on large-scale datasets to create broader learning signals than supervision provides . SIMCLR is based on a metric learning approach called contrastive self-supervision – i.e . learning to distinguish ( dis- ) similar inputs using generated , but weak supervision tasks . However , as Musgrave et al . ( 2020 ) find , “ when evaluating old vs. recent metric learning approaches , while controlling for data and model size , newer methods only marginally improve over the classic contrastive formulation ” . Remarkably , Bansal et al . ( 2020 ) recently showed that adding broader self-supervision rather than increasing data size during large-scale pretraining can substantially boost few-shot performance . Our central question is whether increased ( broader ) pretraining self-supervision also boosts few and zero-shot performance using only small-scale , ‘ task-internal ’ data , instead of resorting to largescale pretraining on orders of magnitude more ‘ task-external ’ data – i.e . Do we really need large datasets for pretraining or just more ( broader ) self-supervised learning signals ? To broaden small data self-supervision , we propose a contrastive self-supervised objective based on labelembedding prediction , where labels are expressed as word embeddings to learn their matching with an input text embedding . For contrastive learning , our method samples positive and negative word input tokens X for self-supervised pretraining , zero and few-shot learning ; and positive and negative classes Y for few-shot to fully supervised fine-tuning . Thus , we propose a model architecture that unifies training from labels Y and inputs X . To increase evaluation robustness , we compare models of the same parameter and data sizes as suggested by Musgrave et al . ( 2020 ) , and evaluate on a challenging learning problem as suggested by Linzen ( 2020 ) ; Hooker ( 2020 ) . Namely , we evaluate against a challenging low-resource , long-tailed , noisy multi-label data settings , where information is always limited , since the long tail grows with data size and because its modeling requires the majority of parameters ( Hooker et al. , 2020b ) . For robust evaluation , we use a typical training , development , test setup and first establish a solid , supervised baseline for many-class multi-label classification that is optimised with a set of generalisation techniques proposed by Jiang et al . ( 2020 ) . For evaluation in supervised , few and zero-shot learning scenarios , we analyse and propose evaluation metric choices which are meaningful across all scenarios for broader performance comparisons . Contributions : 1 We provide a straight-forward method for self-supervised contrastive labelembedding prediction and 2 evaluate it in a challenging , noisy long-tail , low-resource multi-label text prediction scenario . 3 We show that small-scale ‘ data-internal ’ pretraining ( on 8-80MB of text ) not only improves supervised performance , but also strongly boosts few and zero-shot learning by increasing self-supervision amounts for small data , rather than increasing data amounts via the standard large-scale external data pretraining approach . 2 RELATED WORK . Large to Web-scale data pretraining is at the core of state-of-the-art methods in computer vision ( Chen et al. , 2020 ) and language processing ( Rogers et al. , 2020 ; Brown et al. , 2020 ) . However , challenges and disadvantages are increasingly being discussed . ( i ) A requirement of large-scale external text data resources ( Yogatama et al. , 2019 ; Schick & Schütze , 2020a ) , ( ii ) an inability to pretrain recent architectures on small-scale data ( Liu et al. , 2020 ; Melis et al. , 2020 ; Şerbetci et al. , 2020 ) , ( iii ) calls for more challenging evaluation tasks ( Linzen , 2020 ; McCoy et al. , 2019 ) and ( iv ) diminishing returns of pretraining on large supervised datasets ( Wang et al. , 2020b ) . To address issue ( iii ) , challenging evaluations on long-tail prediction ( Chang et al. , 2019 ) , few-shot ( Schick & Schütze , 2020a ) , or zero-shot ( Brown et al. , 2020 ) , were recently shown to benefit from self-supervised pretraining , but to-date , require massive , ‘ task-external ’ , pretraining datasets . ( c ) Remarkably , Bansal et al . ( 2020 ) showed that for large ‘ data-external ’ pretraining , using more self-supervision , not more data , also boosts few-shot performance . This finding inspired us to collect evidence towards a core question : “ Do we need massive data ( signals ) or just more ( diverse ) self-supervised learning signals for pretraining ? ” . We collect evidence by posing three research questions and propose solutions that require designing approaches for issues ( i-iii ) as follows . One , to address issue ( i ) , “ can increasing self-supervision signals during ‘ data-internal ’ pretraining on small data , i.e . without large-scale ‘ data-external ’ pretraining , boost few and zero-shot performance ” ? Two , to address issue ( ii ) , “ what pretraining objectives and models do we chose that work without large training data ” ? Three , to address issue ( iii ) , “ within what challenging learning scenario should we evaluate while incorporating the now standard “ any NLP task as a ‘ text-to-text ’ problem ” paradigm ( Raffel et al. , 2020 ) ” ? Fortunately , existing techniques can be extended to address these issues . For example , supervised label embedding prediction ( pre- ) training enables few and zero-shot learning of subsequent ( unseen ) supervised tasks . However , this requires the first ( pre-training ) task to be supervised , unlike recent large scale self-supervised pretraining methods . Large-scale , self-supervised pretraining and label embeddings can be combined ( Chang et al. , 2019 ) to fine-tune externally pretrained BERT models via label embedding prediction to boost long-tail task performance . However , BERTs ’ contextualized word embeddings did not work as label embeddings ELMOs ’ word embeddings had to be used ( 3.2 ) , further increasing resource requirements . Even worse , when comparing language model pretraining on small text corpora , Transformers ( Wang et al. , 2020a ) largely underperform CNNs and LSTMs ( Merity et al. , 2017 ) . Fortunately , Liu et al . ( 2017 ) established that label-embedding prediction CNNs boost long-tail prediction , even without modern self-supervision or using large ‘ task-external ’ data pretraining . Further , Pappas & Henderson ( 2019 ) ; Zhang et al . ( 2018b ) used supervised text label-embedding ( pre- ) training and investigated transfer to subsequent supervised tasks , though not under long-tail evaluation . Here , label embeddings are average word embeddings over label description words – i.e . label descriptions are required . The former added noise contrastive estimation ( NCE ) ( Ma & Collins , 2018 ) via negative sampling of labels to zero-shot predict rare , unseen classes post supervised pretraining on seen classes . Later , Jiang et al . ( 2019 ) adapted the same idea for zero-shot image classification via supervised pretraining on pairs of ‘ source ’ images and ‘ source ’ text label descriptions . They reduced overfitting by additionally pretraining on pairs of ‘ source ’ image and most similar ‘ zero-shot target class ’ text descriptions – though this is not technically zero-shot learning because sensible target label text descriptions have to be provided , which when unknown ( zero-shot ) , again leads to the long-tail issue . All these approaches are loosely based on Matching Networks by Vinyals et al . ( 2016 ) and add various training objectives . We thus combine the advantages of self-supervised pretraining for large data with supervised label embedding prediction for smaller data to propose a contrastive self-supervised pretraining via label-embedding prediction usable for small data pretraining . We extend the supervised label embedding baseline method by Zhang et al . ( 2018b ) and add four important changes . First , we combine label and word embedding look-up tables into one table , as this pushes input words and label ( -words ) to remain in a shared vector space during training , when predicting dense label ( -word ) embeddings from dense input word embeddings . This ‘ dense-to-dense ’ prediction of words to label ( -words ) follows the current “ any NLP task as a ‘ text-to-text ’ prediction ” paradigm ( Raffel et al. , 2020 ) , but avoids constant dense-to-sparse translation into label distributions via a compute intensive softmax . Secondly , we thus use a noise contrastive estimation ( NCE ) objective ( Ma & Collins , 2018 ) , replacing softmax normalization with negative sampling of ( supervision ) labels . Combining NCE and label embeddings allows predicting arbitrarily large class set ( long-tails ) and unseen classes . While Pappas & Henderson ( 2019 ) used NCE for supervised label pretraining , we add self-supervised pseudo-label ( word ) pretraining . Because labels and input words occupy the same vector space , we can use pseudo-labels ( words ) for self-supervised pretraining by sampling positive words from a current text instance , and negative words from adjacent text instances within a mini-batch . Three , we chose to sample from within a batch to reduce reliance ( training bias ) on knowing or expecting future and past word or label distribution statistics for the whole dataset , since in a zero-shot evaluation scenario unseen label and input word statistics are unknown . This also adds subsequent learning flexibility because no statistics collection preprocessing is required . Fourth , we add k-max pooling as in the CNN long-tail research by Liu et al . ( 2017 ) , because it helps during zero-shot learning . Such label-embedding based self-supervised pretraining has multiple advantages . It does not require large or external resources as in ( i ) . Its small ‘ data-internal ’ self-supervised word pseudo label pretraining addresses issue ( ii ) and enables unsupervised zero-shot learning . It also markedly boosts few-shot performance without requiring task external supervised annotations as in ( i ) or supervised embedding transfer as in Pappas & Henderson ( 2019 ) ; Zhang et al . ( 2018b ) ; Jiang et al . ( 2019 ) . Since label embeddings are a common long-tail prediction technique , which addresses issue ( iii ) , it makes our approach suitable for low-resource , long-tail learning without task external labels or large-scale annotated datasets . Finally , label embedding NCE training allows for ( dense ) ‘ text-totext ’ training , making it applicable to a variety of tasks . We demonstrate the benefits of such a selfsupervised pretraining method and model for self-supervised zero-shot learning ( inputX-efficiency ) §6.4 or few-shot learning ( label Y -efficiency ) §6.3 . coded by the same word embedding layer E 1 , where labels have word IDs for lookup . The text embeddings are then encoded by a sequence encoder T 2 , while c labels are encoded by a label encoder L 3 . Each text has multiple labels , so the text encoding ti is repeated for , and concatenated with , each label encoding l◦i , l . The resulting batch of ‘ text-embedding , label-embedding ’ pairs [ [ ti , l ◦ i,1 ] , . . . , [ ti , l ◦ i , c ] ] 4 is fed into a ‘ matcher ’ classifier 5 that trains a binary cross entropy loss 6 on multiple ( pseudo- ) label ( mis- ) matches { 0 , 1 } for each text instance ti , resulting in a noise contrastive estimation objective ( NCE ) . Words like ‘ measuring ’ provide self-supervised pseudo-labels ( left ) . Positive and negative ( pseudo- ) labels are sampled from their own or other instances in a minibatch . Unlike Zhang et al . ( 2018a ) we use a CNN for 2 , negative sampling and self-supervision .

Summary: This paper proposes a contrastive autoencoder approach that only requires small data to perform a multi-label classification on the long-tail problem. They introduce a matching network to compare text and label embeddings and calculate the probabilities of the label given the input. The proposed idea is very straightforward by combining a matching network with contrastive learning to give broader signals. The goal of this work is to enable zero-shot and few-shot learning with very few resources as a more sustainable approach to machine learning applications.