Self-supervised Contrastive Zero to Few-shot Learning from Small, Long-tailed Text data
Authors: Nils Rethmeier
Isabelle Augenstein
Venue:  (2021)
Keywords: self-supervised pretraining
zero-shot
few-shot
text-to-text
contrastive self-supervised learning
small data
long-tail
multi-label classification
NLP

Abstract: For natural language processing (NLP) ‘text-to-text’ tasks, prevailing approaches heavily rely on pretraining large self-supervised models on massive external datasources.  However, this methodology is being critiqued for: exceptional compute and pretraining data requirements; diminishing returns on both large and small datasets; and importantly, favourable evaluation settings that overestimate performance differences.  The core belief behind current methodology, coined 'the bitter lesson' by R. Sutton, is that 'compute scale-up beats data and compute-efficient algorithms', neglecting that progress in compute hardware scale-up is based almost entirely on the miniaturisation of resource consumption.  We thus approach pretraining from a miniaturisation perspective, such as not to require massive external data sources and models, or learned translations from continuous input embeddings to discrete labels.  To minimise overly favourable evaluation, we examine learning on a long-tailed, low-resource, multi-label text classification dataset with noisy, highly sparse labels and many rare concepts.  To this end, we propose a novel 'dataset-internal' contrastive autoencoding approach to self-supervised pretraining and demonstrate marked improvements in zero-shot, few-shot and solely supervised learning performance; even under an unfavorable low-resource scenario, and without defaulting to large-scale external datasets for self-supervision.  We also find empirical evidence that zero and few-shot learning markedly benefit from adding more 'dataset-internal', self-supervised training signals, which is of practical importance when retrieving or computing on large external sources of such signals is infeasible.

Under review as a conference paper at ICLR 2021
SELF-SUPERVISED CONTRASTIVE ZERO TO FEW-SHOT LEARNING FROM SMALL, LONG-TAILED TEXT DATA
Anonymous authors Paper under double-blind review
ABSTRACT
For natural language processing (NLP) 'text-to-text" tasks, prevailing approaches heavily rely on pretraining large self-supervised models on massive external data sources. However, this methodology is being critiqued for: exceptional compute and pretraining data requirements; diminishing returns on both large and small datasets; and evaluation settings that overestimate performance differences. The core belief behind current methodology, coined 'the bitter lesson' by R. Sutton, is that "compute scale-up beats data and compute-cfficient algorithms'' neglecting that progress in compute hardware scale-up is based near entirely on the minia- turisation of resource consumption. We thus approach pretraining from a minia- turisation perspective, such as not to require massive external data sources and models, and avoid translations from continuous input embeddings to discrete la- bels. To minimise favourable evaluation, we examine learning a challenging long-tailed, low-ress low-resource, multi-label text classification dataset with noisy, highly sparse labels and many rare concepts. To this end, propose using a 'dataset- internal,, self-supervised contrastive autoencoding approach for pretraining that enables marked improvements in zero-shot, few-shot and supervised learning per- formance; even under a challenging, otherwise avoided, low-resource scenario, without defaulting large-scale external datasets as support training signals. Cru- cially, we find evidence that zero and few-shot learning markedly benefit from adding more Kataset-internal', sepe-supe training signals, e.g. when increas- ing self- supervised learning signals via large external sources is infeasible.
INTRODUCTION
The current prevailing approach to supervised and few-shot learning is to use self-supervised pre- training on large-scale 'task-external' data and then fine-tune on end-task labels. Recent studies have found that, thus far, this way of pretraining fails in low-resource settings (Yogatama et al.) 2019 Şerbetci et al. 2020) and that reported performance improvements are caused in part by eval- uation setups that are designed in line with the paradigm that "massive resources are pivotal" to improving language understanding (Linzen] 2020; Schick Schüitze, 2020a Dodge et al. 2020 Brown et al. 2020) or computer vision (Chen et al. 2020). Despite these critiques, the underlying goal of better initialisation of layer weights is a core requirement of successful learning with neural networks, where self-supervised layer- wise pretraining (Bengio al.
2006) was replaced by better layer initialisation (Glorot & Bengio 2010). which was in turn replaced by pretraining on grow- ing amounts of external data (Bojanowski et al. 2017 Devlin al. 2019 Chen et al.) 20200] Brown et al. 2020) - i.e. FastText, BERT, SIMCLR and GPT-3. The latter three approaches require massive compute and data resources, but enable marked learning improvements in few-shot (SIMCLR, GPT- 3) or zero-shot (GPT-3) scenarios compared to models that have several orders of magnitude fewer parameters. There are efforts to reduce model size requirements for few and zero-shot adaptation by orders of magnitude (Schick & Schütze) 2020a by Plank & Rethmeier 2019). with some being increasingly beneficial in scenarios with low input data (X), label resources (Y), and rare events in X, Y. Crucially, such approaches do not simply rely on more data, but creating better ini- tialised input features X. In contrast, approaches like SIMCLR or BERT (Chen et al. 2020 Devlin et al. 2019) use self-supervision via contrastive learning and input masking on large-scale datasets to create broader learning signals than supervision provides. SIMCLR based on a metric learning approach called contrastive self-supervision i.e. learning distinguish (dis-)similar inputs using

---

Under review as a conference paper at ICLR 2021
generated, but weak supervision tasks. However, as Musgrave al. (2020) find, "when evaluating old vs. recent metric learning approaches, while controlling for data and model size, newer meth- ods only marginally improve over the classic contrastive formulation". Remarkably, [aanall et al. (2200) recently showed that adding broader self-superviiion rather than increasing data size during large-Scale pretraining can substantially boost few-shot performance.
Our central question is whether increased (broader) pretraining self-superviison also boosts few and zero-shot performance using only small-scale, 'task-internal" data, instead of resorting to large- scale pretraining on orders of magnitude more 'lask-external' data i.e. Do we really need large datasets for pretraining or just more (broader) self-supervised learning signals? To broaden small data self-supervision, we propose a contrastive self-supervised objective based on label- embedding prediction, where labels are expressed as word embeddings to learn their matching with an input text embedding. For contrastive learning, our method samples positive and negative word input tokens X for self-supervised pretraining, zero and few-shot learning; and positive and nega- tive classes Y for few-shot to fully supervised fine-tuning. Thus, we propose model architecture that unifies training from labels Y and inputs X. To increase evaluation robustness, we compare models of the same parameter and data sizes as suggested by Musgrave et al. (2020), and eval- uate on a challenging learning problem as suggested by [inzen (2020); Hooker (2020). Namely, evaluate against challenging low-resource, long-tailed, noisy multi-label data settings, where information is always limited, since the long tail grows with data size and because its modeling requires the majority of parameters (Hooker et al. (020b). For robust evaluation, we use typi- cal training, development, test setup and first establish a solid, supervised baseline for mans-class multi-label classification that is optimised with set of generalisation techniques proposed by [Jiang et al. (2020). For evaluation in supervised, few and zero-shot learning scenarios, we analyse and pro- pose evaluation metric choices which are meaningful across all scenarios for broader performance comparisons.
Contributions: I We provide straight-forward method for self-supervised contrastive label- embedding prediction and ② evaluate it in challenging, noisy long-tail, low-resource multi-label text prediction scenario. ② We show that small-scale 'data-internal' pretraining (on 8-80MB of text) not only improves supervised performance, but also strongly boosts few and zero-shot learning by increasing self-supervision amounts for small data, rather than increasing data amounts via the standard large-scale external data pretraining approach.
RELATED WORK
Large to Web-scale data pretraining is the core of state-of-the-ar methods in computer vision (Chen et al. (2020) and language processing (Rogers al. 20200 Brown et al.] 2020). However, chal- lenges and disadvantages are increasingly being discussed. (i) requirement of large-scale external text data resources (Yogatama et al.] 2019 [Schick & Schüize 2020a). (ii) an inability to pretrain recent architectures on small-scale data (Liu et 2020 Mclis et al. 2020: Şerbetci et al. 2020), (iii) calls for more challenging evaluation tasks (Linzen] 2020; McCoy et al. 2019) and (iv) dimin- ishing returns of pretraining on large supervised datasets Wang et al. 2020b).
To address issue (iii), challenging evaluations long-tu tail prediction (Chang et al. 2019). few-shot (Schick & Schütze 2020a), or zero-shot (Brown et al. (020). were recently shown to benefit from self-supervised pre- training, but to-date, require massive, "task-ettennll', pretraining dataseís. (c) Remarkably, [Bansal et al. (020)) showed that for large 'data-external' pretraining, using more self-supervision, not more data, also boosts few-shot performance. This finding inspired us to collect evidence towards a core question: "Do we need massive data (signals) or just more (diverse) self-supervised learn- ing signals for pretraining?". We collect evidence by posing three research questions and propose solutions that require designing approaches for issues (i-iii) as follows. One, to address issue (i), "can increasing self-supervision signals during 'data-internal' pretraining on small data, i.e. without large-scale 'data-external" pretraining, boost few and zero-shot performance"? Two, to address issue (ii), "what pretraining objectives and models do we chose that work without large training data"? Three, to address issue (iii,, "within what challenging learning scenario should we evaluate while incorporating the now standard "any NLP task as a 'text-to-text' problem" paradigm (Raffel et al. 2020 "?
2

---

Under review as a conference paper at ICLR 2021
Fortunately, existing techniques can be extended to address these issues. For example, supervised label embedding prediction (pre-)traiming )training enables few and zero-shot learning of subsequent (unseen) supervised tasks. However, this requires the first (pre-training) task to be supervised, unlike recent large scale self-supervised pretraining methods. Large-scale, self-supervised pretraining and label embeddings can be combined (Chang al.] (019 to fine-tune externally pretrained BERT models via label embedding prediction to boost long-tail task performance. However, BERTs' contextu- alized word embeddings did not work as label embeddings ELMOs' word embeddings had to be used (3.2), further increasing resource requirements. Even worse, when comparing language model pretraining on small text corpora, Transformers (Wang et al.] 2020a) largely underperform CNNs and LSTMs (Merity et al.] 2017). Fortunately, [Liu et al. (2017) 7) established that label-embedding prediction CNNs boost long-tail prediction, even without modern self- eeff-supervision or using large "takk-external' data pretraining- Further, [Pappas & Henderson (2019): Zhang et al.] (2018b) used supervised text label-embedding (pre-)training and investigated transfer to subsequent supervised tasks, though not under long-tail evaluation. Here, label embeddings are average word embeddings over label description words - i.e. label descriptions are required. The former added noise con- trastive estimation (NCE) (Ma & Collins 2018 via negative sampling of labels to zero-shot predict rare, unseen classes post supervised pretraining on seen classes. Later, Jiang et al. (2019) adapted the same idea for zero-shot image classification via supervised pretraining on pairs of "source" im- ages and 'source' text label descriptions. They reduced overfitting by additionally pretraining on pairs of 'source' image and most similar 'zero-shot target class' text descriptions though this is not technically zero-shot learning because sensible target label text descriptions have to be provided, which when unknown (zero-shot), again leads to the long-tail issue. All these approaches are loosely based on Matching Networks by Vinyals et al. (2016) and add various training objectives.
We thus combine the advantages of self-supervised pretraining for large data with supervised label embedding prediction for smaller data to propose contrastive self-supervised pretrain- ing via -abe-eemeddiigg prediction usable for small data pretraining. We extend the super- vised label embedding baseline method by Zhang et al.] (2018b) and add four important changes. First, we combine label and word embedding look-up tables into one table, as this pushes input words and
to remain in shared vector space during training, when predicting dense label(-word) embeddings from dense input word embeddings. This "dense-to-dense" prediction of words label(-words, follows the current "any NLP task 'text-to-text' prediction" paradigm RRffel et al.] 2020), but avoids constant dense-to-sparse translation into label distributions via a compute intensive softmax. Secondly, we thus use noise contrastive estimation (NCE) objective (Ma & Collins, 2018 replacing softmax normalization with negative sampling of (supervision) la- bels. Combining NCE and label embeddings allows predicting arbitrarily large class set doog-ttails) and unseen classes. While Pappas & Henderson (2019) used NCE for supervised label pretraining, we add self-supervised pseudo-lahel (word) pretraining. Because labels and input words occupy the same vector space, we can use pseudo-labels (words) for self-supervised pretraining by sampling positive words from a current text instance, and negative words from adjacent text instances within mini-batch. Three, we chose to sample from within a batch to reduce reliance (training bias) on knowing or expecting future and past word or label distribution statistics for the whole dataset, since in zero-shot evaluation scenario unseen label and input word statistics are unknown. This also adds subsequent learning flexibility because no statistics collection preprocessing is required. Fourth, we add k-max pooling in the CNN long-tail research by Liu et al. (2017), because it helps during zero-shot learning.
Such label-embedding based self- self-supervised pretraining has multiple advantages. It does not require large or external resources as in (i). Its small 'data-internal' self-supervised word pseudo label pretraining addresses issue (ii) and enables unsupervised zero-shot learning. It also markedly boosts few-shot performance without requiring task external supervised annotations as in (i) or supervised embedding transfer as in [Pappas & Henderson (2019): Zhang et al.] (2018b); Diang et al. (2019) Since label embeddings are common long-tail prediction technique, which addresses issue (iii), makes our approach suitable for low-resource, long-tail learning without task external labels or seage-scale&ane annotated datasets. Finally, label embedding NCE training allows for (dense) "text-to- text' training, making it applicable to a variety of tasks. We demonstrate the benefits of such a self- supervised pretraining method and model for self-supervised zero-shot learning (input X X-efficiency) 66.4 or few-shot learning (label Y -efficiency) [63
3

---

Under review as a conference paper at ICLR 2021
text sequence encoder
Lrepeat lext lext-encoding for
labels
training labels
measuring
piom
only ane closs match yesnno (BCE)
positive
interaction
text2label matcher net
V1
Y1
Y
pp
y'
negative
interaction
label encoder
pos: pecudo
p-value
fixed mumber of classess/ labels
neg. pscudo
ye
matcher model: A text ('measuring interaction',, ), and positíve ('interaction', R) or negative labels ('p-value") are en- coded by the same word embedding layer O. where labels have word IDs for lookup. The text embeddings are then encoded by sequence encoder T' @, while labels are encoded by label encoder L 3 Each text has multiple labels, so the text encoding t; is repeated for, and concate- nated with, each label encoding L The resulting batch of "text-embedding. label-embedding' pairs R,,l ④ is fed into 'matcher' classifier (5) that trains binary cross entropy loss ④ on multiple (pseudo-)latee Jabel (mis-)matches {0, 1} for each text instance t,, resulting in a noise con- trastive estimation objective (NCE). Words like 'measuring' provide self-supervised pseudo-labels (left). Positive and negative (pseudo-)labed labels are sampled from their own or other instances in a mini- batch. Unlike Zhang et al. (2018a) a CNN for @2, negative sampling and self- self-supervisiin.
Figure 1: Contrastive
[it;, .... [
SELF-SUPERVISED, CONTRASTIVE DENSE-TO-DENSS TEXT PREDICTION
In this section, propose to label- embeeddnb,,,, previously used for supervised learning only (Pappas & Henderson 2019 Zhang et al.] 2018b), and exploit them for self reif-supervieed contrastive pretraining small-scale data. This enables contrastive self-superviser pretraining somewhat sim- ilar to methods used for large-scale models. However, we only use small-scale "task-internal" data for pretraining, which requires orders of magnitude less data and compute than large-scale, 'task- external' pretraining approaches. Most NLP models translate back and forth between discrete words and continuous token embeddings, often using a softmax computation that is limited to predicting classes known at training time. To ease learning from small data, our first core idea is that text input words w; x and labels w²,1 should be mapped into the same word representation space, i.e. drawn from a shared embedding look-up table E, replace dense to sparse translations with embedding- to- no-embedding matching. Thus turns NLP from a discrete 'text-to-text" tasks, as proposed in Raffel al.) (2020), into a 'dense(text)-tor tonnsseseee-----ddese(eeeee task. We thus replace learning instance labels y by their corpus-internally pretrained FastText or randomly initialised word embeddings ! E L, while others (Pappas & Henderson (219) use text deseriptions to form label embeddings as the vector average over description word embeddings. As a result, pretraining word embeddings also pretrains (favourably initialising) label embeeldings. Unknown labels words), in turn, can be inferred via methods like FastText subword embeddings (Bojanowski et al. 2017).
As outlined visually, left to right in Fig. learning multi-label classification then becomes a con- trastive learning problem of matching the word- sord-sequence embedding t; of text i ②, with its la- bel (word-sequence) embeddings 1 = (L,,,.... I,,b Q by feeding e text-vs-label combinations [tt.,,,,...... It. Ieell 4 to a binary classifier M 2 for matching. This means that instead of predicting classes at once, we predict a batch of c, single-class, binary classifications using bi- nary cross entropy ④, where e needs not be constant across instances i. The details of steps I to ④ are as follows. To train a binary classifier, we need both positive and negative labels. Thus, for each text instance w; {Wa....w. we want to classify, we need positive labels w, (w.t...wwt w E R and b negative labels w. {wi .... ....,.. E R²b to form a la- bel selection vector unf {w+ b w- RR++b To indicate positive and negative labels, also need a g sized vector of ones E RS and a b sized zero vector 0 E RR, to get a class indicator I, = {1 + 0} E g-gh+ Both the text (word) indices w; and the label indices wf passed through shared 'word-or-label embedding' look-up-tabe E Q after which they are passed through their respective encoder networks T as text-sequence encoder, L as label en- coder. Thus, the text-encoder produces (single) text embedding vector T(F(w:)) per text
4

---

Under review as a conference paper at ICLR 2021
instance i @2. The label-encoder produces c g + n label embedding vectors (1;) that form a label-embedding matrix L, = [Ij...., - 1,] < T. I(((ww)) + As text-encoder T we use a (CNN--max--ppoolin poonax----ooling--Reeeee (ooolinggRReLUU) sub-network, while the label-encoder is simply an (average-pool) operation, since single label (wi,). e.g. 'multi'-'label" can consist of multiple words. To compare how similar the text-embedding t; is to each label-embedding i we re- peat t, times and combine text and label embeddings to get fext-vs-label-emeed
matrix M. = [[4,1, 4.]..........., ......,,i.. di]] 4 that is passed into the matcher network M 2 to produce batch of probabilities p, (MM
④ As the optimisation loss, we the use binary cross entropy (BCE) between P; and I,, i.e. 1 -i=1 I4a- (..-(g())..) + (1 - I log(1 - Pt.). Sum- ming BCE over positive and negative (pseudo-)labeli) is referred to noise contrastive estimation, as used in representation learning methods across fields (Ma & Collins 2018).
o (M
Via pseudo-label embedding pretraining, a model can predict supervised labels absent prior super- vision. This exploits both transfer learning from inputs and labels. using the matcher as a learned similarity function. Positive labels W." can be supervision labels. Negative labels w, can be sam- pled from the positive labels of other instances w in the same batch, which avoids needing know the label set beforehand. Since labels are words, we can sample positive words from the current and negative words from other text instances to get pseudo-labels. Sampling pseudo llaels provides a straight-forward contrastive, partial autoencoding mechanism usable as selffsupervissin in pre- training or as zero-shot learner. Because both real and pseudo labels are sampled words, the model does not need to distinguish between them. Instead, learning is controlled by an out-of-modee sam- pling routine for real supervision and pscudo self- sulf-superviiion labels. This leads to a second core idea: once inputs X and outputs Y are well initialised, the model θ can also be better initialised by pretraining via self- self-suuervision As a result, we can learn supervised, few and zero-shot tasks in unified manner.
SMALL, LONG-TAILED DENSE-TO-DENSE TEXT (LABEL) PREDICTION
Since it is our goal to research better zero and few-shot learning approuches for small 'text-to-text' pretraining models, we choose a small multi-label question tag prediction dataset as a test bed. We use the "Questions from Cross Validated dataset, where machine learning concepts are tagged per question. This dataset fulfills three requirements: it is small-scale, Long-tailed, and entails solving challenging, noisy 'text-to-text" prediction task. There is currently published baseline for this task. The classes (tags) and input words are highly long-tailed (imbalanced). The first 20% of labels occur in only 7 'head' classes. Tags are highly sparse at most 4 out of 1315 tags are labelled per question. Word embeddings are pretrained with FastText - details in appendix App. A..4 We use the labelled questions part of the dataset, which has 85k questions and 244k labels. What makes this problem particularly challenging is that 80% of the least frequent labels are distributed over 99.5% of classes, as an extreme long tail. The label density (% of active labels per question) only 0.22% or x 2.8/1305 1305 possible classes per instance. For a realistic evaluation setting, we split the dataset diachronically, using the 80% earliest documents for training, the next 10% for development, and the last 10% for testing.
Why not large external pretraining? Real-world, long-tailes datasets are always dominated by a low-leamiig-rrssurce problem for most classes. This makes two things obvious: (A) that model learning cannot simply be solved by using massive data sets as the long-tuil problem grows as well: (B) that studying supervupd pretraining challenging, but smaller, long-taile datasets such as this one, is useful for assessing an approach's 's ability to learn from complex, real-world data. Massive data pretraining masks and thus prevents studying these effects. We thus evaluate the effects of self-supervision in a noisy low-resource setup, also as a response to recent critiques of the evaluation metrics used to assess Web-scale learning (Linzen| 2020 Yogatama et al.) (209)). As McCoy et al. (2019) shows, these evaluation setups are solvable by large-scale pattern overfitting, which, they find, leads 'Clever Hans effect', rather than real task progress.
5

---

Under review as a conference paper at ICLR 2021
5 EXPERIMENTAL SETUP AND METRICS
We want to analyse the benefits of self-uuprrvisoon for fully supervised, (b) few and (c) zero-shot learning noisy low-resource, long-tailed, multi-label classification setting. In this section, we describe suitable evaluation metrics, then discuss results in the next section.
Long-tail evaluation metries and challenges: Long-tail, multi-label classification is challenging to evaluate. Many classification metrics are unsuitable for evaluating long-tailed datasets. They either: (i) misrepresent performance under class imbalance; (ii) do not scale to many classes; or (iii) are only meaningful if the desirable number of classes per instance is known (multi-label clas- sification). For problem (i) ROCAUC AUC is known to overestimate imbalanced performance (Davis & Goadrich 2006 Fernández et al. 2018), e.g. ROCAUC test scores were upwards of .98 for most of our models. For problem (it), measures such as F-score require discretisation threshold search for imbalanced prediction problems, i.e. searching for the optimal threshold per class (on devel- opment set), which becomes computationally infeasible. Simply using a 0.5 probability threshold drives model selection towards balanced prediction, mismatching the long-tail problem. Metrics like precision @k handle problem (i-ii), but require knowledge of k. i.c. problem (ii):: these metrics can only compare a chosen number of labels k and cannot handle cases where the correct num- ber of labels per instance varies or is unknown (label distribution shift). To more reliably measure performance under imbalance (i), to avoid unscalable class decision thresholding (ii), and to not optimise models for a set number of labels k per instance (iii), we use the average-precision (AP) score. is defined as AP = -.. (R,, Rn-1)) 1). where Pn and Rr are the precision and recall at the ruth threshold. AP measures classifier performance over all decision thresholds, is computa- tionally cheaper than threshold search, and allows for a dynamic number of labels per class. This latter property makes this task especially hard. A model has to leam when to predict label. at what rarity, and how many such labels to predict for each instance. We also report the macro-averaged Brier-Score (BS) over all classes, Though more accurate measures exist, computing them is more involved and they require additional evaluation labour when optimising a specific supervised dataset, which is not our goal. For both measures, we use their popular seikit-learn implementationg?
scalable, compute-efficiemt measure of classifier calibration.
A challenging task, even for humans: On the dataset it is hard to guess how many labels per question to tag and how specific they should be, especially without domain knowledge. Out of the different weighting schemes for average precision, we choose AP APiieeo and AP, macroo as they are the most pessimistic (hardest to increase) measures to reduce optimistic evaluation. This choice is motivated by the goal of this work, which is to not simply to push end-task performance, but to use supervised learning scores as proxy to evaluate the effects of pretraining on zero-shot learning as well as data-efficiency and speed of supervised and few-shot learning.
6 RESULTS
In this section, we first analyse a normal and a strong supervised baseline to minimise favourable comparison against subsequently evaluated label-embedding and self-supervision enhanced ap- proaches. Finally, we analyse the benefits of 'dataset-interaal pretraining for few-shot learning, and how the amount of pretraining learning signal and model size affect zero-shot learning. Test scores are reported according to the best dev set average precision score AP Ammiero over all classes.
6.1 BASELINE MODEL RESULTS
In this section, we establish baseline results (BASE) for non-leamnnng majority class baseline (Ze- roR), a common ("weak") CNN baseline trained with binary-cross-entropp and a solid CNN baseline optimised using set of generalisation techniques proposed by [Tang et al. (2020). The ZeroR classi- fier is useful for establishing a baseline performance under class imbalance e.g. if a class is present in only 10% of instances, then 90% accuracy is achieved by simply always predicting zero - i.e. the majority class. When doing so on our get AP and APmacro of 0.2%, since out of the 1315 classes, maximally four classes are active per instance. Importantly, this tells us that: (a) simply learning to predict zeros can not
task, where the class majority is always zero
httpss:/scci kit ti http:p///tttttt.aarrr.ogg/ttaatt e//oeell Leroodeelevvllllee Levallattin

---

Under review as a conference paper at ICLR 2021
Table 1: Supervised prediction results: comparing an optimized baseline (OB) with contrastive methods (CM: 4-12). CMs compare training from scratch vs. pretrain-ffine-tugg vs. self-supervised pretraining for few and zero-shot learning. Given the same hyperparameters (*), all CMs reach sim- ilar supervised end-task performance, while self-supervised CMs produce fundamentally different results for zero and few-swhlee few-shot learning- see subsection details.
AP micro/
Brier score
Training method/ model
learning setup
macro test %
macro
BASE: baselines, only supervised learning (SL)
(0) ZemR
always predict majority per class. (=all zero supervised
00.20/00.20
(1) WB: weak baseline (BCE)
33.75/n.a.
n.a.
optimized baseline (BCE)
supervised
45.01/22.81
0.0015
FROM SCRATCH (8): train from scrateh: supervised (SL.), self+ supervised (S(+S)L)
pretraining
(3) (9) SL+SSLscT: h-params base
supervised self-supervised from scratch
47.13/25.28
0.0028
(4) SLscr: h-parms like (+)
supervised scratch
47.74/26.05
0.0028
PRETRANED (p) FINE-TUNE (f):
supe-sude (SSL) premain fine-tune or SL+SSL)
(5) SSLpre-SSL+SSS h-parms like (*)
SSL pretrain >SL+SSL fine-tune
48,20/25.58
0.0027
(6) SSI.pre-SSI.fn h- h-parms like (*)
SSI. pretrain >SL fine-tune
47.53/25.65
0.0028
FRW-SHOT: few-shot 10% train, 'pretrained then fine-tined (pf) vs from scrotch (s)
SSLpre >SLlin: h-parms like (*)
self pretrain >> 10% supervised fine-tune
38.01/18.31
0.0037
(8) SSL.rr--SL=SSL.nn h-parms like (*)
self pretrain >=10% self+supervised fine-tune
38.25/18.49
0.0038
(9) SLser: h-parms like (+)
10% supervised from scratch
30.46/13.07
0.0032
(10) (*) SL+SSLscr:
10% self-supervised from scratch
30.53/13.28
0.0039
ZERO-SHOt: zero-shot, self-suuee pretrain only
(11) sSLpre->: b-parms. like (*)
self pretrain > zero-hhot
10.26/10.70
0.1139
(12) SSLpre h-param tuning
self pretrain zzero-shot
14.94/14.86
0.0791
score well on under this metric and (b) that this problem setting is challenging. Next, we evaluate both weak and optimised baselines (WB, OB). When using a very small CNN baseline (WB) with max pooling over 10 filters at filter sizes 1-3 that feed into a one-layer classifier, we achieved 33.75%AP mécro on the test set after only tuning the learning rate. When tuning this baseline for parameters known to increase generalisation using a set of such methods suggested by Jiang et al. 2220 we get more solid test score of 45.01 APmicro and an of 22.81 APmasro- The macro result tells us that not all classes perform equally well. Upon closer inspection, we find that model performance worsens with increasing class rarity as expected. While establishing solid baseline, we find expected limitations of model width, max-k pooling and dropout scale-up, and confirma- tion that controlled experiment comparisons that only change one variable at time, do not suffice to find better hyperparameter configurations. For example, when widening lower layer components and observing a decrease in performance, higher layers should also be made wider to accommodate the additional feature information from lower layers which is consistent with findings in Nakkiran et al.] (2020). A more detailed breakdown of this analysis can be found in Table Tab. [ in the ap- pendix App. We explore considerable amount of hyperparameter configurations in effort to compute a solid baseline. This allows for more robust insights and helps to speed up optimisation of the self-supervised models.
6.2 100% SUPERVISION (SL+SSLSCR) AS REFERENCE (*) FOR FEW AND ZERO-SHOT (0%)
Tab. show both: models trained FROM SCRATCH (s), and models that are first PRETRAINED (p) using self-supervised word pscudo-labels from text inputs, and afterwards fine-tuned (f) supervi- sion labels. To fit the supervised end-task (tag prediction), both fine-tuning and training from scratch either: (4) only fit supervision labels (SL) (3) jointly fit supervised labels and self-supervised word pseudo-labels (S(+S)L), as described [3]
However, before analysing results, define controlled experiment setup using fixed, but shared hyperparameter setting '(*) SL+SSLser' as a reference (*). Since SL+SSLscr is the most basic model learning setup that uses both self-supervision and supervision, we use its optimal
7

---

Under review as a conference paper at ICLR 2021
hyperparameters '(*) SL+SSLser' fixed reference configuration for most subsequent learning setups, as indicated by the 'params like (*)' marker. This ensures a more controlled comparison of the effects of pretraining vs. training from scratch, and robust insights on how to design self- supervision during end-task fitting and pretraining. The (*) reference will hence be used for most few and zero-shot settings. When comparing PRETRAINED models with models trained FROM SCRATCH, we see that under comparable hyperparameters. without setting-specti ing, all four learning setups perform similarly within percent point (%p) of each other. We also see that the PRETRAINED (5) model which uses sulf-siinniiiion during both pretraining and fine- tuning performs best. Training FROM SCRATCH using self+supervision SL+SSLser somewhat hurts performance compared to using supervision alone in SLscr. Test scores are reported for the best dev set AP mieroo scores.
parameter tun-
6.3 FEW-SHOT: PRETRAIN FOR BETTER LONG-TAII,, LOW-RESOURCE, FHO-SHOT LEARNING
In this section, present evidence that even in data-limited, long-tailed setting, self-supervised "data-intemaa" pretraining: (a) increases few-shot learning performance of subsequent fine-tuning, while (b) improving learning speed and stability. This demonstrates that small data pretraining
has similar benefits as large-scale pretraining (Brown et al. 2020: Schick & Schütze 2020a). In
Few-shot label efficiency for supervised train from scratch
Few-shot label efficiency during joint self+supervised fine-tuning
0.75
0.5
0.25
0.1
0.76
0.5
0.25
0.1
0.4
0.3
0.2
0.1
epoch
epoch
50
150
200
100
150
200
250
Figure 2: Few-shot learning: Best training from scratch (left) vs. best fine-tuned (right): APmicro.test curves for different few-shot portions: 100%. 75%, 50%, 25% and 10% of train- ing samples. 'Dataset-internal' pretraining via self- (right) markedly improves few-shot learning performance, speed and stability compared to training from scratch (left).
Fig. 2 when using the (*) reference model from Tab. I we now compare training from scratch (4) as before (pretraining off, left), with pretraining via self-supervised word pseudo-labels, and then fine-tuning on the supervised training labels (5) of the end-task (pretraining on). Note that our model architecture (Fig. [ does not distinguish between self-supervised and supervised labels, which means that during self-supervised pretraining, we sample as many word pseudo-labels as real labels during supervised fine-tuning (or when supervising from scratch).
When fine-tuning the pretrained model on an increasingly difficult FEW-SHOT portion of (100%), 75%, 50%, 25% and only 10% of the supervised training data, we see large AP, micro/mmare lest performance improvements compared to training FROM SCRATCH in both Tab. | and Fig. 2 On the right, in Fig. [ we see that the pretrained models start with a higher epoch-0 perfor- mance, train faster, are more stable and achieve a markedly better few-shot end performance than the left-hand "from scratch' setting. This is confirmed by detailed results for the 10% FEW- SHOT setting in Tab. D. where pretrained models (SSLpre-SLLin, SSLpre .38/.18.APPiie iiiiro tacto.test compared to only 130/.13APmiic. mimaommirmmmarrea
achieve for models trained from scratch (compare (7-10). This means that, when using only 10% supervised labels, pretrained models still retain 38.25/ 25/48.20, or roughly 80%, of their fully supervised performance. This pro- vides evidence to answer the underlying question: "Do we really need more data for pretraining or can we simply increase self-supervision?". Very recent work by Bansal et al. (2020) has investi- gated this question for large-scale, self-supervised pretraining, where they showed that increasing self-supervision to create "a richer learning signal" benefits few-shot performance of large mod- els. Our results demonstrate that this is also the case for small-scale, non- Transformer pretrained models, even under a much more challenging anigtailld learning setting than Bansal et al.) (2020) examined. However, to better understand the benefits of using more self- self-supervised training signals
8

---

Under review as a conference paper at ICLR 2021
Zero-shot pretrain perf. by model size and SSL label amount
Few-shot Zero-shot performance (X-efficiency)
larger net, 3.3x labels
3.3k labelss
default like (*)
0.14
AP micioo beat
0.1
0.08
0.04
0.02
epach
opoch
0
50
100 150
100 200 300
400
500
Figure 3: Zero-shot performance by model and signal size: Left plot: When using the same label and parameter amount as for the 'joint self-supervised train from scratch' reference model (*), allowing more self-supervision labels (left middle curve) and widening the network (left top curve) noticeably boosts zero-shot performance (supervised AP micro dev and test). Right: when using less training data text (few-shot on inputs X), zero-shot still works, but we need to wait much longer.
and its relation to model size, we examine the zero-shot performance of our pretraining approach in regards to label (signal) amount, network width and zero-shot X data-efficiency (low-resource zero-shot performance) i.e. zero-shot performance when pretraining on fractions of inputs X to forcibly limit self-supprvision.
6.4 ZERO-SHOT: MORE IS BETTER, FOR "LOW-RESOURCE" ZERO-SHOT PRETRAIN LONGER
In this experiment, we study how the number of self-supervised labels (signal) and the model width used for self-supervised pretraining affects zero-shot performance on the end-task test set. We show results in both Fig. 2 (11, 12) and Tab. (ERO-SSOT). Fig. we see that when using the refer- ence hyperparameter configuration ((*) in Tab. pretraining gets the lowest zero-shot performance. When increasing the number of self-supervised word pseudo-labels from 150 to 500, the model performs better (middle curve), while not using more parameters so increasing self-supervision signals is beneficial. When additionally tripling the network's s sequence and label encoder width and doubling the label match classifier size, zero-shot performance increases even more (top curve). This indicates that for zero-shot learning performance from pretraining, both the amount of training signals and model size have a significant impact. While increased model size has been linked to in- creased zero-shot performance of Web-scale pretrained models like GPT-3 (Brown et al.) (2020), the influence of signal amount zero-shot learning is much less well understood, because large-scale pretraining research often increases training data size when changing self-supervision, as outlined by Liu et al. (2020). Finally, in Fig. 3 we see that when pretraining our model for zero-shot prediction on only portions (100%, 75%. .50%, 25% and 10%) of the training text inputs X, i.c. an increasingly low-resource zero-shot setting, we still converge towards comparable full zero-shot performance (if we had not stopped early). However, each reduction in training size multiplies the required training time when using the same number of self-labels. This provides promising insight into self- supervised pretraining on small datasets, which, if designed appropriately, can be used to pretrain well-initialised models for supervised fine-tuning and and few-shot learning from very small text sizes.
7 CONCLUSION
We showed that label-embedding prediction, modified for sulf-supervised pretraining on a challeng- ing long-tail, low-resource dataset substantially improves low-resource few and zero-shot perfor- mance. We find that increased self-supervision, in place of increased data size or resorting to large- scale pretraining, strongly boosts few and zero-shot performance, even in challenging settings. In future, we envision that the proposed methods could be applied in scenarios where little in-domain (pre-)traine )triininnn data is available, e.g. in medicine (Serbetei al. 2020), and where new labels rapidly emerge at test time, e.g. for hashtag prediction (Ma et al. 2014). The code and data splits will be published on https: / com.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Trapit Bansal, Rishikesh Jha, Tsendsuren Munkhdalai, and Andrew McCallum. Self-supervised meta-learning for few-shot natural language classification tasks. CoRR, abs/2200.08445,, 2020.
http://orrrabb/////////// 08445
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in NewrIPS, 2006, 2006. URL http:://ppppe
wtwp:0044--grddd-alaareeeeeeeeee se otgp
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors
with subword information. TACL, 2017.
@bp/taacl/atiicle/vieee
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005. 14165, 2020. URL htt.:: http::///xxxvvvooo /ebrgg/bb//aa 2005. 14165
Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, Hyrum Anderson, Heather Roff, Gregory C. Allen, Jacob Steinhardt, Carrick Flynn, Seán Ó hEigeartaigh, Simon Beard, Haydn Belfield, Se- bastian Farquhar, Clare Lyle, Rebecca Crootof, Owain Evans, Michael Page, Joanna Bryson, Roman Yampolskiy, and Dario Amodei. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation, 2018.
Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang. and Inderjit Dhillon. X-bert: extreme multi-label text classification with using bidirectional encoder representations from transformers. NewrlPS, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. URL https://arxiv:
org/ aaaaabba22000 05709
Jesse Davis and Mark Goadrich.
The relationship between precision-recall and ROC curves. In Machine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006), Pittsbrrh, Pennsylvania, USA, June 25-29, 2006, pp. 233-240, 2006. doi: 143844.
1143874. URL httpttss://aoi.... org/ 10. 100..11555110 1115/11111444 1143874
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL- HLT. Association for Computational Linguistics, 2019, doi: 10.18653/v1/119-142.. URL
https: //ops::/doi.. www.:.....g..... 11w53//1//1n11 1423
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith.
Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. CoRR, abs/2002.06305, 2020. URL http::://// //arxiv///xvv...///a wwtpoorg///////////oe /LLLLtttpss//axxvv.o///////////p
06305
Alberto Fernández, Salvador García, Mikel Galar, Ronaldo C. Prati, Bartosz Krawezyk, and Francisco Herrera. Learning from Imbalanced Data Sets. Springer, 2018. ISBN 978-3- 319-98073-7. doi: 10. 000100779978-3331--987744444
URI. htttps://cci. org.:0...g11111
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of AISTATS. JMLR, 2010. URL http://procceeiingg....
press/v9/gloree 10a.oorrtttaaa..m........
Sara Hooker. The hardware lottery, 2020. UKLhttpsss//rriivrrrrr/a///////0
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do com- pressed deep neural networks forget?, 2020a.
10

---

Under review as a conference paper at ICLR 2021
Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. Characterising bias in compressed models, 2020b.
Dirk Hovy and Shannon L. Spruit. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Pupers), pp. 591-598, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v 1/P16 2096. URLhttps: www. ac web wwwp//rggaaooooo.. www.anntooooooooooooooo /P16 2096.
Huajie Jiang, Ruiping Wang, Shiguang Shan, and Xilin Chen. Transferable contrastive network for generalized zero-shot learning. In 2019 IEEE/CVE International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South). October 27 November 2. 2019, Pp. URL /oii.ii.....o waw...org/o...... 1 ICCV. 2019. 00986
2019. doi:
10.1109/1CCV2201..0000
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020. URL https: //openreview.oo net. forum? wewrrmmmdd=sssssee Ad=JJ
Tal Linzen. How can we accelerate progress towards human-like linguistic generalizaationnn
In Proceedings of ACL, 2020. URL
acl ael-main. 465/
Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, and Yiming Yang. Deep learning for extreme multi-
label text classification. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, 1145/
pp. 124, 2017. doi: 10. 00.114//077713363000333 3077136. 3080834
URL https: //cit 1145/
Qi Liu, Matt J. Kusner, and Phil Blunsom. A survey on contextual embeddings. (0s/2203.00728,, 2020. URL https: //rtps:/arriv... wttp./rr/aabab...7//....
CoRR,
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL htt.httts:/// http:::///ppeeeevvii www.:eee//eeeeee feotfformm? [d=Bkg3g2R9FX
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for con-
ditional models: Consistency and statistical efficiency. Proceedings of the 2018 Confer-
ence Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31
November 4, 2018, pp. 3698-3707, 2018. doi: 10.18653/11dd18-14005 URL https:
//doi. 10. 18 www.11166//1dd1.
Zongyang Ma, Aixin Sun, Quan Yuan, and Gao Cong. Tagging Your Tweets: A Probabilistic Modeling of Hashtag Annotation in Twitter. In Jianzhong Li, Xiaoyang Sean Wang, Minos N. Garofalakis, Ian Soboroff, Torsten Suel, and Min Wang (eds.), CIKM, pp. 999-1008. ACM, 2014. ISBN 978-1-4503-2598. URL http:://p//bb uni triirrrier... ht/db/cccffccc/////cc///////////
ci cikm2014 html 4Yc11
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguisties, pp. for Computational Linguistics. doi: 10. 10.19653/ URL https::////. www. acl web.
Florence, Italy. July 2019. Association
hrtg/antholooy/PIIC
Gábor Melis, Tomás Kociský, and Phil Blunsom. Mogrifier LSTM. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020. URL
ottpps://openzeview.nt
net www. Sed-sdeeesSSsSSSSSSe.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mix- ture models. In 5th International Conference on Learning Representationss ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings, 2017. URL https:
//openreviews net. fumm d=By 122dxe

---

Under review as a conference paper at ICLR 2021
Margaret Mitchell, Dylan Baker, Nyalleng Moorosi, Emily Denton, Ben Hutchinson, Alex Hanna, Timmit Gebru, and Jamie Morgenstern. Diversity and inclusion metrics in subset selection. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES '20, pp. 117-123, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450371 100. doi: 10.1145/337562..33555322
/tt.itttts:///i..rrg/// 1111//3101.11111135536322 3375583
Kevin Musgrave, Serge J. Belongie, and Ser-Nam Lim. A metric learning reality check. CoRR, abs/2003.08505, 2020. URL htt.hstts:///xxii
abs.///////// /200 08505
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. In ICIR, 2020. URL https://
Nikolaos Pappas and James Henderson. GILE: A generalized input-label embedding for text clas- sification. Trans. Assoc. Comput. Linguistics, 155, 2019. URL httttts:///ttaaaacc.
org/ojs/ windxx//iiiee..///e
Barbara Plank and Nils Rethmeier. Morty: Unsupervised learning of task-specialized word em-
beddings by autoencoding. In ACLL 2019. doi: 00.18653/11/w11-4300. URL
https::/doi.org//
18653/91/w19-4333
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1667, 2020. URL http:
7/jmlz hwwp..orgg/ppers//211// 007/00000...000 html
Brian Rappert and Michael J. Selgelid. On the Dual Uses of Science and Ethics: Principles, Prac- tices, and Prospects. ANU Press, 2013. ISBN 9781925021332. URL httlhhttp///ee jstor. org/ htrgsttable/j.ccthhh
Mark Riedl. Ai democratization in the era of gpt-3. The Gradient, 2020.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What know about how bert works, 2020.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. CoRR, abs/2009.07118 11,1, 2020a.
07118
Timo Schick and Hinrich Schütze. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In Proceedings of AAAI. AAAI Press, 2020b. URL https://aaal
https/iinnxx.ppppaa.eeeeeep phpp/aat/articlee /wwee///viw/eieeeeeeeeee /eiee///////3/
Oguz Necip Şerbetei, Sebastian Möller, Roland Roller, and Nils Rethmeier. Efficare: Better prognostic models via resource-efficient health embeddings. In AMIA Anmual Symposium.
PubMed, 2020. URL https::////e http::/wwwwntt. wwwp:wiwdxiivv..m org http:///cccceeeeeeee//////////// 7020.07.2 20157610
Anna Lowenhaupt Tsing, Heather Anne Swanson, Elaine Gan, and Nils Bubandt. Arts of living on damaged planet: ghosts of the Anthropocene ; Arts of living on a damaged planet: monsters of the Anthropocene. University of Minnesota Press, 2017.
Oriol Vinyals, Charles Blundell, Tim Lillicrap. Koray Kavukeuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, pp. 3630-3638 2016. URL http::::/pppppss.spp.mm oe shotrrr--ooo--tt-------eeee learning
cc /cc/paper/633
Chenguang Wang. Zihao Ye, Aston Zhang, Zheng Zhang, and Alexander Smola. Transformer a diet. CoRR, abs/2002.061 70, 2020a.
wttp://orrrgagb//....o.o 06170
Sinong Wang, Madian Khabsa, and Hao Ma. To pretrain not pretrain: Examining the benefits of pretraining on resource rich tasks. CoRR, abs/2006.08671. 2020b. org/abs/2000 2006 08671
12

---

Under review as a conference paper at ICLR 2021
Sinong Wang, Madian Khabsa, and Hao Ma. To pretrain or not pretrain: Examining the benefits of pretraining on resource rich tasks, 2020c.
Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. Disembodied machine learning: On the illusion of objectivity in nip. anonymous preprint under review, 2020.
Dani Yogatama, Cyprien de Masson d' d'tttmee Jerome Connor, Tomás Kociský, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, and Phil Blunsom. Learning and evaluating general linguistic intelligence. CoRR, 2019. URL http:
Honglun Zhang, Liqiang Xiao, Wenqing Chen, Yongkun Wang, and Yaohui Jin. Multi-task label embedding for text classification. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 4545- 4553, 2018a.
Honglun Zhang, Liqiang Xiao, Wenging Chen, Yongkun Wang, and Yaohui Jin. Multi-task label embedding for text classification. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 444544553, Brussels, Belgium, October-Novente 2018b.
Association for Computational Linguistics. doi: 10. 1/D18 1484. URL htthttp:: www..
ac welweb..o. wweb.org//mthooogg/DD/./ -484
13

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 A BASELINE TUNED USING GENERALISATION TECHNIQUES
Table 2: Building an optimised supervised baseline: using test set generalization techniques as proposed by Jiang et al. (2020). %p denotes absolute percent points. Since parameters cannot be tuned in isolation, %p only reflects drops by deviating from optimal settings once they are found. Details on the explored hyperparameters are found Tab.
Model
variable
observation
optimal parameter, %p drop from not using it
pre-opt NN
learning rate optimized
base setting
33.75% AP Ammicrodest
optimized NN
Optimal parameters +
hase setting
45.01% %p AP mimccrotest. .0015 BSmacro
larger NN
max-k pooling
important
max-3 pooling, 3%p better than max-l pooling
n-gram filter sizes >2 matter
CNN filter size
important
(~~2%p)) comparing same filter amounts
num CNN filters wider classifier
important
100 filters per n-gram size
overfitting
more than I layer classifier lead to overfitting
dropout
on CNN output
improvement
2% better AP mirrro test test, 2% improvement stabilizes learning, but same performance -39% drop A micro.tests despite tuning
on deeper/ wider clf
none, stability
optimizer
ADABOUND
failed
learning rate
lower LR
crucial
LR = 0.0075 for ADAM with cross-entropy batch size = 1024 worked well
batch size
batch size
important
Table Parameters we explored for the optimized baseline. Not all combinations were tried. We tuned in order: learning rate Ir, filter sizes, max-k pooling, tuning embeddings, batch size, classifier depth and lastly tried another optimizer.
14), (1: 57, 29, 10: 14).(1: 285, 2: 145, 70). {1: {1:10, 10:10, 1:10), (1:15, 2:10, 3:53, (1:10), (1:100), (10:100)
Filters
Filter sizes
1, 2,
In
0.01, 0.0075, 0.005, 0.001, 0.0005, 0.0001
bs
1536, 4096
max-k
3,
two layer classifier, "conf"[[f'do' do' None 'out dim': 2048
4196 1024), ('do':None- 0.2)1},
classifier
'conf":[] [['do'.22]]
tune embedding:
True, False
optimizer:
ADAM, ADABOUND by Luo et al. 2019 (very low results)
For the baseline we found optimal hyperparameters to be: Ir=0.0075, filter-sizes=(1: 57, 2: 29, 3: 14 calffoone.layr.cllllle cclassifier, 'conf": [[{ :22] max-k pooling=3, bs=1536, tune embedding-True, oDAM with pytorch defaults. Increasing the filter size, classifier size or depth or using more k decreased dev set performance due increased overfitting. In general the standard multi- label BCE loss overfit much more quickly than the contrastive methods described in { The con- trastive model only differs it was able to use more filters {1: 100, 2: 100, 3: 100), where using only {I: 20, 2: 20, 3: 20} loses 1.5 %p of performance, and that its optimal Ir 0.0005, while the batch size shrinks to 1024 due to increased memory requirements of label matching. This contrastive mod- els optimal matcher classifier is deeper, due to the increased task complexity four layer classiffer,, "con": [f'do': 0.2). d'm': 1024, 'do': 0.1}. ('out.dim', 300, 'do': None), f'out_dim' 1, 'do' None)]].
A.2 LABEL-EMBEDDINGS, PRETRAINING EFFECTS ON THE LONG-TAIL.
In this section we analyse the effects using supervised label-embeddings and self-supervised pre- training with words as laeuds-labels. Plotting the average precision of 1305 would be unreadable.
14

---

Under review as a conference paper at ICLR 2021
(log) class rarity in 5 class buckets (buckets have same combined label frequency)
0-20% head
20-40%
40-60%
60-80%
80-100% tail
10000
1og crass trequency
100
classes: sorted head (common) - tail (rare)
Figure 4: Head and long-tail as 5 label frequency balanced class buckets: We bucket classes by label frequency, into 5 buckets, so that each bucket contains equally many label occurrences i.e. the buckets are balanced and thus comparable. Note the log frequency scale.
Instead, sort classes from frequent to rare and assign them to of five 20% class frequency buckets, such that each bucket has the same amount of positive labels (label occurrences) it. As seen in Fig. 4 this means that the head 0 20% bucket (left, blue) has very few, frequent classes, while tail buckets 20 -
80 100% have increasingly more classes (right, orange, red) that also become increasingly more rare. We bucket classes this balance label frequency between buckets to make them directly comparable.
long-tail performance: BCE vs label-embedding (LE) and self-supervised pretraining (PT)
60
(2) optimized BCE baseline
55
x (4) SLscr (LE, ")
50
(3) SL+SSLscr (LE,
45
(6) SSLpre-SLLin LE,
40
(5) SSLpre- (PT, LE,")
35
30
(5.XL) SLLpre-SLLSSLffn (XL-PT, LE)
25
20
15
0-20% head
20-40%
40-60%
60-80%
80-100% tail
AP macro test for each 20% class buckets: head tail
Figure 5: Long-tail effects of base, label-embeddings and self-supervised pretraining (XL): When reporting Al,, of the five 20% head to tail class bins, the BCE multi-label objective performs worst. Label-embedding NCE (LE, (4-6)) markedly boosts performance, especially on the loig-tail. When using label embeddings for self-supervised pretraining with the same network parameters (*) for all LE models, there is no boost. However, when pretraining with more parameters ((5XLL) - see larger net, 3. labels the zero-shot learning Fig. B) we see substantial long-tail performance boosts (turquoise, upper most curve).
Label-embedding increase long-tail performance: In Fig. 5 can see that the optimized baseline (2) from Tab. [ performs much worse than models that use only the supervised label-embeddings (LE) and methods that also use self-supervised pretraining (PT) via noise contrastive sampling of input words pseudo labels. We also that regarding end-task performance task, training from scratch (LE, pink x) performs only slightly worse than fine-tuning after self- supervised pretraining (purple o). However, also that increasing the model size during
the tag predition
15

---

Under review as a conference paper at ICLR 2021
self- supervueeee pretraining (5.XL) boost performance the long-tail, especially with increasingly tailed or rare (60-100%) classes. Previously, we saw in Fig. thatt the same "larger net, 3.3x labels" model (5.XL) increased zero-shot performance over the default parameters (*), which demonstrates that improved self-supervised zero-shot performance translates into better supervised end-task fine- tuning performance. We also found that increasing the size of non-pretrained (trained from scratch) models did not improve end-task performance, despite hyper-parameter tuning.
This leaves us with two insights for modeling. First, for small-scale, 'data-internal', self-supervised pretraining a larger pretraining model increases long-tail performance, whereas [ooker et al. (2020a) found that compressing larger models first 'forgets' long-tail performance both experi- ments provide evidence that model capacity and long-tail performance are tied. This seems to be the case even for small-scale self-supervised pretraining. i.e. it demonstrates that despite training on small data, we still need increased self-supervision signals and model size to capture long-tail information, which could explain why large-scale pretraining and models perform so well rather than simply assuming them to be overparameterized. Second, this larger pretraining model has an end-task A icro score of 49%, which is only- .8 percent points better than the pretrained model (5) at 2% with default parameters (*), despite showing promising improvements on long-tail classes, which together with the zero and few-shot insights underlines that optimizing for learning insights and analysis other than supervised performance summary metrics can lead to a broader understand- ing of neural learning processes and modeling effects.
The head is learned first (in early epochs), pretraining learns the tail much faster: In Fig. c we compare early epoch training with late (optimal) epoch test scores per class bucket. We see that all models learn the head classes during the first epochs (- dashed line). Methods (5, 5.XL) that use label-embedding (LE) and self-supervised pretraining (PT), start learning the long-tail during the first epoch, while BCE multi-label baseline (2) does not start learning the long-tail until epoch 10, and even then at a much lower performance than the pretrained label-embedding methods. Finally, we see that pretraining a larger model with more self-supervised pretraining signal (5.XL. or "larger net, 3.3x labels" in Fig. 3 increases performance on the long-tail, even during the first epochs.
Long-tail: early epochs vs best (by AP micro dev) late epoch
(2) optimized BCE baseline epoch 10
(2) optimized BCE baseline epoch 82
50
(5) SLLpre (PT. LE, ") epoch
(5) SSLpre-SSL+SSLfm (PT, LE, ') epoch 94
(5.XL) SSLpre (XL-PT. LE) epoch
30
(5.XL) SSLpreeSLL (XL-PT, LE) epoch
35
20
10
0-20% head
20-40%
40-60%
60-80%
80-100% tall
AP macro test per class bucket: head - tail
Figure 6: The long-tail is learned during later epochs: APmacro performance over five frequency balanced 20% head to tail class buckets. All methods (2-5.XL) learn the head classes (0- 20%) first during early epochs. Self-supervised pretraining (5, 5.XL), via pseudo label-embedding NCE (LE, PT), learn the long-tail tail (see 60 100%) even in the first epoch (- dashed line) of supervised learning. The baseline (2) struggles to learn this loig-tail epoch 10 and until its epoch 82 i.e. its optimal dev set score epoch.
Self-supervised label-embedding pretraining bousts long-tail performance: We thus conclude, that self- supervupe pretraining helps us learn the long-tail tail better, and faster i.. even after single epoch of supervised training. This is a useful property in supervised learning scenarios where data or computation cycles are limited. We note that: our pretraining and fine-tuning do not require
16

---

Under review as a conference paper at ICLR 2021
learning rate scheduling normalization layers like BERT RoBERTa (Devlin et al. 2019
Wang et al. 2020c).
Miscellaneous result/ an open future evaluation problem: Finally, since average precision (AP) has no explicit notion of over and under predictions, we plotted over and under predictions per class and over all classes. In standard classification, i.e. discrete, evaluation would have over predic- tions as false positives and under predictions as false negatives. Using continuous measures such as AP has computational advantages and does not limit evaluation single threshold like Fi or accuracy do. However, has no notion of over and under predictions, which especially regarding long-tail issues may have a significant impact. However, plotting over and underpredictions per class (and overall) was only mildly informative. Due to the high label sparsity, we saw close to zero over predictions on average, but all-class average underpredictions were much harder to reduce (op- timize). This observation is reflection of the high label sparsity, i.e. at most 0.3 of labels are active per instance, combined with a long-tail distribution - i.e. many rare events. Under this combination meaningful evaluation of prediction behaviour is hard to analyse in meaningful, concise fashion, because per class plots become large and hard to fit and interpret in a paper, and all-class averages do not reveal class dynamics. We include these observations because we found them instructive to outline the challenges of evaluating long-tail learning and do not include the per class plots, because they would not be readable. We are however happy discuss them upon request.
A.3 FEW-SHOTT: SCRATCII, PRETRAINED, ADDITIONAL SELF+SUPERVISED SCENARIOS
Few-shot label efficiency for supervised train from scratch
Few-shot label efficiency selftsupervised train from scratch
mitro lest
bost
0.4
miro
0.3
50
100
150
200
250
50
100
150
200
Few-shot label-officiency during supervised only fine-tuning Few-shot label efficiency during joint self+super fine-tuning
0.75
0.25
0.4
0.1
opoch
opoch
50 100 150
100 150
250
Figure 7: Few-shot training from scratch (top 2) vs. after pretraining (bottom 2): and using only supervision to fit the end-task (left) vs. jointly using self+supervision (right). Results are in APmicrooeest for different few-shot training set portions (1, 75%. 50%, 25%, 10%). Insight I: self-supervision during end-task fitting makes no learning difference - i.c. when comparing top (or bottom) left (supervised) vsfight (self+supervised) sub-figures, they look nearly the same. Insight 2: Pretraining (bottom figs.) via self-supervision markedly improves few-shot learning performance, speed and stability, independent of fine-tuning via supervision (left) or self+supervised (right).
Few-shot learning increases the long-tail problem. For 10% few shot learn- ing, we train on 6800 instances, so many classes will be unseen at training time We will publish both the parsed data splits and a cleaned code version on Github to encourage experimenting with and extending other low-resource 'dense-to-dens self-supervision methods, additional evaluation metrics and datasets.
Few-shot challenges:
Few-shot, with and without self-supervision as pretraining for joint self+supervised fine tuning: Fig. shows in more detail that the pretrained model (bottom) learns better, and that joint selfvsupervised end-task training (scratch fine-tuned) makes no difference.
17

---

Under review as a conference paper at ICLR 2021
A.4 TEXT PREPROCESSING DETAILS
We decompose tags such as 'p-value' as 'p" and 'value' and split latex equations into command words, as they would otherwise create many long, unique tokens. 10 tag words are not in the input vocabulary and thus we randomly initialise their embeddings. Though we never used this information, we parsed the text and tille and annotated them with 'html-like" title, paragraph and sentence delimiters. The dataset is ordered and annotated by time. Dev and test set are therefore future data compared to the training data, which results in a non non-stationary problem, though we never determined to what extend.
A.5 POTENTIAL ETHICAL CONSIDERATIONS
In this section, we outline potential impacts of our work for machine learning practice, as well as its possible environmental, societal, health and privacy implications. As with any technology there is the dilemma of dual use (Rappert & Selgelid 2013). Below, we briefly discuss beneficial and potential detrimental impacts of this work as we can foresce them Hovy & Spruit (2016); Brundage ct al. 2018).
The main goal of our research is to reduce the hardware and compute requirements of current rep- resentation pretraining methodology for language representations, especially for challenging low low-resource, long-tail problems. Due to the reduction in compute requirements, our methods may help reduce carbon impact and the exhaustion of precious resources like rare metal compared to large-scale pretraining. Consuming less energy and mining less resources for hardware production has major impacts on the environment (Tsing al. 2017). Thus, research community we should take action not to let AI methods become a race for precious metal hardware due to its devastating effects on our shared environment. Further. small-scale pretraining could make access to modern NLP methods easier for machine learning researchers and practitioners, who have less hardware re- source privileges than are required for sate-oof-the-an solutions, or whose language of research does not allow for easy access to Web-scale text collections. This may become even more important as socio-economic factors are likely to play a fundamental role in the future democratisation and fair access to AI technology (Riedl 2020) for economics, health and other key decision making areas. This is especially important as large-scale hardware resources increasingly lead to research and eco- nomic inequalities as described by Hooker (2020); Riedl (2020). Another important advantage of researching more data-efficient methods is that using as little data as needed is a requirement of the GDPR regulations for 'privacy by design' This principle is in direct conflict with the current self- supervised pretraining approaches, which parties who have both access to massive data collections and compute resources predominantly study.
Furthermore, there may be potential implications in better learning of underrepresented and rare events from small or very limited data collections (Mitchell et al.) 2020). When we increase self- supervision during pretraining, i.c. when pretraining on more diverse learning signals than direct supervision can provide, we see a substantial increase increase in few-shot (low-data) performance, which, upon inspection, becomes clear is caused by better retention of rare event performance than direct supervision could provide see Fig. [ However, we did not yet study whether this pretraining reduces increases unwanted data biases (Waseem et al.) 2020), though typical analyses of gender and racial biases may be hard on the current dataset of machine learning questions. Note that we did not chose this data set to solve specific application task, but only as a proxy to study the effects of small-scale pretraining on challenging data.
Better small-scale pretraining could benefit areas like medicine where large pretraining is not as effective or fails for a lack of external data resources (Serbetci et al.) 2020). Due to the usage du- ality of research in general, research into more resource-efficient learning could also cause privacy concerns, enabling easier surveillance, and improved advertisement recommendation can have un- foreseen political, but also economical and even environmental impacts, as the goal of advertisement is increased resource consumption.
Thus, general approach furthering beneficial usage over detrimental applications of dual tech- nology should regard applying ethics principles at every step of reuse of the discussed methods to
https:://ee wedeenwwikkii wi www.aai...gg/////////.
18

---

Under review as a conference paper at ICLR 2021
support its transparent use and public verification and auditing, protect vulnerable groups from harmful applications.
19


Reviews:
[{"review_id": "yvc_OKfwnG-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a contrastive autoencoder approach that only requires small data to perform a multi-label classification on the long-tail problem. They introduce a matching network to compare text and label embeddings and calculate the probabilities of the label given the input. The proposed idea is very straightforward by combining a matching network with contrastive learning to give broader signals. The goal of this work is to enable zero-shot and few-shot learning with very few resources as a more sustainable approach to machine learning applications. \n\nIn general, the idea is interesting, since it leverages the in-task data for distinguishing positive and negative samples. However, the work itself is not entirely new, there is already similar work on this [1]. I could not find enough related work on matching networks [1] and it seems the authors need to give more credit to other seminal work that has been used for many applications for few-shot learning [2]. I would ask the authors to add more relevant work to the paper\n\nThe structure of the paper can be further improved. The naming is not intuitive, such as $(SLpf, S(+S)Lpf)$. The results in Table 1 isn't easy to read (especially the learning setup). \n\nStrengths:\n- A straightforward method for self-supervised contrastive label-embedding prediction using in-task data.\n- This paper focuses on a challenging, noisy long-tail, low-resource multi-label text prediction task\n\nWeaknesses:\n- The key concern about the paper is the lack of comparison to the baselines. Since the proposed method is not the first on this topic, the authors should compare their method with other well known baselines for few-shot learning, such as matching networks and Siamese Networks\n- The paper does not analyze the effectiveness of the approach on the long-tail scenario.\n- No ablation study on the contrastive learning and label embeddings\n\n***Post-rebuttal***\n\n> I want to thank the author for addressing my concerns. However, I still have issues with the evaluation and the clarity of the paper. I think the paper requires another round of revision before it is ready for publication.\n\nReferences\n1. Transferable Contrastive Network for Generalized Zero-Shot Learning 2019 ICCV https://openaccess.thecvf.com/content_ICCV_2019/papers/Jiang_Transferable_Contrastive_Network_for_Generalized_Zero-Shot_Learning_ICCV_2019_paper.pdf\n2. Siamese Neural Networks for One-shot Image Recognition ICML Deep Learning Workshop 2015 https://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf\n3. Matching Networks for One Shot Learning NeurIPS 2016 https://arxiv.org/pdf/1606.04080.pdf", "extra_scores": {}, "extra_text": {}}, {"review_id": "w3f1Tfb_kE8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\nThis paper studied the effect of pretraining from a miniaturization perspective. The proposed, dataset-internal contrastive self-supervised learning benefits in zero- or/and few-shot learning settings. \n\n##########################################################################\nReasons for score: \n \nOverall, my score is marginally below the acceptance threshold. \n\n1. The paper provides a very in-depth survey of recent work on pretraining research and a clear scoping of the problem to be addressed in this work against the other work. I also like the framing of dataset-internal and dataset-external to distinguish between self-supervision and task-supervision, respectively. I also like the list of limitations of the prior work in the Related Work section. \n2. The description of the proposed method in Section 3 was very detailed and easy to follow. \n3. Overall messages and findings from the experiment were clear and convincing to me. \n \nCons: \n \n1. Although the clarity of methodology description and clear motivation compared to prior work, my main concern of this work is the lack of generalizability of the proposed method to other low-resource, long-tail problems. The predicting pseudo, noisy labels itself is not a novel idea. The main novelty of the method comes from predicting new labels from word token space directly, which seems to be very limited to the task itself; free-formed tag labels for text. As described in the conclusion section, the method seems to be only applicable when input text and label tags are in the same place. \n \n2. Another concern is the unclear description of how two binary labels are optimized in a contrastive way. Usually, there should be an additional loss to maximize the difference between two similar-looking examples of different labels in contrastive learning like SIMCLR. I only find that (5) in Figure 1 is a simple aggregation of binary classifiers, rather than contrastive optimization. When the batch set of positive and negative (pseudo) labels are prepared in L_i, does the ordering of the labels matter? Also, again do you pair positive and negative labels for pair-wise optimization? \n \n3. I like to suggest changing the framing of this work from the general findings of pretraining schemes to task-specific findings. That is, Section 4 should be introduced earlier than Section 3, and describe how difficult the task is and how existing pretraining methods fail to achieve good performance. Until section 3, I had a hard time understanding whether the proposed method in Figure 1 is generalizable to any downstream tasks or specific to certain tasks. The partial examples in Figure 1, like “measuring an interaction”, “interaction”, “p-value” are mentioned without even describing what they are. Otherwise, it would be really nice to add more applications in a long-tail, low-data regime, proving that the proposed framework is general enough. \n \n \n##########################################################################\nQuestions during the rebuttal period: \n \nPlease address and clarify the cons above.\nHere are some questions:\n1. In section 3, what do you mean by “Sampling pseudo-labels provides a straight-forward contrastive, partial autoencoding mechanism…”?\n2. In Section 6.2, authors controlled the hyper-parameter for a better-controlled experiment. However, I wonder this is a fair game. Each training technique may have different levels of optimality. So, I guess given a fixed set of parameter ranges, providing the optimal setting might be a more fair setting. Or, at least providing them in the Appendix might be more convincing. \n \n#########################################################################\nSome typos and suggestions for presentation: \n(1) Too many styles in the text (e.g., bold, italic, the numbers) make it difficult to read sometimes. I know they are used to help follow the main points but they are quite distracting.  \n(2) Table Tab. 2 -> Tab. 2\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hv3j1KP2248", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nQuality\n\n\nI like the intended focus of this paper which is to perform self-supervised training of small data for downstream tasks with applications for zero and few-shot learning. However, there are many limitations.\n\n\n(1) The only task considered is multi-class classification. The technique proposed in this paper revolves around learning the label embeddings that would match the input embeddings, which is quite limited to multi-classification and might not carry much value to other tasks. For example, for tasks such as entailment prediction where the labels are `true' or `false', for the model to perform well, most of it comes from the ability to process the input and does not have much to do with the model's understanding of these two labels, which is unlike in multi-class classification where label understanding is crucial. Therefore, the proposed approach seems to be quite limited to the areas of multi-class classification. \n(2) I would love to see more experiments on other tasks besides multi-class classification but wonders how applicable the framework will be. Explanations would be appreciated here.\n(3) Learning label embeddings seems like a good idea and this seems like an ok way to do it. However, label embeddings are somewhat learned already in some pre-trained models such as generative sequence-to-sequence models (T5 or related models). The paper at least need to have these models as baselines for comparison. \n\n\nClarity\n\nThe writing is at an acceptable level. However I find it a bit dense in terms of explanations and might need more illustrative figures. \n\n- I don't really get see the CNN architecture is used here. I have some concerns that the method might not be applicable for other architectures and would like to get some more clarity on this. \n- It is not entirely clear why the prevalent evaluation favors large self-supervised pre-training. *** \n- Too many notations (s, -, +s , SL) that makes it hard to follow in the results section. \n\n\n\n\nOriginality\n\n- The paper seems heavily inspired by SIMCLR so in my opinion the originality is low to moderate. \n\n\nSignificance\n\n\nThe premise of this work is to avoid having to use external datasets for pre-training. However, the prevalent self-supervised training in many work (BERT, RoBerTa, to name a few) do not require labels, so I don't see why we really need to avoid that. The paper can be positioned a bit better if the paper uses these models, and performs additional learning on top with the 'dataset-internal' contrastive learning approach in order to improve performance on long-tail zero and few-shot learning.\n\nHowever the paper does show that it is possible to do well on long-tail zero- and few-shot learning with only small data pretraining, so that is a nice conclusion derived from the experiments. However, I am not entirely sure about the real-world applicability. \n\n\nHigh-level pros and cons\n\n\nPros\n- Long-tail zero and few-shot learning is an interesting direction which the paper explores\n\nCons\n- This paper seems like a probing paper that looks into how well we can do without external pre-training, but might not be so applicable for real-world applications due to reasons above (see discussion in Quality and Significance)\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a self-supervised model based on a contrastive autoencoder that can make use of a small training set for upstream multi-label/class tasks.
Reviewers have several concerns, including the lack of comparisons and justification for the setting, as well as the potentially narrow setting. Overall, I found the paper to be borderline, the cons slightly greater than the pros, so I recommend to reject it.
Decision: Reject