dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgments We would like to thank  S. Albert ,  V. Buth ,  V. da Costa ,  S. Larrède ,  E. Etienne , L. Peaudecerf and  I. Rieger  for monitoring the data, and  A. Bellino  and  M. Delattre  for coordinating the study. We are grateful to the participating doctors, midwives and nurses for patient management. We would also like to thank the  "Centre de Ressources Biologiques", Centre Hospitalier Intercommunal de Créteil , for the management and storage of biospecimens. Table 4. Nested case-control studies aimed to assess the relationship between vitamin D status in early pregnancy and preeclampsia.
off the Indian west coast are also relatively well captured by the model. In contrast, the model tends to substantially overestimate chlorophyll levels off the Somali coast and in the western AS in general. This discrepancy may result from the fact that the model does not represent iron and silicic acid that may be contributing to limiting productivity in this region  (Koné et al. , 2009). Finally, the model captures fairly well the large-scale distribution of oxygen in the AS region  (Fig 4). In particular, the location, size and intensity of the AS oxygen minimum zone is relatively well reproduced. A more quantitative assessment of the model performance is shown using Taylor diagrams  (Taylor, 2001)   Finally, we evaluate the model performance in reproducing observed long term changes. To this end, we binned observations from the World Ocean Database (2018) seasonally on a 5 • x 5 • horizontal grid at each standard depth.
that summarize three important statistics: (i) the correlation coefficient between the model and the observations, (ii) the standard deviation in the model relative to the observations and (iii) the centered root mean square (RMS) with respect to observations (Fig 5). To this end, in-situ observations from the World Ocean Database (2018) were binned temporally and spatially into a 0. 5 • x 0. 5 • seasonal climatology covering the AS region. No spatial interpolation was performed and the model and observations are contrasted at the observation points only. This analysis confirms the relatively good agreement between model and observations. 8 https://doi. org/10. 5194/bg-2020-325 Preprint. Discussion started: 14 September 2020 c Author(s) 2020. CC BY 4. 0 License. More specifically, it shows that surface temperature and salinity, as well as upper (400 m) ocean nitrate and O 2 distributions generally all agree well with observations with correlations above 0.
In this paper, we introduce a state-of-the-art pseudo-labeling technique for few-shot intent detection. We devise a folding/unfolding hierarchical clustering algorithm which assigns weighted pseudo-labels to unlabeled user utterances. We show that our two-step method yields significant improvement over existing solutions. This performance is achieved on multiple intent detection datasets, even in more challenging situations where the number of classes is large or when the dataset is highly imbalanced. Moreover, we confirm these results on the more general text classification task. We also demonstrate that our approach nicely complements existing solutions, thereby providing an even stronger state-of-the-art ensemble method. We make our code publicly available 1 for future research.
Introduction Labeling user utterances as intents is of paramount importance for chatbots. Intent classes are usually engineered by a linguistic team, and these classes are likely to evolve with time. The most frequent situations include new unanticipated intents found by dialogue traces analysis, the need to introduce new intents that were previously handled using web forms, or the urge to split existing intents into several new ones for fine-grained user interactions. Regardless the rationale, each time the intent referential evolves, a significant amount of new labeled data is required to train a decent intent detection model. This calls for a time-consuming and error-prone labeling process -it is therefore expensive. Consequently, intent detection systems need robust models in few-shot settings in order to avoid to repeatedly suffer from labeling user utterances into intents.
A solution to this problem is to generate pseudo-labels for unlabeled utterances. A pseudo-label is a label (intent class) automatically assigned to an unlabeled utterance using the knowledge found in the set of labeled data. The resulting dataset composed of both labeled and pseudo-labeled utterances is then used to train a supervised intent detection model. Most pseudo-labeling methods combine advances in word representations  (Peters et al. , 2018; Bojanowski et al. , 2017; Tissier et al. , 2017)  and then consider unsupervised clustering algorithms  (von Luxburg, 2007)  in order to propagate known labels to unlabeled data (see Section 2). Nonetheless, there are major drawbacks to existing pseudo-labeling algorithms. First, since clustering algorithms are used to partition user utterances representations, it leads to provide a pseudo-label to all unlabeled data.
This is error-prone since cluster shapes for the entire dataset are unknown and hardly predictable in a few shot setting. Consequently, it is much harder to learn a discriminate hyperplane when training the downstream intent detection models since some data are incorrectly labeled. Worse still, this grows exponentially as the number of intents (classes) increases in practice since it is harder to predict correct pseudo-labels, and the hyperplane is more difficult to learn since it requires more discriminative power. A corollary is also that confidence values in pseudo-labels is hard to produce: since cluster shapes are unknown in advance, distances to centroids cannot reliably be used as a factor a confidence in a pseudo-label. In Computer Vision, the same issues exist for the image classification task and therefore end-to-end systems have been 1 https://github.
3). In order to validate our pseudo-label approach, we evaluate its impact on the downstream task of intent detection using four datasets and fine-tuned BERT-based language models (Section 4). Overall, in few-shot configurations and even when the number of classes is large, our proposed method is able to positively impact the intent or text classification tasks accuracies (Section 5), beyond the state-of-the-art. The experiments also exhibits a robustness of our methods to imbalance bias, which is common in practice. Finally, Section 6 concludes.
Related works A growing amount of work has been done on the Conversational AI domain  (Gao et al. , 2018)  over the recent years. There are different types of conversational agents: Question-Answering systems  (Ren et al. , 2018a) , End-to-end Conversational Models  (Vinyals and Le, 2015; Ren et al. , 2018a) , or Task-Oriented bots  (Martin and Jurafsky, 2009). In the latter case, a central part of the process is the understanding of user utterances, a special case of text classification. Convincing approaches to this problem rely on neural networks (Lee and Dernoncourt, 2016), thus requiring large labeled datasets, which are costly to obtain. In practice, as more users interact with the conversational agent, it is easy to collect a large amount of raw data. Having access to a small labeled dataset along with a large unlabeled one calls for semi-supervised learning solutions  (Chapelle et al.
, 2017; Snell et al. , 2017). In our work, we tackle the problem differently: we propagate the knowledge we have on this small amount of data to unlabeled data points, before feeding both truly labeled and pseudo-labeled data to the intent detection system. This intermediate step is of the utmost practical interest: firstly, propagating knowledge on unlabeled points helps us understand how the data is structured. Secondly, in a labeling framework, it allows us to select interesting samples for our in-house linguists to label or correct given low pseudo-label confidence 2  as a human-in-the-loop approach, instead of selecting samples randomly. The more accurate the pseudo-labels, the more efficient it is for linguists. From a broader perspective, it is worth noticing that semi-supervised learning (learning using both labeled and unlabeled data) has also received a lot of attention in computer vision  (Vinyals et al.
On the opposite, splitting clusters until only one label is represented in each cluster improves the confidence when assigning the pseudo-label to previously unlabeled data points. There are several underlying motivations for this method. First, unlike most techniques, this approach does not require any hyper-parameter tuning, making it more robust to different datasets. In practice, when there is not much data to validate methods -this is the case in few-shot learning -, this is very important. Secondly, thanks to its iterative process, it is able to model unusual cluster shapes, which makes it more robust to sentence representations. Thirdly, the tree structure obtained via hierarchical clustering is richer than a more traditional cluster output. Indeed, it helps to understand the clustering process and how data points are linked to each other better.
Experimental setup Alongside with our contribution, the systems under evaluation are nKNN, Proto/Proto++, as introduced in Section 2. We implemented the nKNN baseline, and for the Prototypical-based approaches, we use the code associated to the seminal Prototypical Network paper 3  , using BERT  (Devlin et al. , 2018)  as our sentence encoder. For the Proto++ baseline, to refine prototypes, we use 20 unlabeled samples per class both at training and testing time, as  Ren (2018b)  showed that it yielded better performances. As suggested by Chen (2019), we do not fine-tune Prototypical Networks at test time, as it is reported to decrease performances. Moreover, since Prototypical Networks are an end-to-end approach, it is first impossible to fine-tune such networks using a single shot for each class, and second, the embeddings of utterances for all labels are fine-tuned in the networks themselves.
Few-shot and Cross-validation setup In Few-Shot Learning, datasets are split into training and testing sets  (Sung et al. , 2017) , where there are no overlapping classes between the two sets. Since few-shot regimes are highly dependent on the random initialisation of given shots, it is standard to repeat this process many times to create as many training/evaluation tasks and report average accuracy values. More formally, for each training task, we sample C classes from the training classes C train. For each class c ∈ C, we sample K support samples S c = {(x i , y i = c)} K i=1 , as well as L query samples Q c = {(x i , y i = c)} L i=1. Those samples will be used to train the model for this task. This setup is often named C-way K-shot settings. Test tasks are constructed in the same way, using the test set.
At test time, we will evaluate the quality of our utterance encoder on those new classes C test , disjoint from classes the model trained on. In our experiments, for each dataset, we use a third of classes for each training, validation, and testing stages. We also vary the number of shots K, ranging from 1 to 20, in order to assess the robustness of the different systems. In order to get a sense of consistency in the results, each reported metric is averaged over 10 different cross-validation runs. For each run, the whole training set is used to train the pseudo-labeling model. Only 90% of the test set is selected to assign pseudo-labels to query samples using support samples. The remaining 10% of the test set is used to evaluate the performance of the text classification system which is trained on top of those pseudo-labels.
Datasets To validate the effectiveness of our method, we explore 3 intent detection datasets. Additionally, because our method is not specific to intent detection, we also consider a text classification dataset. We summarize  the main statistics of the different datasets in Table  1. In order to visualize the data, after fine-tuning BERT models on each domain, we compute T-SNE embedding of utterances in Figure  2. Snips This intent detection dataset is composed of 7 evenly distributed intent classes  (Coucke et al. , 2018). Class labels represent different voice commands, such as BookRestaurant, or PlayMusic. This dataset is the biggest one: it contains more than 2,000 sentences for each label, which is not often the case in practice. OOS The OOS dataset, introduced by Larson (2019), was created to assess both intent detection and out-of-scope prediction models.
In our experiments, the out-of-scope class -which has way more utterances as the others -is considered as any other. The dataset is recent, and the high number of classes makes the task challenging. Liu Introduced by  Liu (2019) , this dataset is composed of user utterances collected on Amazon Mechanical Turk. It is used to assess the quality of various Natural Language Understanding Services, on both intent detection as well as entity recognition tasks. In our experiments, this dataset is one of the most unbalanced, ranging from 24 to 5, 920 samples per class. R8 This dataset, mainly used for the more general case of text classification, contains Reuters Newswire articles. Following the work of authors who previously worked on this dataset  (Zhao et al. , 2018) , we used the 8 most frequent classes.
Sentence Encoder For each dataset, we fine-tune the bert-base model  (Wolf et al. , 2019)  on its former task, masked language modeling. Because this is an unsupervised task, we are able to use both labeled and unlabeled utterances during training. As shown by  Sun (2019) , this additional fine-tuning greatly increases the quality of embeddings. This fine-tuned model is then used as a checkpoint upon which prototypicalbased approaches are built. Additionally, because prototypical networks benefit from a classification training step, it would be unfair to prevent the other approaches -nKNN, Fold/Unfold -from benefiting from such a fine-tuning. To account for this difference, we further fine-tune the BERT model obtained earlier on the classification task, using only few shots -BERT-Fit. The encoding part of this model is then used as input for both nKNN and Fold/Unfold methods.
Intent Detection model Given the recent advances in NLP using transfer learning, we tried several methods for the intent detection model. We used pre-trained ELMo  (Peters et al. , 2018) , InferSent  (Conneau et al. , 2017) , Fast-Text  (Bojanowski et al. , 2017)  and BERT  (Devlin et al. , 2018)  models. For ELMo (resp. BERT), we use the last hidden vector of the last (resp. first) token as the sentence embedding. All language models are fine-tuned with a last classification layer. We compare those methods on the intent detection task using a 10-fold validation, but do not report the results here due to space limitations. As in  (Sun et al. , 2019) , we tried several variants for the BERT model -fine-tuning the model in the Masked Language Modeling (MLM) task as well as freezing or not freezing the transformer part.
Results and analysis In Table  2 , we report all results from pseudo-labeling as well as intent detection performance score. In order to measure the quality of pseudo-labels, we compute their F1-score. For the intent detection part, we report the F1-score on the held-out 10% of the dataset. Intuitively, aggregating pseudo-labels from different methods will discard a lot of pseudo-labels. Using 5 shots, aggregating pseudo-labels yields a discard rate of 10. 2%, 25. 1%, 47. 8%, 45. 5% for snips, OOS, Liu and R8 datasets respectively. To measure this trade-off between recall and precision of pseudo-labels, we will further discuss the impacts on the intent detection performances for each dataset. Snips On the Snips dataset, even though our Folding/Unfolding approach outperforms all competitors, those results must be taken lightly.
As we can see, with only 5 shots, we are able to recover pseudolabels and build an intent detection model with a 0. 988 F1-score. Those huge performances clearly hint that this dataset is not challenging at all: the t-SNE projection for this dataset in Figure  2  supports this claim. This means that we can hardly make concluding remarks on how the systems relatively perform on more realistic datasets. Overall, the massive usage of SNIPS to evaluate intent detection techniques in the literature  (Xia et al. , 2018; Goo et al. , 2018)  becomes open to question. OOS On the OOS dataset, our method largely comes out on top, both on pseudo-label and intent detection results. On the intent detection part, our method is better than the aggregated ones, showing that other methods penalize the aggregation.
This finding is very important, as it proves that using an ensemble of various methods do not always lead to better performances. Concerning the prototypical baselines, this is the dataset where the Proto++ is the further away from its former version, Proto. This comes from the fact that this dataset has the highest number of classes: when doing a soft-KMeans step, if the number of classes is high, then it is as much noise for the prototype refinement. Results on this dataset prove that our method is not handicapped when using a large number of classes, making it more robust to real-life scenarios. Additionally, we see that removing the BERT-Fit+nKNN from the aggregation yields better results. This poor performance was not to be seen in the pseudo-label results, as this baseline was performing second behind our approach.
This raises the importance of using this double experiment, and confirms that better pseudo-labels do not directly imply a better model in the downstream task of intent detection. Liu As explained in Section 4. 2, this dataset is very imbalanced. Still, we get the same results as the two previous datasets, which were more balanced: our method is able to come out on top on both pseudolabeling as well as intent detection results. Additionally, the ablation study where we withdraw our method from the aggregation yields the worst results in almost every situation, hinting that we contribute the most to this aggregation. This shows that our method is robust to various situations, including the case of imbalanced datasets. R8 On the R8 dataset, even though we now deal with text classification and much longer sentences, we have similar results as on others intent detection datasets.
Aggregated approach On all datasets, aggregating pseudo-labels obtained from single methods always improves the pseudolabel quality, at the cost of retrieving less pseudo-labels. However, this increment is not fully reflected in the intent detection performance results: most of times, there is a single method which, when discarded from the aggregation, yields better text classification results. This is important to consider in practice, and to have in mind that better pseudo-labels does not necessarily imply better results for the intent detection downstream task. Overall, in the ablation study where we withdraw each single method from the aggregation, the aggregation without our method is very often the worst. This additional study shows that our method contributes the most to the aggregation, as removing it is often the worst case.
Proto and Proto++ We discuss here the performances of prototypical baselines. In our experiments, we showed that the Proto++ variant performs less than its former version, Proto. On each dataset, we also found that finetuning BERT for the classification task tremendously improves the dataset separability -not reported here, due to space limitations. Using a BERT model only fine-tuned on the masked language modeling task directly for the Proto++ also introduces a lot of noise: because embeddings are not well separated, adding 20 unlabeled samples per class adds to much noise for the model to separate classes. This effect is amplified on OOS dataset, as the number of classes is high.
Conclusion In this paper we introduced a new state-of-the-art, hierarchical clustering inspired method for pseudolabeling intent detection datasets. This is of the utmost practical interest in order to help linguists to faster provide new intent detection model as the intent classes repository evolve with time. Our method, being hyper-parameter-free, is robust to various situations, even when the number of classes is high. Overall, it is straightforward to implement and still the best option available for pseudo-labeling user utterances in few-shot classification. Through an ablation study, we also show that our method is complementary to other baselines so that it contributes to aggregated approaches significantly. Moreover, we demonstrate that while Proto++ approach is known to out-perform its former version Proto on image classification, this does not apply for learning few-shot intent detection models.
social skills, 2. nonverbal communication, 3. verbal communication, 4. repetitive and stereotyped behaviours). For each subject, the data was acquired in three separate runs: the first and second (RUN 1 and RUN 2 ) were combined movie watching/ resting-state (RS) acquisitions, with the movie displayed from 5 to 353 s (348 s ¼ 5. 8 min duration), and the RS scanning lasting from 386 to 678 s (292 s ¼ 4. 9 min). The third run (RUN 3 ) was a pure RS acquisition, which lasted for 310 s (5. 2 min). All RS runs were conducted with eyes closed, instructing the subjects not to fall asleep. Subjects were asked to refrain from moving as much as they could, and RUN 3 was not recorded in 4 subjects due to claustrophobia. Part of this data was already previously analysed in a combined EEG/fMRI study of speech processing  (Jochaut et al.
Preprocessing Preprocessing of the data was similar to that of  Bolton et al. (2018). Briefly, we used SPM12 (www. fil. ion. ucl. ac. uk) and a combination of in-house MATLAB scripts and scripts from the DPARSFA toolbox  (Yan and Zang, 2010). Functional volumes were realigned to the first image and co-registered to the structural image, which was segmented to derive the nonlinear deformation field between native and MNI spaces. Voxel-wise functional time courses were detrended, and constant, linear, quadratic, average cerebrospinal fluid (CSF) and average white matter (WM) time courses were regressed out. To generate CSF and WM time courses, we averaged the signals contained in conservative masks from the DPARSFA toolbox. As an additional verification, we also included the regression of the global signal-a historically controversial preprocessing step  (Murphy et al.
4). Mean FD did not differ significantly between the TD and ASD groups (p ¼ 0:08) as assessed by a group Â run ANOVA design; however, there was a significant difference for the percentage of scrubbed out frames (p ¼ 0:0059). This prompted us to perform additional analyses to verify that motion-induced biases remained minimal and did not influence our findings: we thus reran our methodological pipeline on the full dataset using an alternative strategy to scrubbing, wavelet despiking  (Patel and Bullmore, 2015) , where there is no censoring of data points. Despiking was applied prior to the detrending step. The results of these additional analyses can be found in the Supplementary Material, section 5. The data was then regionally averaged using a previously derived functional parcellation of the brain (2-level temporal correlation algorithm from  Craddock et al.