Venue: NAACL 2021

# Few-Shot Text Classification with Triplet Networks,

Data Augmentation, and Curriculum Learning

 Jason Wei\({}^{\circledR}\)  Chengyu Huang\({}^{\circledR}\)  Soroush Vosoughi\({}^{\circledR}\) Yu Cheng\({}^{\circledR}\) Shiqi Xu\({}^{\circledR}\)

\({}^{\circledR}\)ProtagoLab International Monetary Fund

Dartmouth College Microsoft AI

{jason,xu}@protagolabs.com huangchengyu24@gmail.com soroush@dartmouth.edu yu.cheng@microsoft.com

###### Abstract

Few-shot text classification is a fundamental NLP task in which a model aims to classify text into a large number of categories, given only a few training examples per category.

This paper explores data augmentation--a technique particularly suitable for training with limited data--for this few-shot, highly-multiclass text classification setting. On four diverse text classification tasks, we find that common data augmentation techniques can improve the performance of triplet networks by up to 3.0% on average.

To further boost performance, we present a simple training strategy called _curriculum data augmentation_, which leverages curriculum learning by first training on only original examples and then introducing augmented data as training progresses. We explore a _two-stage_ and a _gradual_ schedule, and find that, compared with standard single-stage training, curriculum data augmentation trains faster, improves performance, and remains robust to high amounts of noising from augmentation.

+
Footnote †: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5493–5500 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Traditional text classification tasks such as sentiment classification Socher et al. (2013) typically have few output classes (e.g., in binary classification), each with many training examples. Many practical scenarios such as relation classification Han et al. (2018), answer selection Kumar et al. (2019), and sentence clustering Mnasri et al. (2017), however, have a converse setup characterized by a large number of output classes Gupta et al. (2014), often with few training examples per class. This scenario, which we henceforth refer to as _few-shot, highly-multiclass text classification_, is a common setting in NLP applications and can be challenging due to the scarcity of training data.

Data augmentation for NLP has seen increased interest in recent years Wei and Zou (2019); Qiu et al. (2020). In traditional text classification tasks, it has been shown that although performance improvements can be marginal when training data is sufficient, augmentation is especially beneficial in limited data scenarios Xie et al. (2020). As such, we hypothesize that the few-shot, highly-multiclass text classification scenario is a suitable context for data augmentation.

Based on this motivation, our paper makes two main contributions.

* First, we apply popular data augmentation techniques to the common triplet loss Schroff et al. (2015) approach for few-shot, highly multiclass classification, finding that out-of-the-box augmentation can improve performance noticeably.
* We then propose a simple curriculum learning strategy called _curriculum data augmentation_ and experiment with two schedules, as shown in Figure 1. A _two-stage_ curriculum, which first trains on original data and then introduces aug

Figure 1: Schematic showing the two types of curriculum augmentation that we propose. \(\tau\) is a parameter that controls augmentation temperature (fraction of perturbed tokens).



mented data of fixed temperature (amount of noising), achieves slightly better performance than standard augmentation, while training faster and remaining more robust to high temperatures. A _gradual_ curriculum, which also first trains on original data only but gradually increases augmentation temperature at each subsequent stage, takes longer to converge but improves more than 1% over standard augmentation.

## 2 Curriculum Data Augmentation

**Motivation.** Inspired by human and animal learning, curriculum learning Bengio et al. (2009) posits that neural networks train better when examples are not randomly presented but instead organized in a meaningful order that gradually shows more concepts and complexity. Traditionally, curriculum learning approaches first assume that a range of example difficulty exists in the data and then leverage various heuristics to sort examples by difficulty and train models on progressively harder examples Bengio et al. (2009); Tsvetkov et al. (2016); Weinshall et al. (2018). A newer school of thought, however, has noted that instead of discovering a curriculum in existing data, data can be intentionally modified to dictate an artificial range of difficulty Korbar et al. (2018); Ganesh and Corso (2020)--this is the approach we will take here.

**Our approach.** Unlike data augmentation in computer vision where augmented data undoubtedly resembles original data, in text, data augmentation techniques might introduce linguistic adversity and therefore can be seen as a form of noising Li et al. (2017); Wang et al. (2018), where noised data is harder to learn from than unmodified original data. As such, we can create an artificial curriculum in the data by leveraging controlled application of data augmentation, starting by training on only original data and then adding augmented data with a higher levels of noising as training progresses. Specifically, we propose two simple schedules. **(1) Two-stage** curriculum data augmentation calls for one stage of training with only original data, followed by one stage of training with augmented data of fixed temperature. **(2) Gradual** curriculum data augmentation involves one stage of training with only original data, followed by multiple stages of training with augmented data where the temperature of augmented data (i.e., fraction of perturbed tokens) gradually increases each stage.

## 3 Experimental Setup

We conduct empirical experiments to evaluate curriculum data augmentation on a variety of text classification tasks using a triplet loss model.1

Footnote 1: Code is made publicly available at [https://github.com/jasonwei20/triplet-loss](https://github.com/jasonwei20/triplet-loss).

### Datasets

We will consider four diverse few-shot, highly-multiclass text classification scenarios:

1. [leftmargin=*]
2. **HUFF** (\(c=41\)). The HuffPost Dataset categorizes 200k news headlines from 2012-2018 into 41 categories such as politics, wellness, entertainment, and travel Misra (2018). We use all 41 categories and perform a 70%-30% train-test split by class.
3. **FEWREL** (\(c=64\)). The FewRel dataset contains sentences categorized by a relationship between its specified head and tail tokens such as 'capital of,' 'member of,' and 'birth name' Han et al. (2018). We use all 64 classes given in the posted training set, splitting 100 examples per class into a test set, with the remainder of the examples going into the training set.
4. **COV-C** (\(c=87\)). The COVID-Q dataset classifies questions into 89 clusters where all ques

Figure 2: Example training plot for the HUFF dataset (41 classes, with 5 examples per class) using EDA augmentation Wei and Zou (2019). Our proposed two-stage curriculum (second stage starts at four-thousand updates) trains faster and achieves slightly higher performance compared with standard augmentation while using the same number of updates. Our proposed gradual curriculum (which here linearly increases augmentation temperature \(\tau\) by 0.1 at \(\{4,8,12,16,20\}\)-thousand updates) outperforms both standard augmentation and the two-stage curriculum, but takes longer to converge. Results shown are averaged over thirteen random seeds.

 

tions in a cluster ask about the same thing (Wei et al., 2020). We use the train-test split with three training examples per class as given by the authors. We find that 2 of the 89 classes in the training set actually have only two examples per class instead of the reported three, and so we remove these classes from the training and test sets and use the 87 classes that remain.
4. **AMZN** (\(c=318\)). The Amazon product review dataset aims to categorize a product into a certain class given a review (Yury, 2020). We only consider the 318 'level-3' classes given in this dataset with at least six examples per class.

To balance the class distribution during experiments, we randomly sample \(N_{c}\) examples per class to be used for training, with \(N_{c}\) varying based on the experiment and dataset. Our sampled training sets for COV-C and AMZN have \(N_{c}\!=\!3\) examples per class, and our training sets for HUFF and FEWREL have \(N_{c}\!=\!10\), a common low-resource scenario.2 For all experiments, we use top-1 accuracy (%) as the evaluation metric.

Footnote 2: For COV-C, the given training set size is \(N_{c}\!=\!3\), and for AMZN, \(N_{c}\!=\!3\) is the largest possible such that the training set is balanced by class.

### Triplet Loss Model

For few-shot, highly-multiclass classification, a common approach is the _triplet loss_ classifier (Schroff et al., 2015), first developed for facial recognition and now also used in NLP dos Santos et al. (2016); Ein Dor et al. (2018); Lauriola and Moschitti (2020). Specifically addressing few-shot classification, a triplet loss network minimizes distance between examples with the same label and maximizes distance between examples with different labels. During training, given a triplet of (anchor \(a\), positive example \(p\), and negative example \(n\)), a triplet loss network minimizes:

\[L=\sum_{i}d(a,p)-d(a,n)+\alpha~{}, \tag{1}\]

where \(\alpha\) is a margin enforced between positive and negative pairs, and \(d(\cdot)\) computes the distance between the input encodings of two examples. To sample triplets, we will consider two strategies: _random sampling_, which selects triplets randomly, and _hard negative mining_(Schroff et al., 2015), where triplets are sampled such that \(d(a,p)+\alpha>d(a,n)\). At evaluation time, a triplet loss classifier returns the class of the example in the training set with the smallest distance to a given test example. Indeed, both triplet loss and data augmentation target training with limited data, and so combining them seems particularly promising for the the few-shot classification scenario.

For our model, we use standard BERT-base with average-pooled encodings and then train a two-layer triplet loss network on top of these encodings. Our triplet loss network architecture contains a linear layer with 200 hidden units, tanh activation, a dropout layer with \(p=0.4\), and a final linear layer with 40 hidden units. We use cosine distance, a margin of \(\alpha\!=\!0.4\), a batch size of 64 triplets, and a learning rate of \(2\times 10^{-5}\).

### Augmentation Techniques

We implement EDA Wei and Zou (2019), a popular combination of token-level augmentation techniques (synonym replacement, random insertion, random swap, random deletion) that defines their temperature parameter \(0\leq\tau\leq 1\) as the fraction of perturbed tokens, in SS4.1-4.4, and explore four other techniques in SS4.5.

### Schedules

For the two-stage curriculum, we started by training on original data only, and when validation loss converges, we introduce augmented data of fixed temperature at an augmented to original data ratio of 4:1. For the gradual curriculum, we begin with a temperature of \(\tau\!=\!0.0\) (equivalent to no augmentation) and then linearly increase the temperature by 0.1 every time validation loss plateaus, up to a final temperature of 0.5. Schedules for each dataset are shown in the Appendix. Figure 2 shows an example training plot with our proposed curriculum schedules.

## 4 Results

### Curriculum Data Augmentation

This section compares no augmentation, standard augmentation, and curriculum augmentation for triplet loss networks using two different triplet sampling strategies. Table 1 summarizes these results for five random seeds. We also implement a cross-entropy loss classifier for reference.

For triplet loss using random sampling, a model with no augmentation achieved a mean accuracy across our four datasets of 30.2%, and standard augmentation improved performance noticeably by +1.9%. Two-stage curriculum augmentation, which 

trains for the same number of updates as standard augmentation, achieved a mean accuracy of 32.4%, outperforming standard augmentation by +0.5%. The gradual curriculum further improved +1.0% over the two-stage curriculum.

For triplet loss with hard negative mining, standard augmentation substantially improved +3.0% over no augmentation, as adding in augmented data, which is more difficult to classify, likely helped generate a more diverse set of hard negatives. The two-stage curriculum still maintained small improvement over standard augmentation here, and the gradual curriculum provided an even-stronger boost of +4.4% over no augmentation, possibly because increasing the temperature of augmented data over time facilitated hard-negative mining more so than using a constant temperature.

Notably, the largest gains for all augmentation types were on COV-C (up to +9.4%). We hypothesize that this occurred not necessarily because of COVID-C's smaller data size; rather, there was likely more overfitting to be mitigated by data augmentation as a result of the greater semantic difference between COV-C and the corpus used to pre-train BERT, compared with the other three datasets.

### Ablation: Dataset Size

This ablation investigates how data augmentation performs for different dataset sizes. Figure 3 shows these results for hard negative mining averaged over HUFF and FEWREL, our two datasets where sufficient data is available. The two-stage curriculum outperformed standard augmentation by a small margin, although both dropped in performance at \(N_{c}=20\), consistent with prior findings on the diminished effect of data augmentation for larger datasets Xie et al. (2020); Andreas (2020). The gradual curriculum, on the other hand, maintained relatively robust improvement for all dataset sizes explored.

### Ablation: Augmentation Temperature

Effective curriculum learning necessitates a range of difficulty in training data. In our case, this range is controlled by _augmentation temperature_, a parameter that dictates how perturbed augmented examples are and therefore affects the distribution of difficulty in training examples. When the distribution of difficulty in data is larger, we should expect

Figure 4: Curriculum data augmentation outperforms standard data augmentation for a range of different augmentation temperatures \(\tau\). Whereas standard augmentation performs better at lower \(\tau\), curriculum data augmentation helps even for higher \(\tau\) (e.g., \(\tau\geq 0.2\)).

\begin{table}
\begin{tabular}{l|r r r r|r r}  & HUFF & FEWREL & COV-C & AMZN & \\  & \(c\!=\!41\) & \(c\!=\!64\) & \(c\!=\!87\) & \(c\!=\!318\) & Average & \(\Delta\) \\ \hline Cross-entropy loss & 13.3 \(\pm\) 2.1 & 32.4 \(\pm\) 2.3 & 26.1 \(\pm\) 0.8 & 2.0 \(\pm\) 0.3 & 18.5 & - \\  + standard data augmentation & 16.3 \(\pm\) 2.4 & 33.0 \(\pm\) 1.1 & 24.0 \(\pm\) 1.6 & 2.2 \(\pm\) 0.4 & 18.9 & +0.4 \\ \hline Triplet loss with random sampling & 20.9 \(\pm\) 1.0 & 43.6 \(\pm\) 1.2 & 39.7 \(\pm\) 1.0 & 16.4 \(\pm\) 1.3 & 30.2 & - \\  + standard data augmentation & 22.2 \(\pm\) 1.4 & 44.2 \(\pm\) 1.6 & 45.4 \(\pm\) 1.8 & 16.5 \(\pm\) 1.7 & 32.1 & +1.9 \\  **+ curriculum data augmentation: two-stage** & 22.3 \(\pm\) 1.6 & 44.2 \(\pm\) 1.8 & 46.5 \(\pm\) 1.7 & 17.2 \(\pm\) 1.3 & 32.6 & +2.4 \\  **+ curriculum data augmentation: gradual** & 23.7 \(\pm\) 1.2 & 46.1 \(\pm\) 0.9 & 47.1 \(\pm\) 1.3 & 17.6 \(\pm\) 1.0 & 33.6 & +3.4 \\ \hline Triplet loss with hard negative mining & 21.0 \(\pm\) 1.2 & 44.6 \(\pm\) 1.2 & 39.5 \(\pm\) 1.0 & 16.2 \(\pm\) 0.9 & 30.3 & - \\  + standard data augmentation & 22.6 \(\pm\) 1.8 & 45.0 \(\pm\) 1.6 & 48.2 \(\pm\) 0.9 & 17.4 \(\pm\) 1.7 & 33.3 & +3.0 \\  **+ curriculum data augmentation: two-stage** & 22.6 \(\pm\) 1.8 & 45.7 \(\pm\) 1.4 & 47.6 \(\pm\) 1.3 & 17.9 \(\pm\) 1.1 & 33.5 & +3.2 \\  **+ curriculum data augmentation: gradual** & 23.8 \(\pm\) 0.9 & 47.1 \(\pm\) 1.4 & 48.9 \(\pm\) 0.9 & 18.9 \(\pm\) 0.9 & 34.7 & +4.4 \\ \end{tabular}
\end{table}
Table 1: Accuracy (%) on four diverse highly multiclass classification tasks for no augmentation, standard augmentation, and curriculum augmentation. \(c\): number of classes; \(\Delta\): improvement compared with no augmentation.

Figure 3: Improvement from data augmentation for different dataset sizes (results averaged over HUFF and FEWREL datasets).

 

a greater improvement from curriculum learning.

Figure 4 compares standard and two-stage curriculum augmentation for various temperatures, with results averaged over all four datasets. At low temperature, augmented examples remained pretty similar to original examples, and so the range of difficulty in examples was small and therefore curriculum learning showed little improvement. At higher temperatures, however, augmented examples became quite different from original examples, and so the range of difficulty in examples was much larger and therefore curriculum data augmentation improved over standard augmentation more. Whereas Wei and Zou (2019) recommend \(\tau\in\{0.05,0.1\}\), our curriculum framework liberates us to use much larger \(\tau\) and maintain relatively robust improvements even at \(\tau\in\{0.4,0.5\}\) when standard augmentation is no longer useful.

### Ablation: Curriculum Schedules

The gradual curriculum linearly increases temperature \(\tau\) from 0.0 to 0.5 in six stages, and so to isolate the effect of this curriculum, in this section we compare it with a control schedule (where the \(\tau\) in each stage is decided randomly) and an anti-curriculum schedule (where \(\tau\) linearly decreases from 0.5 to 0.0 in six stages). As expected, these results, shown in Table 2, indicate that the curriculum contributes substantively over the control schedule.

### For Various Augmentation Techniques

As our experiments so far have focused on EDA augmentation (Wei and Zou, 2019), this section explores other common techniques in the curriculum framework: **(1) Token Substitution** replaces words with WordNet synonyms (Zhang et al., 2015); **(2) Pervasive Dropout** applies word-level dropout with probability \(p{=}0.1\)(Sennrich et al., 2016a); **(3) SwitchOut** replaces a token with a randomly token uniformly sampled from the vocabulary (Wang et al., 2018); and **(4) Round-Trip Translation** translates text into another language and then back into the original language (Sennrich et al., 2016b). Figure 5 compares standard and two-stage curriculum results averaged over all datasets. EDA improved performance the most, perhaps because it combines four token perturbation functions, creating more diverse noise compared with using a single operation.

## 5 Related Work and Conclusions

Our work combines curriculum learning, data augmentation, and triplet loss, and is inspired by prior work in these areas. In vision, several papers have proposed reinforcement learning policies for data augmentation (Cubuk et al., 2019; Ho et al., 2019), and hard negative mining (Schroff et al., 2015; Song et al., 2016) itself can be seen as a form of curriculum learning. In NLP, the work of Kumar et al. (2019) is perhaps most similar to ours--they show that sampling strategies are key for improving performance with triplet loss networks. We see our work as the first to explicitly analyze curriculum learning for data augmentation in text.

In closing, we have proposed a curriculum data augmentation framework that is simple yet provides empirical performance improvements, a compelling case for the combination of ideas explored. Our approach exemplifies how data augmentation can create an artificial range of example difficulty that is helpful for curriculum learning, a direction that potentially warrants future research.

## Acknowledgements

We thank Ruibo Liu, Weicheng Ma, Jerry Wei, and Chunxiao Zhou for feedback on the manuscript. We also thank Kai Zou for organizational support.

\begin{table}
\begin{tabular}{l|c c c c|c} Schedule & HUFF & FEWREL & COV-C & AMZN & Avg. \\ \hline Curriculum & 23.7 & 46.1 & 48.1 & 17.6 & **33.63** \\ Control & 23.5 & 45.3 & 46.3 & 17.1 & 33.05 \\ Anti & 23.3 & 44.8 & 46.2 & 17.5 & 32.95 \\ \end{tabular}
\end{table}
Table 2: Gradual curriculum augmentation with three schedules. Curriculum: temperature \(\tau\) increases. Control: \(\tau\) is randomly selected every fifty updates. Anti: decreasing \(\tau\). Results are shown for ten seeds.

Figure 5: Common text data augmentation techniques work better in the curriculum framework (two-stage) than standard single-stage training. A model with no data augmentation achieved a performance of 30.2%.

 



## References

* J. Andreas (2020)Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7556-7566. External Links: Link Cited by: SS1.
* Y. Bengio, J. Louradour, R. Collobert, and J. Weston (2009)Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, New Orleans, Louisiana, USA, pp. 41-48. External Links: ISBN 978-3-319-

pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
* Song et al. (2016) H. O. Song, Y. Xiang, S. Jegelka, and S. Savarese. 2016. Deep metric learning via lifted structured feature embedding. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 4004-4012.
* Tsvetkov et al. (2016) Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Brian MacWhinney, and Chris Dyer. 2016. Learning the curriculum with Bayesian optimization for task-specific word representation learning. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 130-139. Association for Computational Linguistics.
* Wang et al. (2018) Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. SwitchOut: an efficient data augmentation algorithm for neural machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 856-861, Brussels, Belgium. Association for Computational Linguistics.
* Wei and Zou (2019) Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 6382-6388, Hong Kong, China. Association for Computational Linguistics.
* Wei et al. (2020) Jerry Wei, Chengyu Huang, Soroush Vosoughi, and Jason Wei. 2020. What Are People Asking About COVID-19? A Question Classification Dataset. _arXiv preprint arXiv:2005.12522_.
* Weinshall et al. (2018) Daphna Weinshall, Gad Cohen, and Dan Amir. 2018. Curriculum learning by transfer learning: Theory and experiments with deep networks. In _Proceedings of the 35th International Conference on Machine Learning_, volume 80 of _Proceedings of Machine Learning Research_, pages 5238-5246. PMLR.
* Xie et al. (2020) Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. 2020. Unsupervised data augmentation for consistency training.
* Yury (2020) Kashnitsky Yury. 2020. Hierarchical text classification of Amazon product reviews.
* Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_, pages 649-657. Curran Associates, Inc.

 



## Appendix A Appendix

Table 3 shows the training schedules for single-stage, two-stage curriculum, and gradual curriculum training.

All models in standard single-stage training (with and without augmentation) for the same dataset trained for the same number of updates; convergence typically took longer with augmentation compared to without augmentation.

Curriculum two-stage training employs a first stage of only original data and a second stage of augmented data, using the same number of updates as single-stage training in total. We determined the number of updates in the first stage based on when training loss plateaued in the training plot for training with no augmentation.

The gradual curriculum starts with one stage of training with original data only and then increases the augmentation temperature by 0.1 in each of the following five stages. To determine the number of updates in each stage, we examined training plots in preliminary experiments and increased the augmentation temperature (i.e., begun the next stage) whenever training loss plateaued. Since our preliminary experiments already showed relatively strong performance improvements, we did not perform an extensive hyperparameter search or experiment with automatic scheduling, which could further improve performance. As the gradual curriculum trains on more diverse set of augmented data, more updates are needed than in the single-stage and two-stage schedules.

For evaluation, we evaluate our models every 200 updates for COV-C and every 300 updates for HUFF, FEWREL, and AMZN, reporting the highest validation accuracy achieved during training.

In all models, we include 20% original data whenever augmented data is used, in order to prevent catastrophic forgetting.

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline  & HUFF & FEWREL & COVID-C & AMZN \\  & \(c=41\) & \(c=64\) & \(c=87\) & \(c=318\) \\ \hline Single-Stage Training & & & & \\ Updates until convergence, no aug. (approx) & 4,000 & 8,000 & 4,000 & 15,000 \\ Update until convergence, aug. (approx) & 10,000 & 10,000 & 8,000 & 20,000 \\ Total updates & 15,000 & 15,000 & 15,000 & 25,000 \\ \hline Curriculum: Two-Stage Stage & & & & \\ Stage 1 updates & 4,000 & 6,000 & 4,000 & 8,000 \\ Stage 2 updates & 11,000 & 9,000 & 11,000 & 17,000 \\ Total updates & 15,000 & 15,000 & 15,000 & 25,000 \\ \hline Curriculum: Gradual & & & & \\ Stage 1 updates & 6,000 & 6,000 & 6,000 & 10,000 \\ Updates per stage in stages 2-6 & 6,000 & 6,000 & 4,000 & 8,000 \\ Total updates & 36,000 & 36,000 & 26,000 & 50,000 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Training schedules for single-stage training, two-stage curriculum training, and gradual curriculum training.