Introduction
Text classification can be categorized according to the text length of the data, from sentence-level classification BIBREF0 to document-level classification BIBREF1, BIBREF2. One kind of such task is sentiment classification BIBREF3, a subtask of sentiment analysis BIBREF4, BIBREF5 where we are to predict the sentiment/rating given a review written by a user. In some domains, the length of these reviews varies widely. For example, well-known review websites in East Asia such as Naver Movies and Douban Movies provide two channels for users to write reviews, depending on their preferred length. Figure FIGREF4 shows the review channels provided in Naver Movies.
The first channel is a short review channel, which contains large amounts of reviews, and enforces users to write short reviews accompanied by rating labels. Although labeled, these reviews lack expressiveness to extract useful information. In contrast, the second channel is a long review channel, which contains few long and detailed reviews with descriptions of different aspects about the product/service. Despite being more expressive, long reviews are often not accompanied by sentiment labels, which most supervised sentiment classification models would require.
We study the “transferability” from one review channel to the other. That is, we try to answer whether a text classifier trained on a dataset with length $\alpha $ can predict a text with length $\beta $, where $\alpha $ and $\beta $ differ by a large margin (e.g., sentences versus paragraphs). This is an important question because there are scenarios where we may have better and more sufficient labeled text data, but we want to classify unlabeled texts with different length. For long to short transfer, more expressive long reviews can be leveraged for training a sentiment classifier for short and context-sparse reviews. For short to long transfer, large amounts of short reviews can be used as supervision to train a classifier for long reviews.
To motivate the non-triviality of such transfer, we train an out-channel (OC) classifier that uses short texts to predict long texts, and an in-channel (IC) classifier that uses long texts on both training and prediction. We also experiment conversely. We use three kinds of classifiers: bag-of-words (BoW), convolutional neural networks BIBREF6, and BERT multilingual BIBREF7. We calculate the transfer loss BIBREF8, which is the difference between the out-channel and in-channel classifier errors (i.e., $\text{TL}=0$ means trivially transferable). Table TABREF7 shows that, though using a better inductive bias such as CNN and BERT seems to slightly lower TL, it remains significantly high, consistently suggesting that length transfer is non-trivial.
Our first contribution is thus to define a new task called Cross Length Transfer (CLT). CLT is a task similar to Cross Domain BIBREF9 and Cross Lingual Transfer BIBREF10 where the difference between the source and target texts is the text length, having non-trivial influence that is shown in Table TABREF7. Our second contribution is to show that models from similar tasks (e.g. Cross Domain Transfer and Multiple Instance Learning) are not effective for CLT and even yield negative transfer, as we elaborate in Section SECREF10 and empirically show in Section SECREF4. Finally, we present two new models specifically for CLT: a strong baseline called BaggedCNN that treats long texts as bags containing short texts, and a state-of-the-art CLT model called Length Transfer Networks (LeTraNets). LeTraNets enables a two-way encoding scheme using multiple training mechanims, and accepts both short and long text inputs, where one such input is created artificially through concatenation or segmentation. Table TABREF7 shows that LeTraNets has the best transfer loss, and sometimes perform better than in-channel classifier (when TL is less than zero). We test our models using the multiple benchmark datasets we gathered and show that models from other tasks perform worse than our proposed strong baseline and that LeTraNets performs the best among all models. To the best of our knowledge, we are the first to study CLT.
Cross Length Transfer
Cross Length Transfer (CLT) is an unsupervised transfer learning task in which the setting is that the sampling distributions of the training and test data are different because the texts lengths are different (e.g., sentences and paragraphs). Formally, we suppose two sets of texts: a source set $\mathcal {S}$ in which we have labels, and a target set $\mathcal {T}$ in which we want to predict the labels. Moreover, we know that the text length distributions of $\mathcal {S}$ and $\mathcal {T}$ are different, such that an equality case exists as $|\mathcal {S}| = r |\mathcal {T}|$, where $|\mathcal {X}|$ is the mean length of the set $\mathcal {X}$, and $r \ne 1$ is a non-negative rate of difference between two mean lengths. There are two subtasks: long to short transfer where $r>1$ and thus $\mathcal {S}$ contains longer texts, and short to long transfer where $r<1$ and thus $\mathcal {S}$ contains shorter texts. A CLT model should effectively learn to predict the labels of $\mathcal {T}$, on both scenarios. A concrete and simple example is when $\mathcal {S}$ contains labeled sentence reviews and $\mathcal {T}$ contains unlabeled paragraph reviews. A CLT model uses $\mathcal {S}$ for training to effectively predict labels of reviews in $\mathcal {T}$. Also, the same CLT model should be able to do effective prediction vice versa, i.e., when $\mathcal {S}$ are paragraph reviews and $\mathcal {T}$ are sentence reviews. Previous unsupervised transfer learning tasks, i.e. Cross Domain Transfer BIBREF9 and Cross Lingual Transfer BIBREF11, are similar to CLT but have concrete differences. Generally, the goal of these tasks is to map semantic domains, contextually or linguistically, of both $\mathcal {S}$ and $\mathcal {T}$ into a shared space, by aligning the vocabulary BIBREF12, BIBREF13, expanding domain-specific lexicons BIBREF14, BIBREF15, generating labeled samples BIBREF16, and learning to indiscriminate between domains BIBREF17, BIBREF18. These methods are generally symmetric; i.e., even when $\mathcal {S}$ and $\mathcal {T}$ interchange, the same method can be applied easily. However, in CLT, both $\mathcal {S}$ and $\mathcal {T}$ are already in the same contextual and linguistic domains, thus previous methods would not work. Also, CLT brings two new challenges against devising a symmetric model. First, texts with different context richness may have different properties they focus on: hierarchical structures BIBREF2 may be more important for document-level reviews while finding lexical/phrasal cues BIBREF0 may be more important for sentence-level reviews. Second, words on texts with different lengths may have different semantic intensity. For example, “good” may have a very high positive sentiment intensity on short texts, and a relatively low positive sentiment intensity on long ones.
Cross Length Transfer ::: Benchmark Datasets
We provide three pairs of short/long datasets from different domains (movies and restaurants) and from different languages (English and Korean) suitable for the task: Mov_en, Res_en, and Mov_ko. Most of the datasets are from previous literature and are gathered differently The Mov_en datasets are gathered from different websites; the short dataset consists of hand-picked sentences by BIBREF19 from document-level reviews from the Rotten Tomatoes website, while the long dataset consists of reviews from the IMDB website obtained by BIBREF20. The Res_en dataset consists of reviews from Yelp, where the short dataset consists of reviews with character lengths less than 140 from BIBREF21, while reviews in the long dataset are gathered from BIBREF20. We also share new short/long datasets Mov_ko, which are gathered from two different channels, as shown in Figure FIGREF4, available in Naver Movies. Unlike previous datasets BIBREF9, BIBREF22 where they used polarity/binary (e.g., positive or negative) labels as classes, we also provide fine-grained classes, with five classes of different sentiment intensities (e.g., 1 is strong negative, 5 is strong positive), for Res_en and Mov_ko. Following the Cross Domain Transfer setting BIBREF9, BIBREF23, BIBREF24, we limit the size of the dataset to be small-scale to focus on the main task at hand. This ensures that models focus on the transfer task, and decrease the influence of other factors that can be found when using larger datasets. Finally, following BIBREF22, we provide additional unlabeled data for those models that need them BIBREF9, BIBREF23, except for the long dataset of Mov_ko, where the labeled reviews are very limited. We show the dataset statistics in Table TABREF9, and share the datasets here: https://github.com/rktamplayo/LeTraNets.
Cross Length Transfer ::: Possible Existing Solutions ::: Cross Domain Transfer (CDT)
CDT offers models that effectively transfer domain-independent features from two different domains. The most popular non-neural CDT model is Structural Correspondence Learning BIBREF25blitzer2007biographies, a method that identifies feature correspondence from different domains using pivot features. A recent neuralized extension is Neural SCL BIBREF26ziser2017neural, in which an autoencoder module is integrated to SCL. The CDT literature is vast, and we refer the readers to BIBREF27 and BIBREF28 for overviews. Although these models may see improvements due to a possible difference in vocabulary (especially when the review channels are different), these improvements may be marginal since the domain of the datasets is the same.
Cross Length Transfer ::: Possible Existing Solutions ::: Multiple Instance Learning (MIL)
MIL is a task where given the labels of a bag of multiple instances, we are to label the individual instances BIBREF29. In the text classification domain, MIL is often devised as segment-level classification BIBREF30, BIBREF31, where documents are bags and sentences in the documents are segments. The most recent MIL model is the Multiple Instance Learning Network BIBREF32;angelidis2018multiple, where they used attention-based polarity scoring to identify segment labels. MIL models can be used in long to short transfer, where we assume that segment labels in long texts can be used to label short reviews. However, they (a) assume that segments from long data, which rely on inter-sentence semantics, are comparable to self-contained short texts, and (b) are ineffective on short to long transfer because it needs multiple sentences to train components of the model for document-level classification.
Cross Length Transfer ::: Possible Existing Solutions ::: Weak Supervision
A simple yet possible solution for short to long transfer is a three-step approach where we (1) cluster the short texts into several long texts, (2) infer the class labels of the clusters, and (3) use the labeled clusters as weak supervision to create a classifier. Micro Aspect Sentiment Model BIBREF33;amplayo2017aspect does (1) and (2) automatically. For (3), we can train a classifier such as CNNs BIBREF0 to predict labels of long texts. One critical issue of this solution is that since both clustering and class labels are inferred, there is a high chance that at least one of them is incorrect. This thus creates compounding errors that decrease the performance of the model.
Our Models ::: BaggedCNN: A Strong Baseline
We present BaggedCNN, a simple yet strong baseline to the CLT task. BaggedCNN is a model derived from MILNet BIBREF31. MILNet uses CNN to encode segments, BiGRU BIBREF34 to calculate attention weights, and gated polarity to calculate document-level probabilities. We refer the readers to the original paper for more details. We improve using two key modifications: (a) removing the sequential connections (i.e., BiGRU) between segments, and (b) using a single classifier for both the segments and full document. For each document divided into segments $D=\lbrace S_i\rbrace $, BaggedCNN starts by encoding the segments using a CNN classifier called $\text{CNN}_{bag}$. Then, we pool the segment encodings into one vector using attention mechanism. Finally, we use a logistic regression classifier that can be used to classify either the segments or the document. This is possible since the vectors of both segments and document are in the same vector space:
The model is trained differently depending on the transfer task: For long to short transfer, we minimize the cross-entropy loss between the actual and predicted class of the document $\mathcal {L}_d$. For short to long transfer, we minimize the mean cross entropy loss between the actual and predicted class of the segments $\sum \mathcal {L}_{s_i}/n, 1\le i \le n$. Note that BaggedCNN is reduced to a model where average pooling is done instead of the attention mechanism. At test time, we use $y_d$ for classification. While it has been shown that removing the sequential structure in the document level (i.e., BiGRU in the case of MILNet) decreases the performance of the document classifier BIBREF35, BIBREF2, we argue that this removal is effective on the CLT task because of inter-segment independence. That is, sentences in the document are treated similar to short texts. We also show in our experiments that BaggedCNN performs better than MILNet. However, BaggedCNN still fails to consider two things. First, while the model relaxes the strong assumption on similarity between segments and short texts, by removing the sequential connections, most segments cannot be treated as stand-alone short texts. For example, the segment “Yet it is salty.” is not a stand-alone short review. Second, when doing short to long transfer, the input short text is just one segment, thus the model is reduced into a weaker hierarchical CNN classifier.
Our Models ::: LeTraNets: Length Transfer Networks
We improve BaggedCNN by proposing a model called Length Transfer Networks (LeTraNets), as shown in Figure FIGREF16. LeTraNets is composed of two classifiers: a stand-alone CNN classifier with text encoder $\text{CNN}_{lone}$, and BaggedCNN, which includes a segment-level text encoder $\text{CNN}_{bag}$. The $\text{CNN}_{lone}$ encoder is used to capture holistic textual features, while the $\text{CNN}_{bag}$ encoder is used to capture segment-level textual features, assuming there is a bigger text that owns the segments.
For each data instance, LeTraNets accepts two kinds of inputs: a long text $D={w_d}$ and a set of short texts $S={{w_{s_0}},...,{w_{s_n}}}$. However, the task setting only provides either one of long texts or short texts as input. We thus create pseudo-texts from the available text data through the following methods. In the long to short transfer task, we use segments in long texts as pseudo-short texts, as used in BaggedCNN. In the short to long transfer task, we concatenate a random number of short texts to create pseudo-long texts. The latter amounts to a possibly infinite number of long texts we can use for training. The short texts are encoded by both $\text{CNN}_{lone}$ and $\text{CNN}_{bag}$ as $s^{\lbrace l\rbrace }_i$ and $s^{\lbrace b\rbrace }_i$. The long texts are encoded using both $\text{CNN}_{lone}$ and BaggedCNN as $d^{\lbrace l\rbrace }$ and $d^{\lbrace b\rbrace }$:
The encoded long text vectors $d^{\lbrace l\rbrace }$ and $d^{\lbrace b\rbrace }$ and short text vectors $s^{\lbrace b\rbrace }_i$ and $s^{\lbrace l\rbrace }_i$ are used to classify their labels using softmax classifiers specific to the CNN encoders:
Our Models ::: LeTraNets: Length Transfer Networks ::: Training Mechanisms
There are two main issues when training the model in the CLT setting. First, both the stand-alone CNN classifier and BaggedCNN are disconnected, acting as two individual classifiers. Second, the model needs both labels for both short and long text data, but we are only given labels for one kind of data during training for each transfer setting. Solving the second issue is crucial for short to long transfer, as we cannot train the full model if we do not have labels for long data. To this end, we use three training mechanisms below that help mitigate these issues. We connect them on different levels. In the word-level, we use the same word embedding space for both classifiers. Beyond word-level, we use a training mechanism called Joint Training (JT). This concatenates the encoded text vectors, and creates another logistic regression classifier for the concatenated vector. This creates a connection between classifiers at the classification-level.
Beyond word-level, we introduce Prediction Regularization (PR) mechanism to train encoders with no labels. This regularizes the predictions of a weaker classifier based on the predictions of a stronger classifier. We consider BaggedCNN as the stronger classifier for long to short transfer, and $\text{CNN}_{lone}$ as the stronger classifier for short to long transfer. We use Kullback-Leibler divergence as the regularization function.
Finally, using the PR mechanism directly might not work because predictions from the stronger classifier may not be optimized yet. Hence, we use Stepwise Pretraining (SP) mechanism to pretrain specific parts of the model in a step-by-step fashion. First, we pretrain the stronger classifier, then the weaker classifier with PR mechanism, and finally the classifier of the JT mechanism. After pretraining, we train the full model. The training configurations are different depending on the transfer task, which is also shown in Figure FIGREF16. For long to short transfer, we use $p(y^{\lbrace j\rbrace }_{d})$ for the JT mechanism and $R_d$ for the PR mechanism. For short to long transfer, we use $p(y^{\lbrace j\rbrace }_{s_i})$ for the JT mechanism and $R_s$ for the PR mechanism. The final training objective is to minimize the loss function, depending on the text length:
where $\mathcal {L}^{\lbrace a\rbrace }_x$ is the cross-entropy loss between the actual and predicted values of the classifier $p(y^{\lbrace a\rbrace }_x)$, and $\lambda $ is tuned using a development set. At test time, we use $p(y^{\lbrace j\rbrace }_{d})$ and $p(y^{\lbrace j\rbrace }_{s_i})$ to classify the sentiment for long to short and short to long transfer, respectively.
Experiments ::: Experimental Settings
The dimensions of word vectors are set to 300. We use pre-trained GloVe embeddings BIBREF36 to initialize our English word vectors, and pre-trained FastText embeddings BIBREF37 to initialize our Korean word vectors. For all CNNs, we set $h=3,4,5$, each with 100 feature maps, following BIBREF0. We use dropout BIBREF38 on all non-linear connections with a dropout rate of 0.5. We set the batch size to 32. We use stochastic gradient descent over shuffled mini-batches with the Adadelta update rule BIBREF39 with $l_2$ constraint of 3. We experiment with a 5-fold cross-validation on the given source training set and report the average results.
Experiments ::: Comparison Models
We compare our models with the models from similar tasks as discussed in Section SECREF10. Specifically, we compare with (a) Cross Domain Transfer (CDT) models SCL BIBREF9 and NeuSCL BIBREF23, (b) CDT models with a CNN classifier integration BIBREF16 (SCL+CNN and NeuSCL+CNN), (c) a multiple-instance learning (MIL) model MILNet BIBREF31, (d) a weakly supervised model MASM+CNN BIBREF21. We remind that MILNet is only applicable to long to short transfer, and MASM+CNN is only applicable to short to long transfer. We use the available code provided by previous authors. Finally, we also compare with CNN BIBREF0, and a combination of two CNNs (CNNx2) as no-transfer baselines.
Experiments ::: Dataset and Evaluation
We use the datasets described in Table TABREF9 for all our experiments. We use the following evaluation metrics. For all datasets, we use accuracy (Acc) to measure the overall sentiment classification performance. Additionally, for fine-grained datasets, we use root mean squared error (RMSE) to measure the divergence between the predicted and ground truth sentiment scores. Finally, in order to compare models in an integrated manner, we report the average transfer ratio BIBREF40, a version of the transfer loss which is more adaptive to averaging, calculated as the average quotient between the transfer error and the in-domain baseline error, i.e. $\text{TR} = \sum _x e(\mathcal {S}_x,\mathcal {T}_x) / e_b(\mathcal {T}_x,\mathcal {T}_x)$, where $\mathcal {S}_x$ and $\mathcal {T}_x$ are the source and domain of dataset $x$, respectively, $e$ and $e_b$ are accuracy errors from the competing model and the baseline CNN model.
Experiments ::: Long to Short Transfer
We show the results for long to short transfer in the first part of Table TABREF20. Results show that Cross Domain Transfer models do not perform well, which confirms our hypothesis that they are not well suited for this task. MILNet performs well on polarity tasks, but performs poorly on fine-grained tasks, having worse performance than the no-transfer CNN baseline. This shows that although Multiple Instance Learning models are effective in classifying positive or negative sentiments, they are not flexible to fine-grained sentiment intensities, which differs when text lengths are different. On the other hand, BaggedCNN performs better than MILNet, proving that simplifying MIL models work well on CLT. Overall, LeTraNets performs the best among all models, having the best accuracies and RMSEs on all datasets and settings.
Experiments ::: Short to Long Transfer
We report the results for short to long transfer in the second part of Table TABREF20. Results show that Cross Domain Transfer models perform much worse compared to their performance in the long to short transfer task. The weak supervised model MASM+CNN performs the worst, having worse results than the no-transfer CNN baseline on all datasets. BaggedCNN also performs well in this task, even though it does not use its attention mechanism. This shows that BaggedCNN is a very tough-to-beat baseline for the CLT task. Finally, LeTraNets also outperforms all the models on this subtask.
Experiments ::: Transfer Ratio (TR)
Figure FIGREF30 shows the average transfer ratio (TR) of all competing models, where $\text{TR}=1$ means trivially transferable. The figure shows that the CDT models SCL and NeuSCL both obtain a larger transfer ratio compared to the no-transfer CNN baseline. The transfer ratios improve when CNN is integrated into both models, but does not improve much from the baseline. MILNet and BaggedCNN perform comparably on the long to short transfer task, where BaggedCNN performs slightly better. LeTraNets performs the best among the models, having transfer ratios less than 1.1.
Analyses ::: Ablation on Training Mechanisms
We investigate the performance of LeTraNets when the training mechanisms are not used. Specifically, we perform ablation tests on the Joint Training (JT), Prediction Regularization (PR), and Stepwise Pretraining (SP) mechanisms. The results in Table TABREF31 show that LeTraNets performs the best when all training mechanisms are used. Also, when used individually, all the training mechanisms boost up the performance of the model. Hence, we confirm that the training mechanisms help LeTraNets achieve good performance on the task.
Analyses ::: Performance per Text Length
We check the capability of LeTraNets to transfer across text lengths, by looking at its performance as the text length increases. Specifically, we compare the performance per text length of LeTraNets and CNN models, trained on either short texts (LeTraNetsshort and CNNshort) or long texts (LeTraNetslong and CNNlong), on Res_en short/long datasets. Figure FIGREF34 shows the results. CNN performs well when the text length is similar to the training dataset and performs poorly otherwise. LeTraNets, however, performs similarly on all kinds of text lengths although it is trained purely on a dataset of a specific length. More interestingly, LeTraNetsshort performs better than LeTraNetslong on longer texts, and unexpectedly performs worse on shorter texts. This suggests that LeTraNets weakens its ability to classify texts with the same length and improves its ability to classify texts with different length. This property is acceptable in our problem setup since we care on effectively classifying short (or long) texts more, assuming we only have access to long (or short) texts as training data. However, future work should explore on CLT models that perform well on both text lengths.
Analyses ::: On Topic Diversity
Longer texts can discuss diverse topics, while shorter texts are limited to few (or one) topics. In the sentiment classification domain, longer reviews may mention positive sentiments towards an aspect of a product, and then talk about negative sentiments towards another aspect. With this hypothesis, we examine whether LeTraNets can handle longer texts with diverse topics when trained on short texts. Specifically, we compare the performance per topic diversity of LeTraNets and CNN models, trained on short texts of Res_en dataset. We measure topic diversity as the Shannon index BIBREF41 of the topic distribution inferred by an LDA topic model BIBREF42 fit using the unlabeled data. Figure FIGREF36 shows the results. Results indicate that the performance increase of LeTraNets over CNN increases as the diversity of topics increases. This shows that for short to long transfer, LeTraNets is able to handle texts with topics that are more diverse, even when trained on short texts, which tend to have less diverse topics.
Analyses ::: Cross Domain and Length Transfer
Which between domain and text length should we consider to achieve a better performance? To answer this question, we combine Cross Domain Transfer (CDT) and Cross Length Transfer (CLT) into one task: Cross Domain and Length Transfer (CDLT) and compare the performance of CDT and CLT models on the task. We use the Mov_en and Res_en datasets to create four CDLT datasets, and check which between the CDT model NeuSCL+CNN and the CLT model LeTraNets achieves a higher increase in performance. The results are shown in Table TABREF38. We find that NeuSCL+CNN performs worse, obtaining accuracies worse than that of the no-transfer CNN baseline. LeTraNets performs better, obtaining significant increase in performance from the baseline. This shows that solving the non-transferability of length is more important to achieve a more effective sentiment classifier.
Conclusions
We defined a new task called Cross Length Transfer (CLT) to check the transferability across lengths of classification models. We set the grounds by defining the task, providing three benchmark datasets from different domains and languages, and introducing models from related tasks. We proposed two models: a strong baseline model called BaggedCNN, and LeTraNets, a model that improves over the weakness of BaggedCNN. Our multiple experiments show that LeTraNets demonstrates superior performance over all competing models. We aim to apply the CLT to other classification tasks, such as natural language inference BIBREF43, where text length is influential towards overall model performance BIBREF44.
Acknowledgments
We would like to thank the anonymous reviewers for their helpful feedback and suggestions. Amplayo is grateful to be supported by a Google PhD Fellowship. This research was supported by MSIT (Ministry of Science and ICT), Korea, under ITRC program (IITP-2019-2016-0-00464) supervised by IITP. Hwang is the corresponding author.

Question: What dierse domains and languages are present in new datasets?
Answer: movies , restaurants, English , Korean