Title: CLASSIFICATION ON MULTIPLE LEVELS WITH MULTIPLE LABELS

Abstract: Natural language uses words in an associative way to construct sentences: it is not words in isolation, but the appropriate use of hierarchical structures that makes communication successful. We propose a deep learning framework for explicitly tying together the representations between single words and full sentences, resulting in a fluid transfer of knowledge between these two levels of granularity. We construct a multi-head attention mechanism for sentence classification, where the individual attention heads simultaneously learn to perform multi-class sequence labeling. Supervision on individual tokens explicitly teaches the classifier which areas it needs to focus on in each sentence, while the sentence-level objective regularizes the token-level predictions and even enables sequence labeling without token-level training data. Our experiments show that the proposed architecture systematically outperforms its single-task counterparts and exhibits strong transfer capabilities, while also achieving reasonable performance as a zero-shot sequence labeler 1

1. INTRODUCTION:

Natural language has vast syntactic and semantic complexity: it involves using words in an associative way to construct structured sentences. Meaningful expressions are built from other meaningful expressions (Goldberg, 1995) , or, as stated by the Principle of (Semantic) Compositionality (Hirst, 1987) , the meaning of the whole is determined by the meaning of the parts. Neural architectures are often trained end-to-end, expecting the models to independently discover the necessary methods for language composition. However, due to data limitations on most tasks, composition patterns can be difficult to learn automatically. Furthermore, these discovered patterns can pick up noise and bias in the datasets and do not always match the desired model behavior.

We investigate a novel approach to applying the compositionality principle in a deep learning framework, allowing for more direct supervision. Our model learns to perform both sentence classification and token-level sequence labeling while tying these two objectives together in a way that enables them to reinforce each other. The network uses a multi-head attention mechanism to construct a sentence-level representation for multi-class sentence classification. At the same time, the customized attention heads are also used to construct token-level representations for multi-class sequence labeling. The token-level supervision explicitly teaches the classifier which areas it needs to focus on in each sentence, while the sentence-level objective regularizes the sequence labeling predictions. Changpinyo et al. (2018) advocate for the beneficial integration of several, related tasks. In our case, the sentence and word predictions are explicitly tied together, so the network is incentivized to develop an organic relationship between hierarchies that resembles how humans use language. Moreover, as we formulate two distinct tasks based on the same dataset, we effortlessly provide the network with more training examples. We introduce different optimization objectives to satisfy the aim of each task -this implicitly regularizes the weights towards better (i.e., more general) text representations (Ruder, 2017) . Despite these theoretical guarantees, we seek to empirically determine whether (and how) the interaction between the two levels of granularity benefits learning.

1 Code available at https://github-placeholder.

Recent work has shown promising results for directly supervising the internal components of a model. For example, Liu et al. (2016) used alignment annotations to improve the performance of a neural machine translation system. Rei & Søgaard (2019) described an architecture for supervising attention in a binary text classification setting. Barrett et al. (2018) used a related model to guide the network to focus on similar areas as humans, based on human gaze recordings. We build on these ideas and describe a more general framework, extending it to both multi-class text classification and multi-class sequence labeling.

We propose the following training conditions and experimental settings, grouped by the amount of word-level annotation used by our model to guide its learning:

• Fully supervised: the system is provided with full annotations both on the sentence and on the token level. The model has all the information needed to perform very well on each isolated task. However, we are more interested in how the multi-and single-task performances compare. Does the model take advantage of the joint learning regime and the supplemental labeled data to increase its performance on each task? • Semi-supervised: the system is provided with some supervision signal, but only for a subset of the tokens, while sentences are always receiving it in full. We investigate our model's inference abilities and determine the proportion of token annotation that is sufficient for the network to reach as good a performance as the fully supervised one. • Unsupervised: learning sequence labeling without any token-level annotations (zero-shot sequence labeling). In other words, we train a sentence classifier and evaluate it as a sequence labeler. If knowledge can be transferred from a higher, abstract sentence-level to a lower, fine-grained token-level, the system will perform sophisticated word-predictions solely based on the considerably cheaper sentence annotations.

We augment our model with several auxiliary objectives and add a new regularization term to incentivize the construction of distinct tag-specific sub-spaces. As we intertwine the two hierarchical levels, our network exhibits strong transfer capabilities that we validate on three different tasks.

2. THE MULTI-HEAD ATTENTION LABELER (MHAL):

Our architecture is designed to tie the token and sentence representations together. It has two core components: the first one uses Bi-LSTMs to build compact vectors for each word; the second uses a multi-head attention mechanism to obtain two distributions, over the tagset and the sentence labels. The goal of the mechanism is to perform sentence classification. However, because each head acts as a separate label predictor, the system also behaves like a sequence labeler: the scores based on which we make sentence-level predictions are obtained by combining the individual attention weights, which are attached to each word and used to make token-level predictions. Joining multiple levels in this way to detect multiple labels is one of our main contributions.

The first component takes as input a tokenized sentence of length N and maps it to a sequence of vectors [x 1 , x 2 , ..., x N ]. Each vector x i , corresponding to the i th token in a sentence, is the concatenation of its pre-trained word embedding w i with its character-level representation c i , similar to Lample et al. (2016) . Passing each vector x i to a Bi-LSTM (Graves & Schmidhuber, 2005) , we obtain compact token representations z i by merging the hidden states from each direction at every time step and projecting these onto a joint feature space using a tanh activation (equations 1 to 3).

EQUATION

The second component of our architecture is a multi-head attention mechanism (Vaswani et al., 2017) that creates H heads. We set H equal to the size of our tagset and create a correspondence between attention heads and token labels. For each continuous vector representation z i , and for each head h ∈ {1, 2, ..., H}, we obtain three vectors of keys, queries and values (denoted by k ih , q ih , and v ih , respectively) by non-linearly projecting z i onto a different sub-space that is H times smaller Figure 1 : MHAL architecture, taking a sequence of three words. Illustrated for one head h only. We obtain character representations as presented in the dashed rectangle (here, for the word sat).

than the original input size. A critical difference between our approach and the standard multi-head attention mechanism is that we collapse the queries q ih (operating over each token i and each head h) into a single query q h that is specific to head h. We achieve this by averaging q ih across the sentence, as in equation 4. The motivation is to obtain a compressed, individual representation that is shared across the sentence but still encapsulates the identity of a single, specific tag.

We define our attention function as the dot product (denoted by •) between a query and its corresponding key, resulting in the attention evidence scores a ih ∈ R 1 . We apply a sigmoid (σ) activation function2 and normalize the attention scores across the words to obtain the attention weights α ih ∈ [0, 1]. Next, similar to Yang et al. (2016) , we determine the importance of each value vector by multiplying it with its corresponding attention weight. Summing these over the words in a sentence, we get a representation s h ∈ R d , which is further passed through two feedforward layers: the first one is non-linear and projects the sentence representation onto a smaller feature space, while the last one is linear and outputs a scalar sentence score o h for each head h (equations 5 to 8).

q h = 1 N N i=1 q ih (4) a ih = q h • k ih (5) α ih = σ(a ih ) N j=1 σ(a jh ) (6) s h = N i=1 α ih v ih (7) o h = W o tanh(W s s h + b s ) + b o (8)

where W o and W s are weight matrices, and b o and b s are bias vectors.

We need to collect the sentence scores across all heads and make a sentence prediction. The challenge arises as we have to map these H scores (equal to the number of token labels) to the number of sentence labels S, which are not necessarily in direct correspondence. To solve this, we use the fact that datasets typically have a default label that is common between the token and the sentence label sets and that vastly outnumbers the others. We distinguish two situations:

1. H = S: Each sentence label has a corresponding word-tag (and thus, a head associated).

We can concatenate the sentence scores across all heads into a vector õ = [o 1 ; o 2 ; ...; o H ]. 2. H = S and S = 2: The sentence labels are binary, while the token labels are not, so we have to find a correspondence between the heads and the two sentence labels. We concatenate the score obtained for the default head o d (corresponding to the default label) with the maximum score obtained for the non-default heads

o nd : õ = [o d ; o nd ],

where d and nd are the indices of the default and non-default heads, respectively, and

o nd = max h =d (o h ).

We obtain a normalized distribution ỹ over the sentence labels by applying a softmax on the extracted scores õ ∈ R S and predict the label corresponding to the most probable sentence score.

In addition to sentence classification, we also want to make token-level predictions. To achieve this, we treat each attention evidence a ih as the score of an individual word i and head h. It is crucial to note that the sentence scores (and thus, predictions) rely on the attention evidence -we conditioned the sentence label distribution ỹ on the token scores. Thus, the sentence and token-level predictions are intertwined, and we can perform sequence labeling in tandem with sentence classification by re-using the attention evidence scores and re-interpreting them as token-specific predictions. By explicitly tying together these two levels of granularity, we incentivized the network to learn better composition functions and share language features between layers. Finally, we apply a softmax on the concatenation of scores across all heads to obtain the token label distribution ti ∈ R H and predict the tag with the maximum probability.

This subsumes the architectural design of our joint text classifier, to which we refer to as the multihead attention labeler (MHAL), schematically represented in Figure 1 .

3. OPTIMIZATION OBJECTIVES:

Our model can be optimized both as a sentence classifier and as a sequence labeler. Both losses, L sent and L tok , minimize the summation over the categorical cross-entropy between the predicted sentence (or token) label distribution ỹ (or ti ) and its true annotation y (or its true gold tag t i ):

EQUATION

EQUATION

where y (s) j and t (s) ij are binary indicator variables specifying whether sentence s truly is a sentence of label j and token t at position i in sentence s truly is a token of tag type j, respectively.

Recall that the sentence label distribution is based on the attention evidence scores, which represent, in turn, the token scores used for word-level classifications. If we train our model solely as a sentence classifier (by providing only sentence-level annotations), the network will also optimize the token scores because the parameter updates on the sequence labeling task are performed at layers below the sentence classification task. Moreover, the network will learn the important areas of a sentence, combining the scores from individual words to determine the overall sentence label. In this way, our model performs zero-shot sequence labeling, a type of transductive transfer learning (Ruder, 2017) . In contrast, optimizing only the parameters used in the sequence labeling task does not implicitly train the sentence scores since they are situated above it in the architecture. However, when both levels receive supervision, the token signal encourages the network to put more weight on the attention heads indicative of the correct labels.

Previous research in multi-task learning provides significant evidence that including several related tasks along with the core task positively impacts performance due to the regularization and generalization effects of the procedure (Bingel & Søgaard, 2017; Søgaard & Goldberg, 2016; Changpinyo et al., 2018) . Closely following the setting proposed by Rei (2017) , we include language modeling (LM) as a secondary objective, operating both over characters and words. In this way, we inject corpus-specific information into the model as well as syntactic and semantic patterns (Linzen et al., 2016; Marvin & Linzen, 2018) and expect them to lead to more transferable features.

We consider another auxiliary loss, whose purpose is to better wire the two granularity levels, of sentences and words. We call it attention objective because it directly operates on the attention heads corresponding to the correct sentence label, imposing two conditions:

1. There should be at least one word of the same label as the ground-truth sentence. Intuitively, most of the focus should be on the words indicative of the sentence type. 2. There should be at least one word that has a default label. Even if the sentence has a non-default class, it should still contain at least one default word.

These conditions can be formulated as a loss function and then optimized during training:

EQUATION

where

EQUATION

)

and d is the default head, k is the true sentence label, ti is the predicted token label distribution for word i (so ti,h is its value for head h), and, finally, q is the smoothed sentence label distribution (so q j is its value for sentence label j), whose expression is obtained as in Szegedy et al. ( 2016), choosing = 0.15 and K = H.

Lastly, we propose a custom regularization term for the multi-head attention mechanism to motivate the network to learn a truly distinct representation sub-space for each of the query vectors q h . As opposed to the keys and values, which are associated with (possibly reoccurring) words, the queries q h encapsulate the essence of a certain tag. Thus, they need to capture the distinctive features that are specific to a particular head. To push the network towards this goal, we introduce the term R q and calculate it as the average cosine similarity between every pair of queries q h and q i , with h = i (equation 13). R q can be viewed as an orthogonality penalty: it attains a minimum for two query vectors spanning over orthogonal sub-spaces, and a maximum when their sub-spaces coincide. Thus, this technique imposes a wider angle between the queries, encouraging the model to learn unique, diverse, and meaningful vector representations.

R q = 2 H(H -1) H-1 h=1 H i>h q h • q i q h • q i (13)

The final loss function L tot is a weighted sum of our objectives, allowing us to observe the effect of the different components as well as to control the flow of the supervision signal and the importance of each auxiliary task:

L tot = λ sent L sent + λ tok L tok + λ LM L LM + λ attn L attn + λ Rq R q .

4. EXPERIMENTS:

Our model was evaluated on three different datasets: Stanford Sentiment Treebank (SST, Socher et al., 2013) for sentiment analysis, CoNLL-2003 (Tjong Kim Sang & De Meulder, 2003) for named entity recognition (NER), and the First Certificate in English (FCE, Yannakoudakis et al., 2011) for fine-grained grammatical error detection. All of them have already been tokenized and split into train, development, and test sets. In the appendix, we provide some corpus statistics (Table 4 ) and a detailed breakdown of the number of annotated examples available per split and per label (Table 5 ).

Our datasets contain sentences with labeled words, used to make token-level predictions. Thus, they can be phrased as sequence labeling tasks: for SST, identifying the positive, negative and neutral words; for CoNLL-2003, detecting the persons, organizations, locations, miscellaneous, and the non-named entities; for FCE, identifying the errors in content, form, function, orthography, or others as well as the correct words. Given a sequence of labeled words, there is an implicit label for the sentence too. For instance, a sentence containing at least one grammatical mistake makes it ungrammatical overall. SST provides annotations for each sentence in the corpus, using the same labels as for the tagset. For CoNLL-2003 and FCE, there are no pre-existing sentence-level annotations, but we can infer a binary label for each sentence based on the existence of at least one word annotated as an entity or as a grammatical error, respectively. As already mentioned, each dataset has a default label, common between the token and the sentence label sets, and, in our case, it corresponds to neutral phrases, non-named entities, and grammatical words, respectively.

We did not engage in fine-tuning our neural network's components and largely followed the settings proposed by Rei & Søgaard (2019) . We manually searched the best values for our new parameters based on the performance on the development set (see Table 6 in the appendix for a complete list).

To avoid outliers, we ran each experiment with five different random seeds and reported the mean results using the metrics specific to each dataset along with the mean micro-average score (denoted by a subscript µ) of the non-default labels (denoted by a superscript * ), as commonly used in the multi-task learning literature (Changpinyo et al., 2018; Martínez Alonso & Plank, 2017) .

We train our models under the different regimes mentioned in the introduction by assigning corresponding values to the weights λ in the expression of L tot . We distinguish the following models:

• MHAL-sent: single-task sentence classifier; λ sent = 1.0, while all the other λs are zero.

• MHAL-tok: single-task sequence labeler; λ tok = 1.0, while all the other λs are zero.

• MHAL-sent+tok: optimized both as a sentence classifier and a sequence labeler; λ sent = λ tok = 1.0, while all the other λs are set to zero.

• MHAL-joint: just like MHAL-sent+tok, it performs symmetric multi-task learning. In addition, it also sets λ attn = 0.01, λ LM = 0.1, and R q = 0.5. • MHAL-zero: zero-shot sequence labeling where only sentence-level annotation is available for training; setting λ sent = 1.0, λ tok = 0.0, and all the other λs as in MHAL-joint. In Table 1 , we present the performance of our models when trained as sentence classifiers. For SST and FCE, MHAL-zero outperforms MHAL-sent, suggesting that MHAL-zero's additional auxiliary objectives further refine the predictions. Comparing these single-task sentence classifiers to their multi-task counterparts (MHAL-sent+tok and MHAL-joint, which have also received tokenannotated examples), we observe that the F * 1µ score consistently increases for the latter. This finding suggests that the additional information extracted from individually labeled words was successfully transferred to the sentence-level task. Therefore, MHAL-sent+tok and MHAL-joint become exponents of the usefulness of multi-task learning, being our best models.

In Table 2 , we present the performance of our models when trained as sequence labelers. Our zeroshot sequence labeling model MHAL-zero exploits the generality of neural networks, sharing features from a higher to a lower level task, and performs the best amongst other models that do not use token-level supervision. For instance, the random label assignment can be a strong baseline, particularly for highly skewed datasets, such as FCE; MHAL-zero surpasses it, while the simple sentence classifier MHAL-sent does not always do so. MHAL-zero persistently outperforms MHAL-sent by a large margin on all datasets (e.g. it gave a 16.31% boost in F * 1µ on SST). Since the difference between them resides in the activated objectives, these results show that our auxiliary losses introduce a necessary inductive bias, improving MHAL performance. Visualization of the MHAL-zero attention heads can provide a way to understand the patterns learned by the network. Figure 2 shows a sentence from the CoNLL-2003 dataset along with its attention evidence scores, normalized across the heads. The model is able to identify entities in the sentence even when trained with only a binary sentence-level signal. However, this is a difficult task and it does not always assign the correct entity type, here mistaking the person for an organization.

SST CoNLL-2003 FCE P * µ R * µ F * 1µ F 1µ C-P C-R C-F 1 F * 1µ P * 0.5µ R * 0.5µ F * 0.5µ F 0.

By including token-level supervision into the model, MHAL-joint is able to further improve on this result. We include additional visualizations of both MHAL-zero and MHAL-joint in the appendix.

Note that our multi-task models (MHAL-sent+tok, MHAL-joint) register systematic improvements across all our datasets over the single-task sequence labelers, which were trained in isolation (MHAL-tok, MHAL-sent), further emphasizing the effectiveness of sharing information between the two granularity levels. On our way to obtaining the sentence scores, we re-interpreted each attention evidence as a token score. Now, based on the results presented, it follows that wiring each head to a classifier designated for a single label has better stimulated the network to learn shared representations and excel at performing two tasks, under the same architecture. We perform a semi-supervised experiment on MHAL-joint, using the supervision signal of all sentences and only a percentage p of the word-level annotations. In Figure 3 , we present the sequence labeling results of our multi-task MHAL-joint, in comparison to the single-task MHAL-tok, gradually increasing p to allow more tokens to guide learning. We observe that adding as little as 10% of the token-annotated sentences increases MHAL-zero's performance by substantial amounts (38.01% on SST, 61.31% on CoNLL-2003, and 12.15% on FCE) , suggesting that the two tasks are positively influencing each other. Using only 30% of the data, MHAL already approaches its fully-supervised performance, showing that the transfer of knowledge has benefits that flow in both directions between sentences and words. Compared to MHAL-tok, MHAL-joint is systematically better -the multi-task model has the annotated sentences at its disposal and learns to use them to its advantage particularly when receiving weak token supervision signals (steep increases for ≤ 50%). Across all our experiments, we impose two stopping criteria and apply them on the development set:

Dev metric SST CoNLL-2003 FCE P * µ R * µ F * 1µ S-Acc C-P C-R C-F 1 S-F 1 P * µ R * µ F * 0.5µ S-

1. the sentence-level classification performance (S-F * 1µ ), adopted by all models that do not receive any token-level annotation (e.g. MHAL-zero); 2. the token-level classification performance (F * 1µ ), adopted by all models that receive some token annotation (e.g. MHAL-tok, MHAL-joint).

We observed that, even in the case of MHAL-zero, stopping based on the token performance improves the word-level predictions at test time, but usually hurts the sentence predictions. However, as suggested by the results in Table 3 , stopping based on the average of these two metrics generally improves both the token and the sentence predictions.

5. CONCLUSION:

In this paper, we proposed MHAL, a novel model that ties together two hierarchical levels (for sentences and words) to build representations whose richness and utility were evaluated on three different tasks: sentiment analysis, named entity recognition, and grammatical error detection. MHAL's architecture is based on a multi-head attention mechanism that re-interprets each head as an individual label classifier, treating each attention evidence as a token score, based on which we make multi-class word-level predictions. The attention weights also contribute to making sentence predictions, so the two granularity levels are intertwined and incentivized to help each other by sharing knowledge. Therefore, through this design innovation, our model can perform two prediction problems -sequence labeling and sentence classification -using the same, shared architecture.

We proposed training MHAL under various regimes; the results confirm its versatility and robustness. The several auxiliary objectives introduced (attention loss, language modeling, query regularization), whose purpose was to be "bridges" between the two prediction tasks, proved to be beneficial additions. Our zero-shot sequence labeling model learned suggestive patterns for the whole sentence and successfully transferred them to the lower-level task. Helped by their architectural design, our multi-task models (MHAL-sent+tok and MHAL-joint) always outperform their single-task counterparts (MHAL-sent and MHAL-tok). This finding is important as it reveals the benefits of cooperation between related tasks: conditioning the sentence scores on the unnormalized attention weights, which are, in turn, used as token scores, caused a fluid flow of information and the development of an organic relation between granularity levels. Therefore, by (re-)designing joint models, we can bring NLP transfer learning closer to performing structured multi-level text understanding and labeling. 

:

We chose the sigmoid over the softmax to impose a smoother distribution over the tokens attended.

The network usually takes more time to reach the common optimal point when we include the tokenbased stopping criterion. Sentence classification is an easier task than sequence labeling -being predicted at a higher layer in the network hierarchy, it accumulates more information and thus builds solid abstractions, not to mention that it has fewer unique labels. For these reasons, the network falls into a local minimum when guided by the sentence-level performance. However, choosing tokens as a stopping criterion requires annotated development data, which does not generally comply with the framing of our zero-shot learning experiment. Nevertheless, reporting this finding emphasizes that the stopping criterion requires careful consideration -it is responsible for choosing the best performing model used during testing and for driving the application of the learning rate decay. A few performance percentage points could be gained by carefully selecting the stopping metric.

Figures and Tables:

Figure 2: Attention evidence scores, normalized acrossed heads, assigned by MHAL-zero for the words in a sentence from the CoNLL-2003 dataset.

Figure 3: Semi-supervised experiments for SST, CoNLL-2003, and FCE, comparing the sequence labeling performance of the multi-task model MHAL-joint with the single-task model MHAL-tok.

Figure 5: Attention evidence scores, normalized acrossed heads, assigned by MHAL-joint for the words in three sentences from the SST (leftmost), CoNLL-2003 (middle), and FCE (rightmost) datasets.

Sentence classification performance on SST,CoNLL-2003, and FCE datasets.

Table content: <table><tr><td/><td/><td>SST</td><td/><td/><td/><td colspan="2">CoNLL-2003</td><td/><td/><td/><td>FCE</td></tr><tr><td/><td>P  *  µ</td><td>R  *  µ</td><td>F  *  1µ</td><td>Acc</td><td>P  *  µ</td><td>R  *  µ</td><td>F  *  1µ</td><td>F 1µ</td><td>P  *  µ</td><td>R  *  µ</td><td>F  *  1µ</td><td>F 1µ</td></tr><tr><td>Random</td><td colspan="12">38.99 30.81 34.42 31.58 79.11 48.91 60.45 48.91 67.64 51.16 58.26 51.07</td></tr><tr><td>MHAL-sent</td><td colspan="12">71.40 83.75 77.08 70.12 97.16 99.20 98.17 97.06 79.46 89.09 84.00 77.58</td></tr><tr><td>MHAL-zero</td><td colspan="12">71.61 83.80 77.23 71.08 97.18 98.80 97.98 96.80 77.96 93.41 84.99 77.90</td></tr><tr><td colspan="13">MHAL-sent+tok 72.23 83.14 77.30 70.14 97.92 99.09 98.50 97.53 85.61 84.66 85.13 78.92</td></tr><tr><td>MHAL-joint</td><td colspan="12">71.34 84.90 77.53 70.24 97.82 99.13 98.47 97.32 83.58 86.82 85.17 79.50</td></tr></table>

Random 11.01 33.59 16.59 26.41 13.09 20.14 15.87 7.20 2.60 16.78 4.50 16.63 MHAL-tok 87.20 70.90 78.19 92.13 90.75 91.47 91.11 90.31 44.48 15.89 23.34 87.36 MHAL-sent 8.67 13.85 10.62 56.35 13.18 21.95 16.47 9.52 2.46 15.85 4.24 35.33 MHAL-zero 21.60 39.78 26.93 67.26 24.02 27.23 25.51 21.54 4.03 12.00 5.64 60.84 MHAL-sent+tok 87.52 72.35 79.21 92.21 91.05 91.69 91.37 90.77 43.41 17.45 24.47 87.97 MHAL-joint 87.47 73.12 79.65 92.37 91.02 91.75 91.38 90.70 45.66 20.65 28.25 87.98

Table content: <table/>

Sequence labeling performance on SST, CoNLL-2003, and FCE datasets. Metrics starting with Crepresent span-sensitive scores, specific to the NER task.

Table content: <table/>

The effect of the stopping criterion metric during the training of MHAL-zero.

Table content: <table/>

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818-2826, 2016. Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Languageindependent named entity recognition. In Proceedings of the 7th Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142-147. Association for Computational Linguistics, 2003. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural mation Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1480-1489. Association for Computational Linguistics, 2016.Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 180-189. Association for Computational Linguistics, 2011. We list, for sentences and tokens: number of unique labels, proportion of default labels (O), entropy of the label distribution, and entropy of the non-default label distribution (using log 2 ).

Table content: <table><tr><td>A APPENDIX</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Dataset</td><td colspan="4">No. labels sent tok sent Prop. O tok</td><td colspan="4">Full Entropy Non-O Entropy sent tok sent tok</td></tr><tr><td>SST</td><td>3</td><td>3</td><td colspan="5">0.19 0.78 1.509 0.961 0.999</td><td>0.956</td></tr><tr><td>CoNLL-2003</td><td>2</td><td>5</td><td colspan="5">0.20 0.83 0.731 0.979 0.263</td><td>1.929</td></tr><tr><td>FCE</td><td>2</td><td>6</td><td colspan="5">0.37 0.89 0.952 0.775 0.421</td><td>2.288</td></tr><tr><td>Dataset</td><td>Label</td><td/><td colspan="3">Number of sentences</td><td colspan="3">Number of tokens</td></tr><tr><td/><td/><td/><td>Train</td><td>Dev</td><td>Test</td><td>Train</td><td>Dev</td><td>Test</td></tr><tr><td/><td>O</td><td/><td>1,624</td><td>229</td><td>389</td><td colspan="3">128,156 16,684 33,128</td></tr><tr><td>SST</td><td>N P</td><td/><td>3,310 3,610</td><td>428 444</td><td>912 909</td><td>13,384 22,026</td><td>1,740 2,850</td><td>3,488 5,789</td></tr><tr><td/><td>Total</td><td/><td colspan="6">8,544 1,101 2,210 163,566 21,274 42,405</td></tr><tr><td/><td>O</td><td/><td>2,909</td><td>645</td><td>697</td><td colspan="3">169,578 42,759 38,323</td></tr><tr><td/><td>LOC</td><td/><td/><td/><td/><td>8,297</td><td>2,094</td><td>1,925</td></tr><tr><td>CoNLL-2003</td><td>MISC ORG</td><td/><td colspan="3">11,132 2,605 2,756</td><td>4,593 10,025</td><td>1,268 2,092</td><td>918 2,496</td></tr><tr><td/><td>PER</td><td/><td/><td/><td/><td>11,128</td><td>3,149</td><td>2,773</td></tr><tr><td/><td>Total</td><td/><td colspan="6">14,041 3,250 3,453 203,621 51,362 46,435</td></tr><tr><td/><td>O</td><td/><td>10,718</td><td>824</td><td>900</td><td colspan="3">396,479 30,188 35,525</td></tr><tr><td/><td colspan="2">CONTENT</td><td/><td/><td/><td>7,194</td><td>527</td><td>673</td></tr><tr><td/><td>FORM</td><td/><td/><td/><td/><td>8,174</td><td>621</td><td>850</td></tr><tr><td>FCE</td><td>FUNC</td><td/><td colspan="3">17,836 1,384 1,806</td><td>11,084</td><td>888</td><td>1,194</td></tr><tr><td/><td>ORTH</td><td/><td/><td/><td/><td>12,655</td><td>1,126</td><td>1,429</td></tr><tr><td/><td>OTHER</td><td/><td/><td/><td/><td>11,415</td><td>861</td><td>1,116</td></tr><tr><td/><td>Total</td><td/><td colspan="6">28,554 2,208 2,706 447,001 34,211 40,787</td></tr></table>

Statistics of the labeled sentences and tokens, separated by the train, dev or test split.

Table content: <table/>

Peer review (iclr2020):
This paper studies the effects of joint learning of multi-level tasks: sentence classification and sequence labeling using a hybrid architecture of bidirectional LSTM and multi-head attention. They introduce a modification of the multi-head architecture to create a sentence representation and auxiliary objectives to enforce some relationship between the two tasks. The experiments comparing different settings of the models demonstrate that the models jointly learned using the two tasks are somewhat better than individual task-specific models. In addition, the zero-shot model also performs better than the baseline in the sequence labeling task. This paper requires some clarification and more experiment results. I am leaning toward rejecting this paper for the following reasons: It is unclear on the novelty of this paper’s contribution. It has been shown that multi-task learning objective is helpful. While the majority of recent work has focused on transfer learning (from LM objective) and weakly related tasks, the direction of the multi-level learning task has been explored to some extent, such as Hashimoto et al., 2017 (building from bottom-up) and Williams et al., 2017 (weakly supervised lower-level task). I think there should be more effort into distinguishing their findings from the previous. Additionally, the paper should build on top of a well-known architecture for a greater impact. For example, using the multi-head attention architecture to perform classification is discussed in Devlin et al. 2018. A comparison with existing models, both theoretically and empirically, will be very helpful to readers. The proposed method does not seem well justified by the experiment results. As mentioned previously, the paper lacks a comparison with existing models to justified architectural and training choices. Furthermore, the internal comparisons show that the proposed model has a small improvement from the individual models on both sentence-level and token-level tasks. As for the other auxiliary losses, we see all of their effects combined, rather than individual ones. The ablation analysis would be helpful. Finally, I think the zero-shot experiment is interesting. Unfortunately, we cannot draw a solid conclusion here. The MHAL-zero has an additional signal from the weak supervision from the proposed attention loss which is somewhat related to how the sentence labels are created (a big improvement on CoNLL-2003 and FCE). Clarification Questions: 1. How does this model work in the case of H != S and S > 2? 2. How is the LM loss computed? I.e. Which representations are used to predict word distribution? 3. In the semi-supervision experiment, I am not sure how you get “38% on SST, …” for MHAL-zero. Additionally, can you explain “Using only 30% of data, MHAL already approaches its fully-supervised performance”? Hashimoto, Kazuma, et al. *A joint many-task model: Growing a neural network for multiple NLP tasks.* arXiv preprint arXiv:1611.01587 (2016). Williams, Adina, Andrew Drozdov, and Samuel R. Bowman. *Learning to parse from a semantic objective: It works. Is it syntax?* arXiv preprint arXiv:1709.01121 4 (2017). Devlin, Jacob, et al. *Bert: Pre-training of deep bidirectional transformers for language understanding.* arXiv preprint arXiv:1810.04805 (2018).