Question: What are the types of lower layers?
Evidence: Neural network based models have been widely exploited with the prosperities of Deep Learning BIBREF0 and achieved inspiring performances on many NLP tasks, such as text classification BIBREF1 , BIBREF2 , semantic matching BIBREF3 , BIBREF4 and machine translation BIBREF5 . These models are robust at feature engineering and can represent words, sentences and documents as fix-length vectors, which contain rich semantic information and are ideal for subsequent NLP tasks.
One formidable constraint of deep neural networks (DNN) is their strong reliance on large amounts of annotated corpus due to substantial parameters to train. A DNN trained on limited data is prone to overfitting and incapable to generalize well. However, constructions of large-scale high-quality labeled datasets are extremely labor-intensive. To solve the problem, these models usually employ a pre-trained lookup table, also known as Word Embedding BIBREF6 , to map words into vectors with semantic implications. However, this method just introduces extra knowledge and does not directly optimize the targeted task. The problem of insufficient annotated resources is not solved either.
Multi-task learning leverages potential correlations among related tasks to extract common features, increase corpus size implicitly and yield classification improvements. Inspired by BIBREF7 , there are a large literature dedicated for multi-task learning with neural network based models BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 . These models basically share some lower layers to capture common features and further feed them to subsequent task-specific layers, which can be classified into three types:
In this paper, we propose a generalized multi-task learning architecture with four types of recurrent neural layers for text classification. The architecture focuses on Type-III, which involves more complicated interactions but has not been researched yet. All the related tasks are jointly integrated into a single system and samples from different tasks are trained in parallel. In our model, every two tasks can directly interact with each other and selectively absorb useful information, or communicate indirectly via a shared intermediate layer. We also design a global memory storage to share common features and collect interactions among all tasks.
We conduct extensive experiments on five benchmark datasets for text classification. Compared to learning separately, jointly learning multiple relative tasks in our model demonstrate significant performance gains for each task.
Our contributions are three-folds:
These models basically share some lower layers to capture common features and further feed them to subsequent task-specific layers, which can be classified into three types: • Type-I One dataset annotated with multiple labels and one input with multiple outputs. • Type-II Multiple datasets with respective labels and one input with multiple outputs, where samples from different tasks are fed one by one into the models sequentially. • Type-III Multiple datasets with respective labels and multiple inputs with multiple outputs, where samples from different tasks are jointly learned in parallel.
Answer: • Type-I One dataset annotated with multiple labels and one input with multiple outputs. • Type-II Multiple datasets with respective labels and one input with multiple outputs, where samples from different tasks are fed one by one into the models sequentially. • Type-III Multiple datasets with respective labels and multiple inputs with multiple outputs, where samples from different tasks are jointly learned in parallel.