A Unified Architecture for Natural Language Processing:
Deep Neural Networks with Multitask Learning

Ronan Collobert
Jason Weston
NEC Labs America, 4 Independence Way, Princeton, NJ 08540 USA

Abstract
We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similarwords and the likelihood that the sentence
makes sense (grammatically and semantically) using a language model. The entire
network is trained jointly on all these tasks
using weight-sharing, an instance of multitask
learning. All the tasks use labeled data except the language model which is learnt fromunlabeled text and represents a novel form ofsemi-supervised learning for the shared tasks.
We show how both multitask learning andsemi-supervised learning improve the generalization of the shared tasks, resulting in stateof-the-art performance.

1. Introduction
The field of Natural Language Processing (NLP) aims
to convert human language into a formal representation that is easy for computers to manipulate. Current
end applications include information extraction, machine translation, summarization, search and humancomputer interfaces.
While complete semantic understanding is still a fardistant goal, researchers have taken a divide and conquer approach and identified several sub-tasks usefulfor application development and analysis. These range
from the syntactic, such as part-of-speech tagging,chunking and parsing, to the semantic, such as wordsense disambiguation, semantic-role labeling, namedentity extraction and anaphora resolution.
Appearing in Proceedings of the 25 th International Conference on Machine Learning, Helsinki, Finland, 2008. Copyright 2008 by the author(s)/owner(s).

collober@nec-labs.com
jasonw@nec-labs.com

Currently, most research analyzes those tasks separately. Many systems possess few characteristics thatwould help develop a unified architecture which would
presumably be necessary for deeper semantic tasks. In
particular, many systems possess three failings in thisregard: (i) they are shallow in the sense that the classifier is often linear, (ii) for good performance witha linear classifier they must incorporate many handengineered features specific for the task; and (iii) they
cascade features learnt separately from other tasks,
thus propagating errors.
In this work we attempt to define a unified architecture
for Natural Language Processing that learns featuresthat are relevant to the tasks at hand given very limited prior knowledge. This is achieved by training a
deep neural network, building upon work by (Bengio &Ducharme, 2001) and (Collobert & Weston, 2007). We
define a rather general convolutional network architecture and describe its application to many well known
NLP tasks including part-of-speech tagging, chunking,named-entity recognition, learning a language modeland the task of semantic role-labeling.
All of these tasks are integrated into a single system
which is trained jointly. All the tasks except the language model are supervised tasks with labeled training
data. The language model is trained in an unsupervised fashion on the entire Wikipedia website. Training this task jointly with the other tasks comprises anovel form of semi-supervised learning.
We focus on, in our opinion, the most difficult of
these tasks: the semantic role-labeling problem. We
show that both (i) multitask learning and (ii) semisupervised learning significantly improve performanceon this task in the absence of hand-engineered features.
We also show how the combined tasks, and in particular the unsupervised task, learn powerful featureswith clear semantic information given no human supervision other than the (labeled) data from the tasks(see Table 1).

A Unified Architecture for Natural Language Processing

The article is structured as follows. In Section 2 we
describe each of the NLP tasks we consider, and in Section 3 we define the general architecture that we use
to solve all the tasks. Section 4 describes how this architecture is employed for multitask learning on all thelabeled tasks we consider, and Section 5 describes theunlabeled task of building a language model in somedetail. Section 6 gives experimental results of our system, and Section 7 concludes with a discussion of ourresults and possible directions for future research.

2. NLP Tasks
We consider six standard NLP tasks in this paper.
Part-Of-Speech Tagging (POS) aims at labeling
each word with a unique tag that indicates its syntactic role, e.g. plural noun, adverb, . . .
Chunking, also called shallow parsing, aims at labeling segments of a sentence with syntactic constituents
such as noun or verb phrase (NP or VP). Each word
is assigned only one unique tag, often encoded as abegin-chunk (e.g. B-NP) or inside-chunk tag (e.g. INP).
Named Entity Recognition (NER) labels atomic
elements in the sentence into categories such as “PERSON”, “COMPANY”, or “LOCATION”.
Semantic Role Labeling (SRL) aims at giving a semantic role to a syntactic constituent of a sentence.
In the PropBank (Palmer et al., 2005) formalism one
assigns roles ARG0-5 to words that are argumentsof a predicate in the sentence, e.g. the following
sentence might be tagged “[John]ARG0 [ate]REL [theapple]ARG1 ”, where “ate” is the predicate. The precise arguments depend on a verb’s frame and if there
are multiple verbs in a sentence some words might havemultiple tags. In addition to the ARG0-5 tags, there
there are 13 modifier tags such as ARGM-LOC (locational) and ARGM-TMP (temporal) that operate in asimilar way for all verbs.
Language Models A language model traditionallyestimates the probability of the next word being w ina sequence. We consider a different setting: predictwhether the given sequence exists in nature, or not,following the methodology of (Okanohara & Tsujii,2007). This is achieved by labeling real texts as positive examples, and generating “fake” negative text.
Semantically Related Words (“Synonyms”) Thisis the task of predicting whether two words are semantically related (synonyms, holonyms, hypernyms...)which is measured using the WordNet database(http://wordnet.princeton.edu) as ground truth.

Our main interest is SRL, as it is, in our opinion, themost complex of these tasks. We use all these tasks to:(i) show the generality of our proposed architecture;and (ii) improve SRL through multitask learning.

3. General Deep Architecture for NLP
All the NLP tasks above can be seen as tasks assigning labels to words. The traditional NLP approach is:
extract from the sentence a rich set of hand-designed
features which are then fed to a classical shallow classification algorithm, e.g. a Support Vector Machine(SVM), often with a linear kernel. The choice of features is a completely empirical process, mainly basedon trial and error, and the feature selection is task
dependent, implying additional research for each new
NLP task. Complex tasks like SRL then require alarge number of possibly complex features (e.g., extracted from a parse tree) which makes such systems
slow and intractable for large-scale applications.
Instead we advocate a deep neural network (NN) architecture, trained in an end-to-end fashion. The input sentence is processed by several layers of feature
extraction. The features in deep layers of the network
are automatically trained by backpropagation to be relevant to the task. We describe in this section a generaldeep architecture suitable for all our NLP tasks, andeasily generalizable to other NLP tasks.
Our architecture is summarized in Figure 1. The firstlayer extracts features for each word. The second layer
extracts features from the sentence treating it as a sequence with local and global structure (i.e., it is nottreated like a bag of words). The following layers areclassical NN layers.
3.1. Transforming Indices into Vectors
As our architecture deals with raw words and not engineered features, the first layer has to map words into
real-valued vectors for processing by subsequent layersof the NN. For simplicity (and efficiency) we consider
words as indices in a finite dictionary of words D ⊂ N.
Lookup-Table Layer Each word i ∈ D is embedded into a d-dimensional space using a lookup table
LTW (·):LTW (i) = Wi ,where W ∈ Rd×|D| is a matrix of parameters to belearnt, Wi ∈ Rd is the ith column of W and d is theword vector size (wsz ) to be chosen by the user. In
the first layer of our architecture an input sentence{s1 , s2 , . . . sn } of n words in D is thus transformed
into a series of vectors {Ws1 , Ws2 , . . . Wsn } by apply-

A Unified Architecture for Natural Language Processing

ing the lookup-table to each of its words.
It is important to note that the parameters W of thelayer are automatically trained during the learning
process using backpropagation.

LTwK

Convolution Layer#hidden units * (n-2)

Max Over Time

Optional Classical NN Layer(s)Softmax#classes

Figure 1. A general deep NN architecture for NLP. Given
an input sentence, the NN outputs class probabilities for
one chosen word. A classical window approach is a special
case where the input has a fixed size ksz, and the TDNN
kernel size is ksz; in that case the TDNN layer outputs
only one vector and the Max layer performs an identity.

to the idea that a sequence has a notion of order. A
TDNN “reads” the sequence in an online fashion: attime t ≥ 1, one sees xt , the tth word in the sentence.
A classical TDNN layer performs a convolution on agiven sequence x(·), outputting another sequence o(·)
whose value at time t is:n−tX

Lj · xt+j ,

(2)

j=1−t

The lookup table layer maps the original sentence into
a sequence x(·) of n identically sized vectors:(x1 , x2 , . . . , xn ), ∀t xt ∈ Rd .

...

#hidden unitso(t) =
3.2. Variable Sentence Length(d1+d2+...dK)*n

LTw1
...
...
...

We propose to add a feature for each word that encodes
its relative distance to the chosen predicate. For the ith
word in the sentence, if the predicate is at position posp
we use an additional lookup table LT distp (i − posp ).

Lookup Tables...

Classifying with Respect to a Predicate In acomplex task like SRL, the class label of each word in a
sentence depends on a given predicate. It is thus necessary to encode in the NN architecture which predicate
we are considering in the sentence.

sK(1) sK(2) sK(3) sK(4) sK(5) sK(6)

...
...

LTW 1 ,...,W K (i)T = (LTW 1 (i1 )T , . . . , LTW K (iK )T )s1(1) s1(2) s1(3) s1(4) s1(5) s1(6)...

When a word is decomposed into K elements (features), it can be represented as a tuple i ={i1 , i2 , . . . iK } ∈ D1 × · · · × DK , where Dk is the dictionary for the k th -element. We associate to each element a lookup-table LTW k (·), with parameters W k ∈kk
Rd ×|D | where dk ∈ N is a user-specified
vector size.
P
A word i is then embedded in a d = k dk dimensional
space by concatenating all lookup-table outputs:

n words, K features

the cat sat on the mat...

Variations on Word Representations In practice,
one may want to introduce some basic pre-processing,such as word-stemming or dealing with upper andlower case. In our experiments, we limited ourselves toconverting all words to lower case, and represent thecapitalization as a separate feature (yes or no).

Input Sentence
feature 1 (text)
feature 2...
feature K(1)

Obviously the size n of the sequence varies dependingon the sentence. Unfortunately normal NNs are notable to handle sequences of variable length.
The simplest solution is to use a window approach:consider a window of fixed size ksz around each word
we want to label. While this approach works withgreat success on simple tasks like POS, it fails on more
complex tasks like SRL. In the latter case it is common
for the role of a word to depend on words far awayin the sentence, and hence outside of the consideredwindow.
When modeling long-distance dependencies is important, Time-Delay Neural Networks (TDNNs) (Waibel
et al., 1989) are a better choice. Here, time refers

where Lj ∈ Rnhu ×d (−n ≤ j ≤ n) are the parametersof the layer (with nhu hidden units) trained by backpropagation. One usually constrains this convolution
by defining a kernel width, ksz, which enforces∀ |j| > (ksz − 1)/2, Lj = 0 .

(3)

A classical window approach only considers words ina window of size ksz around the word to be labeled.
Instead, if we use (2) and (3), a TDNN considers at thesame time all windows of ksz words in the sentence.
TDNN layers can also be stacked so that one can extract local features in lower layers, and more globalfeatures in subsequent ones. This is an approach typically used in convolutional networks for vision tasks,
such as the LeNet architecture (LeCun et al., 1998).
We then add to our architecture a layer which captures
the most relevant features over the sentence by feeding

A Unified Architecture for Natural Language Processing

the TDNN layer(s) into a “Max” Layer, which takes
the maximum over time (over the sentence) in (2) for
each of the nhu output features.
As the layer’s output is of fixed dimension (independent of sentence size) subsequent layers can be classical
NN layers. Provided we have a way to indicate to ourarchitecture the word to be labeled, it is then able touse features extracted from all windows of ksz words
in the sentence to compute the label of one word ofinterest.

its the dependencies between words it can model. Also
C(·) is itself a NN inside a NN. Not only does one haveto carefully design this additional architecture, but it
also makes the approach more complicated to train
and implement. Integrating all the desired featuresin x() (including the predicate position) via lookuptables makes our approach simpler, more general andeasier to tune.

4. Multitasking with Deep NN

We indicate the word to be labeled to the NN with anadditional lookup-table, as suggested in Section 3.1.
Considering the word at position posw we encode therelative distance between the ith word in the sentence
and this word using a lookup-table LT distw (i − posw ).

Multitask learning (MTL) is the procedure of learning
several tasks at the same time with the aim of mutual
benefit. This an old idea in machine learning; a good
overview, especially focusing on NNs, can be found in(Caruana, 1997).

3.3. Deep Architecture

4.1. Deep Joint Training

A TDNN (or window) layer performs a linear operation over the input words. While linear approaches
work fairly well for POS or NER, more complex tasks
like SRL require nonlinear models. One can add to the
NN one or more classical NN layers. The output of thelth layer containing nhul hidden units is computed withol = tanh(Ll · ol−1 ), where the matrix of parameters
Ll ∈ Rnhul ×nhul−1 is trained by backpropagation.

If one considers related tasks, it makes sense that features useful for one task might be useful for other ones.
In NLP for example, POS predictions are often used asfeatures for SRL and NER. Improving generalizationon the POS task might therefore improve both SRL
and NER.

The size of the last (parametric) layer’s output olast
is the number of classes considered in the NLP task.
This layer is followed by a softmax layer (Bridle, 1990)
which makes sure the outputs are positive and sum to
1, allowing us to interpret the outputs of the NN asprobabilities for each class. The ith output is given by
P lastlast
eoi / j eoj . The whole network is trained withthe cross-entropy criterion (Bridle, 1990).
3.4. Related Architectures
In (Collobert & Weston, 2007) we described a NN
suited for SRL. This work also used a lookup-table togenerate word features (see also (Bengio & Ducharme,
2001)). The issue of labeling with respect to a predicate was handled with a special hidden layer: its output, given input sequence (1), predicate position posp ,and the word of interest posw was defined as:o(t) = C(t − posw , t − posp ) · xt .
The function C(·) is shared through time t: one could
say that this is a variant of a TDNN layer with a kernel width ksz = 1 but where the parameters are conditioned with other variables (distances with respect
to the verb and word of interest).
The fact that C(·) does not combine several words inthe same neighborhood as in our TDNN approach lim-

A NN automatically learns features for the desired
tasks in the deep layers of its architecture. In the caseof our general architecture for NLP presented in Section 3, the deepest layer (consisting of lookup-tables)
implicitly learns relevant features for each word in the
dictionary. It is thus reasonable to expect that whentraining NNs on related tasks, sharing deep layers inthese NNs would improve features produced by thesedeep layers, and thus improve generalization performance. The last layers of the network can then betask specific.
In this paper we show this procedure performs very
well for NLP tasks when sharing the lookup-tables ofeach considered task, as depicted in Figure 2. Trainingis achieved in a stochastic manner by looping over thetasks:
1. Select the next task.
2. Select a random training example for this task.
3. Update the NN for this task by taking a gradientstep with respect to this example.
4. Go to 1.
It is worth noticing that labeled data for training eachtask can come from completely different datasets.
4.2. Previous Work in MTL for NLP
The NLP field contains many related tasks. This
makes it a natural field for applying MTL, and sev-

A Unified Architecture for Natural Language Processing
Lookup Tables
LTw2

LTw3

Lookup Tables
LTw‘ 2

LTw1

Convolution

Convolution

Max

Max

Classical NN Layer(s)Finally, the authors of (Musillo & Merlo, 2006) madean attempt at improving the semantic role labelingtask by joint inference with syntactic parsing, but theirresults are not state-of-the-art. The authors of (Sutton& McCallum, 2005b) also describe a negative result at
the same joint task.

Classical NN Layer(s)Softmax

Softmax

Task 1

Task 2

Figure 2. Example of deep multitasking with NN. Task 1
and Task 2 are two tasks trained with the architecture
presented in Figure 1. One lookup-table (in black) is shared(the other lookup-tables and layers are task specific). The
principle is the same with more than two tasks.

eral techniques have already been explored.
Cascading Features The most obvious way toachieve MTL is to train one task, and then use thistask as a feature for another task. This is a very common approach in NLP. For example, in the case of
SRL, several methods (e.g., (Pradhan et al., 2004))
train a POS classifier and use the output as features
for training a parser, which is then used for building
features for SRL itself. Unfortunately, tasks (features)are learnt separately in such a cascade, thus propagating errors from one classifier to the next.
Shallow Joint Training If one possesses a dataset labeled for several tasks, it is then possible to train thesetasks jointly in a shallow manner: one unique model
can predict all task labels at the same time. Using thisscheme, the authors of (Sutton et al., 2007) proposed a
conditional random field approach where they showed
improvements from joint training on POS tagging andnoun-phrase chunking tasks. However the requirementof jointly annotated data is a limitation, as this is often
not the case. Similarly, in (Miller et al., 2000) NER,
parsing and relation extraction were jointly trained ina statistical parsing model achieving improved performance on all tasks. This work has the same joint labeling requirement problem, which the authors avoided
by using a predictor to fill in the missing annotations.
In (Sutton & McCallum, 2005a) the authors showedthat one could learn the tasks independently, hence
using different training sets, by only leveraging predictions jointly in a test time decoding step, and still obtain improved results. The problem is, however, that
this will not make use of the shared tasks at training
time. The NN approach used here seems more flexible
in these regards.

5. Leveraging Unlabeled Data
Labeling a dataset can be an expensive task, especiallyin NLP where labeling often requires skilled linguists.
On the other hand, unlabeled data is abundant andfreely available on the web. Leveraging unlabeled data
in NLP tasks seems to be a very attractive, and challenging, goal.
In our MTL framework presented in Figure 2, there isnothing stopping us from jointly training supervised
tasks on labeled data and unsupervised tasks on unlabeled data. We now present an unsupervised task
suitable for NLP.
Language Model We consider a language model
based on a simple fixed window of text of size ksz using our NN architecture, given in Figure 2. We trained
our language model to discriminate a two-class classification task: if the word in the middle of the input
window is related to its context or not. We construct
a dataset for this task by considering all possible ksz
windows of text from the entire of English Wikipedia(http://en.wikipedia.org). Positive examples arewindows from Wikipedia, negative examples are thesame windows but where the middle word has been
replaced by a random word.

XX

We train this problem with a ranking-type cost:max (0, 1 − f (s) + f (sw )) ,(4)s∈S w∈D

where S is the set of sentence windows of text, D is thedictionary of words, and f (·) represents our NN architecture without the softmax layer and sw is a sentence
window where the middle word has been replaced by
the word w. We sample this cost online w.r.t. (s, w).
We will see in our experiments that the features (embedding) learnt by the lookup-table layer of this NN
clusters semantically similar words. These discoveredfeatures will prove very useful for our shared tasks.
Previous Work on Language Models (Bengio &Ducharme, 2001) and (Schwenk & Gauvain, 2002) already presented very similar language models. However, their goal was to give a probability of a word
given previous ones in a sentence. Here, we only wantto have a good representation of words: we take advantage of the complete context of a word (before and af-

A Unified Architecture for Natural Language Processingter) to predict its relevance. Perhaps this is the reason
the authors were never able to obtain a good embedding of their words. Also, using probabilities imposes
using a cross-entropy type criterion and can require
many tricks to speed-up the training, due to normalization issues. Our criterion (4) is much simpler inthat respect.
The authors of (Okanohara & Tsujii, 2007), like us,
also take a two-class approach (true/fake sentences).
They use a shallow (kernel) classifier.
Previous Work in Semi-Supervised Learning
For an overview of semi-supervised learning, see(Chapelle et al., 2006). There have been several usesof semi-supervised learning in NLP before, for example in NER (Rosenfeld & Feldman, 2007), machinetranslation (Ueffing et al., 2007), parsing (McClosky
et al., 2006) and text classification (Joachims, 1999).
The first work is a highly problem-specific approach
whereas the last three all use a self-training type approach (Transductive SVMs in the case of text classification, which is a kind of self-training method). Thesemethods augment the training set with labeled examples from the unlabeled set which are predicted by themodel itself. This can give large improvements in amodel, but care must be taken as the predictions areof course prone to noise.
The authors of (Ando & Zhang, 2005) propose a setup
more similar to ours: they learn from unlabeled dataas an auxiliary task in a MTL framework. The maindifference is that they use shallow classifiers; however
they report positive results on POS and NER tasks.
Semantically Related Words Task We found it
interesting to compare the embedding obtained witha language model on unlabeled data with an embedding obtained with labeled data. WordNet isa database which contains semantic relations (synonyms, holonyms, hypernyms, ...) between around
150, 000 words. We used it to train a NN similar tothe language model one. We considered the problem
as a two-class classification task: positive examples arepairs with a relation in Wordnet, and negative examples are random pairs.

6. Experiments
We used Sections 02-21 of the PropBank dataset version 1 (about 1 million words) for training and Section 23 for testing as standard in all SRL experiments.
POS and chunking tasks use the same data split via
the Penn TreeBank. NER labeled data was obtainedby running the Stanford Named Entity Recognizer (aTable 1. Language model performance for learning an embedding in wsz = 50 dimensions (dictionary size: 30, 000).
For each column the queried word is followed by its index in
the dictionary (higher means more rare) and its 10 nearest
neighbors (arbitrary using the Euclidean metric).
france
454
spain
italy
russiapolandenglanddenmark
germany
portugal
swedenaustriajesus
1973
christgod
resurrection
prayeryahwehjosephus
moses
sin
heavensalvation

xbox
6909
playstation
dreamcastpsNUMBER
snes
wii
nes
nintendo
gamecube
psp
amiga

reddish11724
yellowish
greenishbrownishbluishcreamywhitishblackishsilvery
greyishpaler

scratched29869
smashedrippedbrushedhurledgrabbedtossedsqueezedblastedtangledslashedCRF based classifier) over the same data.
Language models were trained on Wikipedia. In allcases, any numeric number was converted as “NUMBER”. Accentuated characters were transformed totheir non-accentuated versions. All paragraphs containing other non-ASCII characters were discarded.
For Wikipedia, we obtain a database of 631M words.
We used WordNet to train the “synonyms” (semantically related words) task.
All tasks use the same dictionary of the 30, 000 mostcommon words from Wikipedia, converted to lower
case. Other words were considered as unknown andmapped to a special word.
Architectures All tasks were trained using the NN
shown in Figure 1. POS, NER, and chunking tasks
were trained with the window version with ksz = 5.
We chose linear models for POS and NER. For chunking we chose a hidden layer of 200 units. The language
model task had a window size ksz = 11, and a hidden
layer of 100 units. All these tasks used two lookuptables: one of dimension wsz for the word in lower
case, and one of dimension 2 specifying if the first letter of the word is a capital letter or not.
For SRL, the network had a convolution layer withksz = 3 and 100 hidden units, followed by anotherhidden layer of 100 hidden units. It had three lookuptables in the first layer: one for the word (in lowercase), and two that encode relative distances (to theword of interest and the verb). The last two lookuptables embed in 5 dimensional spaces. Verb positions
are obtained with our POS classifier.
The language model network had only one lookuptable (the word in lower case) and 100 hidden units.
It used a window of size ksz = 11.
We show results for different encoding sizes of the wordin lower case: wsz = 15, 50 and 100.

A Unified Architecture for Natural Language Processing
Table 2. A Deep Architecture for SRL improves by learning auxiliary tasks that share the first layer that represents words
as wsz-dimensional vectors. We give word error rates for wsz=15, 50 and 100 and various shared tasks.
wsz=15
16.54
15.99
16.42
16.67
15.46
14.42
16.46
16.45
16.33
15.71
14.63

SRL
SRL + POS
SRL + Chunking
SRL + NERSRL + Synonyms
SRL + Language model
SRL + POS + Chunking
SRL + POS + NER
SRL + POS + Chunking + NER
SRL + POS + Chunking + NER + Synonyms
SRL + POS + Chunking + NER + Language model

wsz=15

Wsz=100

22

19

18

22

SRL
SRL+POS
SRL+CHUNKSRL+POS+CHUNK
SRL+POS+CHUNK+NER
SRL+SYNONYMS
SRL+POS+CHUNK+NER+SYNONYMS
SRL+LANG.MODEL
SRL+POS+CHUNK+NER+LANG.MODEL

21

20

19

18

20

19

18

17

17

17

16

16

16

15

15

141

11

21

31

141

SRL
SRL+POS
SRL+CHUNKSRL+POS+CHUNK
SRL+POS+CHUNK+NER
SRL+SYNONYMS
SRL+POS+CHUNK+NER+SYNONYMS
SRL+LANG.MODEL
SRL+POS+CHUNK+NER+LANG.MODEL

21

Test Error

20

Test Error

SRL
SRL+POS
SRL+CHUNKSRL+POS+CHUNK
SRL+POS+CHUNK+NER
SRL+SYNONYMS
SRL+POS+CHUNK+NER+SYNONYMS
SRL+LANG.MODEL
SRL+POS+CHUNK+NER+LANG.MODEL21

wsz=100
18.40
16.53
16.48
17.21
15.17
14.46
16.41
16.29
16.27
15.48
14.50

Wsz=5022

Test Error

wsz=50
17.33
16.57
16.39
17.29
15.17
14.30
15.95
16.89
16.36
14.76
14.44

15

3.5

6

Epoch

8.5

11

Epoch

13.5

16

18.5

141

3.5

6

8.5

11

13.5

Epoch

Figure 3. Test error versus number of training epochs over PropBank, for the SRL task alone and SRL jointly trainedwith various other NLP tasks, using deep NNs.

Results: Language Model Because the language
model was trained on a huge database we first trained
it alone. It takes about a week to train on one computer. The embedding obtained in the word lookuptable was extremely good, even for uncommon words,
as shown in Table 1. The embedding obtained bytraining on labeled data from WordNet “synonyms”
is also good (results not shown) however the coverage
is not as good as using unlabeled data, e.g. “Dreamcast” is not in the database.
The resulting word lookup-table from the language
model was used as an initializer of the lookup-tableused in MTL experiments with a language model.
Results: SRL Our main interest was improving SRL
performance, the most complex of our tasks. In Table 2, we show results comparing the SRL task alone
with the SRL task jointly trained with different combinations of the other tasks. For all our experiments,
training was achieved in a few epochs (about a day)over the PropBank dataset as shown in Figure 3. Testing takes 0.015s to label a complete sentence (given oneverb).

All MTL experiments performed better than SRL
alone. With larger wsz (and thus large capacity) the
relative improvement becomes larger from using MTL
compared to the task alone, which shows MTL is agood way of regularizing: in fact with MTL results
are fairly stable with capacity changes.
The semi-supervised training of SRL using the language model performs better than other combinations.
Our best model performed as low as 14.30% in perword error rate, which is to be compared to previously published results of 16.36% with an NN architecture (Collobert & Weston, 2007) and 16.54% for astate-of-the-art method based on parse trees (Pradhan
et al., 2004)1 . Further, our system is the only one not
to use POS tags or parse tree features.
Results: POS and Chunking Training takes about
30 min for these tasks alone. Testing time for labeling a complete sentence is about 0.003s. We obtainedmodest improvements to POS and chunking results us1
Our loss function optimized per-word error rate. We
note that many SRL results e.g. the CONLL 2005 evaluation use F1 as a standard measure.

A Unified Architecture for Natural Language Processing

ing MTL. Without MTL (for wsz = 50) we obtain
2.95% test error for POS and 4.5% (91.1 F-measure)for chunking. With MTL we obtain 2.91% for POS
and 3.8% (92.71 F-measure) for chunking. POS error
rates in the 3% range are state-of-the-art. For chunking, although we use a different train/test setup to the
CoNLL-2000 shared task (http://www.cnts.ua.ac.
be/conll2000/chunking) our system seems competitive with existing systems (better than 9 of the 11
submitted systems). However, our system is the only
one that does not use POS tags as input features.

Gildea, D., & Palmer, M. (2001). The necessity of parsing
for predicate argument recognition. Proceedings of the40th Annual Meeting of the ACL, 239–246.
Joachims, T. (1999). Transductive inference for text classification using support vector machines. ICML.
LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998).
Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86.
McClosky, D., Charniak, E., & Johnson, M. (2006). Effective self-training for parsing. Proceedings of HLTNAACL 2006.

Note, we did not evaluate NER error rates because
we used non-gold standard annotations in our setup.
Future work will more thoroughly evaluate these tasks.

Miller, S., Fox, H., Ramshaw, L., & Weischedel, R. (2000).
A novel use of statistical parsing to extract information from text. 6th Applied Natural Language Processing
Conference.

7. Conclusion

Musillo, G., & Merlo, P. (2006). Robust Parsing of the
Proposition Bank. ROMAND 2006: Robust Methods inAnalysis of Natural language Data.

We proposed a general deep NN architecture for NLP.
Our architecture is extremely fast enabling us to takeadvantage of huge databases (e.g. 631 million words
from Wikipedia). We showed our deep NN could beapplied to various tasks such as SRL, NER, POS,
chunking and language modeling. We demonstrated
that learning tasks simultaneously can improve generalization performance. In particular, when training
the SRL task jointly with our language model ourarchitecture achieved state-of-the-art performance in
SRL without any explicit syntactic features. This is
an important result, given that the NLP community
considers syntax as a mandatory feature for semantic
extraction (Gildea & Palmer, 2001).

References
Ando, R., & Zhang, T. (2005). A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data. JMLR, 6, 1817–1853.
Bengio, Y., & Ducharme, R. (2001). A neural probabilistic
language model. NIPS 13.
Bridle, J. (1990). Probabilistic interpretation of feedforward classification network outputs, with relationships
to statistical pattern recognition. In F. F. Soulié and
J. Hérault (Eds.), Neurocomputing: Algorithms, architectures and applications, 227–236. NATO ASI Series.
Caruana, R. (1997). Multitask Learning. Machine Learning, 28, 41–75.
Chapelle, O., Schlkopf, B., & Zien, A. (2006). Semisupervised learning. Adaptive computation and machinelearning. Cambridge, Mass., USA: MIT Press.
Collobert, R., & Weston, J. (2007). Fast semantic extraction using a novel neural network architecture. Proceedings of the 45th Annual Meeting of the ACL (pp. 560–
567).

Okanohara, D., & Tsujii, J. (2007). A discriminative language model with pseudo-negative samples. Proceedingsof the 45th Annual Meeting of the ACL, 73–80.
Palmer, M., Gildea, D., & Kingsbury, P. (2005). The
proposition bank: An annotated corpus of semanticroles. Comput. Linguist., 31, 71–106.
Pradhan, S., Ward, W., Hacioglu, K., Martin, J., & Jurafsky, D. (2004). Shallow semantic parsing using support
vector machines. Proceedings of HLT/NAACL-2004.
Rosenfeld, B., & Feldman, R. (2007). Using Corpus Statistics on Entities to Improve Semi-supervised Relation Extraction from the Web. Proceedings of the 45th Annual
Meeting of the ACL, 600–607.
Schwenk, H., & Gauvain, J. (2002). Connectionist language modeling for large vocabulary continuous speech
recognition. IEEE International Conference on Acoustics, Speech, and Signal Processing (pp. 765–768).
Sutton, C., & McCallum, A. (2005a). Composition of conditional random fields for transfer learning. Proceedingsof the conference on Human Language Technology and
Empirical Methods in Natural Language Processing, 748–
754.
Sutton, C., & McCallum, A. (2005b). Joint parsing andsemantic role labeling. Proceedings of CoNLL-2005 (pp.
225–228).
Sutton, C., McCallum, A., & Rohanimanesh, K. (2007).
Dynamic Conditional Random Fields: Factorized Probabilistic Models for Labeling and Segmenting Sequence
Data. JMLR, 8, 693–723.
Ueffing, N., Haffari, G., & Sarkar, A. (2007). Transductive
learning for statistical machine translation. Proceedingsof the 45th Annual Meeting of the ACL, 25–32.
Waibel, A., abd G. Hinton, T. H., Shikano, K., & Lang,
K. (1989). Phoneme recognition using time-delay neuralnetworks. IEEE Transactions on Acoustics, Speech, andSignal Processing, 37, 328–339.