Human Language Technology:
Applica6on to Informa6on AccessLesson 10
Deep learning for NLP:
Mul6lingual Word Sequence Modeling
December 15, 2016

EPFL Doctoral Course EE-724
Nikolaos Pappas
Idiap Research Ins6tute, Mar6gny

Outline of the talk
1. Recap: Word Representa6on Learning
2. Mul6lingual Word Representa6ons• Alignment models• Evalua6on tasks
3. Mul6lingual Word Sequence Modeling• Essen6als: RNN, LSTM, GRU• Machine Transla6on• Document Classiﬁca6on
4. Summary* Figure from Lebret's thesis, EPFL, 2016

Nikolaos Pappas

2 /88

Disclaimer
Research highlights rather than in-depth analysis• By no means exhaus6ve (progress too fast!)• Tried to keep most representa6ves• Focus on feature learning and two major NLP tasks••

Not enough 6me to cover other exci6ng tasks:• Ques6on answering• Rela6on classiﬁca6on• Paraphrase detec6on• Summariza6on
Nikolaos Pappas

3 /88Recap: Learning word
representa6ons from text•

Why should we care about them?
• tackles curse of dimensionality• captures seman6c and analogy rela6ons of words• captures general knowledge in an unsupervised way
king - man + woman ≈ queen

Nikolaos Pappas

4 /88Recap: Learning word
representa6ons from text•

How can we beneﬁt from them?
• study linguis6c proper6es of words• inject general knowledge on downstream tasks• transfer knowledge across languages or modali6es• compose representa6ons of word sequences

Nikolaos Pappas

5 /88Recap: Learning word
representa6ons from text•

Which method to use for learning them?
• neural versus count-based methods➡ neural ones implicitly do SVD over a PMI matrix➡ similar to count-based when using the same tricks• neural methods appear to have the edge (word2vec)➡ eﬃcient and scalable objec6ve + toolkit➡ intui6ve formula6on (=predict words in context)

Nikolaos Pappas

6 /88Recap: Con6nuous Bag-ofWords (CBOW)

Nikolaos Pappas

7 /88Recap: Con6nuous Bag-ofWords (CBOW)

Nikolaos Pappas

8 /88Recap: Learning word
representa6ons from text•

What else can we do with word embeddings?

dependency-based embeddings: Levy and Goldberg 2014
• retroﬁjed-to-lexicons embeddings: Faruqui et al. 2014
• sense-aware embeddings: Li and Jurafsky 2015
• visually-grounded embeddings: Lazaridou et al. 2015
• mul6lingual embeddings: Gouws et al 2015•

Nikolaos Pappas

9 /88Outline of the talk
1. Recap: Word Representa6on Learning
2. Mul6lingual Word Representa6ons• Alignment models• Evalua6on tasks
3. Mul6lingual Word Sequence Modeling• Essen6als: RNN, LSTM, GRU• Machine Transla6on• Document Classiﬁca6on
4. Summary* Figure from Gouts et al., 2015.

Nikolaos Pappas

10 /88

Learning cross-lingual word
representa6ons•

Monolingual embeddings capture seman6c, syntac6c
and analogy rela6ons between words•

Goal: capture this rela6onships two or more languages* Figure from Gouts et al., 2015.

Nikolaos Pappas

11 /88Supervision of cross-lingualalignment methods• Parallel sentences for MT: Guo et al., 2015

high

Sentence by sentence and word alignments• Parallel sentences: Gouws et al., 2015

Sentence by sentence alignments• Parallel documents: Søgaard et al., 2015

Documents with topic or label alignments

Annotation
cost• Bilingual dicHonary: Ammar et al., 2016

Word by word transla6ons• No parallel data: Faruqui and Dyer, 2014

Really!

low
Nikolaos Pappas

12 /88

Cross-lingual alignment with noparallel data•

Nikolaos Pappas

13 /88

Cross-lingual alignment withparallel sentences•

Nikolaos Pappas

14 /88

Cross-lingual alignment withparallel sentences(Gows et al., 2016)
Nikolaos Pappas

15 /88

Cross-lingual alignment withparallel sentences for MT

Nikolaos Pappas

16 /88Uniﬁed framework for analysis
of cross-lingual methods• Minimize monolingual objec6ve• Constraint/Regularize with bilingual objec6ve

Nikolaos Pappas

17 /88

Evalua6on: Cross-lingual documentclassiﬁca6on and transla6on(Gows et al., 2015)

Nikolaos Pappas

18 /88

Bonus: Mul6lingual visual
sen6ment concept matching
concept = adjec6ve-noun-phrase (ANP)(Pappas et al., 2016)

Nikolaos Pappas

19 /88

Mul6lingual visual sen6ment
concept ontology(Jou et al., 2015)

Nikolaos Pappas

20 /88

Word embedding model(Pappas et al., 2016)

Nikolaos Pappas

21 /88

Mul6lingual visual sen6ment
concept retrieval(Pappas et al., 2016)

Nikolaos Pappas

22 /88

Mul6lingual visual sen6ment
concept clustering(Pappas et al., 2016)

Nikolaos Pappas

23 /88

Mul6lingual visual sen6ment
concept clustering(Pappas et al., 2016)

Nikolaos Pappas

24 /88

Discovering interes6ng clusters:
Mul6lingual(Pappas et al., 2016)

Nikolaos Pappas

25 /88

Discovering interes6ng clusters:
Western vs. Eastern(Pappas et al., 2016)(Pappas et al., 2016)

Nikolaos Pappas

26 /88

Discovering interes6ng clusters:
Monolingual(Pappas et al., 2016)

Nikolaos Pappas

27 /88

Evalua6on: Mul6lingual visual
sen6ment concept analysis•

Aligned embeddings are bejer than transla6on inconcept retrieval, clustering and sen6ment predic6on

Nikolaos Pappas

28 /88

Conclusion•

Aligned embeddings are cheaper than transla6on andusually work bejer than it in several mul6lingual or
crosslingual NLP tasks without parallel data•

document classiﬁca6on Gows et al., 2015•

named en6ty recogni6on Al-Rfou et al., 2014•

dependency parsing Guo et al., 2015•

concept retrieval and clustering Pappas et al., 2016

Nikolaos Pappas

29 /88

Outline of the talk
1. Recap: Word Representa6on Learning
2. Mul6lingual Word Representa6ons• Alignment models• Evalua6on tasks
3. Mul6lingual Word Sequence Modeling• Essen6als: RNN, LSTM, GRU• Machine Transla6on• Document Classiﬁca6on* Figure from Colah’s blog, 2015.

4. Summary
Nikolaos Pappas

30 /88

Language Modeling•

Computes the probability of a sequence of words orsimply “likelihood of a text”: P(w1, w2, …, wt)•

N-gram models with Markov assump6on:•

Where is it useful?
• speech recogni6on• machine transla6on• POS tagging and parsing•

Nikolaos Pappas

What are its limitaHons?
• unrealis6c assump6on• huge memory needs• back-oﬀ models
31 /88

Recurrent Neural Network (RNN)•

Neural language model:•

What are its main limitaHons?
• vanishing gradient problem (error doesn’t propagate far)• fail to capture long-term dependencies• tricks: gradient clipping, iden6ty ini6aliza6on + ReLus
Nikolaos Pappas

32 /88

Long Short Term Memory (LSTM)•

Long-short term memory nets are able to learn longterm dependencies: Hochreiter and Schmidhuber 1997
Simple RNN:

* Figure from Colah’s blog, 2015.

Nikolaos Pappas

33 /88

Long Short Term Memory (LSTM)•

Long-short term memory nets are able to learn longterm dependencies: Hochreiter and Schmidhuber 1997
• Ability to remove or add informa6on to the cellstate regulated by “gates”* Figure from Colah’s blog, 2015.

Nikolaos Pappas

34 /88

Gated Recurrent Unit (GRU)•

Gated RNN by Chung et al, 2014 combines the forget
and input gates into a single “update gate”• keep memories to capture long-term dependencies• allow error messages to ﬂow at diﬀerent strengths* Figure from Colah’s blog, 2015.

zt: update gate — rt: reset gate — ht: regular RNN update
Nikolaos Pappas

35 /88

Deep Bidirec6onal Models• Here RNN but it applies to LSTMs and GRUs too(Irsoy and Cardie, 2014)

Nikolaos Pappas

36 /88

Convolu6onal Neural Network (CNN)
Typically good for images• Convolu6onal ﬁlter(s) is (are)applied every k words:••

Similar to Recursive NNs but without
constraining to gramma6cal phrases
only, as Socher et al., 2011
• no need for a parser (!)• less linguis6cally mo6vated ?

(Collobert et al., 2011)(Kim, 2014)
Nikolaos Pappas

37 /88

Hierarchical Models• Word-level and sentence-level modeling withany type of NN layers(Tang et al., 2015)
Nikolaos Pappas

38 /88Ajen6on Mechanism for
Machine Transla6on•

Chooses “where to look” or learns to assign a relevanceto each input posi6on given encoder hidden state forthat posi6on and the previous decoder state•

learns a sou bilingual alignment model(Bahdanau et al., 2015)
Nikolaos Pappas

39 /59Ajen6on Mechanism forDocument Classiﬁca6on
Operates on input word sequence (or intermediate
hidden states: Pappas and Popescu-Belis 2016)• Learns to focus on relevant parts of the input withrespect to the target labels• learns a sou extrac6ve summariza6on model•(Pappas and Popescu-Belis, 2014)
Nikolaos Pappas

40 /59Outline of the talk
1. Recap: Word Representa6on Learning
2. Mul6lingual Word Representa6ons• Alignment models• Evalua6on tasks
3. Mul6lingual Word Sequence Modeling• Essen6als: RNN, LSTM, GRU• Machine Transla6on• Document Classiﬁca6on* Figure from Colah’s blog, 2015.

4. Summary
Nikolaos Pappas

41 /88RNN encoder-decoder for
Machine Transla6on
GRU as hidden layer• Maximize the log likelihoodof the target sequence
given the source sequence:••

WMT 2014 (EN→FR)(Cho et al., 2014)
Nikolaos Pappas

42 /88Sequence to sequence learningfor Machine Transla6on• LSTM hidden layers instead of GRU• 4 layers deep instead of shallow encoder-decoder(Sutskever et al., 2014)
Nikolaos Pappas

43 /88Sequence to sequence learningfor Machine Transla6on•

WMT 2014 (EN→FR)•

PCA projec6on of the hidden state of the last encoder layer(Sutskever et al., 2014)
Nikolaos Pappas

44 /88

Jointly learning to align andtranslate for Machine Transla6on
LimitaHon: can we compress all
the needed informa6on in thelast encoder state?
• Idea: use all the hidden statesof the encoder• length propor6onal to thatof the sentence!
• compute a weighted averageof all the hidden states•(Bahdanau et al., 2015)
Nikolaos Pappas

45 /88

Jointly learning to align andtranslate for Machine Transla6on•

WMT 2014 (EN→FR)(Bahdanau et al., 2015)
Nikolaos Pappas

46 /88

Eﬀec6ve approaches toajen6on-based NMT• Global and local ajen6on• Input-feeding approach• Stacked LSTM instead of single-layer(Luong et al., 2015)
Nikolaos Pappas

47 /88

Mul6-source NMT• Train p(e|f, g) model

directly on trilingual data
• Use it to decode e givenany (f, g) pair
• Take local-ajen6on NMT
model and concatenate
context from mul6ple
sources(Zoph and Knight, 2016)
Nikolaos Pappas

48 /88

Mul6-source NMT• Mul6-source training improves over individual

French English and German English pairs• Best: basic concatena6on with ajen6on(Zoph and Knight, 2016)
Nikolaos Pappas

49 /88

Mul6-source NMT• Mul6-source training improves over individual

French English and German English pairs• Best: basic concatena6on with ajen6on(Zoph and Knight, 2016)
Nikolaos Pappas

50 /88

Mul6-target NMT• Mul6-task learning framework for mul6ple target

language transla6on• Op6miza6on for one to many model(Dong et al., 2015)
Nikolaos Pappas

51 /88

Mul6-target NMT• Improves over NMT

and moses baselines
over WMT 2013 test
• but also on larger
datasets• Faster and bejer
convergence inmul6ple languagetransla6on(Dong et al., 2015)
Nikolaos Pappas

52 /88

Mul6-way, Mul6lingual NMT••

Encoder-decoder model with
mul6ple encoders and decoders
shared across pairs• share knowledge across langs• universal space for all langs• good for low-resource langs
Ajen6on is pair speciﬁc, hence
expensive O(L^2)• instead share ajen6on acrossall pairs!

Figure: n_th encoder and m_th decoder at 6mestep t / φ makes encoder & decoder states compa6blewith the ajen6on mechanism / f_adp makes context vector compa6ble with the decoder→ all these transforma6ons to support diﬀerent types of encoders/decoders for diﬀerent languages!
(Firat et al., 2016)
Nikolaos Pappas

53 /88

Mul6-way, Mul6lingual NMT• Consistent improvements for low-

resource languages• the lower the training data thebigger the improvement• In large-scale translaHon improves
only translaHon to English• hypothesis: EN appears always assource or target language for all
pairs → bejer decoder ?

(Firat et al., 2016)
Nikolaos Pappas

54 /88

Mul6-way, Mul6lingual NMT• Consistent improvements for low-

resource languages• the lower the training data thebigger the improvement• In large-scale translaHon improves
only translaHon to English• hypothesis: EN appears always assource or target language for all
pairs → bejer decoder ?

(Firat et al., 2016)
Nikolaos Pappas

55 /88

Google’s Neural MachineTransla6on System “Monster”• An encoder, a decoder and an ajen6on network• Plus 8-layer deep with residual connec6ons• Plus reﬁnement with Reinforcement Learning• Plus sub-word units…Plus….

(Wu et al., 2016)
Nikolaos Pappas

56 /88

Google’s Neural MachineTransla6on System “Monster”•

EN->FR training takes 6 days on 96GPUS !!!! and 3 more days for reﬁnement...

(Wu et al., 2016)
Nikolaos Pappas

57 /88

Future of NMT and otherpossibili6es• MulH-task learning: Training

mul6ple pairs of languages
jointly and with other tasks→ Image cap6oning,
Speech recogni6on !

•Larger context: Modeling larger sequences than sentences as
in document classiﬁca6on will be key• understanding long-term dependencies• leveraging structural informa6on of the input• being able to reason over it to solve any task
→ Eﬀec6ve Ajen6on / Memory?
(Luong, Cho, Manning tutorial, 2016)
Nikolaos Pappas

58 /88

Outline of the talk
1. Recap: Word Representa6on Learning
2. Mul6lingual Word Representa6ons• Alignment models• Evalua6on tasks
3. Mul6lingual Word Sequence Modeling• Essen6als: RNN, LSTM, GRU• Machine Transla6on• Document Classiﬁca6on* Figure from Colah’s blog, 2015.

4. Summary
Nikolaos Pappas

59 /88Paragraph vectors forDocument Classiﬁca6on

Learning vectors of paragraphs inspired by word2vec• trained without supervision on a large corpus• preferably similar domain as the target• Two methods: with or without word ordering•(Le et al., 2014)
Nikolaos Pappas

60 /88Paragraph vectors forDocument Classiﬁca6on
Learned paragraph vectors + logis6c regression• Outperformed previous method on sentence-level anddocument-level sen6ment classiﬁca6on•(Le et al., 2014)
Nikolaos Pappas

61 /88

Convolu6onal neural network
for Document Classiﬁca6on
Used mul6ple ﬁlter widths• Dropout regulariza6on (randomly dropping por6on of
hidden units during back-propaga6on)•(Kim et al., 2014)
Nikolaos Pappas

62 /88

Convolu6onal neural network
for Document Classiﬁca6on•

Not all baseline methods used drop-out though(Kim et al., 2014)
Nikolaos Pappas

63 /88Modeling and Summarizing Documentswith a Convolu6onal Network•

Similar to Kim et al, 2014 however diﬀerent• K-max pooling instead of max pooling• Two layers of convolu6ons(Denil et al., 2014)
Nikolaos Pappas

64 /88Modeling and Summarizing Documentswith a Convolu6onal Network(Denil et al., 2014)
Nikolaos Pappas

65 /88Modeling and Summarizing Documentswith a Convolu6onal Network(Denil et al., 2014)
Nikolaos Pappas

66 /88Modeling and Summarizing Documentswith a Convolu6onal Network(Denil et al., 2014)
Nikolaos Pappas

67 /88

Gated recurrent neural network
for Document Classiﬁca6on(Tang et al., 2015)
Nikolaos Pappas

68 /88

Gated recurrent neural network
for Document Classiﬁca6on(Tang et al., 2015)
Nikolaos Pappas

69 /88

Standard Pipeline for Document
Classiﬁca6on
Feature engineering: BOW, n-grams, topic models, etc.
• Feature learning: auto-encoders, convolu6onal,
recurrent, recursive NNs•(Pappas and Popescu-Belis, 2014)
Nikolaos Pappas

70 /88

Mul6ple-instance Learning forDocument Classiﬁca6on(Pappas and Popescu-Belis, 2014)
Nikolaos Pappas

71 /88

How to combine vectors?
Structural assump6ons(Pappas and Popescu-Belis, 2014)
Nikolaos Pappas

72 /88

Joint learning of an instancerelevance mechanism and a classiﬁer(Pappas and Popescu-Belis, 2014)
Nikolaos Pappas

73 /88

Joint diﬀeren6able objec6ve forsolving with SGD(Pappas and Popescu-Belis, 2014)
Nikolaos Pappas

74 /88Observa6ons on aspect ra6ng
predic6on

The proposed mechanism issuperior than alterna6ves• all text regions are useful
but to a diﬀerent extent• Beneﬁt regardless of the input
features used
• Reaches state-of-the-art
without using:• structured output learning• segmented text•

(Pappas and Popescu-Belis, 2014)
Nikolaos Pappas

75 /88

Comparison with neuralnetwork models

This mechanism can be used as a parametric pooling func6on of NNs• opera6ng on intermediate hidden states• Works bejer than Dense, GRU neural methods + average pooling• Outperforms RCNN and uses far less parameters•(Pappas and Popescu-Belis, 2016)
Nikolaos Pappas

76 /88

Hierarchical ajen6on networks
for Document Classiﬁca6on•

Very similar hierarchicalstructure as Tang et al., 2015
except average pooling• ajen6on mechanism at theword and document levels(Yang et al., 2016)
Nikolaos Pappas

77 /88

Hierarchical ajen6on networks
for Document Classiﬁca6on(Yang et al., 2016)
Nikolaos Pappas

78 /88

Reﬂec6ons on Mul6lingualDocument Classiﬁca6on•

What are the present limitaHons?
• Current evalua6on datasets contain small number oftarget classes and examples• RCV1/RCV2 → 6,000 documents, 2 langs, 4 labels• TED corpus → 12,078 documents, 12 langs, 15 labels• Requires the labels to be common across languages• Data are not enough to train SOA neural architectures•

ObservaHon: currently there are several domains which
support mul6ple languages but only monolingualclassiﬁca6on is possible
Nikolaos Pappas

79 /88

New dataset: Deutsche Welle
corpus (600k docs, 8 langs)

Nikolaos Pappas

80 /88

Conclusion
Mul6lingual word embeddings are useful for tasks where
there is lack of parallel data
• Word sequence modeling is advancing quickly with theestablishment of neural methods• Machine Transla6on• Document Classiﬁca6on• MulHlingual Neural Machine TranslaHon• is useful for low-resourced languages• transfers knowledge in large-scale se}ng• MulHlingual Document ClassiﬁcaHon• several large resources available but with disjoint labels• could possibly beneﬁt from NMT lessons•

Nikolaos Pappas

81 /88References (1/3)• Le, Quoc V., and Tomas Mikolov. "Distributed Representa6ons of Sentences and Documents." In ICML, vol. 14, pp.

1188-1196. 2014.
• Kim, Yoon. "Convolu6onal neural networks for sentence classiﬁca6on." arXiv preprint arXiv:1408.5882, 2014.
• Denil, Misha, Alban Demiraj, Nal Kalchbrenner, Phil Blunsom, and Nando de Freitas. "Modelling, visualising andsummarising documents with a single convolu6onal neural network." arXiv preprint arXiv:1406.3830, 2014.
• Yang, Zichao, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. "Hierarchical ajen6on networks for
document classiﬁca6on." In Proceedings of the 2016 Conference of the North American Chapter of the Associa6on for
Computa6onal Linguis6cs: Human Language Technologies. 2016.
• Tang, Duyu, Bing Qin, and Ting Liu. "Document modeling with gated recurrent neural network for sen6ment
classiﬁca6on." In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.
1422-1432, 2015.
• Firat, Orhan, Kyunghyun Cho, Baskaran Sankaran, Fatos T. Yarman Vural, and Yoshua Bengio. "Mul6-way, mul6lingual
neural machine transla6on." Computer Speech & Language, 2016.
• Pappas, Nikolaos, Miriam Redi, Mercan Topkara, Brendan Jou, Hongyi Liu, Tao Chen, and Shih-Fu Chang. "Mul6lingual
visual sen6ment concept matching.” In Interna6onal Conference of Mul6media Retrieval, 2016.
• Pappas, Nikolaos, and Andrei Popescu-Belis. "Explaining the stars: Weighted mul6ple-instance learning for aspectbased sen6ment analysis." In Conference on Empirical Methods in Natural Language Processing, 2014.
• Pappas Nikolaos, Andrei Popescu-Belis. “Explicit Document Modeling through Weighted Mul6ple-Instance Learning”,
Under review.
• Yoav Goldberg. “A primer on neural network models for natural language processing” arXiv preprint:1510.00726, 2015.
• Ian Goodfellow, Aaron Courville, and Joshua Bengio. “Deep learning”. Book in prepara6on for MIT Press., 2015.

Nikolaos Pappas

82 /88References (2/3)• Wu, Yonghui, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun et al.

"Google's Neural Machine Transla6on System: Bridging the Gap between Human and Machine Transla6on." arXiv
preprint arXiv:1609.08144, 2016.
• Zoph, Barret, and Kevin Knight. "Mul6-Source Neural Transla6on." arXiv preprint arXiv:1601.00710, 2016.
• Dong, Daxiang, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. "Mul6-task learning for mul6ple language transla6on."
In Proceedings of the 53rd Annual Mee6ng of the ACL and the 7th Interna6onal Joint Conference on Natural Language
Processing, pp. 1723-1732. 2015.
• Luong, Minh-Thang, Hieu Pham, and Christopher D. Manning. "Eﬀec6ve approaches to ajen6on-based neural machinetransla6on." arXiv preprint arXiv:1508.04025, 2015.
• Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. "Neural machine transla6on by jointly learning to align andtranslate." arXiv preprint arXiv:1409.0473, 2014.
• Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. "Sequence to sequence learning with neural networks." In Advances inneural informa6on processing systems, pp. 3104-3112. 2014.
• Cho, Kyunghyun, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, andYoshua Bengio. "Learning phrase representa6ons using RNN encoder-decoder for sta6s6cal machine transla6on." arXiv
preprint arXiv:1406.1078, 2014.
• Irsoy, Ozan, and Claire Cardie. "Deep recursive neural networks for composi6onality in language." In Advances in Neural
Informa6on Processing Systems, pp. 2096-2104. 2014.
• Chung, Junyoung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. "Empirical evalua6on of gated recurrent neuralnetworks on sequence modeling." arXiv preprint arXiv:1412.3555 (2014).
• Hochreiter, Sepp, and Jürgen Schmidhuber. "Long short-term memory." Neural computa6on 9, no. 8 (1997): 1735-1780.
• Levy, Omer, and Yoav Goldberg. "Dependency-Based Word Embeddings." In ACL (2), pp. 302-308. 2014.

Nikolaos Pappas

83 /88References (3/3)• Pappas, Nikolaos, Miriam Redi, Mercan Topkara, Brendan Jou, Hongyi Liu, Tao Chen, and Shih-Fu Chang. "Mul6cultural

Visual Concept Retrieval and Clustering.”, under review, 2016.
• Klemen6ev, Alexandre, Ivan Titov, and Binod Bhajarai. "Inducing crosslingual distributed representa6ons of words."
2012.
• Bengio, Yoshua, and Greg Corrado. "Bilbowa: Fast bilingual distributed representa6ons without word alignments." 2014.
• Hermann, Karl Moritz, and Phil Blunsom. "Mul6lingual distributed representa6ons without word alignment." arXiv
preprint arXiv:1312.6173, 2013.
• Faruqui, Manaal, and Chris Dyer. "Improving vector space word representa6ons using mul6lingual correla6on."
Associa6on for Computa6onal Linguis6cs, 2014.
• Søgaard, Anders, Željko Agić, Héctor Marˆnez Alonso, Barbara Plank, Bernd Bohnet, and Anders Johannsen. "Inverted
indexing for cross-lingual nlp." In The 53rd Annual Mee6ng of the Associa6on for Computa6onal Linguis6cs and the 7thInterna6onal Joint Conference of the Asian Federa6on of Natural Language Processing (ACL-IJCNLP 2015), 2015.
• Ammar, Waleed, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A. Smith. "Massively
mul6lingual word embeddings." arXiv preprint arXiv:1602.01925, 2016.
• Guo, Jiang, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. "Cross-lingual dependency parsing based on
distributed representa6ons." In Proceedings of the 53rd Annual Mee6ng of the Associa6on for Computa6onal
Linguis6cs and the 7th Interna6onal Joint Conference on Natural Language Processing, vol. 1, pp. 1234-1244, 2015.
• Lazaridou, Angeliki, Nghia The Pham, and Marco Baroni. "Combining language and vision with a mul6modal skip-grammodel." arXiv preprint arXiv:1501.02598, 2015.
• Li, Jiwei, and Dan Jurafsky. "Do mul6-sense embeddings improve natural language understanding?." arXiv preprint
arXiv:1506.01070 (2015).
• Faruqui, Manaal, Jesse Dodge, Sujay K. Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. "Retroﬁ}ng word vectors to

Nikolaos Pappas

84 /88

Resources (1/2)➡ Online courses

Coursera course on “Neural networks for machine learning” by Geoﬀrey Hinton
hjps://www.coursera.org/learn/neural-networks• Coursera course on “Machine learning” by Andrew Ng
hjps://www.coursera.org/learn/machine-learning• Stanford CS224d “Deep learning for NLP” by Richard Socher
hjp://cs224d.stanford.edu/•➡ Conference tutorials

Richard Socher and Christopher Manning, “Deep learning for NLP”, EMNLP 2013
tutorial.
hjp://nlp.stanford.edu/courses/NAACL2013/• David Jurgens and Mohammad Taher Pilehvar, “Seman6c Similarity Fron6ers: From
Concepts to Documents”, EMNLP 2015 tutorial.
hjp://www.emnlp2015.org/tutorials.html#t1• Mitesh M Kharpa, Sarath Chandar, “Mul6lingual and Mul6modal Language
Processing”, NAACL 2016 tutorial.
hjp://naacl.org/naacl-hlt-2016/t2.html•

Nikolaos Pappas

85 /59Resources (2/2)➡ Deep learning toolkits• Theano hjp://deeplearning.net/souware/theano• Torch hjp://www.torch.ch/• Tensorﬂow hjp://www.tensorﬂow.org/• Keras hjp://keras.io/➡ Pre-trained word vectors and codes• Word2vec toolkit and vectors
hjps://code.google.com/p/word2vec/• GloVe code and vectors
hjp://nlp.stanford.edu/projects/glove/• Hellinger PCA
hjps://github.com/rlebret/hpca• Online word vector evalua6on
hjp://wordvectors.org/
Nikolaos Pappas

86 /59