Towards Language Agnostic Universal Representations

When a bilingual student learns to solve word problems in math , we expect the student to be able to solve these problem in both languages the student is fluent in , even if the math lessons were only taught in one language . However , current representations in machine learning are language dependent . In this work , we present a method to decouple the language from the problem by learning language agnostic representations and therefore allowing training a model in one language and applying to a different one in a zero shot fashion . We learn these representations by taking inspiration from linguistics , specifically the Universal Grammar hypothesis and learn universal latent representations that are language agnostic ( Chomsky , 2014 ; Montague , 1970 ) . We demonstrate the capabilities of these representations by showing that models trained on a single language using language agnostic representations achieve very similar accuracies in other languages . 1 INTRODUCTION . Anecdotally speaking , fluent bilingual speakers rarely face trouble translating a task learned in one language to another . For example , a bilingual speaker who is taught a math problem in English will trivially generalize to other known languages . Furthermore there is a large collection of evidence in linguistics arguing that although separate lexicons exist in multilingual speakers the core representations of concepts and theories are shared in memory ( Altarriba , 1992 ; Mitchel , 2005 ; Bentin et al. , 1985 ) . The fundamental question we ’ re interested in answering is on the learnability of these shared representations within a statistical framework . We approached this problem from a linguistics perspective . Languages have vastly varying syntactic features and rules . Linguistic Relativity studies the impact of these syntactic variations on the formations of concepts and theories ( Au , 1983 ) . Within this framework of study , the two schools of thoughts are linguistic determinism and weak linguistic influence . Linguistic determinism argues that language entirely forms the range of cognitive processes , including the creation of various concepts , but is generally agreed to be false ( Hoijer , 1954 ; Au , 1983 ) . Although there exists some weak linguistic influence , it is by no means fundamental ( Ahearn , 2016 ) . The superfluous nature of syntactic variations across languages brings forward the argument of principles and parameters ( PnP ) which hypothesizes the existence of a small distributed parameter representation that captures the syntactic variance between languages denoted by parameters ( e.g . head-first or head-final syntax ) , as well as common principles shared across all languages ( Culicover , 1997 ) . Universal Grammar ( UG ) is the study of principles and the parameters that are universal across languages ( Montague , 1970 ) . The ability to learn these universalities would allow us to learn representations of language that are fundamentally agnostic of the specific language itself . Doing so would allow us to learn a task in one language and reap the benefits of all other languages without needing multilingual datasets . We take inspiration from the UG hypothesis and learn latent representations that are language agnostic which allow us to solve downstream problems in new languages without the need of any language-specific training data . We do not make any claims about the Universal Grammar hypothesis , but simply take inspiration from it . 2 RELATED WORK . Our work attempts to unite universal ( task agnostic ) representations with multilingual ( language agnostic ) representations ( Peters et al. , 2018 ; McCann et al. , 2017 ) . The recent trend in universal representations has been moving away from context-less unsupervised word embeddings to contextrich representations . Deep contextualized word representations ( ELMo ) trains an unsupervised language model on a large corpus of data and applies it to a large set of auxiliary tasks ( Peters et al. , 2018 ) . These unsupervised representations boosted the performance of models on a wide array of tasks . Along the same lines McCann et al . ( 2017 ) showed the power of using latent representations of translation models as features across other non-translation tasks . In general , initializing models with pre-trained language models shows promise against the standard initialization with word embeddings . Even further , Radford et al . ( 2017 ) show that an unsupervised language model trained on a large corpus will contain a neuron that strongly correlates with sentiment without ever training on a sentiment task implying that unsupervised language models maybe picking up informative and structured signals . In the field of multilingual representations , a fair bit of work has been done on multilingual word embeddings . Ammar et al . ( 2016 ) explored the possibility of training massive amounts of word embeddings utilizing either parallel data or bilingual dictionaries via the SkipGram paradigm . Later on an unsupervised approach to multilingual word representations was proposed by Chen & Cardie ( 2018 ) which utilized an adversarial training regimen to place word embeddings into a shared latent space . Although word embeddings show great utility , they fall behind methods which exploit sentence structure as well as words . Less work has been done on multilingual sentence representations . Most notably both Schwenk & Douze ( 2017 ) and Artetxe et al . ( 2017 ) propose a way to learn multilingual sentence representation through a translation task . We train downstream models using language agnostic universal representations on a set of tasks and show the ability for the downstream models to generalize to languages that we did not train on . 3 OPTIMIZATION PROBLEM . Statistical language models approximate the probability distribution of a series of words by predicting the next word given a sequence of previous words . p ( w0 , ... , wn ) = n∏ i=1 p ( wi | w0 , ... , wi−1 ) where wi are indices representing words in an arbitrary vocabulary . Learning grammar is equivalent to language modeling , as the support of p will represent the set of all grammatically correct sentences . Furthermore , let jα represent a particular language . Let pjα ( · ) represent the language model for the jαth language and wjα represents a word from the jα th language . Let kjα represent a distributed representation of a specific language along the lines of the PnP argument ( Culicover , 1997 ) . UG , through the lens of statistical language modeling , hypothesizes the existence of a factorization of pjα ( · ) containing a language agnostic segment . The factorization used throughout this paper is the following ( ◦ denotes function composition ) : b = u ◦ ejα ( w jα 0 , ... , w jα i ) ( 1 ) pjα ( wi | w0 , ... , wi−1 ) = e−1jα ◦ h ( b , kjα ) ( 2 ) s.t . d ( p ( b | jα ) || p ( b | jβ ) ) ≤ ( 3 ) The distribution matching constraint d , insures that the representations across languages are common as hypothesized by the UG argument . Function ejα : Ni → Ri×d is a language specific function which takes an ordered set of integers representing tokens ( token meaning wjαi ) and outputs a vector of size d per token . Function u : Ri×d → Ri×d takes the language specific representation and attempts to embed into a language agnostic representation . Function h : ( Ri×d , Rf ) → Ri×d takes the universal representation as well as a distributed representation of the language of size f and returns a language specific decoded representation . e−1jα maps our decoded representation back to the token space . For the purposes of distribution matching we utilize the GAN framework . Following recent successes we use Wasserstein-1 as our distance function d ( Arjovsky et al. , 2017 ) . Given two languages jα and jβ the distribution of the universal representations should be within with respect to the W1 of each other . Using the Kantarovich-Rubenstein duality ( Arjovsky et al. , 2017 ; Villani , 2008 ) we define d ( p ( b | jα ) || p ( b | jβ ) ) = sup ||fα , β ||L≤1 Ex∼p ( b|jα ) [ fα , β ( x ) ] − Ex∼p ( b|jβ ) [ fα , β ( x ) ] ( 4 ) where L is the Lipschitz constant of f . Throughout this paper we satisfy the Lipschitz constraint by clamping the parameters to a compact space , as done in the original WGAN paper ( Arjovsky et al. , 2017 ) . Therefore the complete loss function for m languages each containing N documents becomes : max θ m∑ α=1 N∑ i=0 log pjα ( w jα i,0 , ... , w jα i , n ; θ ) − λ m2 m∑ α=1 m∑ β=1 d ( p ( b | jα ) || p ( b | jβ ) ) λ is a scaling factor for the distribution constraint loss . 4 UG-WGAN . Our specific implementation of our factorization and optimization problem we denote as UGWGAN . 4.1 ARCHITECTURE . Each function described in the previous section we implement using neural networks . For ejα in equation 1 we use a language specific embedding table followed by a LSTM ( Hochreiter & Schmidhuber , 1997 ) . Function u in equation 1 is simply stacked LSTM ’ s . Function h in equation 2 takes input from u as well as a PnP representation of the language via an embedding table . Calculating the real inverse of e−1jα is non trivial therefore we use another language specific LSTM whose outputs we multiply by the transpose of the embedding table of ejα to obtain token probabilities . For regularization we utilized standard dropout after the embedding layers and layer-wise locked dropout after each LSTM ’ s layer ( Srivastava et al. , 2014 ; Gal & Ghahramani , 2016 ) . The critic , adopting the terminology from Arjovsky et al . ( 2017 ) , takes the input from u , feeds it through a stacked LSTM , aggregates the hidden states using linear sequence attention as described in DrQA ( Chen et al. , 2017 ) . Once we have the aggregated state we map to a m ×m matrix from where we can compute the total Wasserstein loss . A Batch Normalization layer is appended to the end of the critic ( Ioffe & Szegedy , 2015 ) . The α , βth index in the matrix correspond to the function output of f in calculating W1 ( p ( b | jα ) || p ( b | jβ ) ) . 4.2 TRAINING . We trained UG-WGAN with a variety of languages depending on the downstream task . For each language we utilized the respective Wikipedia dump . From the wikipedia dump we extract all pages using the wiki2text1 utility and build language specific vocabularies consisting of 16k BPE tokens ( Sennrich et al. , 2015 ) . During each batch we uniform sample random documents from our set of languages which are approximately the same length , therefore a batch will be mixed with respect to language . We train our language model via BPTT where the truncation length progressively grows from 15 to 50 throughout training . The critic is updated 10 times for every update of the language model . We trained each language model for 14 days on a NVidia Titan X . For each language model we would do a sweep over λ , but in general we have found that λ = 0.1 works sufficiently well for minimizing both perplexity and Wasserstein distance .

Summary: This paper introduced a GAN-based method to learn language universal representations without parallel data. The model architecture is analogous to an autoencoder. The encoder is a compound of language-universal mapper plus a language-specific LSTM. For decoding, another language-universal module first map language-universal representation back to language-specific embedding space, then another LSTM decoder generates the original sentence. The authors used GAN to encourage intermediate representation to be language-universal. The authors tested the proposed method on zero-shot semantic analysis and NLI tasks and showed nice results.