Towards Language Agnostic Universal Representations

When a bilingual student learns to solve word problems in math , we expect the student to be able to solve these problem in both languages the student is fluent in , even if the math lessons were only taught in one language . However , current representations in machine learning are language dependent . In this work , we present a method to decouple the language from the problem by learning language agnostic representations and therefore allowing training a model in one language and applying to a different one in a zero shot fashion . We learn these representations by taking inspiration from linguistics , specifically the Universal Grammar hypothesis and learn universal latent representations that are language agnostic ( Chomsky , 2014 ; Montague , 1970 ) . We demonstrate the capabilities of these representations by showing that models trained on a single language using language agnostic representations achieve very similar accuracies in other languages . 1 INTRODUCTION . Anecdotally speaking , fluent bilingual speakers rarely face trouble translating a task learned in one language to another . For example , a bilingual speaker who is taught a math problem in English will trivially generalize to other known languages . Furthermore there is a large collection of evidence in linguistics arguing that although separate lexicons exist in multilingual speakers the core representations of concepts and theories are shared in memory ( Altarriba , 1992 ; Mitchel , 2005 ; Bentin et al. , 1985 ) . The fundamental question we ’ re interested in answering is on the learnability of these shared representations within a statistical framework . We approached this problem from a linguistics perspective . Languages have vastly varying syntactic features and rules . Linguistic Relativity studies the impact of these syntactic variations on the formations of concepts and theories ( Au , 1983 ) . Within this framework of study , the two schools of thoughts are linguistic determinism and weak linguistic influence . Linguistic determinism argues that language entirely forms the range of cognitive processes , including the creation of various concepts , but is generally agreed to be false ( Hoijer , 1954 ; Au , 1983 ) . Although there exists some weak linguistic influence , it is by no means fundamental ( Ahearn , 2016 ) . The superfluous nature of syntactic variations across languages brings forward the argument of principles and parameters ( PnP ) which hypothesizes the existence of a small distributed parameter representation that captures the syntactic variance between languages denoted by parameters ( e.g . head-first or head-final syntax ) , as well as common principles shared across all languages ( Culicover , 1997 ) . Universal Grammar ( UG ) is the study of principles and the parameters that are universal across languages ( Montague , 1970 ) . The ability to learn these universalities would allow us to learn representations of language that are fundamentally agnostic of the specific language itself . Doing so would allow us to learn a task in one language and reap the benefits of all other languages without needing multilingual datasets . We take inspiration from the UG hypothesis and learn latent representations that are language agnostic which allow us to solve downstream problems in new languages without the need of any language-specific training data . We do not make any claims about the Universal Grammar hypothesis , but simply take inspiration from it . 2 RELATED WORK . Our work attempts to unite universal ( task agnostic ) representations with multilingual ( language agnostic ) representations ( Peters et al. , 2018 ; McCann et al. , 2017 ) . The recent trend in universal representations has been moving away from context-less unsupervised word embeddings to contextrich representations . Deep contextualized word representations ( ELMo ) trains an unsupervised language model on a large corpus of data and applies it to a large set of auxiliary tasks ( Peters et al. , 2018 ) . These unsupervised representations boosted the performance of models on a wide array of tasks . Along the same lines McCann et al . ( 2017 ) showed the power of using latent representations of translation models as features across other non-translation tasks . In general , initializing models with pre-trained language models shows promise against the standard initialization with word embeddings . Even further , Radford et al . ( 2017 ) show that an unsupervised language model trained on a large corpus will contain a neuron that strongly correlates with sentiment without ever training on a sentiment task implying that unsupervised language models maybe picking up informative and structured signals . In the field of multilingual representations , a fair bit of work has been done on multilingual word embeddings . Ammar et al . ( 2016 ) explored the possibility of training massive amounts of word embeddings utilizing either parallel data or bilingual dictionaries via the SkipGram paradigm . Later on an unsupervised approach to multilingual word representations was proposed by Chen & Cardie ( 2018 ) which utilized an adversarial training regimen to place word embeddings into a shared latent space . Although word embeddings show great utility , they fall behind methods which exploit sentence structure as well as words . Less work has been done on multilingual sentence representations . Most notably both Schwenk & Douze ( 2017 ) and Artetxe et al . ( 2017 ) propose a way to learn multilingual sentence representation through a translation task . We train downstream models using language agnostic universal representations on a set of tasks and show the ability for the downstream models to generalize to languages that we did not train on . 3 OPTIMIZATION PROBLEM . Statistical language models approximate the probability distribution of a series of words by predicting the next word given a sequence of previous words . p ( w0 , ... , wn ) = n∏ i=1 p ( wi | w0 , ... , wi−1 ) where wi are indices representing words in an arbitrary vocabulary . Learning grammar is equivalent to language modeling , as the support of p will represent the set of all grammatically correct sentences . Furthermore , let jα represent a particular language . Let pjα ( · ) represent the language model for the jαth language and wjα represents a word from the jα th language . Let kjα represent a distributed representation of a specific language along the lines of the PnP argument ( Culicover , 1997 ) . UG , through the lens of statistical language modeling , hypothesizes the existence of a factorization of pjα ( · ) containing a language agnostic segment . The factorization used throughout this paper is the following ( ◦ denotes function composition ) : b = u ◦ ejα ( w jα 0 , ... , w jα i ) ( 1 ) pjα ( wi | w0 , ... , wi−1 ) = e−1jα ◦ h ( b , kjα ) ( 2 ) s.t . d ( p ( b | jα ) || p ( b | jβ ) ) ≤ ( 3 ) The distribution matching constraint d , insures that the representations across languages are common as hypothesized by the UG argument . Function ejα : Ni → Ri×d is a language specific function which takes an ordered set of integers representing tokens ( token meaning wjαi ) and outputs a vector of size d per token . Function u : Ri×d → Ri×d takes the language specific representation and attempts to embed into a language agnostic representation . Function h : ( Ri×d , Rf ) → Ri×d takes the universal representation as well as a distributed representation of the language of size f and returns a language specific decoded representation . e−1jα maps our decoded representation back to the token space . For the purposes of distribution matching we utilize the GAN framework . Following recent successes we use Wasserstein-1 as our distance function d ( Arjovsky et al. , 2017 ) . Given two languages jα and jβ the distribution of the universal representations should be within with respect to the W1 of each other . Using the Kantarovich-Rubenstein duality ( Arjovsky et al. , 2017 ; Villani , 2008 ) we define d ( p ( b | jα ) || p ( b | jβ ) ) = sup ||fα , β ||L≤1 Ex∼p ( b|jα ) [ fα , β ( x ) ] − Ex∼p ( b|jβ ) [ fα , β ( x ) ] ( 4 ) where L is the Lipschitz constant of f . Throughout this paper we satisfy the Lipschitz constraint by clamping the parameters to a compact space , as done in the original WGAN paper ( Arjovsky et al. , 2017 ) . Therefore the complete loss function for m languages each containing N documents becomes : max θ m∑ α=1 N∑ i=0 log pjα ( w jα i,0 , ... , w jα i , n ; θ ) − λ m2 m∑ α=1 m∑ β=1 d ( p ( b | jα ) || p ( b | jβ ) ) λ is a scaling factor for the distribution constraint loss . 4 UG-WGAN . Our specific implementation of our factorization and optimization problem we denote as UGWGAN . 4.1 ARCHITECTURE . Each function described in the previous section we implement using neural networks . For ejα in equation 1 we use a language specific embedding table followed by a LSTM ( Hochreiter & Schmidhuber , 1997 ) . Function u in equation 1 is simply stacked LSTM ’ s . Function h in equation 2 takes input from u as well as a PnP representation of the language via an embedding table . Calculating the real inverse of e−1jα is non trivial therefore we use another language specific LSTM whose outputs we multiply by the transpose of the embedding table of ejα to obtain token probabilities . For regularization we utilized standard dropout after the embedding layers and layer-wise locked dropout after each LSTM ’ s layer ( Srivastava et al. , 2014 ; Gal & Ghahramani , 2016 ) . The critic , adopting the terminology from Arjovsky et al . ( 2017 ) , takes the input from u , feeds it through a stacked LSTM , aggregates the hidden states using linear sequence attention as described in DrQA ( Chen et al. , 2017 ) . Once we have the aggregated state we map to a m ×m matrix from where we can compute the total Wasserstein loss . A Batch Normalization layer is appended to the end of the critic ( Ioffe & Szegedy , 2015 ) . The α , βth index in the matrix correspond to the function output of f in calculating W1 ( p ( b | jα ) || p ( b | jβ ) ) . 4.2 TRAINING . We trained UG-WGAN with a variety of languages depending on the downstream task . For each language we utilized the respective Wikipedia dump . From the wikipedia dump we extract all pages using the wiki2text1 utility and build language specific vocabularies consisting of 16k BPE tokens ( Sennrich et al. , 2015 ) . During each batch we uniform sample random documents from our set of languages which are approximately the same length , therefore a batch will be mixed with respect to language . We train our language model via BPTT where the truncation length progressively grows from 15 to 50 throughout training . The critic is updated 10 times for every update of the language model . We trained each language model for 14 days on a NVidia Titan X . For each language model we would do a sweep over λ , but in general we have found that λ = 0.1 works sufficiently well for minimizing both perplexity and Wasserstein distance .

Summary: This paper starts with the bold aim of extracting Montague's universal grammar from multiple languages. In order to do so, the authors train multiple language models where each LM is explicitly factorized into language-specific and language-independent representations. The authors then apply the GAN framework to the language-independent parts to enforce all languages to share the same latent space. The claim here is that the language independent parameters capture the essence of universal grammar. The authors show that their framework enables effective zero-shot learning of tasks over new languages (for example sentiment classifier learned on top of English data generalize to Chinese when trained on the universal grammar embeddings).