tributionalism. Based upon those aspects, we advanced that distributionalism
is less about co-occurrence of words in context than about *simultaneous and
articulated discrimination* between terms and contexts. The figure of *analogy*
corresponding to it provided then a possible alternative to that of co-occurence
to qualify that mechanism from the perspective of individual speakers. However,
drawing from the formal principles governing the mechanisms of MMs, and in
particular, from the dual spaces corresponding to row and column vectors of
matrices, we proposed a formal characterization of that articulated discrimi-
nation by resorting to the notion of *(bi-)duality*: terms and contexts stand in
relations of duality in such a way that the relatedness between two terms is
established through bi-duality, that is, through the duality the first maintains
with the dual contexts of the second. We also suggested that such a mechanism
could be pushed further by considering the duality between a term and the type
of its dual contexts established by bi-duality.

As a consequence of this characterization, the notion of context appeared as
a strictly *formal* one, less related to the cognitive capacities of individuals than
to an internal dimension of the organization of language itself. The conceptual
framework for the analysis of language is thus freed from the figure of individ-
ual subjects since language is less about subjective use than about relatively
autonomous regularities in the global organization of language before which in-
dividuals are left with the possibility of a strategy. This provides a plausible
explanation of why word embedding models are capable of deriving significant
linguistic content without the need for any more human intervention than the
one implied in raw linguist corpora.

To address the other two initial questions of our inquiry, we noted that if
the joint derivation of semantic and syntactic properties might seem surprising,
that should not be attributed to a new capacity of NN models but to a neglected
possibility within MMs due to the mainly semantic orientation governing their
development. We identified *three epistemological obstacles associated with that
semantic orientation*, namely the *almost complete exclusion of syntax* from the
original interests of MMs, the *privilege granted to words as fundamental units
of linguistic analysis* and the *disregard for possible underlying structures of lan-
guage* in favor of an organization of the linguistic space in terms of similarities.
We showed then how the first two aspects were intimately connected: accounting
for syntactic content in the framework of MMs requires to consider terms and
contexts at sub- and supra-lexical levels, and conversely, introducing the latter
naturally induces the establishment of syntactic regularities. We suggested then
that, if NN word embeddings were able to grasp semantic and syntactic contents
as the result of one and the same procedure, the reason might reside in their
capacity of taking advantage of this neglected possibility of MMs.

The abandon of the lexical privilege and the corresponding consideration of
syntactic content were for us the principal hint to understand why the embed-
ding space can be informed by an emergent structure. To follow that lead, we
entered into “structuralist soil” by examining Harris’s original distributional-
ism. In accordance with our interpretation, Harris’s formulations show that,
given that the very sense of a distributional description is to free the scientific
account of language from the external viewpoint of semantics, the link between
the analysis of linguistic distribution and the derivation of structural features
is necessary. Indeed, outside a semantic approach, lexicality has no privilege
for distributional analysis, which takes place essentially at the phonological and
morphological levels. Unconcerned with semantics and standing on phonologi-
cal or morphological features, *distribution is necessarily structural distribution*.
Significantly, we found that, despite a natural lack of technical complexity, the
key analytic instruments of Harris's distributionalism are highly convergent with
the mechanisms of modern MMs, namely a term-context matrix with dimen-
sionality reduction techniques. However, we saw that the latent space derived
through those techniques provides a complete series of formal units structurally
related to each other rather than a set of dimensions to measure approximate
similarity. Harris's theory was not free of difficulties, nonetheless. Indeed, by
making such a clear-cut distinction between semantics and distributional fea-
tures, Harris precluded the possibility of conceiving a direct relation between
those structured formal units and any semantic content. This is what led us
to conclude our inquiry by assessing the structuralist theoretical background of
Harris's distributionalism.

We found the answer to that last open issue in an idea stemming from the
singular approach to language of European structuralism (i.e. that of Saussure
and Hjelmslev), which we called “the structuralist hypothesis”, namely the idea
that *meaning is the effect of structure*. Such hypothesis is based on the fact
that linguistic structure is not the result of the composition of units but of
unmotivated *decomposition or segmentation* of an originally shapeless material
continuum. Such segmentation can only be done through the intervention of a
second heterogeneous continuum operating a discriminating action on the first
one, while receiving from the latter a comparable action in return. Structure is
then always double, since it is the result of a process of simultaneous segmenta-
tion between two heterogeneous systems or planes, of which one can be thought
of as the content or meaning of the other. This is why the analysis of segmen-
tation of one plane—which is what distributional analysis in fact achieves—can
capture meaningful features belonging to the other one, carved, as it were, in
the form of the former. Moreover, an overview of Hjelmslev’s development of
Saussure’s program suggested the way in which that structuralist conception of
language could be related to the basic formal principles of the contemporary
models under study.

From word2vec's word contexts and implicit matrix factorization to Saus-
sure's syntagmatic and paradigmatic relations and Hjelmslev's categories, through
MMs, SVD and Harris's element-environment charts, we have found the action
of one and the same mechanism underlying the capacity of linguistic analysis
to grasp significant features of language: the *duality between terms and con-
texts*, and a principle of *equivalence between terms and between contexts related
through bi-duality*. Interestingly, that mechanism tends to characterize bottom-
up approaches to language, that is, approaches in which the organization of
language is not supposed to be given from without, but derived from within the
practice of language itself, as an emergent structure. NN vector representations
(whether it is of words, characters or contexts) appear then only as the latest
tool capturing the effects of that mechanism of duality and bi-duality in lan-
guage. Compared to previous perspectives of the same nature, its importance
lies in that, with almost minimal presupposed structure (let alone computational
costs), it can account for those effects as something more than mere semantic
similarity satisfying most needs of MMs; but also, in that it can actually pro-
vide a surprising amount of semantic content, the derivation of which classic
structuralist theories could, at best, only conjecture.
As we have suggested, that basic mechanism of (bi-)duality is to be at-
tributed to language itself rather than to the linguistic models studying it. This
was a common conception among the authors we have studied, from Landauer
affirming that the model “is the real-thing” (Landauer et al., 2007, p. 8), to
Harris stating that “the position of the speakers is after all similar to that of
the linguist” (Harris, 1970, p. 779), to Saussure, for whom, the units of lan-
guage “are not given from the outside, language must draw them from within
itself” (Maniglier, 2006, quoted in p. 207, our translation). This is why we
need to think that embeddings not only provide a successful technique, but also
convey a whole image of language, or in other terms a general but elementary
conception of what language is, of what it means to speak and write.

Animated by (bi-)duality as a fundamental principle governing the linguis-
tic practice of speakers, language appears more like a *game* than like a tool.
(Bi-)duality and the regularities attached to it are in this sense like the rules
of chess or of Go, which are independent of the will of individual players, who
cease to be players of those games as soon as they stop submitting to those
rules. Rather than an instrumental use, subjects develop a *strategy within lan-
guage*, which far from violating its regularities, reinforces them in a way models
like word embeddings and MMs in general can efficiently identify. The elements
of that game are not to be confused with the words as elementary material
units whose identity is determined by their referential meanings, since both the
identity and the meaning of words are the result of the fundamental duality be-
tween terms and contexts, as two internal dimensions of linguistic *formal units*.
Yet units are not necessarily lexical, and against a purely semantic approach of
meaning, we were confronted with only a difference of degree between the syn-
tactic and the semantic levels. Finally, as units are not given from the outset,
linguistic practice and the production of meaning is less the result of a com-
position of words than the effect of the simultaneous *segmentation* of pairs of
material continuums into formal units of different levels. From this perspective,
if the linguistic subject computes, it is a strategy of segmentation rather than
a compositional rule that it computes.

(Bi)-duality rather than co-occurrence, game rather than tool, strategy rather
than use, form rather than substance, segmentation rather than compositional-
ity. This is the image of language that we have been able to draw behind word
embeddings by tracing back the meaning of its underlying mechanisms. That
image contrasts with many of the principles generally orienting the present de-
velopment of the very field that begets it. Our hope is that, by bringing that
underlying image to the surface and making explicit its connections with the
tradition from which it stems, we will contribute to providing novel orientations
to the field, in adequacy with the nature of its latest advances. In particular,
the image of language we have derived can allow us to relativize the predom-
inantly semantic perspective of the field in favor of the derivation of complex
interconnected structures at different linguistic levels. In this sense, one of the
clearest hints offered by our inquiry is that a segmentation procedure should
be an integral part of the process by which structural features are derived,
and more precisely, of the construction of the representation of the elementary
units of language. Since, as we have seen, structural features are intimately
connected with the segmentation of units at all levels, the consideration of seg-
mentation for the construction of vector representations should help capturing
the structural features those vectors will be expected to represent. By pre-
supposing the segmentation of words and discarding other possible levels of
segmentation, classical word embedding models rely only on partial information
for the reconstruction of linguistic structure. Incidentally, that presupposition
also relativizes the unsupervised character of the learning for, as insignificant as
it may seem, word segmentation already involves a great amount of linguistic
content (that is precisely the content word embeddings are able to grasp). By
including segmentation in the analytic procedure, one would be contributing to
achieving a truly unsupervised model, while increasing its capabilities at the
same time, since the structure of the segmentation bears a direct relation with
the linguistic structures the model is intended to capture. The recent orienta-
tions of the field, obtaining remarkable results with character and context-based
embeddings, tend to confirm this idea.

Yet the consequences of reconstructing the alternative image of language
implied in the new NN language technologies can transcend the limits of the NLP
field, and attain other regions of knowledge and research, such as social sciences,
epistemology and even logic or computer science, in which the notion of language
occupies a crucial position. With respect to those fields, the emergence of a new
image of language can provide the opportunity of significant conceptual shifts
which could be, moreover, accompanied by a fruitful transfer of methodologies,
introducing in those fields analytical tools from NLP and enriching the latter
with the specific treatment language receives in each of the former. In this
respect, we need to recognize a major stake in the possibility, provided by that
new image, of understanding language and meaning independently of purely
individual practices, and yet not resorting to absolute principles valid without
restriction that would make the analysis of actual practices inconsequential.
At equal distance from those two positions, the image of language we have
reconstructed in the previous pages allows us to envisage language as a collective
playground, as a reservoir where the significant distinctions resulting from a
collective construction of signs are deposited as the most intimate treasure of a
culture—to borrow a Saussurean metaphor. It is, after all, the image of those
cultures that models like word embeddings give us the means to depict.

References

Marco Baroni and Alessandro Lenci. Distributional memory: A general framework for corpus-based semantics. *Computational Linguistics*, 36(4):673–721, 2010. doi: 10.1162/coli_a_00016.

Marco Baroni, Georgiana Dinu, and Germán Kruszewski. Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In *Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 238–247. Association for Computational Linguistics, 2014. doi: 10.3115/v1/P14-1023.

Y. Bengio. Neural net language models. *Scholarpedia*, 3(1):3881, 2008. doi: 10.4249/scholarpedia.3881. revision #140963.

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. *J. Mach. Learn. Res.*, 3:1137–1155,
March 2003. ISSN 1532-4435. URL http://dl.acm.org/citation.cfm?id=944919.944966.

David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent Dirichlet Allocation. *J. Mach. Learn. Res.*, 3:993–1022, March 2003. ISSN 1532-4435.

J. Blitzer, K.Q. Weinberger, L. K. Saul, and F. C. N. Pereira. Hierarchical distributed representations for statistical language modeling. In *Advances in Neural and Information Processing Systems*, volume 17, Cambridge, MA, 2005. MIT Press.

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomáš Mikolov. Enriching word vectors with subword information. *CoRR*, abs/1607.04606, 2016. URL http://arxiv.org/abs/1607.04606.

John A. Bullinaria and Joseph P. Levy. Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and svd. *Behavior Research Methods*, 44(3):890–907, Sep 2012. ISSN 1554-3528. doi: 10.3758/s13428-011-0183-8. URL https://doi.org/10.3758/s13428-011-0183-8.

Joan L. Bybee and Paul J. Hopper, editors. *Frequency and the emergence of linguistic structure*. Number 45 in Typological studies in language. Benjamins, Amsterdam, 2001. ISBN 978-90-272-2947-2 978-1-58811-027-5 978-1-58811-028-2 978-90-272-2948-9. OCLC: 216471429.

Rudolf Carnap. *Logical syntax of language*. Routledge, London, 2001. ISBN 978-0-415-22553-3. OCLC: 916122384.

John Caron. Computational information retrieval. chapter Experiments with LSA Scoring: Optimal Rank and Basis, pages 157–169. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2001. ISBN 0-89871-500-8.

Nick Chater, Alexander Clark, John A. Goldsmith, and Amy Perfors. *Empiricism and language learnability*. Oxford University Press, Oxford, United Kingdom, first edition edition, 2015. ISBN 978-0-19-873426-0. OCLC: ocn907131354.

Juntian Chen, Yubo Tao, and Hai Lin. Visual exploration and comparison of word embeddings. *Journal of Visual Languages and Computing*, 48:178–186, 2018. ISSN 1045-926X. doi: https://doi.org/10.1016/j.jvlc.2018.08.008.

Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In *Proceedings of the 25th International Conference on Machine Learning*, ICML '08, pages 160–167, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-205-4. doi: 10.1145/1390156.1390177. URL http://doi.acm.org/10.1145/1390156.1390177.

G. E. Dahl, D. Yu, L. Deng, and A. Acero. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. *IEEE Transactions on Audio, Speech, and Language Processing*, 20(1):30–42, Jan 2012. ISSN 1558-7916. doi: 10.1109/TASL.2011.2134090.
Scott C. Deerwester, Susan T. Dumais, Thomas K. Landauer, George W. Fur-
nas, and Richard A. Harshman. Indexing by Latent Semantic Analysis. JA-
SIS, 41(6):391–407, 1990.

Gilles Deleuze. *Difference and repetition*. Columbia University Press, New York, 1994. ISBN 0-231-08158-8.

Simon Dennis, Tom Landauer, Walter Kintsch, and Jose Quesada. Introduction to Latent Semantic Analysis. Slides from the tutorial given at the 25th annual meeting of the cognitive science society. Boston, 2003.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805.

J. L. Elman. Finding structure in time. *Cognitive Science*, 14:213–252, 1990.

Jeffrey L. Elman, editor. *Rethinking innateness: a connectionist perspective on development*. Neural network modeling and connectionism. MIT Press, Cambridge, Mass, 1996. ISBN 978-0-262-05052-4.

John Rupert Firth. A synopsis of linguistic theory 1930–1955. In *Studies in Linguistic Analysis*, pages 1–32. Blackwell, Oxford, 1957.

Gottlob Frege. Function and concept. In Brian McGuinness, editor, *Collected papers on mathematics, logic, and philosophy*, pages 137–156. Basil Blackwell, 1984.

Jila Ghomeshi, Ray Jackendoff, Nicole Rosen, and Kevin Russell. Contrastive focus reduplication in english (the salad-salad paper). *Natural Language & Linguistic Theory*, 22(2):307–357, May 2004. ISSN 1573-0859. doi: 10.1023/B:NALA.0000015789.98638.f9. URL https://doi.org/10.1023/B:NALA.0000015789.98638.f9.

Jean-Yves Girard. Locus solum: from the rules of logic to the logic of rules. *Mathematical Structures in Computer Science*, 11(3):301–506, 2001.

Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. Analogy-based de-
tection of morphological and semantic relations with word embeddings: what
works and what doesn't. In *SRW@HLT-NAACL*, pages 8–15. The Association
for Computational Linguistics, 2016.

Arthur M. Glenberg and Sarita Mehta. Constraint on covariation: It's not
meaning. *Special issue of the Italian Journal of Linguistics*, 1(20):241–264,
2008.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. *Deep Learning*. MIT Press, Cambridge MA, London UK, 2016.

Bob Hale and Crispin Wright. *A Companion to the Philosophy of Language*. Blackwell Publishers, Oxford, 1997. ISBN 0-631-16757-9.

William L. Hamilton, Jure Leskovec, and Dan Jurafsky. Diachronic word em-
beddings reveal statistical laws of semantic change. CoRR, abs/1605.09096,
2016. URL http://arxiv.org/abs/1605.09096.
Zellig Harris. *Structural linguistics*. University of Chicago Press, Chicago, 1960.
ISBN 0226317714 0226217714.

Zellig Harris. Distributional structure. In *Papers in Structural and Transfor-
mational Linguistics*, pages 775–794. Springer, Dordrecht, 1970.

Alain Herreman. Analyser l'analyse, décrire la description. texto! Textes & Cultures, 16(2), 2011. URL http://www.revue-texto.net/index.php?id=2875.

John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 4129–4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1419.

Geoffrey E. Hinton. Learning distributed representations of concepts. In *Proceedings of the Eighth Annual Conference of the Cognitive Science Society*, pages 1–12. Hillsdale, NJ: Erlbaum, 1986.

Louis Hjelmslev. *La catégorie des cas*. Wilhelm Fink Verlag, Munchen, 1935.

Louis Hjelmslev. *Prolegomena to a Theory of Language*. Wawerly Press, Baltimore, 1953.

Louis Hjelmslev. *Résumé of a Theory of Language*. Number 16 in Travaux du Cercle linguistique de Copenhague. Nordisk Sprog-og Kulturforlag, Copenhagen, 1975. ISBN 0-299-07040-9.

Louis Hjelmslev. *Système linguistique et changement linguistique*. Classiques Garnier, Paris, November 2016. ISBN 978-2-406-06468-8. doi: 10.15122/isbn.9782406064701.

Thomas Hofmann. Probabilistic Latent Semantic Indexing. In *Proceedings of the 22Nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval*, SIGIR '99, pages 50–57, New York, NY, USA, 1999. ACM. ISBN 1-58113-096-1. doi: 10.1145/312624.312649. URL http://doi.acm.org/10.1145/312624.312649.

Thomas Hofmann. Unsupervised learning by probabilistic Latent Semantic Analysis. *Mach. Learn.*, 42(1-2):177–196, January 2001. ISSN 0885-6125. doi: 10.1023/A:1007617005950. URL https://doi.org/10.1023/A:1007617005950.

Jeremy Howard and Sebastian Ruder. Fine-tuned language models for text classification. *CoRR*, abs/1801.06146, 2018. URL http://arxiv.org/abs/1801.06146.

Google inc. word2vec, https://code.google.com/archive/p/word2vec/, 2013.
URL https://code.google.com/archive/p/word2vec/.

Navdeep Jaitly, Patrick Nguyen, Andrew Senior, and Vincent Vanhoucke. Application of pretrained deep neural networks to large vocabulary speech recognition. In *Proceedings of Interspeech 2012*, 2012.
Stefan Jansen. Word and phrase translation with word2vec. CoRR, abs/1705.03127, 2017. URL http://arxiv.org/abs/1705.03127.

Dan Jurafsky and James H. Martin. *Speech and Language Processing*. Prentice Hall, Upper Saddle River, 2 edition, 2008.

Jean-Louis Krivine. Typed lambda-calculus in classical Zermelo-Fraenkel set theory. *Archive for Mathematical Logic*, 40(3):189-205, 2001.

Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. Statistically significant detection of linguistic change. CoRR, abs/1411.3315, 2014. URL http://arxiv.org/abs/1411.3315.

T. K. Landauer, P. W. Foltz, and D. Laham. An Introduction to Latent Semantic Analysis. *Discourse Processes*, 25:259-284, 1998.

Thomas K Landauer. On the computational basis of learning and cognition: Arguments from LSA. volume 41 of *Psychology of Learning and Motivation*, pages 43-84. Academic Press, 2002. doi: https://doi.org/10.1016/S0079-7421(02)80004-4. URL http://www.sciencedirect.com/science/article/pii/S0079742102800044.

Thomas K. Landauer, Danielle S. McNamara, Simon Dennis, and Walter Kintsch, editors. *Handbook of Latent Semantic Analysis*. Lawrence Erlbaum Associates, Mahwah, New Jersey, USA, 2007.

Alessandro Lenci. Distributional semantics in linguistic and cognitive research. *From context to meaning: distributional models of the lexicon in linguistics and cognitive science*, Italian Journal of Linguistics, 1(20):1-31, 2008.

Alessandro Lenci. Distributional models of word meaning. *Annual Review of Linguistics*, 4(1):151-171, 2018. doi: 10.1146/annurev-linguistics-030514-125254. URL https://doi.org/10.1146/annurev-linguistics-030514-125254.

Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In *Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2*, NIPS'14, pages 2177-2185, Cambridge, MA, USA, 2014a. MIT Press.

Omer Levy and Yoav Goldberg. Linguistic regularities in sparse and explicit word representations. In *Proceedings of the Eighteenth Conference on Computational Natural Language Learning, CoNLL 2014*, Baltimore, Maryland, USA, June 26-27, 2014, pages 171-180, 2014b.

Omer Levy, Yoav Goldberg, and Ido Dagan. Improving distributional similarity with lessons learned from word embeddings. *TACL*, 3:211-225, 2015.

Jiwei Li, Xinlei Chen, Eduard H. Hovy, and Dan Jurafsky. Visualizing and understanding neural models in NLP. In *HLT-NAACL*, pages 681-691. The Association for Computational Linguistics, 2016.

Alvin M. Liberman. Some results of research on speech perception. *The Journal of the Acoustical Society of America*, 29(1):117-123, 1957. doi: 10.1121/1.1908635.
Tal Linzen. Issues in evaluating semantic spaces using word analogies. CoRR, abs/1606.07736, 2016. URL http://arxiv.org/abs/1606.07736.

S. Liu, P. Bremer, J. J. Thiagarajan, V. Srikumar, B. Wang, Y. Livnat, and V. Pascucci. Visual exploration of semantic relationships in neural word embeddings. *IEEE Transactions on Visualization and Computer Graphics*, 24 (1):553–562, Jan. 2018. ISSN 1077-2626. doi: 10.1109/TVCG.2017.2745141. URL doi.ieeecomputersociety.org/10.1109/TVCG.2017.2745141.

Kevin Lund and Curt Burgess. Producing high-dimensional semantic spaces from lexical co-occurrence. *Behavior Research Methods, Instruments, & Computers*, 28(2):203–208, Jun 1996. ISSN 1532-5970. doi: 10.3758/BF03204766. URL https://doi.org/10.3758/BF03204766.

Thang Luong, Hieu Pham, and Christopher D. Manning. Bilingual word representations with monolingual quality in mind. In *Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, VS@NAACL-HLT 2015, June 5, 2015, Denver, Colorado, USA*, pages 151–159, 2015.

Brian MacWhinney, editor. *The emergence of language*. Carnegie Mellon symposia on cognition. Lawrence Erlbaum Associates, Mahwah, NJ, 1999. ISBN 978-0-8058-3010-1 978-0-8058-3011-8.

Patrice Maniglier. *La vie énigmatique des signes*. Léo Scheer, Paris, 2006.

Patrice Maniglier. Milieux de Culture. In Waldir Beividas, Ivan Carlos Lopes, and Sémir Badir, editors, *Cem Anos com Saussure, Textos de Congresso Internacional*, volume 2. Annablume editora, Sao Paulo, 2016.

Christopher D. Manning and Hinrich Schütze. *Foundations of Statistical Natural Language Processing*. MIT Press, Cambridge, MA, USA, 1999. ISBN 0-262-13360-1.

Anthony M. McEnery and Anita Wilson. *Corpus Linguistics: An Introduction*. Edinburgh University Press, Edinburgh, 2001.

Tomáš Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781, 2013a. URL http://arxiv.org/abs/1301.3781.

Tomáš Mikolov, Quoc V. Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. CoRR, abs/1309.4168, 2013b. URL http://arxiv.org/abs/1309.4168.

Tomáš Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeff Dean, Quoc Le, and Thomas Strohmann. Learning representations of text using neural networks. NIPS deep learning workshop 2013 slides, 2013c.

Tomáš Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. CoRR, abs/1310.4546, 2013d. URL http://arxiv.org/abs/1310.4546.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In *Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 746–751. Association for Computational Linguistics, 2013e.

George A. Miller and Walter G. Charles. Contextual correlates of semantic similarity. *Language and Cognitive Processes*, 6(1):1–28, 1991. doi: 10.1080/01690969108406936.

Andriy Mnih and Geoffrey Hinton. Three new graphical models for statistical language modelling. In *Proceedings of the 24th International Conference on Machine Learning*, ICML ’07, pages 641–648, New York, NY, USA, 2007. ACM. ISBN 978-1-59593-793-3. doi: 10.1145/1273496.1273577. URL http://doi.acm.org/10.1145/1273496.1273577.

Charles W. Morris. *Foundations of the Theory of Signs*. The University of Chicago Press, Chicago, 1938.

Maximilian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. CoRR, abs/1705.08039, 2017. URL http://arxiv.org/abs/1705.08039.

Michael A. Nielsen. *Neural Networks and Deep Learning*. Determination Press, 2015. URL http://neuralnetworksanddeeplearning.com/.

Arvid Österlund, David Ödling, and Magnus Sahlgren. Factorization of latent variables in distributional semantic models. In *Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing*, pages 227–231, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1024.

D. Park, S. Kim, J. Lee, J. Choo, N. Diakopoulos, and N. Elmqvist. Conceptvector: Text visual analytics via interactive lexicon building using word embedding. *IEEE Transactions on Visualization and Computer Graphics*, 24 (1):361–370, Jan 2018. ISSN 1077-2626. doi: 10.1109/TVCG.2017.2744478.

Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In *EMNLP*, volume 14, pages 1532–1543, 2014.

Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. CoRR, abs/1802.05365, 2018. URL http://arxiv.org/abs/1802.05365.

Yonggang Qiu and Hans-Peter Frei. Concept based query expansion. In *Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval*, SIGIR ’93, pages 160–169, New York, NY, USA, 1993. ACM. ISBN 0-89791-605-0. doi: 10.1145/160688.160713. URL http://doi.acm.org/10.1145/160688.160713.

Alec Radford. Improving language understanding by generative pre-training. 2018.
Malka Rappaport Hovav and Beth Levin. The syntax-semantics interface. In Shalom Lappin and Chris Fox, editors, *The Handbook of Contemporal Semantic Theory*, chapter 19, pages 593–624. Wiley-Blackwell, 2015. ISBN 978118882139. doi: 10.1002/978118882139.ch19. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/978118882139.ch19.

François Rastier. *Arts et sciences du texte*. Presses Universitaires de France, Paris, 2001.

François Rastier, Marc Cavazza, and Anne Abeillé. *Semantics for Descriptions: From Linguistics to Computer Science*. The University of Chicago Press, Chicago, 2001.

David E. Rumelhart and James L. McClelland. *Parallel Distributed Processing*. MIT Press, Cambridge, 1986.

Magnus Sahlgren. *The Word-Space Model: Using Distributional Analysis to Represent Syntagmatic and Paradigmatic Relations between Words in High-Dimensional Vector Spaces*. PhD thesis, Stockholm University, Stockholm, Sweden, 2006.

Magnus Sahlgren. The distributional hypothesis. *Special issue of the Italian Journal of Linguistics*, 1(20):33–53, 2008.

Gerard Salton. *The SMART retrieval system: Experiments in automatic document processing*. Prentice-Hall, Upper Saddle River, NJ, 1971.

Gerard Salton, A. Wong, and C. S. Yang. A vector space model for automatic indexing. *Communications of the ACM*, 18(11):613–620, 1975.

Ferdinand de Saussure. *Course in General Linguistics*. McGraw-Hill, New York, 1959. Translated by Wade Baskin.

Jürgen Schmidhuber. Deep learning in neural networks: An overview. *Neural Networks*, 61:85–117, 2015. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2014.09.003.

Tobias Schnabel, Igor Labutov, David M. Mimno, and Thorsten Joachims. Evaluation methods for unsupervised word embeddings. In *Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015*, pages 298–307, 2015.

H. Schütze. Dimensions of meaning. In *Proceedings of the 1992 ACM/IEEE Conference on Supercomputing, Supercomputing '92*, pages 787–796, Los Alamitos, CA, USA, 1992. IEEE Computer Society Press. ISBN 0-8186-2630-5.

Hinrich Schütze. Word space. In *Advances in Neural Information Processing Systems 5, [NIPS Conference]*, pages 895–902, San Francisco, CA, USA, 1993. Morgan Kaufmann Publishers Inc. ISBN 1-55860-274-7.

Hinrich Schütze. Automatic word sense discrimination. *Comput. Linguist.*, 24 (1):97–123, March 1998. ISSN 0891-2017.
Holger Schwenk and Jean luc Gauvain. Connectionist language modeling for large vocabulary continuous speech recognition. In *In International Conference on Acoustics, Speech and Signal Processing*, pages 765–768, 2002.

Lutfi Kerem Senel, Ihsan Utlu, Veysel Yücesoy, Aykut Koç, and Tolga Çukur. Semantic structure and interpretability of word embeddings. *CoRR*, abs/1711.00331, 2017. URL http://arxiv.org/abs/1711.00331.

Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In *Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1162.

Daniel Smilkov, Nikhil Thorat, Charles Nicholson, Emily Reif, Fernanda B. Viégas, and Martin Wattenberg. Embedding projector: Interactive visualization and interpretation of embeddings. *CoRR*, abs/1611.05469, 2016.

Donald P. Spence and Kimberly C. Owens. Lexical co-occurrence and association strength. *Journal of Psycholinguistic Research*, 19(5):317–330, Sep 1990. ISSN 1573-6555. doi: 10.1007/BF01074363. URL https://doi.org/10.1007/BF01074363.

Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. Word representations: A simple and general method for semi-supervised learning. In *Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics*, pages 384–394, Uppsala, Sweden, July 2010. Association for Computational Linguistics.

Peter D. Turney. A uniform approach to analogies, synonyms, antonyms, and associations. In *Proceedings of the 22Nd International Conference on Computational Linguistics - Volume 1*, COLING '08, pages 905–912, Stroudsburg, PA, USA, 2008. Association for Computational Linguistics. ISBN 978-1-905593-44-6. URL http://dl.acm.org/citation.cfm?id=1599081.1599195.

Peter D. Turney and Patrick Pantel. From frequency to meaning: Vector space models of semantics. *CoRR*, abs/1003.1141, 2010.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2019.