To be Fair: a Case for Cognitively-Inspired Models of Meaning
Authors: Simon Preissner, Aurélie Herbelot (Università di Trento, 2019)
Topic: Distributional Semantics

Abstract
In the last years, the cost of Natural Language Processing algorithms has become more and more evident. That cost has many facets, including training times, storage, replicability, interpretability, equality of access to experimental paradigms, and even environmental impact. In this paper, we review the requirements of a ‘good’ model and argue that a move is needed towards lightweight and interpretable implementations, which promote scientific fairness, paradigmatic diversity, and ultimately foster applications available to all, regardless of financial prosperity. We propose that the community still has much to learn from cognitively-inspired algorithms, which often show extreme efficiency and can ‘run’ on very simple organisms. As a case study, we investigate the fruit fly’s olfactory system as a distributional semantics model. We show that, even in its rawest form, it provides many of the features that we might require from an ideal model of meaning acquisition.

Introduction
In recent years, the Natural Language Processing (NLP) community has seen an increase in the popularity of expensive models requiring enormous computational resources to train and run. The cost of such models is multi-faceted. From the point of view of shaping the scientific community, they create a huge gap between researchers in wealthy institutions and those with less resources and they often make replication prohibitive. From the point of view of applicability, they make the end-user dependent on high-tech hardware which they may not afford, or on cloud services which may have problematic privacy side-effects (and are not available to those with poor Internet access). Training such models can often take a long time and extraordinary amounts of energy, generating CO2 emissions disproportionate to the models’ improvements (Strubell et al., 2019). From a pure modelling point of view, finally, complexity often comes with a loss of interpretability, which weakens theoretical insights. Whilst we appreciate that a part of NLP is focused on engineering applications rather than modelling natural language proper, the linguists and cognitive scientists in the community have a duty to provide transparent, explanatory simulations of particular phenomena. Such considerations call for smaller and more interpretable systems. In this paper, we offer an example investigation into one of the most widely used techniques in NLP: the vectorial representation of word meanings. Our starting point is the set of requirements that should be fulfilled by an ideal model of lexical acquisition, which is expressed in QasemiZadeh et al. (2017): (A) high performance on fundamental lexical tasks, (B) efficiency, (C) low dimensionality for compact storage, (D) amenability to incremental learning, (E) interpretability. As we will show in §2, stateof-the-art systems still fail to integrate all those points. (A-D) are however basic features of humans and animal cognition. It seems, therefore, that we should find inspiration in algorithms from cognitive science, which in turn would allow us to derive interpretability (E) from the clear underpinnings of biological or psychological theories. We propose that a good place to find appropriate algorithms is the natural world, as many organisms display core cognitive abilities such as incremental learning, generalization or classification, which many NLP systems need to emulate. Such faculties develop in extremely simple systems, which are good contenders for the type of models we advocate here. One success story from ‘algorithmic’ cognitive science is based on the neural architecture of the fruit fly’s olfac- tory system, which clusters patterns of chemicals into categories of smells (Stevens, 2015), and has inspired the so-called Fruit Fly Optimization Al- gorithm (Pan, 2011; here: Fruit Fly Algorithm or ‘FFA’). The FFA has been implemented as a lightweight neural algorithm that performs ran- dom indexing for locality-sensitive hashing (LSH) (Dasgupta et al., 2017). This LSH algorithm has successfully been applied to various tasks, partic- ularly in information retrieval and for data com- pression (Andoni and Indyk, 2008). As a simple LSH algorithm, the FFA compresses data while preserving the notion of similarity of the origi- nal data, which is one of the core mechanisms involved in constructing vector representations of word meaning. To our knowledge, it has however never been taken as the basis for building distribu- tional semantic models from scratch, even though it seems to naturally fulfill a number of require- ments of those models. In the following, we present the FFA and show how it can be adapted to create vector spaces of word meaning (§4). We then apply the FFA in an incremental setup (§5) and assess its worth as a model, according to the various criteria high- lighted above (§6), including a possible interpre- tation of the FFA’s output.

Conclusion
We started this paper suggesting that NLP should explore a different class of algorithms for its most fundamental tasks. We argued that it is worth investigating cognitively-inspired architectures, which may not (yet) perform at state-of-theart level, but give us insights into potentially more plausible ways to model linguistic faculties in the mind. We also made a case for ‘small’ and ‘fair’ systems, in reach of all researchers and end-users. As illustration, we have explored what the olfactory system of a fruit fly can do for the representation of word meanings. The algorithm is certainly ‘fair’ in terms of complexity and required resources. Being based on an actual cognitive mechanism, it naturally encodes requirements such as (processing and storage) efficiency. Its simplicity lends itself to incremental learning and interpretability. Performance on a relatedness data set highlights that the raw model successfully captures latent concepts in the data but would probably require an extra attention layer, as indicated by the stronger results obtained on additionally preprocessed data. We hope to have demonstrated that such study is accessible to all, and actually sheds insights into the minimal components of a model in a way that more complex systems do not achieve. We particularly draw attention to the fact that the interesting behaviour of the fruit fly with respect to interpretability and incrementality makes it a worthy competitor for other distributional models – or at the very least, a source of inspiration.