Can a Fruit Fly Learn Word Embeddings?
Authors: Yuchen Liang
Chaitanya Ryali
Benjamin Hoover
Leopold Grinberg
Saket Navlakha
Mohammed J Zaki
Dmitry Krotov
Venue: ICLR 2021 Poster (2021)
Keywords: neurobiology
neuroscience
fruit fly
locality sensitive hashing
word embedding
sparse representations

Abstract: The mushroom body of the fruit fly brain is one of the best studied systems in neuroscience. At its core it consists of a population of Kenyon cells, which receive inputs from multiple sensory modalities. These cells are inhibited by the anterior paired lateral neuron, thus creating a sparse high dimensional representation of the inputs. In this work we study a mathematical formalization of this network motif and apply it to learning the correlational structure between words and their context in a corpus of unstructured text, a common natural language processing (NLP) task. We show that this network can learn semantic representations of words and can generate both static and context-dependent word embeddings. Unlike conventional methods (e.g., BERT, GloVe) that use dense representations for word embedding, our algorithm encodes semantic meaning of words and their context in the form of sparse binary hash codes. The quality of the learned representations is evaluated on word similarity analysis, word-sense disambiguation, and document classification. It is shown that not only can the fruit fly network motif achieve performance comparable to existing methods in NLP, but, additionally, it uses only a fraction of the computational resources (shorter training time and smaller memory footprint). 

Published as a conference paper at ICLR 2021
CAN A FRUIT FLY LEARN WORD EMBEDDINGS?
Yuchen Liang" * RPI
Chaitanya K. Ryali
Benjamin Hoover
Department of CS UC San Diego
MIT-IBM Watson Al Lab IBM Research benj Denjamin.hooverreeeee www..oooovveeeb..oooommmm
MIT-IBM Watson AI Lab
liangy7 70rpi. edu
rckri mokrishn@eng.o ucsd. edu
Leopold Grinberg
Saket Navlakha
IBM Research 1grinbel lm. com
Cold Spring Harbor Laboratory navlakh khaechhla.......c...cc..ccc..h edu
Mohammed J. Zaki
Dmitry Krotov
Department of CS RPI
MIT-IBM Watson Al Lab
IBM Research krotovê bm com
zakiêcs. rpi. edu
ABSTRACT
The mushroom body of the fruit fly brain is one of the best studied systems in neuroscience. At its core it consists of a population of Kenyon cells, which re- ceive imputs from multiple sensory modalities. These cells are inhibited by the anterior paired lateral neuron, thus creating a sparse high dimensional represen- tation of the inputs. In this work study a mathematical formalization of this network motif and apply it to learning the correlational structure between words and their context a corpus of unstructured text, a common natural language pro- cessing (NLP) task. We show that this network can learn semantic representations of words and can generate both static and context-dependent word embeddings. Unlike conventional methods (e.g., BERT, Glo Ve) that use dense representations for word embedding. our algorithm encodes semantic meaning of words and their context in the form of sparse binary hash codes. The quality of the learned rep- resentations is evaluated on word similarity analysis, word-sense disambiguation, and document classification. It is shown that not only can the fruit fly network motif achieve performance comparable to existing methods in NLP, but, addition- ally, it uses only fraction of the computational resources (shorter training time and smaller memory footprint).
INTRODUCTION
Deep learning has made tremendous advances in computer vision, natural language processing and many other areas. While taking high-level inspiration from biology, the current generation of deep learning methods are not necessarily biologically realistic. This raises the question whether biolog- ical systems can further inform the development of new network architectures and learning algo- rithms that can lead to competitive performance on machine learning tasks or offer additional in- sights into intelligent behavior. Our work is inspired by this motivation. We study a well-established neurobiological network motif from the fruit fly brain and investigate the possibility of reusing it for solving common machine learning tasks NLP. We consider this exercise as a toy model exam- ple illustrating the possibility of "reprogramming" of naturally occurring algorithms and behaviors (clustering combinations of input stimuli from olfaction, vision, and thermo-hydn sensory system) into target algorithm of interest (learning word embeddings from raw text) that the original bio- logical organism does not naturally engage in.
The mushroom body (MB) is major of the brain responsible for processing of sensory infor- mation in fruit flies. It receives inputs from a set of projection neurons (PN) conveying information
Yuchenn Liang AT Horizons Scholar, part of the Rensselaer-IBM AI Research Collaboration (ATRC).

---

Published as a conference paper at ICLR 2021
from several sensory modalities. The major modality is olfaction (2], but there also inputs from the PN responsible for sensing temperature and humidity [29], as well as visual inputs [45 These sensory inputs are forwarded to a population of approximately 2000 Kenyon cells (KCs) through set of synaptic weights [26]. KCs are reciprocally connected through an anterior paired lateral (APL.) neuron, which sends strong inhibitory signal back KCs. This recurrent network effectively im- plements winner-takes-all competition between KCs, and silences all but a small fraction of top activated neurons (8). This is the network motif that we study in this paper; its schematic is shown in Fig. [ KCs also send their outputs to mushroom body output neurons (MBONs), but this part of the MB network is not included into our mathematical model.
anterior paired lateral neuron (APL)
Kenyon calls (KC)
t
projection neurons (PN)
offaction
thermo-hydro
vision
sensory system
Figure I: Network architecture. Several groups of PNs corresponding to different modalities send their activ- ities to the layer of KCs, which are inhibited through the reciprocal connections the API. neuron.
Behaviorally, it is important for fruit fly to distinguish sensory stimuli, e.g., different odors. If fruit fly senses a smell associated with danger, it's best to avoid it; if it smells food, the fruit fly might want to approach it. The network motif shown in Fig. [ is believed be responsible for clustering sensory stimuli so that similar stimuli elicit similar patterns of neural responses at the level of KCs allow generalization, while distinct stimuli result in different neural responses, to allow discrimination. Importantly, this biological network has evolved to accomplish this task in very efficient way.
In computational linguistics there is long tradition 19 of using distributional properties of linguis- tie units for quantifying semantic similarities between them, as summarized in the famous quote by JR Firth: "a word is characterized by the company it keeps" [14] This idea has led to powerful tools
such as Latent Semantic Analysis [9] topic modelling (3), and language models like word2vec (30), GloVe (34), and, more recently, BERT [10] which relies on the Transformer model 44. Specifi- cally word2vec models are trained to maximize the likelihood of a word given its context, Glo models utilize global word-word co-occurence statistics, and BERT uses deep neural network with attention to predict masked words (and the next sentence). As such, all these methods utilize the correlations between individual words and their context in order to learn useful word embeddings.
In our work we ask the following question: can the correlations between words and their contexts be extracted from raw text by the biological network of KCs, shown in Fig. D Further, how do the word representations learned by KCs differ from those obtained by existing NLP methods? Although this network has evolved process sensory stimuli from olfaction and other modalities and not to "understand" language it uses a general purpose algorithm to embed inputs (from different modalities) into a high dimensional space with several desirable properties. which we discuss below.
Our approach relies on a recent proposal that the recurrent network of mutually inhibited KCs can be used as a "biological" model for generating sparse binary hash codes for the input data presented at the projection neuron layer [8) It was argued that matrix of random weights projecting from PN layer into the KCs layer leads to the highly desirable property of making the generated hash codes locality sensitive, i.e., placing similar inputs close to each other in the embedding space and pushing distinct stimuli far apart. A subsequent study [39] has demonstrated that the locality sensitivity of the hash codes can be significantly increased, compared to the random case, if the matrix of weights from PN to KCs is learned from data. The idea of using the network of KCs with random projections for NL.P tasks has also been previously explored in [37] see discussion in section 6
2

---

Published as a conference paper at ICLR 2021
Biologically, there is ongoing debate in the neuroscience community regarding whether these projections are random. For instance, [3] argues for the random model, while [47] presents evidence of the non-random structure of this network, which is related to the frequency of presented odors. Since the goal of our work is to build a useful Al system and not mimic every detail of the biolog- ical system, we adopt the data-driven synaptic weight strategy even if fruit flies may random projections, As is clearly demonstrated in [39], learned synapses lead to better performance.
Our main contributions in this work are the following:
1. Inspired by the fruit fly network, we propose an algorithm that makes possible generate binary (as opposed to continuous) word embeddings for words and their context. We sys- tematically evaluate the performance of this algorithm on word similarity task, word-sense disambiguation, and document classification.
2. We demonstrate that our binary embeddings result in tighter and better separated clusters concepts compared to continuous GloVe embeddings, and stand in line with clustering properties of binarized versions of Glo Ve.
3. We show that training the fruit fly network requires order of magnitude smaller compute time than training the classical NLP architectures, like BERT, at the expense of relatively small decrease in classification accuracy.
2 LEARNING ALGORITHM
Consider a training corpus. Each sentence can be decomposed into collection of w-grams of consecutive words. If the word tokens come from a predefined vocabulary of size Nvac, the input to the algorithm is a vector of size 2o This vector consists of two blocks: the context (the first Nvoc elements), and the target (the remaining Nyee elements); see Fig. 2 In this work w is assumed to be an odd integer, and the target word is assumed to be the center of the w-gram. The target word
Apple stock rises on optimism for the new iPhone.
context
target
0
0
0
0
0
1
Apple
Apple
stock
The encoding method. The input vector consists of two blocks separated by the (thick) blue line. Assuming w 3. center word "stock" is the target word and the two flanking words form context. The w-gram is highlighted in light blue.
Figure 2:
is one-hot encoded in the target block, and the context words are binary encoded as a bag of words in the context block (no positional information is used). The window w slides along the text corpus, and for each position generates a training vector v {o(") 1 where the index A enumerates different w-grams. and index i enumerates positions in the context-target vector. These training vectors are passed to the learning algorithm. The goal of the algorithm to learn correlations between the context and the target blocks.
2.1 MATHEMATICAL FORMULATION
Mathematically, the objective of the training algorithm is to distribute set of context-target pairs among K buckets, so that similar pairs end up in similar buckets. In order to achieve this, the learning algorithm takes two inputs: a set of training vectors v E [0, 1|2N... and a vector of occurrence probabilities p = {p- = f(x mod 12 E 22 where fj is the probability of observing word j in the training c The learning can be formalized as a minimization of the
2N
In our notation vector f has Nov elements, while vector p has 2.Voc elements. Given that index i runs from to 2Nw.. notation p: f(x mod Noc) is mathematical way to concatenate two vectors f into twice longer vector p.
3

---

Published as a conference paper at ICLR 2021
energy function,
(39
for additional details, defined by
Wa p. V /p)
E=
Σ AEdata
where P -- max
W ns vA)
(I)
1/2
W p: W
#
In this equation W E RKx22 is matrix of synaptic connections, given as W- = [W = [Wai 3. projecting from PN layer (individual neurons the layer are denoted by the index i) the KC layer (individual neurons in the KC layer are denoted by the index (4). There are 2Nw neurons in the PN layer and K neurons in the KC layer. The inner product (X. Y) Σ Ei as a contraction over index i of PN cells. In the numerator the energy function the binary encoded w-gram is divided by the probabilities of occurrences of individual words element-wise that the numerator can be written as
2N X,Y, is defined
2N Σ
W
W
Pi
Probabilities p are calculated based on the frequencies words the training corpus. The vocab- ulary contains Nvoe most frequent words in the corpus, thus all the elements of pæ non-zero and the element-wise division is well defined.
Intuitively, the goal of the training algorithm is to adjust the weights of the neural network so that they are aligned with w-grams that are frequently present in the training corpus. We rely on the assumption that semantically related w-grams share several "core" words, while few individual words might be substituted by synonyms//ntonyms. The minimization of the energy function (T) is accomplished by the iterative update of the weights satisfying the following learning rule 25 39 17
AW μi Eg [ W
I
( W
W
Pt
Pj
In this equation the activation function is equal to one for maximally driven hidden unit (Kenyon cell), and is equal to zero otherwise
[ = 6ymns
where X = arg max
(3)
The learning rate is denoted by E, and ón, is a Kronecker delta symbol.
2.2 BIO-HASHING
After learning is complete the hash codes for the inputs can be generated in the following way. Given the binary encoded w- w-gram v
H 1, if (W n, vA) in the top l of all KCs activations
(4)
0, otherwise
This is a crude mathematical approximation of the biological computation performed by the PN- KC-APL neural network 8 33]. An input v generates an input current (W,,. v into the KC neurons using feedforward weights Wpi The recurrent network of KCs and the APL neuron silences all but a small fraction of KCs. Those cells that remain active are assigned state while the rest of the KCs are assigned the inactive state O.
Notice, that equation makes possible generate the hash codes for both individual words (static word embeddings like word2vec and Glo and phrases (similar to Transformer models). In the static case, the input V has all zeros the context block and a one-hot encoded word in the target block. In the context-dependent case, both blocks have binary encoded input words. Importantly, both context-dependent and static embeddings are mapped into the same space of sparse binary hash codes (a vector of K elements, with l ones in it). We show below that these hash codes capture semantic meaning of the target word and the context in which it is used. For the rest of the paper refer to the parameter is in equation (4) as the hash length.
In order to provide an intuition behind the learning algorithm defined by the energy function [ and weight update rule (2) and connect it to some of the existing methods machine learning, consider
4

---

Published as a conference paper at ICLR 2021
the limit when all the words have equal probabilities in the training corpus, p; N. In this limit the energy function (I) reduces to the familiar spherical K K-means clustering algorithm [TT] In this limit the weights of each KC correspond to the centroids of the clusters of context-target vectors. The hashing rule (4) assigns active state 1 to the & closest centroids (and inactive state 0 to the remaining ones), defined with respect to cosine similarity distance. In this simple limit the learning algorithm that we use can be viewed as biologically plausible implementation of this classical algorithm. For real datasets the probabilities of words are different, thus this correspondence does not hold. Notice that division by the probability appears only in the expression for the energy, but not in the definition of ji in equation (T). Equivalently, division by p; appears in the second bracket of equation 22. but not in the argument of the activation function g[æn Thus, in the general case (for different word probabilities p.) our algorithm is not equivalent to spherical K-means on context-target vectors rescaled by the probabilities. Rather, in the general case, the closest centroid is found for given context-target vector (via the definition of in equation ( no pi involved), but the "updates of the position" of that centroid are computed by enhancing the contributions of rare words (small p:) and suppressing the contributions of frequent words (large pt). Empirically, we have found that division by the probabilities improves performance our method compared to the case of spherical K means (when the factor 1/p is removed from the algorithm).
EMPIRICAL EVALUATION
The KC network shown in Fig. [ was trained on the OpenWebText Corpus [15], which is a 32GB corpus of unstructured text containing approximately 6B tokens. The details of the training protocols and the hyperparameters are reported section 7 in the supplement.
3.1 STATIC WORD EMBEDDINGS EVALUATION
Our aim here is to demonstrate that the sparse embeddings obtained by the fruit fly network motif are competitive with existing state-of-the-ard word embeddings such as Glo Ve (34 and word2vec (30)
and commonly used binarization tools for these continuous embeddings. We show this by evaluating the semantic similarity of static word embeddings. Several common benchmark datasets are used:
WS353 [13], MEN [4. RW (28), SimLex [21] RG-65 (38), Mturk [18]. These datasets contain pairs of words with human-annotated similarity scores between them. Following previous work [43 42), model similarity score for binary representations is evaluated as sim(11, U22 = (n11 + Roo)/n, where R11 (noo) is the number of bits 21 and 02 that are both (0), and n is the length of 21 Cosine similarity is used for real-valued representations. Spearman's 's correlation coefficient calculated between this similarity and the human annotated score. The results are reported in Table
Dataset
Ours
GloVe
word2vec
SOTA
MEN
56.6
69.5
75.5
81.3
12
WS353
63.7
64.0
66.5
81.0
18
SIMLEX
21.0
31.5
41.7
56.0
40
RW
39.4
46,8
61.3
61.7
36
RG
69.0
74.2
75.4
83.3
20
Mturk
56.1
57.5
69.8
72.7
18
Table 1: Evaluation on word similarity datasets via Spearman's rank correlation coefficient. Both GloVe and word2vec use 300d pretrained embeddings. Hyperparameter settings for our model: K 400, w = 11. Results for our algorithm are reported only for fixed hash length, k = 51. See Table 7 for results as a function of hash length.
We observe that our word embeddings demonstrate competitive performance compared to GloVe, but worse performance than word2vec. At the same time, our embeddings are binary, as opposed to GloVe and word2vec, which are represented by continuous vectors. Thus, it is more appropriate to compare them with commonly used binarized versions of the continuous embeddings. Specifically, compare the performance of fruit fly embeddings with number of state-of- -ftthe-rrt binarization methods such as: LSH/SimHash 7 (random contractive projections followed by binarization based on sign), RandExp 8 (random expansive projections followed by k-winner take all binarization),
5

---

Published as a conference paper at ICLR 2021
(iterative quantization), SH (spectral hashing) 46 PCAH [66 (PCA followed by bina- rization based on sign). The complete evaluation of all these methods for varying hash length is presented in Section & please see Tables 7 g 9 for binarization of pretrained Glo pretrained word2vec, and GloVe trained on OpenWeb'len
ITQ [16]
In Table we also include evaluation from NLB, "Near-Lossless Binarization" (43] (autoencoder--aase based binarization) for the hash lengths where those results are available. Here we only present a short summary of those results for a specific (small)
hash length k 4 in Table 2
Dataset
Ours
LSH
RandExp
ITQ
SH
PCAH
MEN
34.0
16.9/35.5/23.6
27.5/24.2/28.4
0.1/9.2/26.9
9,417.2/23.8
12.5/5.3/26.0
WS353
43.2
8.2/26.0/20.2
20.9/23.5/30.5
-6.6/16.0/55.9
15,4/3.3/18.1
6.4/17.3/21.2
SIMLEX
13.4
6.8/17.0/8.0
10,4/17.6/10.1
7.0/3.3/7.3
9.31-3.6/12.1
4.4/-2.9/11.5
RW
11.0
10.8/21
19.9/24.7/22.00
13.7/17.4/24.5
22.6/14.6/19.7
12.4/15.0/19.7
RG
24.0
21. 6/225///
36.6/30.4/28.7
-17.5/32.8/11.4
4.5/18.0/39.8
1.9/20.8/45.0
Mturk 44.0
16.0/33.1/18.3
29.3/22.7/28.3
9.9/22.5/26.3
18.9/21.9/2..3
15.5/23.6/24.9
Table 2: Comparison to common binarization methods. This table is simplified version (for hash
length k = 4) of the complete evaluation for a range of hash lengths reported in Tables 7 Each binarization technique was evaluated on three continuous embeddings: pretrained Glo Ve, pretrained word2vec, GloVe trained on Open WebText (the same dataset that was used for training our fruit fly embeddings), format: pretrained GloVe/ pretrained word2vec/ GloVe on OWT. Hyperparameter settings for our model: K = 400, w = 11. Best result in bold; second best underlined.
It is clear from Table 2 that fruit fly embeddings outperform existing methods for word embedding discretization on WS353 and Mturk, and demonstrate second best result (after LSH binarization of word2vec) MEN. In general (see Tables 7 powerful compared to existing methods at small hash lengths (see = 4, 8 in the aforementioned tables). These results indicate that the fruit fly network can learn meaningful binary semantic rep- resentations directly from raw text. We also note that an added advantage of binary embeddings is that they require only fraction (approx. 3%) of the memory footprint required for continuous word embeddings (assuming they have the same length), since real value requires 32-bits per vector element, whereas boolean value requires only 1-bit.
789. 9. find that fruit fly embeddings are particularly
3.2
WORD CLUSTERING
A nice aspect of binary embeddings is that they result in tighter and better separated clusters than continuous embeddings. To evaluate this property for our method we started with hash codes for individual words and performed agglomerative clustering via complete link, using the cosine dis- tance as the metric. The clustering algorithm was terminated at 200 clusters (we experimented with possible choices of this parameter, such as 200, 500, 1000, 2000, 3000, 5000, and arrived at similar conclusions). We repeated the same analysis for continuous GloVe, binarization of Glo Ve embeddings via autoencoder-like like method [43), and simple discretization method of Glo when one declares the largest & elements of each word vector to be 1 and assigns O to the remaining elements (for A: 50, 75, 120, 200). The results for the inter-cluster similarity vs. intra-cluster similarity shown in Fig. 3 (panel A). It is clear from this scatter plot that the average distance between the points within a cluster is smaller (higher similarity) for all considered binary embeddings compared to Glo embeddings. At the same time, the distance between the closest clusters is larger or equal (smaller similarity) for the fruit fly embeddings and naive discretizations with & < 120. We also observe that the clusters lose detail (i.e., both intra- and inter-cluster similarity increases) as the bi- narization threshold gets higher (shown for Glove). However, our embeddings maintain a balance between intra- and inter-clustering similarity, and thus still capture fine-grained cluster information. For instance, inspecting the semantic structure of the clusters obtained this way, an example of the hierarchical clustering diagram (lower part of the tree containing 42 leaves) is shown in Fig. 3 (panel B). We elearly observe semantically coherent clusters resulting from the fruit fly word embeddings.
3.3 CONTEXT-DEPENDDRT WORD EMBEDDINGS
Here, we evaluate the effectiveness of our fruit fly inspired approach for contextual word embed- dings, opposed static (or context-independent) embeddings from above. We use the WiC (35)

---

Published as a conference paper at ICLR 2021
max cosine simieniy between 3wo cipsest clssers
1.0
B
renpect)
Haghnopsst
0.7
pa
0.6
fly
readway
(eadwyy)))
0.5
GloVe
Tissier et al.
beatt!
GloVe 50
GloVe_ 75
(alread))
0.3
GleVe_ 120
(ndderdaal) upicie())
6.0
0.1
0.2
0.3
0.4
0.5
0.6 0.7
0.8
average cosine similanty within the cluster
Figure 3: Panel average cosine similarity between the points within cluster vs, maximum cosine similar- ity (minimal distance) to a point from the closest cluster. Solid lines correspond to mea for the individual clusters. Numbers next to GloVe in the legend correspond to the number of largest elements in the word vector that are mapped to 0 under the naive discretization procedure. Panel B: example of cluster generated by the agglomerative clustering for our method, the integer number associated with each node corresponds to the number of daughter leaves in that cluster. The root node corresponds to "interchange (42)".
and SCWS benchmarks for the evaluation of context-sensitive word embeddings for word sense disambiguation. Both the datasets comprise pairs of sentences that contain target word, and the task is to determine whether the two target words share similar semantic meaning the corre- sponding contexts. The WiC dataset is modeled the target words have the same sense, and indicating that they mean different things. The SCWS dataset is modeled rank prediction task, since for each pair of sentences and target words, it reports the average human similarity scores (from 10 Amazon Mechanical Turkers per pair).
binary prediction task, with denoting that
word in context
10 nearest neighbor words in the hash code space
design apple latest iphone filling sweet applé pie recipe money in bank checking account boat the bank of the river
design, web, features, graphics, radeon, android, apps, ios, apple, nvidia
chocolate, sweet, crispy, noodles, syrup, coconut, cheese, sauce, butter, cinnamon money, credit, loans, account, cash, funds, savings, paying, banks, pension
river, lake, near, island, creek, canyon, valley, mountains, area, shore
Figure 4:
For every word (highlighted in green) in context (left), 10 nearest neighbor words in the hinary hashing space are shown (right). Context allows the algorithm disambiguate the target word's meaning.
Before presenting quantitative results, qualitatively examine how the fruit fly network performs on context sentence pairs for target words "apple" and "bank" in Fig, 4 We show the top = 10 nearest neighbor words for the context dependent target word. These examples clearly indicate that the "correct" sense of the word has been found ("apple" the device manufacturer has different nearest neighbors from the fruit, and "bank" the financial institution from the natural feature).
For the quantitative comparison, we contrast our method against contextual embeddings from BERT (10), GloVe (34), word2vec (30] and Word2Sense (33). For BERT we use the 768- dimensional embeddings from the uncased-large model, for GloVe and word2vec we use the 300- dimensional embeddings, and for Word2Sense we use the sparse 2250-dimensional pretrained em- beddings. Since BERT outputs contextual embeddings for each word in sentence, we simply compute the cosine similarity between the embedding vectors for the target words for each pair of instances. For Glo Ve/wor, we use context window of size centered at each of the target words and compute the average embedding for each window and compute the cosine similarity be- tween the two window vectors. Similar approach is used for Word2Sense, but the similarity between two embeddings is based on the Jensen-Shane Shannon divergence [33. For the fruit fly network, given the effectiveness of the top-q nearest neighbor words (as seen in Fig. 4 we devise two component scoring function. The first component is the dot-product between the context-depender hash codes for the target words plus length context blocks, denoted Jast. The second is the number of
7

---

Published as a conference paper at ICLR 2021
common contextual nearest neighbors of the two target words among the top-q neighbors of each (scaled to be between and 1), denoted Jan- The final score is given as J = Jot + (1-a) -nns where [o, 1] is a hyperparameter. For all the methods, we predict WiC pair to be positive if the score is above threshold value 0. For SCWS, the ranking is proportional to the scores above 0. with the rest scored as zero. The hyperparameter is tuned for all the methods independently. Finally, for a fair comparison, all methods use the same 20k vocabulary.
We report the performance of our context-dependent word embeddings for both SCWS and WiC in Table 3 and Table respectively. For both benchmarks we report the results from a 5-fold cross- validation study, where each fold (in turn) is used development set, and the remaining four folds as the test set. We select the optimal hyperparameters (including 0, a, q, k, w) for all the methods using only the first fold; no training done since we evaluate only the pretrained embeddings. The tables report the Spearman rank correlation on SCWS, and the accuracy on WiC.
Method BERT
mean
std
Method BERT
mcan
std
56.8
0.54
61.2
0.22
word2vec (w = 0)
56.7
0.005
word2vec (w 5)
51.3
0,004
GloVe (w = 3)
40.9
1.3
Word2vec (w 0)
50.0
0.003
GloVe (w 0)
54.4
0.10
GloVe (wo 7)
54.9
0.26
Word2Sense (w- 3)
41.4
0.01
Glo (w = 0)
50.1
0.25
Word2Sense (w = 0)
54.2
0.008
Word2Sense (w 7)
56.5
0.004
Ours (w =0) 0)
49.1
0.36
Word2Sense (w 0)
50.0
0,003
Ours 21)
57.7
0.27
Table 3: SCWS dataset: mean and std for
Spearman rank correlation. The best window value is also shown.
Table 4: WiC dataset: mean and std for accu- racy. The best window value is also shown.
On SWCS (Table 3). we see that the context-independent embeddings (using w = 0) are better for GloVe, Word2Sense and our method, with word2vec yielding the best results. The reason is that about 86.5% of the word pairs in SCWS are different words, and can be distinguished without looking the context. Unlike SCWS, the WiC benchmark uses the same target word (with only minor variations in some cases) in both contexts, and therefore a context-independent approach is not expected perform well. Indeed, WiC (Table we clearly observe that context-independeml vectors (w = 0) are not very good, and our method, that uses the joint scoring function J combining both the hash code and nearest neighbor scores, better than context-dependent GloVe (w 7),
word2vec (w 5) and Word2Sense (also w = 7).
Dataset
Ours
Glove
NLB(256bits)
NLB(512bits)
Word2vec
BERT
20Newsgroup
78.2
77.9
61.6
64.1
77.3
78.6
SST-2
77.1
78.3
76.3
78.6
80.7
90.8
WOS-11967
83.8
84.2
70.6
72.8
84.8
86.7
TREC-6 90.4
89.0
85.2
88.8
90.9
94.0
Table 5: Accuracy for document classification task. We use 300d pretrained models for Glo Ve and word2vee, and pretrained bert-large-nncased model for BERT. For NLB, 300d GloVe embeddings were binarized into 256 and 512 bits. For our model, hash length 30 is used. For fair comparison, all models use the same vocabulary of 20k words.
3.4 DOCUMENT CLASSIFICATION
We also compare our binary embeddings with GloVe 34 Word2vec 31,. BERT 110 and Near-
Lossless Binarization 44 on document classification tasks. The benchmarks we use are 20 News- groups [T] Stanford Sentiment Treebank [II, WOS--1997 (24) and TREC-6 datasets [27]. The 20 Newsgroups dataset contains around 18,000 documents, partitioned evenly into 20 different groups; the Stanford Sentiment Treebank dataset contains movie reviews reflecting their sentiment ei- ther positive or negative; WOS- 11967 dataset contains 1 11967 documents with 35 categories which include parents categories; and TREC-6 dataset consists of open-domain, fact-based questions di- vided into broad semantic categories. We use the TextCNN [3] classifier that uses all the different embeddings mentioned above. For fair comparison, we use the same model parameters (e.g., ker- nel size, filter dimension) while testing different embeddings. The results in Table 5 show how our sparse binary encodings are competitive with other methods.
8

---

Published as a conference paper at ICLR 2021
device
K
batch-size
GPU mem
time
V100 3
400
2000 3
122MB
17m
70-
Cemmmmnn is rank cont. coeff.
V100 x 3
400
x 3
150MB
8m
60
V100 x 3
600
2000 3
232MB
24m
50
V100 3
600
10000
267MB
11.5m
CPU 44cores
400
2000
76m
CPU 44cores
400
10000
25m
Speeal
22
Table 6: Training time (per epoch) and memory footprint of our method on GPUs and CPUs. For the GPU implemen- tation, three V100 GPUs interconnected with 100GB/s (bi- directional) NVLink were used. For the CPU implemen- tation, the computation was done on two 22-core CPUs. CPU memory is 137GB. The results are reported for win- dow 11.
HCR5
Mturk
training time. min
Figure 5:
Spearman's 's correlation on word similarity datasets (see Sec- tion B.I) vs. training time. Each point one epoch.
COMPUTATIONAL COMPLEXITY
The computational complexity of our method can be evaluated by analyzing equations 2 for the weight updates. In these equations A is sparse vector, which has only w non-zero elements in it. Thus, for a minibatch of size [BS]. the computational complexity evaluating the dot product with weights is K w |BS). Additionally, the argmax operation requires K |BS] operations. We will assume that the largest parameters in our model are the size of the corpus |A| 10t0, and the size of the vocabulary Nov x 10 10 Additionally
large minibatches [B|| Nvoe- Calculation of the second term in (2) requires O(K Novee) operations addition to K w [BS] operations for calculating the dot-product for each data point. Since the algorithm has go
the entire corpus, this computation needs to be repeated / times per epoch. Thus, the overall computational
ens//BSs) ~ K |A| w per epoch. Thus, in the leading order it does not grow with the size of the vocabulary, which is a nice computational feature.
complexity of our method is O (w + /
From the practical perspective, typical wall-clock training time and memory requirements per epoch shown in Table 6 As is shown in Fig. 3 accurate solutions obtained after about 3 epochs; improvements beyond that are relatively small. Thus, our algorithm capable of producing competitive models in a couple of hours. Contrast this with approximately 24 hours training time for GloVe [34]: days of training on 16 TPUs for BERTBASE: and 4 days on 64 TPUs for BERT
I0 (the last two numbers assume training corpus of size 250B tokens vs. 6B tokens considered in this paper). The record breaking training time of 47 minutes for BERT requires the use of 1472 NVIDIA V100 GPUs each with 32GB of memory and a specialized DGX server architecture 332. In our own experiments, we trained GloVe embedding on OWT corpus using the same vocabulary of 20k words that we used for the fruit fly embeddings. The wall-clock training time was approximately 10 hours 16 threads, see details in Section [I These are substantially larger computational resources than those required for training the fruit fly network.
DISCUSSION AND CONCLUSIONS
In this work asked the intriguing question whether the core computational algorithm of one of the best studied networks in neuroscience- the network of KCs in the fruit fly brain can be repurposed for solving a well defined machine learning task, namely, learning word embeddings from text. We have shown that, surprisingly, this network can indeed learn the correlations between the words and their context, and produce high quality word embeddings. On the semantic similarity task the fruit fly word embeddings outperform common methods for binarizing continuous SOTA word embeddings (applied to Glo word2vee, and GloVe trained on OWT) at small hash lengths. On the word-in-context task the fruit fly network outperforms GloVe by almost 3%, word2vec by more than 6%, but looses to BERT by 3.5%, see Table The small gap in classification accuracy compared with BERT, however, is outweighed by the benefit of requiring significantly smaller computational resources to obtain these fruit fly embeddings, as we have explained in Section 4 see Table 6 We view this result as example of general statement that biologically inspired algorithms might be more compute efficient compared with their classical (non- biolooigog counterparts, even if they slightly lose in terms of accuracy.

---

Published as a conference paper at ICLR 2021
REFERENCES
[ 20NewsGrroups. 20 newsgroups dataset, 1995. URL http::ttp://etm /pop::://ppoplll e. csail .mmt.cc. .mit.edu//
jremnie/2ONewsrropps///
[2] Alexander Shakeel Bates, Philipp Schlegel, Ruairi JV Roberts, Nikolas Drummond, Imaan FM Tamimi, Robert Gillies Turnbull, Xincheng Zhao, Elizabeth C. Marin, Patricia Demetria Popovici, Serene Dhawan, al. Complete connectomic reconstruction of olfactory projec- tion neurons in the fly brain. BioRxiv, 2020.
[3] David M Blei, Andrew Y Ng. and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan)993-10022 2003.
[4] Elia Bruni, Nam Khanh Tran, and Marco Baroni. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49(1):1-47, January 2014. ISSN 1076-9757.
[5] Sophie JC Caron, Vanessa Ruta, LF Abbott, and Richard Axel. Random convergence of olfac- tory inputs in the drosophila mushroom body. Nature, 13774777 2013.
[6] Sophie Jeanne Cecile Caron, Jinzhi Li, Brennan Dale Mahoney, and Miles Solomon Jacob. Two parallel pathways convey distinct visual information the drosophila mushroom body. bioRxiv, 2020.
[7] Moses S. Charikar. Similarity Estimation Techniques from Rounding Algorithms. In Annual ACM Symposium Theory of Computing, pp. 380-388, 2002. ISBN 118-1-58113-4953 doi: 10. 10.1145/599907.509965
[8] Sanjoy Dasgupta, Charles F. Stevens, and Saket Navlakha. A neural algorithm for a fundamen- tal computing problem. Science, 358(6364):793-795 2017.
[9] Scott Deerwester, Susan T Dumais, George W Furnas, Thomas Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for in- formation science, 4 41(6):391-407, 1990.
[10] Jacob Devlin, Ming-Wer Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint urXiv:1810.04805. 2018.
[11] Inderjit S Dhillon and Dharmendra Modha. Concept decompositions for large sparse text data using clustering. Machine learning, 42(1-2):1433173 2001.
[12] András Dobó. A comprehensive analysis of the parameters in the creation and comparison of feature vectors in distributional semantic models for multiple languages. PhD thesis, szte, 2019.
[13] Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. Placing search in context: The concept revisited. ACM Trans. Inf. Syst., 20 (1), 2002. ISSN 1046-8188.
[14] John Firth. synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis, 1957.
[15] Aaron Gokaslan and Vanya Cohen. Open Corpus.
http:://kkylion007
g vithub.ion http:ppeeeeeeeexxoororropoo
2019.
[16] Yunchao Gong and Svetlana Lazebnik. Iterative quantization: A procrustean approach to learn- ing binary codes. CVPR 20/ 1, pp. 817-824. IEEE, June 2011. ISBN 978- -47---4577- doi: 10.1109/CVPR.2011.. 1.5595542..
[17] Leopold Grinberg. John Hopfield, and Dmitry Krotov. Local unsupervised learning for image analysis. arXiv preprint (xXiv:190.0089) 2019.
[18] Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. Large-scale learning of word relatedness with constraints. KDD, pp. 1406-14144 2012.
[19] Zellig S Harris. Distributional structure. Word, 10(2-3):146-162 1954.
10

---

Published as a conference paper at ICLR 2021
[20] Samer Hassan Hassan and Rada Mihalcea. Semantic relatedness using salient semantic analy- sis. In Twenty-Fifth AAAI Conference on Artificial Intelligence. Citeseer, 2011.
[21] Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665 695, December 2015.
[22] Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Ng. Improving Word Representations via Global Context and Multiple Word Prototypes. In Annual Meeting of the Association for Computational Linguistics (ACL), 2012.
[23] Yoon Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408. 5882, 2014.
[24] Kamran Kowsari, Donald E Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi,, Matthew Gerber, and Laura E Barnes. Hditex: Hierarchical deep learning for text classification. In Machine Learning and Applications (ICMLA), 2017 16th IEEE International Conference on. IEEE, 2017.
[25] Dmitry Krotov and John J. Hopfield. Unsupervised learning by competing hidden units. Pro- ceedings of the National Academy of Sciences, 116(16)7733-773,, 2019.
[26] Feng Li, Jack Lindsey, Elizabeth C Marin, Nils Otto, Marisa Dreher, Georgia Dempsey, Ildiko Stark, Alexander Shakeel Bates, Markus William Pleijzier, Philipp Schlegel, al. The con- nectome of the adult drosophila mushroom body: implications for function. bioRxiv, 2020.
[27] Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference Computational Linguistics, 2002.
[28] Thang Luong, Richard Socher, and Christopher Manning, Better word representations with recursive neural networks for morphology. In Conference on Computational Natural Language Learning, pp. 104-113, 2013.
[29] Elizabeth Marin, Ruairí JV Roberts, Laurin Büld, Maria Theiss, Markus Pleijzier, Tatevik Sarkissian, Willem J Laursen, Robert Gillies Turnbull, Philipp Schlegel, Alexander Shakeel Bates, et al. Connectomics analysis reveals first, second, and third order thermosensory and hygrosensory neurons in the adult drosophila brain. BioRxiv, 2020.
[30] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv: 1301.3781, 2013.
[31] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Dis- tributed representations of words and phrases and their compositionality. In C. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26, pp. 3111-3119. Curran Asso- ciates, Inc., 2013. WttLLhttts:///poceeeeee ing.eeddding....... neur i ps. cc./paper/2013/f1ee Paper.
9aa4
[32] Shar Narasimhan. Nvidia clocks world's fastest bert training time and largest transformer nvidia.
based model, paving path for advanced conversational ai.
com/training bert wwwttthhhhe gathh/pps/// 2019.
[33] Abhishek Panigrahi, Harsha Vardhan Simhadri, and Chiranjib Bhattacharyya. Word2sense: sparse interpretable word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5692-5705, 2019.
[34] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Empirical methods in natural language processing (EMNLP), pp- 1532-1543, 2014.
[35] Mohammad Taher Pilehvar and Jose Camacho- Wic: the word-in-conl in-ciieetntett dataset for evaluating context-sensitive meaning representations. arXiv preprint arXiv: arXivv/808.09/11 2018.
11

---

Published as a conference paper at ICLR 2021
[36] Mohammad Taher Pilehvar, Dimitri Kartsaklis, Victor Prokhorov, and Nigel Collier. Card-660- Cambridge rare word dataset-a reliable benchmark for infrequent word representation models. arXiv preprint 1808: 093008 2018.
[37] Simon Preissner and Aurélie Herbelot. To be fair: case for cognitively-inspired models of meaning, In CLiC-iI, 2019.
[38] Herbert Rubenstein and John B. Goodenough. Contextual correlates of synonymy. Communi- cations of the ACM, 8(10)627 1965.
[39] Chaitanya K. Ryali, John J. Hopfield, Leopold Grinberg, and Dmitry Krotov. Bio-Inspired Hashing for Unsupervised Similarity Search. arXiv preprint arXiv:2001.04907 2020.
[40] Roy Schwartz, Roi Reichart, and Ari Rappoport. Symmetric pattern based word embeddings for improved word similarity prediction. In Proceedings of the nineteenth conference on com- putational natural language learning, pp. 258-267, 2015.
[41] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sen- timent treebank. In Proceedings of the 2013 conference on empirical methods in natural lan- guage processing, pp. 1631-1642, 2013.
[42] RR Sokal. A statistical method for evaluating systematic relationships. Univ. Kansas, Sei. Bull., 38: 1409-1488,, 1958.
[43] Julien Tissier, Christophe Gravier, and Amaury Habrard. Near-lossless binarization of word embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7104-7111, 2019.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, pp. 5998 6008, 2017.
[45] Katrin Vogt, Yoshinori Aso, Toshihide Hige, Stephan Knapek, Toshiharu Ichinose, Anja B Friedrich, Glenn C Turner, Gerald M Rubin, and Hiromu Tanimoto. Direct neural pathways convey distinct visual information drosophila mushroom bodies. Elife, 5:e 2016
[46] Yair Weiss, Antonio Torralba, and Rob Fergus. Spectral hashing. Advances in neural informa- tion processing systems, 11:1733-1760 2008.
[47] Zhihao Zheng. Feng Li, Corey Fisher, Iqbal J Ali, Nadiya Sharifi, Steven Calle-Schuler, Joseph Hsu, Najla Masoodpanah, Lucia Kmecova, Tom Kazimiers, et al. Structured sampling of olfactory input by the fly mushroom body. bioRxxv, 2020.
12

---

Published as a conference paper at ICLR 2021
6 APPENDIX A. RELATED WORK.
Our work builds on several ideas previously discussed in the literature. The first idea is that fruit fly olfactory network can generate high quality hash codes for the input data random [8] and data- driven [39] cases. There are two algorithmic differences of our approach compared to these previous studies. First, our network uses representational contraction, rather than expansion when we go from the PN layer to the KCs layer. Second. [8] (3) construct hash codes for data coming from a single modality (e.g.. images, or word vectors). while the goal of the present paper is to learn correlations between two different "modalities": : target word and its context. The second idea pertains to the training algorithm for learning the PN-KCs synapses. We use a biologically plausible algorithm of 25 to do this, with modifications that take into account the wide range of frequencies of different words in the training corpus (we discuss these differences in section 2.1). Also, similarly to B 33) the algorithm of [5 used for learning the representations of the data, and not correlations between two types of data (context and target) as we do in this paper.
Another closely related work 377 uses the network of KCs with random weights for generating binary hash codes for individual words. There are several differences compared to our approach. First, our system the synaptic weights from PNs to KCs are learned and not random. We have found that learning these weights improves the performance compared to the random case. Second, unlike (37 (and unlike fruit flies), in our system the number of KCs is smaller than the number of PNs, so there is no representational expansion expansion essential for the system of [37], which uses random weights. Finally, our algorithm uses different encoding scheme at the level of PNs, see Fig. 2
we move into the "mushroom body". This
APPENDIX B. TRAINING PROTOCOLS AND HYPERPARAMETBA CHOICES.
The fruit fly network was trained on the Open Corpus [5]. which is a 32GB corpus of unstructured text containing approximately 6B tokens. Individual documents were concatenated and split into sentences. A collection of w-grams were extracted from each sentence by sliding a window of size along each sentence from the beginning the end. Sentences shorter than were removed. The vocabulary was composed of Nvoe 20000 most frequent tokens in the corpus.
Training was done for Nepoch- At each epoch all the w-grams were shuffled, organized in mini- batches, and presented to the learning algorithm. The learning rate was linearly annealed starting from the maximal value Eg at the first epoch to nearly zero the last epoch.
The training algorithm has the following hyperparameters: size of the KC layer K, window w, overall number of training epochs Nepoch, initial learning rate Eg, minibatch size, and hash length k. All models presented in this paper were trained for Napoch = 15. The optimal ranges of the hyperparameters are: learning rate is En R 10-1 5. 10-4; K R 200 - 600; w z 9 - 15; minibatch size 2000 15000; hash length is is reported for each individual experiment.
8 APPENDIX C. COMPARISON WITH BINARIZED GLOVE AND WORD2VEC.
Our aim here is to demonstrate that the fruit fly word embeddings are competitive with existing state-of-the-art binarization methods applied to GloVe and word2vec embeddings. We show this by evaluating the semantic similarity of static word embeddings, using several common benchmark datasets: WS353 [13), MEN [4, RW (28), SimLex (2I). RG-65 (38), and Mturk (18). These datasets contain pairs of words with human- annotated similarity scores between them. Specifically, we com- pare with GloVe 34 word embeddingg trained on Wiki2014 14 and Gigaword 5, Glo Ve embeddings trained on Open WebText Corpus [15] and word2vec e
Since our representations are binary (in contrast GloVe and word2vec), we binarize GloVe and word2vec embeddings and report their performance using number of common hashing methods, LSH/SimHHash 77 (random contructive projections followed by binarization based on sign), Ran- dExp ( (random expansive projections followed by k-winner take all binarization), ITQ [16] (it-
pretrrined embeddings: https://nlp.stanfrr..dd wtw.anfoord.uuu//ppcctt//o/ooooo
pretrained embeddings: http://coddegoogle. www.golg....mmcccc////ooeeeeeee.
13

---

Published as a conference paper at ICLR 2021
Method
Hash Length (k)
Hash Length
4
8
16
32
64
128
4
8
64
128
MEN (69.
099.5/88.1 I)
WS353
144.0/4.77
Ours
34,0
49.9
55.9
56.7
55.3
51.3
43.2
52.1
55.3
57.4
60.3
51.7
I.SH
16.9
23.7
35.6
42.6
53.6
63.4
8.2
20.7
30.0
34.7
43.9
50.3
RandExp
27.5
37.7
46.6
57.6
67.3
71.6
20.9
32.9
41.9
48.4
57.6
61.7
ITQ
0.1
7.7
10.5
16.5
30.4
50.5
-6.6
-6.1
-2.4
-4.4
6.1
24.8
SH
9.4
17.0
22.9
37.6
$2.9
65.4
15.4
14.1
19.5
32.3
43.1
58.4
PCAH
12.5
21.8
27.6
39.6
53.4
6.4
6.3
49.8
62.6
NLB
46.1
63.3
30.1
44.9
SIMLEX
(1.5729.8
RW (46.8/31.4
Ours
13.4
16.5
22.8
22.1
21.1
17.0
11.0
22.6
25.8
36.9
38
35.2
LSH
6.8
11.9
17.0
21.2
26.8
30.9
10.8
16.3
21.8
27.8
36.3
45.0
RandExp
10.4
17.2
22.8
28.5
32.4
35.2
19.9
21.3
30.9
40.5
47.6
53.3
ITQ
7.0
1.6
4.3
5.5
11.8
18.2
13.7
5.3
6.6
6.9
12.5
26.5
SH
9.3
15.6
15.9
17.0
23.1
31.2
22.6
21.5
24.3
28.8
36.1
45.8
PCAH
4.4
10.3
11.0
17.3
24.1
31.6
12.4
16.7
21.5
30.3
36,9
44.4
NLB
20.5
31.4
25.1
34.3
RG (74.2/67.6)
Mturk (57.5/61.9
Ours
24.0
40.4
51.3
62.3
63.2
55.8
44.0
49.0
52.2
60.1
57.7
55.2
LSH
21.2
35.4
44.6
55.1
63.1
70.1
16.0
23.1
33.2
35.6
42.7
55.5
RandExp
36.6
49.0
49.5
66.1
69.6
70.9
29.3
35.8
41.4
50.4
59.0
61.6
ITQ
-17.5
-8.9
26.3
41.7
50.5
66.2
9.9
7.8
10.1
17.7
32.8
47.3
SH
4.5
5.8
20.3
42.9
61.3
72.6
18.9
17.6
27.5
35.45
48.1
57.9
PCAH
1.9
19.8
40.9
53.3
68.2
13.5
15.1
27.1
41.7
46.5
56.2
Table 7: Evaluation on word similarity datasets. For each dataset and hash length, the best (sec- ond best) score is in bold (underlined). The performance for GloVe embeddings is reported next to the name of each dataset the format 300d/100d. Spearman's rank correlation coefficient is reported for common baselines that binarize GloVe (300d) embeddings together with our results. Hyperparameter settings for our algorithm: K 400, w 11.
erative quantization), SH (spectral hashing) [46 PCAH [16] (PCA followed by binarization based on sign). Where available, we include evaluation from NLB, "Near-Lossless Binarization" " [43 (autoencoder-based binarization).
Following previous work 43 model similarity score for binary representations is evaluated as sim(01, 12) (M11 rego) (nn))/, where 711 (nop) is the number of bits in U1 and 122 that are both (0), and n is the length of 01,2. Cosine similarity is used for real-valued representations. The results
report performance across a range of hash lengths {4, 8, 16, 32, 64, 128) For methods that incorporate randomness (LSH, RandExp, ITQ),, we report the average across runs. ITQ, SH and PCAH Tables and g were trained using the top 400k most frequent words. dataset that our method trained on) using the same vocabulary as our method uses.
reported in Tables 7 8 and For each dataset,
our method to GloVe trained on OpenWebTen. (same
Our binary word embeddings demonstrate competitive performance compared to published meth- ods for GloVe and word2vee binarization, and our algorithm can learn meaningful binary semantic representations directly from raw text. Importantly, our algorithm does not require training GloVe or word2vec embeddings first before binarizing them.
APPENDIX D. DETAILS OF TECHNICAL IMPLEMENTATION.
From the practical perspective, efficient implementation of the learning algorithm the fruit fly network requires the use of sparse algebra, atomic updates, and block-sparse data access. Our algorithm is implemented CUDA as back-end, while python is used as an interface with the main functions.
The typical memory footprint of our approach is very small. About 100 270MB GPU memory is allocated for the operators W pu, A and temporary fields; while approximately 140GB CPU memory nceded to store the input data, array of random numbers for shuffle operations and shuffled indices. For GPU implementation, the model data is stored in the GPU's memory, while the input data
14

---

Published as conference paper at ICLR 2021
Method
Hash Length (k)
Hash Length (k)
4
8
16
32
64
128
32
64
128
MEN
(75.5)
WS353
(66.5)
Ours
34.0
49.9
55,9
56.7
55.3
51.3
43.2
52.1
55.3
57.4
60.3
51.7
LSH
35.5
42.5
53.6
63.4
68.4
72.2
26.0
34.7
43.9
50.3
56.0
58.6
RandExp
24.2
34.6
45.8
57.5
66.1
71.7
23.5
34.3
37.3
48.0
57.6
63.7
ITQ
9.2
13.3
25.1
41.5
57.6
68.5
16.0
18.1
22.5
30.2
43.9
54.8
SH
7.2
15.8
31.3
46.9
62.3
69.4
3.3
9.6
22.7
34.1
50.0
54.7
PCAH
5.3
18.6
37.7
52.0
63.9
71.6
24.9
38.5
42.0
52.1
59.3
SIMLEX
(41.7)
RW
(61.3)
Ours
13.4
16.5
22.8
22.1
21.1
17.0
11.0
22.6
25.8
36.9
38.6
35.2
LSH
17.0
21.2
26.8
30.9
34.4
35.1
21.8
27.8
36.3
45.0
49.6
52.1
RandExp
17.6
24.4
29.2
32.6
38.0
39.8
24.7
27.7
39.8
46.8
52.3
55.6
ITQ
3,25
5.7
6.2
14.9
23.1
31.5
17.4
15.7
19.1
33.5
45.6
53.4
SH
-3.6
3.6
10.4
17.0
23.7
32,4
14.6
22.8
28.7
37.9
43.5
52.4
PCAH
-2.9
2.5
11.8
17.0
24.0
36.0
15.0
21.5
28.8
35.4
46.4
50.6
RG (75.4)
Mturk (69.8)
Ours
24.0
40.4
51.3
62.3
63.2
55.8
44.0
49.0
52.2
60.1
57.7
55.2
LSH
44.6
55.1
63.1
70.1
76.4
75.8
33.1
35.6
42.7
55.5
58.6
62.4
RandExp
30.4
42.0
48.6
59.1
70.2
74.6
22.7
34.8
42.0
45.9
57.9
61.2
ITQ
32.8
49.7
31.5
55.9
62.2
71,6
22.5
21.3
42.3
46.9
59.3
60.7
SH
18.0
30.6
36.0
48.8
56.9
75.8
21.9
27.4
41.8
51.2
58.8
58.0
PCAH
20.8
22.9
40.6
36.5
59.0
71.2
23.6
34.4
45.5
55.7
64.2
60.5
Table 8: Evaluation word similarity datasets, analogous to Table dings.
for 300d word2vec embed-
Method
Hash Length (k)
Hash Length (k)
4
8
16
32
64
128
4
8
16
32
64
128
MEN
(76.4)
WS353
(72.2)
Ours
34.0
49.9
55.9
56.7
55.3
51.3
43.2
52.1
55.3
57.4
60.3
51.7
LSH
23.6
29.1
37.4
49.6
60.6
67.0
20.2
29.0
35.5
47.5
53.3
61.4
RandExp
28.4
40.3
52.3
62.5
67.7
71.0
30.5
40.0
48.1
57.9
63.3
67.5
ITQ
26.9
33.9
46.3
56.1
64.1
70.3
25.9
33.7
44.5
67.6
SH
23.8
28.7
44.1
54.7
62.1
69.7
18.1
25.7
40.1
51.8
60.9
62.9
PCAH
26.0
30.1
46.3
57.9
67.5
72.4
21.2
30.5
43.8
50.7
61.1
59.9
SIMLEX
(34.0)
RW
(54.5)
Ours
13.4
16.5
22.8
22.1
21.1
17.0
11.0
22.6
25.8
36.9
38.6
35.2
LSH
8.0
16.8
19.0
24.8
26.7
32.9
16.2
21.0
26.1
33.6
40.8
47.0
RandExp
10.1
17.3
23.4
26.6
29.7
31.3
22.0
28.8
34.1
43,9
46.3
51.5
ITQ
7.3
13.8
14.4
20.9
25.3
30.3
24.5
26.8
34.8
43.2
49.1
51.5
12.1
14.2
17.5
26.4
36.0
19.7
24.8
32.9
38.7
46.7
PCAH
11.5
13.8
16.4
22,6
31.1
38.6
19.7
24.8
32.9
38.7
45.4
46.7
RG (78.7)
Mturk
(71.1)
Ours
24.0
40.4
51.3
62.3
63.2
55.8
44.0
49.0
52.2
57.7
55,2
LSH
25.5
24.9
34.6
62.1
61.8
73.5
18.3
31.3
31.4
42.9
56.5
60.7
RandExp
28.7
45.6
47.3
63.7
67.8
70.8
28.3
41.3
50.1
56.5
65.4
67.1
ITQ
21.4
32.7
50.4
57.7
67.6
70.3
26.3
41.4
53.2
61.2
67.1
68.9
SH
39.8
45.6
50.0
50.2
62.3
68.6
20.3
35.9
51.9
61.9
59.1
61.3
PCAH
45.0
50.0
49.2
46.8
66.6
24.9
40.7
55.7
64.3
64.4
60.5
Table Evaluation on word similarity datasets, analogous to Table The 300d GloVe embeddings trained from scratch on the same Open WebText dataset our algorithm.
stored in the CPU memory. The parallelization strategy in our implementation is based on two aspects. First, each minibatch of data is divided d into smaller sub-minibatches which are processed on different GPUs. Second, all the operations (dense-sparse matrix multiplications, arg max operation, and weight updates) are executed in parallel using multiple threads.
15

---

Published as a conference paper at ICLR 2021
10 APPENDIX E. QUALITATIVE EVALUATION OF CONTEXTUAL EMBEDDINGS.
In order to evaluate the quality of contextualized embeddings we have created an online tool, which we are planning to release with the paper, that allows users to explore the representations learned by model for various inputs (context-targer pairs). For a given query the tool returns the word cloud visualizations for each of the four top activated Kenyon cells. We show some examples of the outputs produced by this tool in Fig. Each query is used to generate bag of words input vector v This vector is then used to compute the activations of KCs using Wy,.v Top four KCs with the
highest activations are selected. The corresponding four weight vectors are used to generate four probability distributions of individual words learned by those KCs by passing the weights through softmax function. For example, for one of those vectors with index ja, the probability distribution is computed prob; SM(W, pa)) These probability distributions for the top four activated KCs are visualized as word clouds. In computing the softmax only the target block of the weight vector was used (we have checked that using only the context block gives qualitatively similar word clouds).
Query: Entertainment industry shares rise following the premiere of the mass destruction weapon documentary
ECOOSON
Harodacten
Boodilest
sroducts
goods
nuclear
amiin
weapons mass
shootings
hoasing
value
reand
half liss chemical third
sundance cannes
prfone
market all stock currency share
Ictcole
incarceraion
Query: European Court of Human Rights most compelling cases
Enstum
soviet- union eanpees
innohing
mat wand
human
court <NUM>
rights
supreme Abdhe
SUBPOCSO
J01
Query: Senate majority leader discussed the issue with the members of the committee
mitch- meconnell
-ose democratic party
congress
white
family
Mae leally
Figure 6: Examples of three queries and corresponding word cloud visualization for top four activated KCs (by each query).
The results indicate that the fruit fly network indeed has learned meaningful representations. Con- sider for example the first query. The sentence: "Entertainment industry shares rise following the premiere of the mass destruction weapon documentary" results in the four top activated KCs shown in Fig. 0 The top activated KC has the largest weights for the words "weapon", "mass", etc. The second activated KC sensitive to the words "market", "stock", etc. This illustrates how the fruit fly network processes the queries. In this example the query refers to several distinct combinations of concepts: "weapon of mass destruction", "stock market", "movie industry". Each of those concepts has dedicated KC responsible for it. As one can see the responses are not perfect. For example in this case one would expect to have the 4-th highest activated KC, which is responsible for the "movie industry" concept have a higher activation than the 3-rd highest KC, which is responsible for the types of "weapons of mass destruction". But overall all the concepts picked by the KCs are meaningful and related to the query.
16

---

Published as a conference paper at ICLR 2021
11 APPENDIX F. DETAILS OF GLOVE RETRAINING
To directly compare our method to GloVe, we trained an uninitialized GloVe model on the same WpenWebText corpus using the code provided by the original GloVe authors [34 This model was optimized to have the same vocab size as our model (the 20k most frequent tokens), used an embedding size of 300, and a window size of 15. The model was trained for 180 iterations at about minutes, 20 seconds per iteration on 16 threads, resulting in the total training time of approximately 10 hours.
12 ACKNOWLEDGEMENTS
We are thankful to L.Amini, S.Chang. D.Cox, J.Hopfield, Y.Kim, and H.Strobelt for help- ful discussions. This work was supported by the Rensselaer-IBM AI Research Collaboration (http://airc.rpi.edu)) part of the IBM Al Horizons Network (http://ibmm. bit.:/ii////oaa
(htt:://ibm.biz/AHHorzi (illoiioon)//
17


Reviews:
[{"review_id": "mEyuIFKkWdH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe authors present a formalization of a simple biological network (the mushroom body of the fruit fly), that allows very efficient “biologically inspired” word embeddings. They train this network to generate both static (context-independent) and context dependent embeddings, and evaluate these embeddings using several metrics comparing mainly to GloVe embeddings, and to some extent to BERT embeddings. Although the results are sometimes inferior, they are overall comparable, and importantly achieved at significantly lower computational resources. The main contribution of this work is not this specific network formalization (which is nice), but rather demonstrating that formalizing biological networks can generate more efficient algorithms, that achieve results comparable to the complex algorithms used ubiquitously.\n\nStrengths:\n+ The approach of the paper, namely generating word embeddings using a formalization of a well-known biological network, is interesting and inspiring\n+ The authors evaluate the embeddings using both intrinsic and extrinsic evaluation methods, which is important\n+ The learned embeddings are surprisingly strong, given the small and computationally efficient network. Furthermore, the embeddings seem to separate concepts well. These results support the idea that correlations between words and contexts, can indeed be extracted from raw text (in an unsupervised manner) by this network. I find this important because although the biological network of the fruit fly clearly did not evolve to perform NLP tasks, the fact that it evolved to process input from several modalities seems to allow it to learn efficient embeddings. \n\nWeaknesses:\n- I would have liked to see comparison of the resulting embeddings to more than GloVe. E.g. see https://arxiv.org/pdf/1901.09785.pdf, who discuss intrinsic and extrinsic evaluations of embeddings and compare to many embeddings. For example, GloVe is actually not the best for word similarity. \n- For the extrinsic tasks describes in the document classification section, I would have liked to see a comparison to BERT as well, not just GloVe (or even other embeddings). I think these tasks can indeed use context.  \n- I think it would also be useful to evaluate on more extrinsic tasks, although given the space constraints I understand this may be difficult.\n- Finally, it would have been good to see more analysis of the embeddings, e.g. by doing error analysis in one or more of the evaluation methods.\n\nRecommendation:\nI vote for accepting this paper. I very much like the approach of looking at biological networks and trying to learn from them how one can efficiently perform computations. I would have liked to see a more in-depth comparison and analysis in some places, but the results as-is are promising, interesting and inspiring enough to be of interest to the ICLR community.\n\nQuestions and minor comments:\n1. In Figure 2, the legend reads to me as if the context words are in light blue, but in the figure both target word and context words are in light blue.\n2. I think it would be helpful to make a comment that context in this paper is essentially Bag Of Words, as there is no positional information.\n3. In the semantic similarity section, the authors refer to the supplement for binarized GloVe embedding results. I looked at the supplemental results and I found them compelling. I recommend at least saying the conclusion in the main paper (that your binary word embeddings are competitive with binarized GloVe embeddings), rather than just sending the reader to the supplement. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LxDr8fD0h85", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Although the paper does not say so, my understanding is that the proposed word embedding method actually first perform Kmeans-like clustering on the context vector shown in Figure 2. In the binary word embedding of each word, we set the dimensions corresponding to the k closest cluster centers to 1 and 0 otherwise. Most parts of the paper are describing how the simple word embedding method is related to the neural system of a fruit fly and showing the method achieves comparable performance compared with GloVe in word similarity tasks, context-sensitive word similarity datasets, and document classification tasks.\n\nPros:\n1. It is interesting to see the connection between a biological neural system and word embedding.\n2. As far as I know, the method is novel.\n3. Such a simple and biologically plausible method could reach reasonable performance. \n\nCons:\n1. The experiments have not demonstrated that the method is a useful word embedding method for NLP researchers. Some of the experiment designs, comparisons, and presentations are misleading.\n2. Some related work and comparisons are missing.\n3. The authors actually do not explain why this method works well in machine learning perspectives (similar to a biological neural system is not a very strong explanation for many people).\n\nClarity:\nThe paper is easy to understand, but it does not explain why the method works from the ML perspective.\n\nOriginality:\nAs far as I know, the method is novel.\n\nSignificance of this work:\nThis might have a large impact on the neuroscience field.\n\nThe method is compared with GloVe most of the time. First, GloVe actually does not perform well in word similarity tasks compared with word2vec/SGNS [3]. In terms of document classification, the results are mixed and I think the authors should try more datasets to know which method is better. In addition, GloVe is trained on a different dataset, so the result is not very comparable. Second, GloVe is a dense method. The proposed method should be compared with binary word embedding or compressed word embedding methods such as [1,2]. It will be even better if you can show that it is better than sparse representation such as PPMI in [3], even though PPMI is not binary and uses much more dimensions (you might want to use the dot product values instead of binary value in Equation (4) to improve your method and make the comparison fairer). \n\nAs I mention at the beginning, I believe the word embedding is actually a binary projection to Kmeans-like cluster centers in the context vector space. Here is why. Let's first assume W_{mu} and v^A are normalized and ignore the normalization term /p to build the connection. Then, the l2 distance is (1 - dot product). Equation (1) tells us that we are minimizing the l2 distance between v^A and its closest cluster center (i.e., W_{mu}). Finally, Equation (4) clearly describes the top k projection. If I am right, I think adding this perspective will make readers understand the method better. If I am wrong, please explain why and try to explain what the method is actually doing in some machine learning perspectives. \n\nIf I am right, we will find the description section 3.2 might be misleading. The main reason that the average cosine similarity within the cluster is higher might be because the binarization projection makes the word embeddings locally collapse to small cluster centers, which makes the word embeddings cannot capture the fine-grained statistics details. You can visualize the word embedding in a 2D space to see if I am right. If I am right, this is not an advantage in my opinion.\n\nIf the authors could show that the proposed method is state of the art compared with binary word embeddings in a fair setting (e.g., train on the same dataset), I will vote acceptance. Otherwise, I am very likely to keep my weak rejection vote.\n\nMinor:\n1. The definition of p before Equation (1) is unclear. Why do you have a mod here?\n2. The optimization method in Equation (2) is unclear. What does dt mean? Is t the time step in iterative optimization? Is this a kind of gradient descent? I guess the differential equation comes from neural science and it needs more explanations to make NLP researchers understand. It would be better if the author can compare with other optimization methods such as gradient descent and EM algorithm for Kmeans clustering.\n\n\n[1] Tissier, Julien, Christophe Gravier, and Amaury Habrard. \"Near-lossless binarization of word embeddings.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.\n[2] Wang, Yuwei, et al. \"Biological Neuron Coding Inspired Binary Word Embeddings.\" Cognitive Computation 11.5 (2019): 676-684.\n[3] Levy, Omer, Yoav Goldberg, and Ido Dagan. \"Improving distributional similarity with lessons learned from word embeddings.\" Transactions of the Association for Computational Linguistics 3 (2015): 211-225.", "extra_scores": {}, "extra_text": {}}, {"review_id": "F472BKufg8y", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a simplified model of a biological system, and asks the question of whether it can learn unsupervised word embeddings. The objective function and its optimization are clear and unreasonable. (Note: I do not have the expertise to comment on whether this is an accurate characterization of the working of the biological system.) The authors connect the optimization problem to other recent learnings in this area such as sparse learnt projection -- this perspective is appreciated. \n\nTo establish the quality of the embeddings, the authors compare them to GloVe and BERT embeddings on word specific tasks like similarity and contextual tasks on datasets like SCWS. In these comparisons, the quality of embeddings is relatable to GloVe, and inferior to BERT. There is also a small qualitative demonstration of the embeddings (Fig. 3) This tells us that the embeddings are meaningful, and their quality is comparable to Glove, if not the newer generation embeddings. I think this is a good outcome for the model they develop.\n\n For these reasons, I am generally in favor of accepting the paper. However, I have reservations about particular aspects, and request the authors to edit the following content.\n1. SOTA claims. GloVe is a great candidate to compare a new methodology to, but is not SOTA for unsupervised embeddings. There are many other unsupervised non-DL methods (e.g. Word2vec) which do better than GloVe on specific tasks. It is important to clarify and cite appropriate SoTA for each task as of the time of writing.\n\n2. I am not sure the comparison to GloVe for contextual evaluation is the most appropriate. There are other works (e.g. Word2Sense, ACL'19 and citations therein) for stronger contextual embeddings. Please bechmark against this.\n\n3. I feel the comparison with BERT is mostly irrelevant. I think comparison with GloVe and related work are sufficient to show that the new methodology has something going for it. I would not have expected it to compete with BERT like models nor serve all their use cases. Therefore, claims that training is faster than BERT are also mostly unnecessary. \n\n4. I would instead place more emphasis on qualitive and quantitative evidence that the embeddings capture contextual meanings, polysemy etc. Consider expanding section 3.2 with more evidence.\n\nWith these changes, I would make a stronger recommendation.\n\nRevised rating to 7 after revision.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work is likely to lead to more connections between machine learning and neuroscience at a fine-grained level where ML methods can help explain and understand neural circuits.

To encourage this, it would be helpful if authors described the biology of the PN-KC-APL network and the known constraints over possible formalizations of that network. The authors present one formalization, but little discussion is given toward the design space for such models. Are there other possible ways to describe the PN-KC-APL network? Are all alternate ways to do so equivalent to the model presented here? What properties are unknown and how could they affect the formalization presented here?

Overall, reviewers agree this is a good submission.
Decision: Accept (Poster)