Venue: EMNLP 2017

# Sound-Word2Vec:

Learning Word Representations Grounded in Sounds

Ashwin Vijayakumar1, Ramakrishna Vedantam2 and Devi Parikh1

###### Abstract

To be able to interact better with humans, it is crucial for machines to understand sound - a primary modality of human perception. Previous works have used sound to learn embeddings for improved generic semantic similarity assessment. In this work, we treat sound as a first-class citizen, studying downstream 6textual tasks which require aural grounding. To this end, we propose _sound-word2vec_ - a new embedding scheme that learns specialized word embeddings grounded in sounds. For example, we learn that two seemingly (semantically) unrelated concepts, like _leaves_ and _paper_ are similar due to the similar _rustling_ sounds they make. Our embeddings prove useful in textual tasks requiring aural reasoning like text-based sound retrieval and discovering Foley sound effects (used in movies). Moreover, our embedding space captures interesting dependencies between words and onomatopoeia and outperforms prior work on aurally-relevant word relatedness datasets such as AMEN and ASLex.

## 1 Introduction

Sound and vision are the dominant perceptual signals, while language helps us communicate complex experiences via rich abstractions. For example, a novel can stimulate us to mentally construct the image of the scene despite having never physically perceived it. Indeed, language has evolved to contain numerous constructs that help depict visual concepts. For example, we can easily form the picture of a _white, furry_ cat with _blue_ eyes via. a description of the cat in terms of its visual attributes Lampert et al. (2009); Parikh and Grauman (2011).

Need for Onomatopoeia.However, how would one describe the auditory instantiation of cats? While a first thought might be to use audio descriptors like _loud_, _shrill_, _husky etc._ as mid-level constructs or "attributes", arguably, it is difficult to precisely convey and comprehend sound through such language. Indeed, Wake and Asahi (1998) find that humans first communicate sounds using "onomatopoeia" - words that are suggestive of the phonetics of sounds while having no explicit meaning _e.g.__meow, tic-toc._ When asked for further explanation of sounds, humans provide descriptions of potential sound sources or impressions created by the sound (_pleasant, annoying,_ etc.)

Need for Grounding in Sound.While onomatopoeic words exist for commonly found concepts, a vast majority of concepts are not as perceptually striking or sufficiently frequent for us to come up with dedicated words describing their sounds. Even worse, some sounds, say, musical instruments, might be difficult to mimic using speech. Thus, for a large number of concepts there seems to be a gap between sound and its counterpart in language Sundaram and Narayanan (2006). This becomes problematic in specific situations where we want to talk about the heavy tail of concepts and their sounds, or while describing a particular sound we want to create as an effect (say in movies). To alleviate this, a common literary strategy is to provide metaphors to more relatable exemplars. For example, when we say, "He thundered angrily", we compare the person's angry speech to the sound of thunder to convey the seriousness of the situation. However, without this grounding in sound, thunder and anger both appear to be seemingly unrelated concepts in terms of semantics.



Contributions.In this work, we learn embeddings to bridge the gap between sound and its counterpart in language. We follow a retrofitting strategy, capturing similarity in sounds associated with words, while using distributional semantics (from word2vec) to provide smoothness to the embeddings. Note that we are not interested in capturing phonetic similarity, but the grounding in sound of the concept associated with the word (say "rustling" of leaves and paper.) We demonstrate the effectiveness of our embeddings on three downstream tasks that require reasoning about related aural cues:

1. Text-based sound retrieval - Given a textual query describing the sound and a database containing sounds and associated textual tags, we retrieve sound samples by matching text (Sec. 5.1) 2. Foley Sound Discovery - Given a short phrase that outlines the technique of producing Foley sounds1, we discover other relevant words (objects or actions) which can produce similar sound effects (Sec. 5.2)

Footnote 1: Foley sounds are sound effects (typically ambient sounds) that are added to movies in the post-production stage to make actions or situations appear more realistic. These sounds are generally created using easily available proxy objects that mimic the sound of the true situation being depicted. For example, sound of breaking celery sticks is used to create the effect of breaking bones.

3. Aurally-relevant word relatedness assessment on AMEN and ASLex Kiela and Clark (2015) (Sec. 5.3)

We also qualitatively compare with word2vec to highlight the unique notions of word relatedness captured by imposing auditory grounding.

## 2 Related Work

Audio and Word Embeddings.Multiple works in the recent past Bruni et al. (2014); Lazaridou et al. (2015); Lopopolo and van Miltenburg (2015); Kiela and Clark (2015); Kottur et al. (2016) have explored using perceptual modalities like vision and sound to learn language embeddings. While Lopopolo and van Miltenburg (2015) show preliminary results on using sound to learn distributional representations, kiela2015multitask build on ideas from Bruni et al. (2014) to learn word embeddings that respect both linguistic and auditory relationships by optimizing a joint objective. Further, they propose various fusion strategies to combine knowledge from both the modalities. Instead, we "specialize" embeddings to exclusively respect relationships defined by sounds, while initializing with word2vec embeddings for smoothness. Similar to previous findings Melamud et al. (2016), we observe that our specialized embeddings outperform language-only as well as other multi-modal embeddings in the downstream tasks of interest.

In an orthogonal and interesting direction, other recent works Chung et al. (2016); He et al. (2016); Settle and Livescu (2016) learn word representations based on similarity in their pronunciation and not the sounds associated with them. In other words, phonetically similar words that have near identical pronunciations are brought closer in the embedding space (_e.g., flower_ and _flour_). Sundaram and Narayanan (2006) study the applicability of onomatopoea to obtain semantically meaningful representations of audio. Using a novel word-similarity metric and principal component analysis, they find representations for sounds and cluster them in this derived space to reason about similarities. In contrast, we are interested in learning word representations that respect aural-similarity. More importantly, our approach learns word representations for in a data-driven manner without having to first map the sound or its tags to corresponding onomatopoecic words.

Multimodal Learning with Surrogate Supervision.Kottur et al. (2016) and Owens et al. (2016) use a surrogate modality to induce supervision to learn representations for a desired modality. While the former learns word embeddings grounded in cartoon images, the latter learns visual features grounded in sound. In contrast, we use sound as the surrogate modality to supervise representation learning for words.

## 3 Datasets

Freesound.We use the freesound database Font et al. (2013), also used in prior work Kiela and Clark (2015); Lopopolo and van Miltenburg (2015) to learn the proposed sound-word2vec embeddings. Freesound is a freely available, collaborative dataset consisting of user uploaded sounds permitting reuse. All uploaded sounds have human descriptions in the form of tags and captions in natural language. The tags contain a broad set of relevant topics for a sound (_e.g., ambience, electronic, birds, city, neverb_) and 

captions describing the content of the sound, in addition to details pertaining to audio quality. For the text-based sound retrieval task, we use a subset of 234,120 sounds from this database and divide it into training (80%), validation (10%) and testing splits (10%). Further, for foley sound discovery, we aggregate descriptions of foley sound production provided by sound engineers (epicsound, accessed 23-Jan-2017; Singer, accessed 23-Jan-2017) to create a list of 30 foley sound pairs, which forms our ground truth for the task. For example, the description to produce a foley "driving on gravel" sound is to record the "crunching sound of plastic or polyethene bags".

_AMEN and ASLex._ AMEN and ASLex [6] are subsets of the standard MEN [5] and SimLex [9] word similarity datasets consisting of word-pairs that "can be associated with a distinctive associated sound". We evaluate on this dataset for completeness to benchmark our approach against previous work. However, we are primarily interested in the slightly different problem of relating words with similar auditory instantiations that may or may not be semantically related as opposed to relating semantically similar words that can be associated with some common auditory signal.

## 4 Approach

We use the Freesound database to construct a dataset of tuples \(\{s,T\}\), where \(s\) is a sound and \(T\) is the set of associated user-provided tags. We then aim to learn an embedding space for the tags that respects auditory grounding using sound information as cross-modal context - similar to word2vec [12] that uses neighboring words as context / supervision. We now explain our approach in detail.

_Audio Features and Clustering._ We represent each sound \(s\) by a feature vector consisting of the mean and variance of the following audio descriptors that are readily available as part of Freesound database:

* computed as given in [1].
* computed according to [1].
* Dissonance: It measures the perceptual roughness of the sound [5].
* Zero-crossing Rate: It is the percentage of sign changes between consecutive signal values and is indicative of noise content.
* Spectral Spread: This feature is the concatenation of the \(k\)-order moments of the spectrum, where \(k\in\{0,1,2,3,4\}\).
* Pitch Salience: This feature helps discriminate between musical and non-musical tones. While, pure tones and unpitched sounds have values near 0, musical sounds containing harmonics have higher values [13].

We then use \(K\)-Means algorithm to cluster the sounds in this feature space to assign each sound to a cluster \(C(s)\in\{1,\ldots K\}\). We set \(K\) to 30 by evaluating the performance of the embeddings on text-based audio-retrieval on the held out validation set. Note that the clustering is only performed once, prior to representation learning described below.

_Representation Learning._ We represent each tag \(t\in T\) using a \(|\mathcal{V}|\) dimensional one-hot encoding denoted by \(\mathbf{v}_{t}\), where \(\mathcal{V}\) is the set of all unique tags in the training set (the size of our dictionary). This one-hot vector \(\mathbf{v}_{t}\) is projected into a \(D\)-dimensional vector space via \(W_{P}\in\mathbb{R}^{|\mathcal{V}|\times D}\), the _projection matrix_. This projection matrix computes the representation for each word in \(\mathcal{V}\). The idea of our approach is to use \(W_{P}\) to accurately predict cluster assignments (for sounds associated with words), which enforces grounding in sound. For each data-point, we obtain the summary of the tags \(T\), by averaging the projections of all tags in the set as \(\frac{1}{|T|}\sum_{t\in T}W^{\prime}_{P}\mathbf{v}_{t}\). We then transform the so obtained summary representation via a

Figure 1: The model used to learn the proposed sound-word2vec embeddings. The projection matrix \(W_{P}\) containing that is used as the sound-word2vec embedding is learned by training the model to accurately predict the cluster assignment of the sound.

 

linear layer (with parameters and pass the output through the softmax function to obtain a distribution, over the K sound clusters. We perform maximum-likelihood training for the correct cluster assignment , optimizing for parameters and:

(1)

We use SGD with momentum to optimize this objective which essentially is the cross-entropy between cluster assignments and . We set to be consistent with the publicly available word2vec embeddings.

_Initialization._ We initialize with word2vec embeddings Mikolov et al. (2013) trained on the Google news corpus dataset with ~3M words. We fine-tune on a subset of 9578 tags which are present in both Freesound as well as Google news corpus datasets, which is 55.68% of the original tags in the Freesound dataset. This helps us remove noisy tags unrelated to the content of the sound.

In addition to enlarging the vocabulary, the pretraining helps induce smoothness in the sound-word2vec embeddings - allowing us to transfer semantics learnt from sounds to words that were not present as tags in the Freesound database. Indeed, we find that word2vec pre-training helps improve performance (Sec. 5.3). Our use of language embeddings as an initialization to fine-tune (specialize) from, as opposed to formulating a joint objective with language and audio context Kiela and Clark (2015) is driven by the fact that we are interested in embeddings for words grounded in sounds, and not better generic word similarity.

## 5 Results

_Ablations._ In addition to the language-only baseline word2vec Mikolov et al. (2013), we compare against tag-word2vec - that predicts a tag using other tags of the sound as context, inspired by Font et al. (2014). We also report results with a randomly initialized projection matrix (sound-word2vec(r) to evaluate the effectiveness of pretraining with word2vec.

_Prior work._ We compare against previous works Lopopolo and van Miltenburg (2015) and Kiela and Clark (2015). While the former uses a standard bag of words and SVD pipeline to arrive at distributional representations for words, the latter trains under a joint objective that respects both linguistic and auditory similarity. We use the openly available implementation for Lopopolo and van Miltenburg (2015) and re-implement Kiela and Clark (2015) and train them on our dataset for a fair comparison of the methods. In addition, we show a comparison to word-vectors released by Kiela and Clark (2015) in the supplementary material. All approaches use an embedding size of 300 for consistency.

### Text-based Sound Retrieval

Given a textual description of a sound as query, we compare it with tags associated with sounds in the database to retrieve the sound with the closest matching tags. Note that this is a purely textual task, albeit one that needs awareness of sound. In a sense, this task exactly captures what we want our model to be able to do - bridge the semantic gap between language and sound. We use the training split (Sec. 3) to learn the sound-word2vec vectors, validation to pick the number of clusters (K), and report results on the test split. For retrieval, we represent sounds by averaging the learnt embeddings for the associated tags. We embed the caption provided for the sound (in the Freesound database) in a similar manner, and use it as the query. We then rank sounds based on the cosine similarity between the tag and query representations for retrieval. We evaluate using standard retrieval metrics - Recall@{1,10,50,100}. Note that the entire testing set (\(\approx\)10k sounds) is present in the retrieval pool. So, recall@100 corresponds to obtaining the correct result in the top of the search results, which is a relatively stringent evaluation criterion.

_Results._ Table. 1 shows that our sound-word2vec embeddings outperform the baselines. We see that specializing the embeddings for sound using our two-stage training outperforms prior workKiela and Clark (2015) and Lopopolo and van Miltenburg (2015)), which did not do specialization. Among our approaches, tag-word2vec performs second best - this is intuitive since the tag distributions implicitly capture auditory relatedness (a sound may have tags cat and meow), while word2vec and sound-word2vec(r) have the lowest performance.

 



### Foley Sound Discovery

In this task, we evaluate how well embeddings identify matching pairs of target sounds (_flapping bird wings_) and descriptions of Foley sound production techniques (_rubbing a pair of gloves_). Intuitively, one expects sound-aware word embeddings to do better at this task than sound-agnostic ones. We setup a ranking task by constructing a set of original Foley sound pairs and decoy pairs formed by pairing the target description with every word from the vocabulary. We rank using cosine similarity between the average word-vectors in each member of the pair. A good embedding is one in which the original Foley sound pair has the lowest rank. We use the mean rank of the Foley sound in the dataset for evaluation. We transfer the embeddings from Sec. 5.1 to this task, without additional training.

ResultsWe find that Sound-word2vec performs the best with a mean rank of 34.6 compared to other baselines tag-word2vec (38.9), sound-word2vec(r) (114.3) and word2vec (189.45). As observed previously, the second best performing approach is tag-word2vec. Lopopolo and van Miltenburg (2015) and Kiela and Clark (2015) perform worse than tag-word2vec with a mean rank of 48.4 and 42.1 respectively. Note that random chance gets a rank of \((|{\cal V}|+1)/2=4789.5\).

### Evaluation on AMEN and ASLex

AMEN and ASLexKiela and Clark (2015) are subsets of the MEN and SimLex-999 datasets for word relatedness grounded in sound. From Table 2, we can see that our embeddings outperform Kiela and Clark (2015) on both AMEN and ASLex. These datasets were curated by annotating concepts related by sound; however we observe that relatedness is often confounded. For example, _(river, water)_, _(automobile, car)_ are marked asurally related however they do not stand out asurally-related examples as they are already semantically related. In contrast, we are interested in how onomatopoeic words relate to regular words (Table 3), which we study by explicit grounding in sound. Thus while we show competitive performance on this dataset, it might not be best suited for studying the benefits of our approach.

## 6 Discussion and Conclusion

We show nearest neighbors in both sound-word2vec and word2vec space (Table 3) to qualitatively demonstrate the unique dependencies captured due to auditory grounding. While word2vec maps a word (say, _apple_) to other semantically similar words (other fruits), similar 'sounding' words (_chips_) or onomatopoeia (_munch_) are closer in our embedding space. Moreover, onomatopoeic words (say, _boom_ and _slam_) are mapped to relevant objects (_explosion_ and _door_). Interestingly, parts (_e.g._, _lock, latch_) and actions (_closing_) are also closer to the onomatopoeic query - exhibiting an understanding of the auditory scene.

ConclusionIn this work we introduce a novel word embedding scheme that respects auditory grounding. We show that our embeddings provide strong performance on text-based sound retrieval, Foley sound discovery along with intuitive nearest neighbors for onomatopoeia that are tasks in text requiring auditory reasoning. We hope our work motivates further efforts on understanding and relating onomatopoeic words to "regular" words.

#### Acknowledgments

We thank the Freedman team and Frederic Font in particular for helping us with the Freewood API by Leo thank Kushis Gupta and Stefan Lee for fruitful discussions. This work was funded in part by NSF CAREER, ONR Grant N00014-16-1-2713, ONR YI, Susan Fallbrouchig, Alain Distinguished Investigatorio, Google Faculty Research Award, Amazon Academic Research Award to DP.

\begin{table}
\begin{tabular}{c c c c c} \hline Embedding & \multicolumn{4}{c}{Recall} \\  & 01 & 010 & 850 & 810.02 \\ \hline weed2vec & 6.47\({}_{100}\) & 14.25\({}_{100}\) & 21.72\({}_{100}\) & 26.03\({}_{100}\) \\ tag-word2vec & 6.95\({}_{100}\) & 15.02\({}_{100}\) & 22.43\({}_{100}\) & 22.71\({}_{100}\) \\ sound-word2vec & 6.94\({}_{100}\) & 15.03\({}_{100}\) & 22.94\({}_{100}\) & 22.51\({}_{100}\) \\ (Lepeppoyden van Moltenburg, 2015) & 6.48\({}_{100}\) & 15.09\({}_{100}\) & 21.82\({}_{100}\) & 26.89\({}_{100}\) \\ (Kai et al., 2015) & 6.53\({}_{200}\) & 15.21\({}_{100}\) & 21.92\({}_{100}\) & 22.74\({}_{100}\) \\ sound-word2vec & **7.11\({}_{100}\)** & **15.80\({}_{100}\)** & 23.44\({}_{100}\) & **25.67\({}_{100}\)** \\ \hline \end{tabular}
\end{table}
Table 1: Text-based sound retrieval (_higher_ is better). We find that our sound-word2vec model outperforms all baselines.

\begin{table}
\begin{tabular}{c c c} \hline Embedding & \multicolumn{2}{c}{Spearman Correlation \(\rho_{*}\)} \\  & AMEN & ASLex \\ \hline (Loppolo and van Miltenburg, 2015) & 0.410\({}_{100}\) & 0.237\({}_{100}\) \\ (Kiela and Clark, 2015) & 0.648\({}_{100}\) & 0.366\({}_{100}\) \\ sound-word2vec & **0.674\({}_{100}\)** & 0.05 & **0.391\({}_{100}\)** \\ \hline \end{tabular}
\end{table}
Table 2: Comparison to state of the art AMEN and ASLex datasets Kiela and Clark (2015) (_higher_ is better). Our approach performs better than Kiela and Clark (2015).





## References

* Akkermans et al. (2009) Vincent Akkermans, Joan Serra, and Perfecto Herrera. 2009. Shape-based spectral contrast descriptor. In _Proc. of the Sound and Music Computing Conf.(SMC)_.
* Bruni et al. (2014) Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. _Journal of Artificial Intelligence Research (JAIR)_.
* Chung et al. (2016) Yu-An Chung, Chao-Chung Wu, Chia-Hao Shen, Hung-yi Lee, and Lin-Shan Lee. 2016. Audio word2vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder. _CoRR_.
* EPCSSound accessed 23-Jan-2017. The guide to sound effects.
* Font et al. (2014) Frederic Font, Sergio Oramas, Gyorgy Fazekas, and Xavier Serra. 2014. Extending tagging ontologies with domain specific knowledge. In _Proceedings of the International Semantic Web Conference_.
* Font et al. (2013) Frederic Font, Gerard Roma, and Xavier Serra. 2013. Freesound technical demo. In _Proceedings of the 21st ACM International Conference on Multimedia_.
* Ganchev et al. (2005) Todor Ganchev, Nikos Fakotakis, and George Kokkinakis. 2005. Comparative evaluation of various mfcc implementations on the speaker verification task. In _Proceedings of the SPECOM_.
* He et al. (2016) Wanjia He, Weiran Wang, and Karen Livescu. 2016. Multi-view recurrent neural acoustic word embeddings. _arXiv preprint arXiv:1611.04496_.
* Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. _Computational Linguistics_.
* Kiela and Clark (2015) Douwe Kiela and Stephen Clark. 2015. Multi-and cross-modal semantics beyond vision: Grounding in auditory perception. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Kottur et al. (2016) Satwik Kottur, Ramakrishna Vedantam, Jose M. F. Moura, and Devi Parikh. 2016. Visual word2vec (vis-w2v): Learning visually grounded word embeddings using abstract scenes. In _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Lampert et al. (2009) Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. 2009. Learning to detect unseen object classes by betweenclass attribute transfer. In _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Lazaridou et al. (2015) Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. 2015. Combining language and vision with a multimodal skip-gram model. _arXiv preprint arXiv:1501.02598_.
* Koppolo and van Miltenburg (2015) Alessandro Lepopolo and Emiel van Miltenburg. 2015. Sound-based distributional models. _IWCS 2015_.
* Melamud et al. (2016) Oren Melamud, David McClosky, Siddharth Patwardhan, and Mohit Bansal. 2016. The role of context types and dimensionality in learning word embeddings. _arXiv preprint arXiv:1601.00893_.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in Neural Information Processing Systems (NIPS)_.
* Owens et al. (2016) A. Owens, J. Wu, J. H. McDermott, W. T. Freeman, and A. Torralba. 2016. Ambient Sound Provides Supervision for Visual Learning. _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Parikh and Grauman (2011) Devi Parikh and Kristen Grauman. 2011. Relative Attributes. In _Proceedings of IEEE International Conference on Computer Vision (ICCV)_.
* Plomp and Levelt (1965) Reinier Plomp and Willem Johannes Maria Levelt. 1965. Total consonance and critical bandwidth. _The journal of the Acoustical Society of America_.
* Ricard (2004) Julien Ricard. 2004. Towards computational morphological description of sound. _DEA pre-thesis research work, Universitat Pompeu Fabra, Barcelona_.
* Settle and Livescu (2016) Shane Settle and Karen Livescu. 2016. Discriminative acoustic word embeddings: Recurrent neural network-based approaches. _arXiv preprint arXiv:1611.02550_.
* Singer (2017) Philip R. Singer. accessed 23-Jan-2017. Art of foley.
* Sundaram and Narayanan (2006) Shiva Sundaram and Shrikanth Narayanan. 2006. Vector-based representation and clustering of audio using onomatopoeia words. In _In Proceedings of AAAI 2006 Fall Symposia_.
* Wake and Asaihi (1998) Sanae Wake and Toshiyuki Asaihi. 1998. Sound retrieval with intuitive verbal expressions. In _Proceedings of the 5th International Conference on Auditory Display (ICAD)_.