# A General Framework for Visualization of Sound Collections in Musical Interfaces
## 2 Background  
Creative stages of music production are typically driven by musical intuitions and auditory cues. In this context, dealing with labels and file systems can be disruptive, which hinders the use of large collections of sounds. Non-verbal interfaces for interacting with large collections of sounds (from very short segments in the order of a hundred milliseconds to longer audio recordings such as music loops) are valuable, although still rare.  
A good deal of research has focused on automatic audio analysis for creating interactive systems based on 2D plots of sound collections. One basic strategy is to use two perceptual sound descriptors (for example pitch and amplitude) as the two axes of the plot. Several works have implemented this idea in interfaces that allow the user to choose the descriptor for each axis [10; 11]. These systems suffer from several limitations. First, the direct use of sound descriptors often requires an understanding of concepts related with signal processing and psychoacoustics. Moreover, there is no assurance that a given sound collection will have an interesting variation along a given set of descriptors. For example, a pitch descriptor may be irrelevant for a collection obtained from drums or environmental sounds. In general, each collection will most likely have its own sonic dimensions beyond a particular choice of descriptors. Second, such descriptors are typically obtained from a frame-level representation, which means they may vary significantly over time. A single value (typically the average over a sequence of frames) is relevant for a very short sound, but it will not be as useful for longer samples.  
Automatic descriptors that can capture the diversity in sound collections typically span more than two dimensions. At the same time, different sounds can be related on the basis of non-linear functions of these descriptors. Thus, several works have explored non-linear dimensionality reduction to create automatic low-dimensional maps of sound collections.  
An early example of non-linear dimensionality reduction for sound can be found in research on perception of musical instrument timbre [12], where Multi-Dimensional Scaling (MDS) was used to find _timbre spaces_. The same concept has been more recently implemented using Variational Autoencoders (VA) [13]. This concept is limited to the view of timbre as a static quality of pitched sounds and thus not generally applicable to sound collections.  
Another early trend was the use of Self-Organizing Maps (SOM) for browsing collections of drum sounds [14] and sound effects [15]. SOMs have the property of naturally producing grid layouts, but they are mostly interpreted as clusters, where multiple data points can be assigned to the same output location.  
Most modern non-linear dimensionality reduction approaches are based on neighbour graphs. The IsoMap algorithm [16], based on applying MDS to distances derived from the k-nearest neighbors (KNN) graph, was used in [17]. In [18], the algorithm was extended using the Hungarian algorithm to map the results to grid layouts. More recently, several works have focused on newer dimensionality reduction algorithms. t-distributed Stochastic Network Embedding (t-SNE) [19] uses a probabilistic view of the distances in the original high-dimensionality space to build the neighbourhood graph. This algorithm was used for visualizing sounds from the Freesound database in [5], and for visualising textural sounds in [7]. Uniform Manifold Approximation and Projection (UMAP) [20] is a similar algorithm based on concepts from mathematical topology. The results are similar to t-SNE, but it is significantly faster. It has been used in [6; 21].