MusCat: A Music Browser Featuring Abstract Pictures andZooming User Interface
1st Author

2nd Author

3rd Author

1st author's affiliation
2nd author's affiliation
3rd author's affiliation
1st line of address
1st line of address
1st line of address
2nd line of address
2nd line of address
2nd line of address
Telephone number, incl. country code Telephone number, incl. country code Telephone number, incl. country code

1st author's email address

2nd E-mail

ABSTRACT
Today many people store music media files in personal computersor portable audio players, thanks to recent evolution ofmultimedia technologies. The more music media files thesedevices store, the messier it is to search for tunes that users wantto listen to. We propose MusCat, a music browser to interactively
search for the tunes according to features, not according to
metadata (e.g. title, artist name). The technique firstly calculates
features of tunes, and then hierarchically clusters the tunes
according to the features. It then automatically generates abstract
pictures, so that users can recognize characteristics of tunes moreinstantly and intuitionally. It finally visualizes the tunes by using
abstract pictures. The technique enables intuitive music selection
with the zooming user interface.

1. INTRODUCTION
Recently many people listen to the music by using personal
computers or portable players. Numbers of tunes stored in our
computers or players quickly increase due to the increase of sizesof memory devices or hard disk drives. User interfaces therefore
become more important for users to easily select tunes users wantto listen to.
Here we think that the procedure to search for the tunes would be
more enjoyable, if we develop a technique to interactively search
for tunes, not based on metadata but based on features. We
usually select tunes based on their metadata, such as titles, artist
names, and album names. On the other hand, we may want toselect tunes based on musical characteristics depending on
situations. For example, we may want to listen to graceful tunes atquiet places, loud tunes at noisy places, danceable tunes at
enjoyable places, and mellow tunes at night. Or, we may want toselect tunes based on feelings. For example, we may want toselect “something speedy/slow” or “something major/minor”
tunes based on feelings. We think features are often more
informative rather than metadata, to select tunes based on
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.
SAC’11, March 21-25, 2011, TaiChung, Taiwan.
Copyright 2011 ACM 978-1-4503-0113-8/11/03…$10.00.

3rd E-mailsituations or feelings. However, it is not very intuitive if we showfeature values of tunes just as numeric characters. We think
illustration of features may help intuitive tune selection.
This paper presents MusCat, a music browser featuring abstract
pictures and zooming user interface. It visualizes collections oftunes by abstract pictures, based on features, not based on
metadata. The technique presented in this paper firstly calculates
features of tunes, and hierarchically clusters the tunes according
to the features. It then automatically generates abstract pictures
for each tune and cluster, so that users can recognize
characteristics of tunes more instantly and intuitionally. It finallydisplays the tunes and their clusters by using the abstract pictures.
We apply an image browser CAT [1] to display a set of abstract
pictures. CAT supposes that hierarchically clustered pictures are
given and representative pictures are selected for each cluster.
CAT places the set of pictures onto a display space by applying a
rectangle packing algorithm that maximizes the display space
utilization. CAT provides a zooming graphical user interface,which displays representative pictures of all clusters whilezooming out, and pictures in the specific clusters while zooming
in, to effectively browse the pictures. We call the user interface“MusCat”, as an abbreviation of “Music CAT”, which enables
intuitive music selection with its zooming user interface.

2. RELATED WORK
2.1 Music and Sensitivity
There have been several works on expression of musical features
as adjectives. For example, Yamawaki et al. [2] presented thatpeople recognize three pairs of adjectives: heavy - light, speedy slowly, and powerful - unpowerful, as impression of music, in apart of their study of the correspondence analysis. Similarly, ourtechnique also assigns two pairs of adjectives to musical features.

2.2 Color and Sensitivity
There have been several studies on the relationship between
colors and sensitivity, and our work is on the top of such studies.
Color system [3] expresses impressions of colors by arranging
them onto a two dimensional sensitivity space, which has warmcool and soft-hard axes. The color system supposes to placehomochromous and polychromous colors onto the sensitivity
space: it arranges homochromous colors onto a limited region of
the sensitivity space, while it arranges polychromous colors
covering whole the sensitivity space. Therefore, we think that

impression of music can be adequately expressed by using
polychrome colors rather than by homochromous colors.

2.3 Combination of Tunes and Pictures
There have been several techniques on coupling tunes andpictures. MIST [4] is a technique to assign icon pictures to tunes.
As a preparation, MIST requires users to answer the questions
about conformity between sensitive words and features of sample
tunes or icons. MIST then learns correlations among the sample
tunes, and couples the tunes and icons based on the learning
results.
Kolhoff proposed Music Icons [5], a technique to select abstract
pictures suited the tunes based on their features. The techniquepresented in this paper is also based on the features of tunes, but it
focuses on the automatic generation of abstract pictures.

2.4 Feature-based Music Analysis andRetrieval
There have been several techniques for music analysis andretrieval. For example, Lie et al. [6] presented a hierarchicalframework that automates the task of mood detection fromacoustic music data, by following some music psychologicaltheories in western cultures. It extracts three feature sets, intensity,
timbre, and rhythm, to represent the characteristics of music clips.

2.5 User Interface for Music
User interface is very important for interactive music retrieval,and therefore many techniques have been presented. Goto et al.
presented Musicream [7], which enables enjoyable operations togroup and select tunes. Lamere et al. [8] presented “Search inside
the Music,” which applies a music similarity model and a 3D
visualization technique to provide new tools for exploring andinteracting with a music collection.

Figure 1. Zooming operation of an image browser CAT.

2.6 Image Browser

3. PRESENTED TECHNIQUE

Image browser is an important research topic, because numbers ofpictures stored in personal computers or image search engines are
drastically increasing. CAT (Clustered Album Thumbnail) [1] is atypical image browser that supports an effective zooming userinterface. CAT supposes that hierarchically clusters pictures are
given and representative pictures are selected for each cluster.
CAT firstly packs thumbnails of the given pictures in each cluster,and encloses them thumbnail by rectangular frames to represent
the clusters. CAT then packs the rectangular clusters and encloses
them by larger rectangular frames. Recursively repeating theprocess from the lowest to the highest level of hierarchy, CAT
represents the hierarchically clustered pictures.

Our current implementation uses acoustic features (MFCC: MelFrequency Cepstrum Coefficient) for clustering, because we
assumed that acoustic-based division is intuitive to briefly selecttunes based on situation, such as “quiet tunes for quiet spaces”, or“loud tunes for noisy spaces.” On the other hand, it uses musicalfeatures for abstract image generation, because we assume that
more information may be needed to select specific tunes fromclusters of similarly sounding tunes.

In addition, CAT has a zooming user interface, as shown in
Figure 1. CAT displays representative images instead ofrectangular frames while zooming out, as the initial configuration.
On the other hand, CAT displays image thumbnails whilezooming in the specific clusters. CAT enables to intuitively
search for interesting images, by briefly looking at the all
representative images, then zooming into the clusters displayed asinterested representative images, and finally looking at the
thumbnail images in the clusters.

Our technique consists of the following four steps: (1) calculationof features from music media files, (2) clustering of tunes basedon the features, (3) generation of abstract pictures, and (4)visualization of tunes by using abstract pictures.

3.1 Music Feature Extraction
There have been various techniques to extract music features, andsome of them have been components of commercial products oropen source software. Our current implementation uses features
calculated by Marsyas [9] and MIRtoolbox [10]. It uses means
and standard deviations of nine bands of MFCC calculated by
Marsyas, for clustering and abstract image generation for clusters.
We preferred to use Marsyas for MFCC calculation just because it
had more variables as results of MFCC calculation. Also, ourimplementation uses normalized features shown in Table 1
calculated by MIRtoolbox, for abstract image generation for tunes.
It may be sometimes difficult to express musical characteristics
by single feature value, because features may change gradually or
suddenly during a tune. Our current implementation calculates
features from randomly selected 15 seconds of the tunes. In the
future, we would like to extend our implementation so that we canselect the most characteristic features from whole parts of thetunes.
Table 1. Music features we apply in our experiments.
FeatureExplanation

RMS energy

Root-mean-squareenergyrepresents a volume of the tune.

which

Tempo

Tempo (in beats per minute).

Roll off

Frequency which takes 85% of totalenergy, by calculating the sum of energyof lower frequencies.

Brightness

Percentage of energy of 1500Hz or
higher frequency.

Spectral irregularity

Variation of tones.

Roughness

Percentage of energy of disharmonicfrequency.

Mode

Difference of energy between major andminor chords.

We believe the approach is effective as discussed below. First
reason is that visual and music words are often related. Actually,
many music works depict scenery that artists looked and imagined.
Also, some painting artists expressed emotion while listening to
the music as abstract pictures [8]. Second reason is that humans
have synesthesia [11]. Impression of colors is related toimpressions of sound, because some people have perception socalled “the colored hearing,” which associates colors by listening
to the music. Third reason is that impressions of sounds andcolors are often expressed by same adjectives. Therefore, we
think that music can express through the abstract picturesconsidering colors to visualize the music.
This section describes our implementation to automaticallygenerate abstract pictures. Currently it is just designed based on
our subjective, but we do not limit the abstract pictures to thefollowing design.

3.3.1 Example of Color Assignment
Our technique firstly assigns colors to the objects in abstract
pictures. As mentioned previously, this technique uses
polychrome colors because the polychrome color arrangement can
express impression richer than the monochrome color
arrangement. The technique selects colors of abstract pictures
based on color image scale [3]. It is a color system that distributes
combination of three colors in a two dimensional space, so called“sensibility space”, which has the warm-cool and the soft-hardaxes, as shown in Figure 3. The technique distributes tunes into
this sensibility space, and assigns the colors corresponding in thesensitivity space to the tunes.

3.2 Clustering
Next, the technique hierarchically clusters music files based onmeans and standard deviations of MFCC values. There are manyclustering techniques for multi-dimensional datasets (e.g., nearest
neighbor, furthest neighbor, group average, centroid, median,
Ward.) We experimentally applied various clustering techniques
for our own collection of tunes, and compared the results bycarefully looking at the dendrograms. As a result of ourobservation, we selected Ward method as a clustering algorithm,
because it successfully divides a set of tunes into evenly sizedclusters. Figure 2 shows the comparison of dendrogram among
three clustering algorithms.

soft (more Roll off)

warm

cool

(more Mode)(less Mode)

Figure 2. Comparison of dendrogram. (Left) group averagemethod. (Center) median method. (Right) Ward method.

3.3 Abstract Picture Generation for Tunes
Our technique generates abstract pictures to express tunes, anddisplays so that users can intuitively select the tunes. It generates
the abstract pictures for each tune, and for each cluster. It
calculates the average of feature values for each cluster in order,
to generate the abstract pictures of the clusters.

hard (less Roll off)
Figure 3. Sensitivity space based on color image scale.

Here, let us discuss which features match to the warm-cool andthe soft-hard axes. We feel major chords express positive
impression similar to bright and warm colors, while on the other
hand, minor chords express negative impression similar to dark
and cold colors. Based on the feeling, we assign Mode is to warmcool axis. Similarly, we assign Roll off to soft-hard axis. We thinklisteners often use substitute adjective words such as “light” or“soft” for the impression of music, and often these impression isrelated to frequency-based tone balances.
Our current implementation places many sample colors onto thesensitivity space. Calculating Mode and Roll off of a tune, ourimplementation places the tune onto the sensitivity space, andselects the color closest to the tune. Here, let the position of thetune as (m wc , m sh ) , and the position of a color set as (c wc , c sh ) .
Our implementation selects the color set that satisfies thefollowing formula:min( (m wc − c wc )2 + (m sh − c sh )2 )We assign RMS energy to the gradation layer. We would like torepresent power, weight, and broadening by the gradation, andthink that RMS energy is the most suitable feature for thisrepresentation.
We assign Tempo, Spectral irregularity, and Roughness, to thegeneration of orthogonally arranged circles. We would like torepresent frequency of rhythm by the number of circles,
irregularity and variation of music by irregularity of circles. We
think Tempo, Spectral irregularity, and Roughness are the mostsuitable features for this representation.
We assign Brightness to the number of randomly placed stars. We
expect many of users will associate bright music from the stars,
and therefore we think Brightness is the most suitable feature forthis representation.
After generating the three layers, the technique finallysynthesizes the three layers to complete the abstract picture
generation, as shown in Figure 5.

3.3.2 Example of Abstract Picture Design
This section describes an example of design of abstract pictures
based on music features. Our design first generates the following
three layers, 1) gradation layer, 2) a set of circles, and 3) a set ofstars, as shown in Figure 4.

Figure 5. Abstraction picture synthesis from three layers ofimages.

3.4 Abstract Picture Generation for Clusters
Our technique generates another design of abstract pictures forclusters. It simply represents mean values of nine bands of MFCC
as colored squares. Figure 6 shows an illustration how ourtechnique generates abstract pictures. Our implementation defines
nine colors for the bands based on the soft-hard axes shown in
Figure 3. It assigns softer colors to higher bands, and hardercolors to lower bands. It calculates the average values of the meanvalues of the tunes for each cluster, and calculates the sizes of thecolored squares as proportional to the average values. The
pictures denote acoustic textures of tunes in the clusters.

Figure 4. Automatic generation of three layers of images
based on music features.

Figure 6. Illustration of abstract picture generation forclusters.

3.5 Image Browser CAT as a Music Browser
The technique displays a set of abstract picture by applying theimage browser CAT [1]. We extend CAT so that we can use CAT
as a music browser, where we call the extended CAT as“MusCat,” as an abbreviation of “Music CAT”.
Figure 7 is an example snapshot of MusCat. Initially MusCatshows all abstract pictures of clusters while zooming out, asshown in Figure 7(Lower). When a user prefers a picture andzooms in it, MusCat turns the abstract pictures for clusters toabstract pictures for tunes, as shown in Figure 7(Upper). Users
can select a tune and play it by double-clicking the corresponding
picture.
Figure 7 shows characteristic clusters (a), (b), and (c). The
abstract image of cluster (a) denotes that the lowest frequency
band (MFCC 0) of its tunes is respectively large. The abstract
images of the three tunes in the cluster (a) have size-varying andun-aligned circles, and respectively more stars. These images
denote that the tunes in cluster (a) have loud low and highfrequency sounds, and respectively more disharmonic sounds.
Actually, two of the three tunes are dance music that have loudBass Drum beats, and backing of electric disharmonic sounds.
The abstract image of cluster (b) denotes that low and highfrequency bands (MFCC 0, 1, 2, 7, and 8) are extremely small.
The abstract images of the two tunes in the cluster (b) havealigned circles, and less number of starts. Colors of circles aredifferent between the two images. Actually, the two tunes are
Japanese traditional folk music played by old wood wind
instruments, without bass part of high-tone percussions. One of
the tunes plays major scale, and the other plays minor scale. That
is why colors of two pictures of the tunes are much different.
The abstract image of cluster (c) denotes that specific two
frequency bands (MFCC 1 and 3) are large, but others are quite
small. The abstract images of the two tunes in the cluster (c) havesmaller number of well-aligned, equally-sized circles. Actually,
the two tunes are slow female vocal songs, and these backings are
only piano or Japanese traditional strings. That is why the abstract
image of the cluster denotes that specific two frequency bands arelarge, and the abstract images of the tunes have less number ofcircles.
As above mentioned, users of MusCat can select clusters oftunes based on acoustic features by specifying the abstract imagesof clusters, and then select tunes based on musical features. We
think this order is reasonable: users can firstly narrow down thetunes based on their situations: for example, quiet tunes for quiet
places, loud tunes for noisy places, and so on. They can selectfeatures of tunes in the specific clusters based on their feelings.
Though our original concept of MusCat supposes to show
different abstract images for clusters and tunes, it is also possible
to generate abstract images of tunes based on MFCC, as well as
those of clusters. Figure 8 shows an example of abstract images oftunes generated based on MFCC.

Zoom inFigure 8. Example of visualization by MusCat. Abstract
images of tunes are generated similar to those of clusters.

4. EXPERIMENTS
This section describes our experiments using the presented
technique. We used Marsyas [9] and MIRtoolbox [10] for feature
extraction, and R package for clustering. We implemented
abstract picture generation module in C++ and executed with
GNU gcc 3.4. We implemented MusCat in Java SE and executedwith JRE 1.6. We applied 88 tunes which are categorized into 11
genres (pops, rock, dance, jazz, latin, classic, march, world, vocal
music, Japanese, and a cappella), provided by RWC Music
Database [12].
We had a user evaluation with 15 examinees to examine thevalidity of abstract pictures. We also asked the examinees to playwith MusCat and give us comments or suggestions.

4.1 Suitability of Abstract Picture of Tunes
We showed 12 tunes and their abstract pictures to the examinees.
We then asked to evaluate the suitability of the pictures for thetunes by 5-point scores, where “5” denotes suitable, and “1”denotes unsuitable. Table 2 denotes the statistics of the evaluation.
Table 2. Evaluation of abstract pictures.
1 (suitable)

2

3

4

5 (unsuitable)

5. CONCLUSION AND FUTURE WORK
We presented MusCat, a music browser applying abstract picturesand zooming user interface. The technique uses features to clustertunes, and to generate abstract pictures, so that users canrecognize tunes more instantly and intuitionally without listening.

Tune 1

1

11

2

1

0

Tune 2

1

3

7

4

0

Tune 3

0

0

4

7

4

Tune 4

2

3

2

7

1

Tune 5

3

5

6

1

0

Tune 6

9

4

2

0

0

Tune 7

4

9

0

2

0

Tune 8

0

3

4

2

6

Tune 9

0

6

4

3

2

Tune 10

1

4

8

2

0

Tune 11

3

4

5

3

0

Tune 12

1

4

4

5

1

[Abstract picture:] Our current abstract picture design is just an
example, and therefore we think there may be better designs.
Some of our examinees pointed that color is more important for
the impression of pictures, than shapes and properties of objects.
However, we have not yet found the best scheme to assign threecolors to gradation, circles, and star. We need to discuss better
schemes to assign the three colors. Another discussion is moodbased design of abstract pictures, since current design directly
represents feature values.

This experiment obtained good evaluations for several tunes (e.g.
Tune 1, 6, 7); on the other hand, it obtained relatively bad
evaluations for other several tunes (e.g. Tune 3, 4, 8, 9, 12). Next
section discusses the reasons of the results with the comments of
examinees.

[User Interface:] Current version of MusCat just plays the musicby click operations, and simply indicates text information. We
would like to extend the functionality. We would like to develop
to show more metadata information of the selected tunes. Also,
we would like to develop to play a set of tunes in the selectedclusters by one click operation.

4.2 Feedback from Examinees

6. REFERENCES

This section introduces free comments from examinees.
We asked examinees to give us any comments during theexperiments introduced in Sections 4.1, and 4.2. Examinees
commented as follows:yyColors were dominant for the first impressions of pictures.
Gradation of pictures was unremarkable because of thecolor arrangement.
ySome examinees might associate colors of music genre byfashion; for instance, colors of rock were black and white.
We think the above comments are key points to improve theevaluation of users. We will discuss about them more in the future.
We also asked examinees to give any comments on usability of
MusCat. Many examinees gave us positive comments that
MusCat was useful to use when they wanted to select tunes
according to intuition or emotion, especially for unknown tunes.
Some of them suggested us that MusCat can be an alternative of
shuffle play mechanism of music player software.
We also got some constructive suggestions from examinees. Someof them suggested us to indicate the metadata or feature values oftunes they selected, even though they selected the tunes according
to the impression of abstract pictures. We think it is interesting to
add more effective pop-up mechanism for the selected tunes toindicate such information. Some other examinees commented thatthey might loss which part they are zooming in. We would like to
add a navigation mechanism to solve the problem.

Following are our potential future work:[Music feature:] Our current implementation calculates featuresfrom randomly selected 15-second segment of a tune. We would
like to calculate features from all 15-second segments of a tune
and select the most preferable or characteristic features from thecalculation results.

[1]

A. Gomi, R. Miyazaki, T. Itoh, J. Li: “CAT: A Hierarchical
Image Browser Using a Rectangle Packing Technique,” 12th
International Conference on Information Visualization,
pp.82-87, 2008.

[2]

K. Yamawaki, H. Shiizuka: “Characteristic recognition of
the musical piece with correspondence analysis,” Journal ofKansei Engineering, Vol. 7, No. 4, pp. 659–663, 2008.

[3]

S. Kobayashi: Color System, Kodansha, Tokyo, 2001.

[4]

M. Oda, T. Itoh, “MIST: A Music Icon Selection TechniqueUsing Neural Network”, NICOGRAPH International, 2007.

[5]

P. Kolhoff, J. Preub, J. Loviscach: “Music Icons: Procedural
Glyphs for Audio Files,” IEEE SIBGRAPI, pp. 289-296,
2006.

[6]

L. Lie, D. Liu, H. Zhang: “Automatic mood detection andtracking of music audio signals,” IEEE Transactions on
Audio, Speech, and Language Processing, Vol.14, pp. 5-18,
2006.

[7]

M. Goto, T. Goto: Musicream: “New Music Playback
Interface for Streaming, Sticking, Sorting, and Recalling
Musical Pieces,” Proceedings of the 6th International
Society for Music Information Retrieval, pp.404-411, 2005.

[8]P. Lamere, D. Eck: “Using 3D Visualizations to explore anddiscover music,” Proceedings of the 8th International
Society for Music Information Retrieval, pp.173-174, 2007.

[9]

Marsyas, http://marsyas.info/[11] J. Harrison: SYNAESTHESIA The Strangest Thing, shin-yosha, 2006.
[12] RWC Music Database, http://staff.aist.go.jp/m.goto/RWCMDB/[10] O. Lartillot: “MIRtoolbox,”
http://www.jyu.fi/hum/laitokset/musiikki/en/research/coe/m
aterials/mirtoolbox

Zoom in(b)(a)

Figure 7. Example of visualization by MusCat. (Upper) MusCat
displays abstract pictures of tunes while zooming in. (Lower)
MusCat displays abstract pictures of clusters while zooming
out. In the cluster (a), the lowest frequency band (MFCC 0) is
respectively large. In the cluster (b), low and high frequency
bands (MFCC 0, 1, 2, 7, and 8) are extremely small. In thecluster (c), specific two frequency bands (MFCC 1 and 3) arelarge, but others are quite small.

(c)