## A Social Semiotic Theory of Synesthesia? - A Discussion Paper

## Abstract

After a brief survey of ideas about synesthesia in philosophy, science and the arts, this paper explores the common qualities of the parameters of colour, graphic shape (including typography), timbre and texture, hypothesizes a number of points of correspondence and argues for their semiotic importance in the contemporary multimodal communication of identity.

## Keywords

Identity, metaphor, multimodality, parametric analysis, social semiotics, synesthesia.

## 1. Introduction

Strictly speaking, synesthesia is a neurological condition in which the 'stimulation of one sensory modality automatically triggers a perception in a second modality in the absence of any direct stimulation to this second modality' (Harrison/Baron-Cohen 1997: 3) - we might, for instance, see the letter a and simultaneously see the colour blue, as vividly as we see the shape of the a , even though in reality the a is not blue.

This phenomenon had been noted for a long time, for instance by John Locke, in his Essay Concerning Human Understanding of 1690 (quoted in Harrison/Baron-Cohen 1997: 4):

A studious blind man who had mightily beat his head about a visible object, and made use of the explications of his books and friends, to understand the names of light and colours, which often came his way, betrayed one day that he now understood what scarlet signifi  ed. Upon which his friend demanded what scarlet was? The blind man answered, it was like the sound of a trumpet.

But it was in the late 19 th  century, when the new sciences of neurology and psychology began to fl  ourish, that synesthesia received extensive scientifi  c attention, often focusing on correspondences between the letters of the alphabet and colours, and between music and colour. From the 1930s onwards, interest in synesthesia began to wane. Behaviourism reigned and conclusions based on self-reporting, rather than on outwardly perceivable symptoms, were no longer acceptable. In the late 1980s, however, interest revived, mostly as a result of the work of the American neuroscientist Richard Cytowski (1989) - and today brain scanning has provided the missing proof, allowing us to see the two simultaneous 'stimulations' light up distinct areas in the brain (Aleman et al, 2001).

But the term synesthesia, which derives from the Greek words for 'together' ( syn ) and 'perceiving' ( aisthano ) has also been used for combinations of 'sensory stimulations' that are deliberately produced, rather than involuntary triggered. This too has a long history. The ancient Greeks conceived of the cosmos as consisting of eight concentric spheres, each associated with a distinct colour and musical note, thus creating correspondences, not only between colours and musical

* Theo Van Leeuwen University of Southern Denmark Department of Language and Communication Campusvej 55 DK-5230 Odense M leeuwen@sdu.dk

105

notes, but also between sensory perceptions and the order of the cosmos. Even earlier, Eastern systems such as the Chinese feng shui did much the same. In more recent Western history artists have been in the forefront of the search for correspondence. Rimbaud's 'Sonnet of the Vowels' of 1871 is a well-known example:

I have invented the colours of the vowels - A black, E white, I red, O blue, U green - I have ordered the form and movement of each consonant, and with instinctive rhythm I have fl  attered myself that I have invented a poetic verb, accessible one day or another to every sense

Correspondences between music and colour go back to the 'colour harpsichord' of Louis-Bertrand Castel in 1735, with matched twelve colours with the tones of the then new twelve-tone system - the colours popped up on paper strips when the relevant keys were played. More recently, Scriabin wrote his Prometheus (1911) for piano, organ, orchestra, chorus and a colour keyboard which projected colours that corresponded not to individual notes but to keys (red for C, yellow for D, blues for E and F sharp, etc) on a screen behind the orchestra (van Campen 2008: 50-52). Today computer algorithms convert music into visuals and 'coloured hearing' is accessible to every computer user, for instance on Windows Media Player.

The artists who, in 1911, formed the Blaue Reiter ('Blue Rider') art movement, sought to create Gesamtkunstwerke ('total art works') that combined music, dance and theatrical production. Kandinsky was a member and contributed a theatrical piece called Der Gelbe Klang ('The Yellow Sound') and in his teaching at the Bauhaus he encouraged students to experiment with the relationships between visual form, colour, touch, temperature, sound and energy (van Campen, 2006: 56). Contemporary designers do much the same thing. Haverkamp (2013: 17) for instance, encourages product designers to 'optimize the alignment of the auditory perception and the visual appearance of products with one another' and to ask questions like 'Which colour scale best represents the timbre of engine noise?', or 'Which indicator sound harmonizes ideally with the visual features of the indicator switch lever?'

Neurologists have traditionally  rejected  such  'artistic  aspirations  to  sensory  fusion'  (along with  'cross-modal  association  and  metaphoric  language')  as  'pseudo  synesthesia'  (Cytowski 1997: 20), and Huisman (quoted in van Campen 2008: 146) has suggested the term 'synchronesthesia', to distinguish it from 'genuine synesthesia'. Nevertheless, scientifi  c and artistic interest in synesthesia grew at the same time, inspired by the same changes in the semiotic landscape - and not all neurologists agree with the strict separation between 'genuine' and 'pseudo' synesthesia. Some have revitalized a line of thought which has existed since antiquity, when Aristotle (2008: 425a27) asked a question that is still of central interest to multimodality researchers: how can humans perceive a unity in the multitude of sensory impressions? His answer was that there is a ' koine aisthesis' , a 'common sense' which perceives common qualities in the input from different senses. This insight, which was indeed 'common sense' in the Middle Ages, returned, in a new way, in the Romantic era, for instance in the work of Herder (2002 [1772]) who proposed that those common qualities are feelings - it is feelings, emotions, Herder thought, which create the affective, synesthetic bond between the senses. A similar link between perception and affect had emerged in aesthetics (the term was invented by Alexander Baumgarten in the mid 18 th  century and also derives from aisthano , 'perceiving').

In the work of late 19 th  century scholars such as Féré, Bleuler and Lehman and Flournoy (cf Marks 1997), and, more recently, in the work of neuroscientists such as Cytowski and Marks, these ideas re-emerge: synesthesia might be explained by 'elemental perceptual qualities', 'form constants' which are 'rather abstract', yet 'provoke reactions' (Cytowic 1997: 29). Not much progress has been made, however, in establishing just what these elemental qualities are, although 'brightness' (corresponding to pitch in sound and lightness in colour) has been mentioned as a candidate since the late 1920s (von Hornbostel 1931). In this context, the idea that synesthesia is a unique condition affecting only a few people, is beginning to be watered down. According to Marks (1997: 87-88):

Synesthesia is not an isolated phenomenon, separated from non-synesthetic perception and thought. Rather, synesthesia is a cross-modal manifestation of meaning in its purely sensory, and in one sense its strongest form.

As a result, assessments of the prevalence of synesthesia have changed radically. Originally stating that it affects 1 in 100,000 people, Cytowic later reduced the ratio to 1 in 25000 (1997: 33) and more recently synesthesia researchers have proposed ratios of 1 in 500, 1 in 200 and even 1 in 100 (van Campen 2008: 128). Incidence of synesthesia in children had long been reported to be higher than in adults, with estimates in the range of 40 to 50 % (Marks 1997: 88). We are all synesthetes, Marks concludes, but it may be that in adulthood 'it becomes valuable, indeed necessary, for the child to transfer the meanings from the perceptual-synesthetic to the verbal realm', so that, for most adults, synesthesia becomes 'vestigial' (Marks 1997: 89).

Two themes emerge. Firstly, the link between synesthesia and emotion. Neuroscientists like Cytowic and Marks situate synesthesia in the limbic brain, which handles emotion, memory and attention, and 'easily overwhelms thinking' (Cytowic 1997: 31). Secondly, despite the idea of 'common qualities', synesthetic perception continues to be regarded as idiosyncratic - different synesthetes link letters and colours, or sounds and colours, in different ways. This fi  ts in with the traditional Kantian view of aesthetics, in which aesthetic appreciation is seen as different from rational thought - as subjective, a matter of taste.

As a social semiotician, one cannot help noticing how closely these new scientifi  c developments fi  t, not just the simultaneous explorations of artists, but also the way public communication has developed since the 1920s - its increasing multimodality, its appeal to emotion rather than reason, its emphasis on aesthetic qualities, today even in everyday documents like invoices and business reports (cf Van Leeuwen 2015).

## 2. Parametric systems

In multimodality studies, a number of semiotic modes have, in recent years, been described as parametric rather than binary systems. They happen to be the kind of modes which have played a  key role in the literature on synesthetics - colour (Kress/Van Leeuwen 2002; Van Leeuwen 2011), vocal and musical timbre (Van Leeuwen 1999, 2008), graphic shape, including typography (Van Leeuwen, 2006, 2010; Johannessen 2010) and texture (Djonov/Van Leeuwen 2013). It is thus possible that the descriptive frameworks multimodality researchers have developed for these modes can help identify some of the 'common qualities' that have eluded the neuroscientists.

What is a parametric system? Until recently, the meaning potentials of semiotic modes have been described as lexico-grammatical systems of binary choices. Thus the mood system of English allows the expression of three basic interactions (cf fi  gure 1a) - fi  rst of all there is a binary choice between 'offering information', as expressed by the indicative mood, and 'demanding' something; then there is the binary choice between 'demanding information', as expressed by the interrogative mood, and 'demanding goods and services', i.e. demanding that people do something, as expressed by the imperative mood (the logical possibility of 'offering goods and services' has no realization in the mood system as such). In their grammar of visual communication, Kress and Van Leeuwen (2006) extended this to visual communication. If someone looks at the viewer from within a picture, they argued, something is symbolically 'demanded' from the viewer (sympathy, respect, erotic attraction, etc). If this is not the case the picture simply 'offers information' about what is represented. This binary choice can then lead to further binary choices, specifying further kinds of demands, for instance by means of facial expressions and/or gestures.

Offering information

Demanding information

Demanding Demanding goods & services Light Dark Intense In studying the semiotic mode of colour, however, Kress and Van Leeuwen (2002) found it impossible to construct such a system of binary choices. Colour meanings, they argued, derive from a set of simultaneously present, rather than sequentially selected choices. To model this (cf fi  gure 1b) they turned, not to grammar, but to phonology as a source of inspiration, more specifi  cally to Jakobson's 'distinctive feature' theory (Jakobson and Halle 1971). According to this theory, speech sounds are characterized by a number of simultaneously present features. A [p], for instance, is unvoiced and labial and plosive, amongst other things. Colours, similarly, should be described, not only on the basis of hue, of whether they are red or green or blue, but by a range of features, including, for instance, 'value' (how light or dark they are), 'saturation', how intense or pale they are, and several other features. A colour is never only red, but always red and dark and desaturated and shiny and plain, and so on. And these parameters are graded (as indicated by the double-headed arrows in fi  gure 1b). A colour is not either dark or light, it has a value somewhere on the analogue scale from dark to light.

Figure 1a. Binary (lexicogrammatical) system

<!-- image -->

Figure 1b. Parametric (expression) system

But Kress and Van Leeuwen also diverged from 'distinctive feature' theory. For Jakobson distinctive features function only to distinguish phonemes, and hence words, from each other. In themselves they are regarded as meaningless. For Kress and Van Leeuwen, expression modes (which they term media , using this term in the way artists use it, hence not only for technological media, cf Kress/Van Leeuwen 2001) make meaning in their own right. It is true that speech sounds express lexicogrammar, but they also add another layer of meaning. The speaker's voice might be high-pitched or low-pitched, for instance, and rough or smooth, and this will convey something about that speaker, or perhaps about the organization s/he works for, or about the genre of speech s/he is engaged in. Similarly, typography does not just function to realize linguistic meaning, it also adds meaning - by being bold or light, roman or italic, rounded or angular, and so on. Colours, too, can serve to express other systems. In PowerPoint, for instance (cf Van Leeuwen 2011), colour schemes help to distinguish foreground from background, title from main copy, bullet points from hyperlinks, and so on, but different colour schemes can be chosen to do so, and by choosing a particular palette further meaning is added.

The meanings expressed in this way tend to be identity meanings. They say something about the speaker or writer, or about the organization s/he works for, or about the kind of communication s/he is involved in, and they do so in a way that has aesthetic and emotive resonance. It is no accident that such identity meanings are becoming increasingly important in the age of 'branding', when the communication of identities and of the values they stand for, is no longer used only for products, but also for all kinds of organizations, commercial and non-commercial, and even for individuals, who, like companies, seek to style themselves in accordance with life-style values and attributes (Machin/Van Leeuwen 2008).

How do identity meanings come about? Here Kress and van Leeuwen borrowed from Lakoff/ Johnson (1980) for whom meaning derives from metaphors that are based on common experience, whether bodily experience, experience in the handling of materials, or experience of interactions with people. The meaning of light and dark, for instance, rests on our common experience of night and day, which is a source of a wide range of metaphors, both in everyday language and in visual communication. Precisely how light and dark informs our understanding in a particular case will depend on the context. Light colours may signify lightheartedness in Hollywood comedies or divine revelation in a Rembrandt painting, and these are only two of the many possibilities. As for the handling of materials, experience tells us why graphic shapes, including letterforms, may be irregular - for instance because they are produced by children who have not yet mastered writing or drawing, or because a writer is intoxicated or unable to see what s/he is writing. Thus irregularity can become a metaphor for childish innocence and playfulness, or for rebelliousness or the deviance of law breaking, to mention just two possibilities. It follows that the basis of at least some of these metaphors, namely those that are based on physical experiences shared by all humans, can, at least in principle, be universally understood - an important thought for communicators engaged in global communication.

The parameters of individual expression media such as colour, graphic shape, timbre and texture have now begun to be described, and this allows us to look for correspondences. Irregularity, for instance, can exist, not only in graphic shapes, but also in textures, in music and other sounds, and in colours which may either be plain and even or vary in darkness, saturation, etc. in irregular ways. We can therefore reframe the issue of synesthetic correspondences from a semiotic point of view and ask what correspondences there may exist between the parameters of different expression media and hence between the meanings that can be expressed by them. To do so, we will fi  rst review the parameters of the individual expression media and then look for possible parallels between them.

## 3. Colour

Beginning with colour, the following parameters have so far been proposed, some of them already well-established, others less so (Kress/Van Leeuwen 2002; Van Leeuwen 2011):

## Value

Value is the scale from light to dark. As already mentioned, the difference between light and dark is a key human experience and can give rise to many metaphors in which light usually carries positive and dark negative values - though we should not forget that white can also be the colour of 'bloodless' death and ghostly things and black the colour of elegant dresses and luxury items.

## Saturation

Saturation is the scale from intense to pale, from, for instance, the fullest red to a grey in which just a tinge of redness remains. Whatever meanings are built on this, saturation will always involve degrees of intensity. But a lessening of intensity need not be negative - it can also suggest modesty and restraint.

## Purity

Purity is the scale from maximum purity to maximal 'mixedness' or 'hybridity'. Impure colours are colours that belie description, colours we can only describe as, for instance, 'a slightly reddish green', or a 'bluish purple'. Thus impurity, in context, can come to suggest hybridity, ambiguity, ambivalence, and purity its opposite.

## Luminosity, lustre and luminescence

Colours that have brilliance, that glow from within without emanating from an actual light source are said to be luminous. They suggest radiance, glowing - which can then, in context, become more specifi  c: one can radiate health, for instance, or pride and glory. Closely related, but not quite the same, is lustre, the scale from matt to shiny, which can also create radiance (or its opposite) and which, through the ages, has been associated with luxury and opulence. This, again, differs from luminescence, colours that are actually created by light sources. They have always existed but are of increasing importance in our age of the vibrant colours of neon lights and television and computer screens.

## Modulation

Modulation is the scale from fl  at, plain colour to varied tints and shades. While plain colour is even and unchanging, modulated colour varies, usually irregularly, in brightness, saturation, purity and so on. Depending on the context, this can be seen as subtle and nuanced, or as fussy, or smudgy, just as plain colour can be seen as basic and unrealistic, or as bringing out the essence and spirit of a given colour.

## Temperature

Temperature is the scale from warm to cold, as expressed by hue - the scale from red to blue. Our experience tells us that red can always be associated with energy - whether it is the red of passion or the red of danger or any other red, and blue with its opposite, with some form of rationality, relaxation or serenity.

## Differentiation

Finally there is differentiation, the scale from monochrome to the use of a maximal variety of different colours, which is more a feature of colour palettes than of individual colours (though all colour parameters can characterize individual colours as well as colour palettes), and whatever differentiation can come to mean in a given context, it will always convey a sense of variety, abundance and extroversion about it - the context will then fi  ne-tune this broad meaning potential.

As mentioned, every colour displays some value, some degree, of each of these parameters, and it is in their synthesis, diffi  cult to express in words, yet immediately graspable by the eye, that colour meaning resides, just as it is in the combination of the ingredients, in their different proportions, that the taste of a dish resides. Clearly, the parametric approach can generate a great variety of colours, and hence unlock great creative potential and realize a great variety of identity traits from energetic to laidback, from exuberant to restrained, from bold and plain-speaking to subtle and nuanced, and so on. As such it differs from other ways in which colours can mean - from the fi  xed colour codes of specifi  c domains of meaning such as traffi  c signs or the maps of railway systems, and from the conventionalized colours and colour schemes whose metaphorical origins (or sometimes practical origins, e.g. the cost or provenance of the materials from which certain dyes were originally made) we have forgotten, such as the purple and gold of royalty (and of many a chocolate box), the regulation colours of uniforms, the pink and blue of gender differentiation, and so on. All these continue to play a role as well, but the parametric approach, with its potential for creative meaning making and creative interpretation is particularly well suited to an age in which there is a constant need for new identities that are both unique and able to communicate values that will resonate in specifi  c communities.

## 4. Graphic shape

The parameters of graphic shape were initially developed in work on typography (Van Leeuwen 2006). But it soon became apparent that the same parameters are at work in the design of letter forms and in the design of abstract graphic shapes such as abstract paintings or decorative patterns (Van Leeuwen 2010) - like letterforms, these, too, may be angular or rounded, regular or irregular, and so on. The parameters below are adapted from Van Leeuwen (2006).

## Weight

Weight refers to a scale from bold to light. Both letterforms and abstract graphic shapes can be bold, heavy or light - the terms themselves are synesthetic metaphors. Bold shapes can be assertive and daring, solid and substantial, light shapes more timid or insubstantial. But lightness can also be elegant and subtle, and boldness overbearing and domineering. It depends on the contexts and the way these meanings are valued in those contexts.

## Expansion

Expansion refers to a scale that runs from maximally condensed to maximally expansive - from narrow, compressed shapes, closely huddled together, to expansive and widely spaced shapes or letterforms. Its meaning potential derives from our experience of space, and from the values we attach to different kinds of spaces. Maximally condensed forms make maximal use of limited space, packing the available space with content, so to speak. Expansion provides room to move, room to breathe but can also become empty and vacant and alienating. Expansion is often related to class. The covers of popular magazines are packed with content and cheap restaurants pile the food on the plate, while the covers of elite magazines leave large areas blank and expensive restaurants serve small delicacies on large plates.

## Slope

In typography, slope refers to the difference between cursive, sloping, 'script'-like typefaces and upright typefaces. Even when a cursive typeface does not evoke a particular era, sloping scripts will remind of handwriting and upright scripts of printing, which can suggest the handmade versus the machine-made, the authentic and individual versus the mass-produced replica. Sloping shapes are also unstable in terms of gravity, which give them a latent energy that can express some form of dynamism, as in many of the constructivist designs of the 1920s.

## Curvature

Curvature refers to the degree to which a shape is angular or curved, rounded. Our understanding of this derives in part from our experience of trace making, of writing and drawing. We know that  producing  straight  angular  forms  requires  brisk,  decisive  movements,  and  round  forms  a more gradual, fl  uid control of movement. But it also can be based on our experiential and cultural associations with essentially round or essentially angular objects. Human-made technological objects are often angular, and this may signify rationality, functionality, technological precision and so on, while roundness can be smooth, soft, natural, organic, maternal and so on (though we should not forget that sharp, spiky forms exist in nature as well). Clearly the fi  eld of possibilities is wide but not limitless, and it will of course be narrowed down by the context in which curvature appears.

## Connectivity

Connectivity refers to whether graphic shapes are connected or separate and self-contained. In typography it is associated with handwriting, and it shares much of its meaning potential with 'slope'. But it can also apply to abstract graphic shapes. External disconnection - disconnection between the different shapes in a design, forms, can suggest atomization, fragmentation, and external connection wholeness or integration, belonging together. Internal disconnection, disconnection between the parts of a shape may look unfi  nished and sloppy - or easy-going. Internal connection may look neat and fi  nished - or formal and buttoned-up.

## Orientation

Orientation refers to the degree to which graphic forms are either oriented towards the horizontal axis, by being comparatively fl  attened, or towards the vertical axis, stretching upwards and/or downwards. The meaning potential of this is ultimately based on our experience of gravity and of walking upright. Horizontal orientation can be solid and well grounded (no chance of toppling over), while vertical orientation can suggest upwards aspiration but also instability.

## Regularity

Many forms have deliberate irregularities, such as ragged edges, distressed textures or irregularities between repeated forms, e.g. between different occurrences of a given letter in the same text. This can be based on our experience of trace making, or on cultural connotation. Irregularity can signify the hand-made, hence the individual and the authentic, and regularity machine-made perfection. Irregularity can suggest a refusal or inability to produce neat, regular forms, hence rebellion, deviance, or disability. It can also suggest 'out of the box thinking', playfulness and caprice. Its ubiquity, also in computer-generated designs, where, on closer inspection, irregularities may have their own regularities, shows the importance of these values in contemporary communication.

It needs again to be stressed that graphic shapes can also be conventional, based on culturally well-established associations - in the case of typography, think, for instance, of copperplate, or the use of a typewriter font; and in the case of more or less abstract graphic shapes, think of semi-abstract fl  oral motifs and arabesques. Parametric design is a form of semiosis that is on the one hand based on the objective affordances of the graphic shapes and/or on our experience in producing these shapes, but on the other hand open and creative in how it uses these affordances and experiences in making meaning and in interpretation.

## 5. Timbre

The timbres of the human voice and musical instruments and, indeed, of any other kind of sound, can also be understood as the product of parametric systems (Van Leeuwen 1999, 2008). The inventory below is based on the human voice, but equally applies to musical instruments, sound effects and ambient sounds.

## Tension

Tension is the scale from tense to lax. When we tense our vocal musculature, our voice becomes higher, sharper and brighter, because in their tensed state the walls of the throat cavity dampen the sound less than they would in their relaxed state. The resulting sound not only is tense, it also means 'tense'. Our experience allows us to recognize tension in our own voice and in the voice of others, and to know when voices are likely to become tense - when we are nervous, or intimidated, or threatened, for instance - or when we are excited. This then allows a metaphorical transfer from the domain of experience to the domain of more abstract ideas and values and identities which have some component of tension in their meaning.

## Pitch range

Our experience tells us that our pitch range tends to fl  atten when we feel fl  at and listless, or bored, but also when we are speaking in a soft, intimate voice - some of those sensuous Brazilian bossa nova songs have melodies with a very fl  at range. When we are excited, on the other hand, our energy level and hence our pitch range increases - songs with a wide pitch range are the staple fare of patriotic hymns and other songs that seek to energize people into action.

Our experience of pitch range tells us two other things as well: men's voices are, on average, lower than those of women and children, and smaller resonating chambers (e.g. violins) produce higher sounds than larger resonating chambers (e.g. double basses). As a result low voices (and other low sounds, e.g. rumbling noises in horror fi  lms) can be threatening. Pitch range also has a gendered meaning potential. Men use the higher regions of their pitch range to assert themselves and to dominate, while women use the lower end of their pitch range for this purpose. But it is hard to be low and loud at the same time, so women face a diffi  cult choice. They must either speak low (which is assertive) and soft (which is intimate) - which can evoke the 'dangerous woman'

stereotype, or high ('belittling themselves') and loud ('being assertive') which may be considered 'shrill'. In either case the dominant norms of the public, assertive, 'masculine' voice are at odds with the dominant norms of the private, intimate, 'feminine' voice.

## Loudness

Loudness relates to power. The more powerful people or institutions are, the more noise they are allowed to make: 'Thunder, the voice of God, migrated fi  rst to the cathedral, then to the factory and the rock band', Murray Schafer wrote (1977: 179). Loudness also relates to social distance, both literally and fi  guratively. At close range we speak softly, for instance when we have an intimate conversation. At long range we project our voice and speak louder, for instance when we address a meeting or a classroom full of students. Loudness can therefore also suggest relationships, for instance on the scale from intimacy to formality.

## Rough, breathy and trembling sounds

In rough voices we can hear other things beside the tone of the voice itself - hoarseness, harshness, rasp, grit. The opposite of the rough voice is the smooth voice from which all noisiness is eliminated. Much of the effect of roughness comes from the a-periodic vibration of the vocal cords which causes noise in the spectrum. As this is more audible in the lower pitches, it is more easily heard in male voices and lower female voices. Perhaps this is why rough voices are common in male singing and highly valued in cultural contexts that encourage assertiveness and enterprise (Lomax 1968: 192). Again, the meaning of roughness lies in what it is - rough. Our experience tells us that roughness comes from wear and tear, whether as a result of smoking and drinking, hardship and adversity, or old age. But roughness is valued differently in different contexts. In Western classical music perfection and polish is highly valued, in African American music roughness. The same applies to popular music: the lived through voice of Tom Waits can be contrasted with Leonard Cohen's pleasant croon, the nasal trademark of Marianne Faithfull to Celine Dion's warm head voice.

In the breathy voice, another sound mixes in with the tone of the voice itself - breath. Its metaphor potential derives from our experience of what can make our voice breathy - exertion of some kind, or excitement, or sexual arousal. It often combines with a soft voice, suggesting intimacy. Advertisers use it to give their message a sensual, erotic appeal, and singers may use it for the same reason.

The meaning potential of vibrato, the trembling voice, derives from what we know makes our voice tremble - emotion. It plays a key role in the musical expression of emotion. Strings, for instance, are particularly good at producing vibrato sounds and hence universally used to 'pull the heartstrings', to present and represent love and romance. But other emotions, too, can make us tremble, and vibrato, sometimes extending into tremolo, is also used, for instance, in horror movie music, to express fear and uncertainty.

## Articulation

Some vowels are 'frontal', articulated with the tongue in the front of the mouth (e.g. the [i] of heed ), others are articulated in the back of the oral cavity (e.g. the [a] of hard ). But frontality and its opposite can also be overall articulatory settings expressing a quality of being 'upfront', 'confronting' or, of 'holding back', 'not coming out with it'.

The same can be said for 'aperture'. Some vowels are produced with the mouth comparatively closed and the oral cavity therefore comparatively small, others with the mouth comparatively open and the oral cavity therefore larger. The [i] of heed and the [u] of hood are less open, for instance, than the [a] of hard . This too can become a vocal 'setting' and part of the overall timbre, expressing some kind of 'openness' or 'closure'. In Star Wars, The Phantom Menace (2000), the treacherous Viceroy of Naboo, a character with an inscrutable fi  sh-like physiognomy, not only has a vague Chinese accent, but also speaks with a stiff jaw and an almost-closed mouth, using a breathy, hollow-sounding 'faucalized' voice.

All these parameters can also be applied to the sound of musical instruments and other sounds. Trumpets, too, can sound rough or smooth; saxophones can wail or whisper; drums can sound tense (when played with sticks) or lax (when played with mallets). Gates can sound smooth, welloiled and relatively low in pitch, or tense, high-pitched and squeaky. Even electronic sounds can be understood in these terms.

## 6. Texture

This section will deal with textural parameters that can be perceived by touch (Djonov/Van Leeuwen 2013):

## Liquidity

All surface textures have a value on a scale that runs from wet to dry. In our experience, liquidity associates positively with water and life, as in human skin, which should not be too dry, and in beauty products and foodstuffs, and negatively with rot and decay, as in the texture of rotting wood. Dryness may connote ageing - but also cleanliness and comfort, as in freshly laundered towels.

## Viscosity

Surfaces may also be sticky to different degrees. This can have negative associations, as in the term 'sticky' itself, associations with contamination, for instance, as in unclean kitchens, but it can also suggest 'grip', support and safety, as used in the ergonomic design of bicycle handle bars or the base of mouse pads.

## Temperature

Temperature is a rich source of metaphors. 'Cool' is often associated with rationality and the intellect, but it can also become 'cold' and signify a lack of affect, while warm often signifi  es affect, intimacy, and can move up the scale to the 'heat' of passion. The temperature of textured objects does not necessarily result from heating or cooling. Even under the same external conditions of temperature, different substances (say, glass or metal as opposed to wood) may have subtly different temperatures.

## Relief

Surfaces also vary in the degree to which parts of them extend below and above a horizontal plane. In the absence of such extensions, a surface is fl  at. The greater their number and frequency across space, the more relief the surface has. Relief may be caused by protuberations, pimples, bulges, bumps or embossments, or by indentations, pits, pockmarks and other depressions. Relief surfaces offer resistance to movement, and this may be engaging - there is something to explore. Flat surfaces may lack such engagement, but, on the positive side, they may feel smooth, suggesting, for instance, youth and unblemished purity. The meaning of relief will depend on a variety of factors - the smoothness or roughness of the protuberations or indentations, their size, their density, the regularity or irregularity of their distribution. A blemished skin will not please the fi  nger, but a roughly plastered white wall may suggest rural simplicity and authenticity.

## Density

Another quality of surfaces is density, which is determined by the distribution of a texture's elements in space, as in the difference between fi  nely and coarsely woven fabrics. Interpretations of density and sparsity will vary according to context. Sparsity may evoke thriftiness and freedom but also lack of durability and strength, while density may suggest solidity, the ability to withstand wear and tear, as well as high quality.

## Rigidity

Surfaces may also display resistance to the pressure of touch to various degrees: the more resistant the surface, the harder the texture. Rigidity is a rich source of experiential metaphor. Softness may be weak and submissive or sensitive and accommodating. Hardness may be strong, stable and durable, or unforgiving and harsh.

## Heterogeneity

If a texture has a single value for each of the qualities proposed so far, it is homogeneous. The greater the number of qualities that change their values across a surface and the more frequent these changes are, the more heterogeneous its texture.

## Regularity

Regular textures have predictable tactile patterns - regular indentations or regular alternations between smooth and rough. Irregular textures are characterized by unevenly distributed variations, as, e.g. in the bark of a tree. This will afford a similar range of interpretations as regularity of colour and graphic shape: hand-made or natural versus machine-made, hence authentic or natural, rebellious, playful and so on.

## Roughness

Rough textures combine a number of parameters: rough surfaces are always relief, sparse and hard, while smooth surfaces are always relatively fl  at, dense and non-viscous. They therefore share some of the meaning potential of the primary qualities that each comprises. Rough surfaces, such as the bark of an old tree, are often found in nature, and roughness (like sparseness) may result from the wear and tear of natural forces or human use or abuse. Depending on context, this may be interpreted as natural and authentic or as worn-out, old and weak. Smooth surfaces may be slick and slippery or richly silken and satiny.

## 7. Common qualities

On the basis of this account, a number of common qualities can be proposed:

## Energy

At least two colour parameters have been associated with 'energy' or 'intensity' (I take these to be closely related, as it takes energy to produce intensity) - temperature and saturation. Intensity was also associated with bold shapes, and energy with oblique, sloping shapes. But the energy of oblique shapes is only latent and perhaps does not impress the viewer as such unless associated with boldness, as, for instance, in the Nike swoosh. Energy and intensity can also be expressed by aural parameters of which we know from experience that they involve increased vocal effort - pitch range, loudness and tension. This experience can then also be brought to bear on our perception of increased pitch, loudness and tension in other sounds. It can fi  nally be argued that some textural parameters require more effort to be perceived than others, for instance rigidity (the harder the material, the more energy is needed to squeeze it) and weight (the heavier an object, the more energy is needed to lift it).

This brief discussion shows that common qualities may be realized by different parameters of the same expression medium, whether separately or in combination, as clusters. Parametric correspondences are fl  uid and fl  exible, and this enhances their creative potential, their ability to create ever new expressions of what are, deep down, the same values. We should not try to make the system neater than it is.

## Brightness

As has been long recognized in the synesthesia literature, two expression media can clearly realize 'brightness' - colour, through the parameter of value (but also through the parameters of lu- minosity and luminescence), and timbre, through the parameter of pitch: the higher the pitch, the brighter the sound. Perhaps 'frontality' can also realize brightness, both in specifi  c speech sounds (the frontal [i]) for instance appears brighter than the back [a]) and in the use of frontality as a vocal setting. Musical instruments, too, can be more or less bright, as a result of their pitch register and of the formant structures that result from the shapes of their resonators and the way they are played.

However, it is not obvious how brightness might be realized in graphic shapes and textures, and this shows that a given common quality may not be realizable in every expression medium, as has long been recognized in the synesthesia literature - 'certain combinations of synesthesia almost never occur (for example, touch to hearing)' (Harrison/Baron-Cohen 1997: 3).

## Roughness

It would appear that all expression media can, one way or another, express roughness. Graphic shapes can have ragged edges or irregularly textured areas. Sounds can be smooth or rough - 'noisy', with hoarseness, breath, rasp, grit, friction audible along with the actual tone. Aural roughness can also derive from a lack of blending between different instruments and/or voices playing or singing the same note. As for texture, surfaces can clearly feel smooth or rough to the touch. And perhaps it can be argued that colour modulation can also be rough, as when colour is applied with rough brush strokes.

This shows that the search for common qualities can prompt a rethinking of the parameters of individual expression media. Irregularity for instance, fi  rst discussed in relation to graphic shapes (Van Leeuwen 2006), turns out to include two distinct parameters - roughness (irregularity of the contours and the visual texture of the areas they enclose) and irregularity of the shapes of recurring letterforms or other shapes, or their distribution across a surface. Colour modulation, similarly, turns out to include not only modulation itself, but also the neatness or roughness of the way the different nuances of the same colour are applied, something which has been explored extensively in modern art, from the impressionists to the abstract expressionists.

## Regularity

Regularity can therefore be redefi  ned as the degree to which different instances of a given colour, shape, sound or texture are identical or different in form, and the way in which colours, shapes, sounds and textures are distributed across space and time. At the core of regularity, therefore, are the notions of predictability and constancy, notions which closely correlate with patterns of attention. In the case of graphic shapes, regularity refers to the degree to which repeated letterforms or other graphic shapes are identical or different in form, and the degree to which letterforms and shapes are distributed regularly across the surface. In the case of texture, regularity refers to the degree to which relief patterns create regular touch impressions; in the case of colour to the regularity of colour modulations; and in the case of sound to the degree in which a sustained tone varies over time in parameters such as loudness, roughness, frontality and aperture.

From this we can see that some common qualities pertain to the way other parameters are applied or distributed, something that was already discussed in Djonov/Van Leeuwen (2013), with respect to texture. We could therefore distinguish between 'simple' and 'complex' parameters.

## Differentiation

Differentiation would then be another complex parameter. In the case of colour it refers to the scale from monochrome to a maximally wide variety of colours. That it was not recognized in the case of graphic shape perhaps stems from the fact that the parametric system of graphic shape was originally worked out in relation to typography, which has a distinct set of 26 graphic shapes. But the shapes of abstract paintings and decorative patterns may well be either all stars or circles, or vary. In the case of sound, the timbre of a given sound can blend the different voice registers of a choir, or the different instruments in an orchestra, and, ever since Star Wars , movie sound effects are increasingly blended as well. In the case of touch, fi  nally, we already noted that relief patterns can consist of a single type of indentation or protuberation, or a variety of relief patterns, as e.g. in Braille.

Considerations of this kind show that thinking about expression media from the perspective of other expression media can reveal parameters that were initially overlooked.

## Connection

Connection is yet another complex parameter. We have seen that it can be applied to graphic shapes, but it has also been discussed in relation to sound (Van Leeuwen, 1999) where it is realized by the contrast between legato and staccato, which is audible, not only through the connection between sounds, but also in the timbre of the individual sounds themselves. In the case of colour and texture, the connection parameter is not immediately evident, but colours and textures (e.g. differences in textural density or roughness) may either fl  ow into each other, merge at the boundaries between them, or be separated by quite distinct boundaries.

## Expansion

It is possible to fi  nd 'expansion' not only in graphic shapes, but also in other expression media density in the case of texture, for instance, and aperture (with friction and obstruction on the other end of the scale) in the case of sound. 'Opening out' and 'closing in', 'extroversion' and 'introversion' can also be proposed as common qualities.

## Acuteness

Angularity versus roundness has been recognized as a parameter of graphic shape. Can it be recognized in other expression media as well? It is certainly the case that textures can feel sharp or blunt to the touch, and that sounds can be piercing and shrill, sharp and stabbing as well as mild and soothing. We even speak of sharpness in relation to taste ('sharp cheese') but whether it can be applied to colour is not clear.

Figure 2 provides an overview of the proposals discussed in this section. It shows that for some parameters no plausible common qualities have been found - purity and lustre in the case of colour; orientation in the case of graphic shape; vibrato in the case of sound; liquidity and viscosity in the case of texture. It is of course possible to 'translate' these parameters into other expression media, but this would probably be iconic (e.g. wavy lines to indicate vibrato, or fl  uid legato and repetitive melodies to indicate liquidity), rather than based on the visible, audible and touchable affordances of the expression media themselves, and on the tactile experiences of articulating (and in the case of touch) perceiving them.

## 8. Conclusion

It was the aim of this paper to systematically compare the inventories of the parameters of expression media, so as to explore the possibility of common qualities between them and to make a start with the development of a social semiotic take on the question of synesthesia. Two things can be said about the result. Firstly, I have called this paper a discussion paper - it is meant to open a discussion. No doubt my proposals will need to be amended and refi  ned, and in the end it will only be their application to actual multimodal analyses which can prove their value. The question which Aristotle asked nearly two and a half thousand years ago, is still fundamental for multimodal discourse analysis: How can humans perceive a unity in the multitude of sensory impressions? So far no method has been developed to analyze the integration of different expression media, and if this exploration can make a fi  rst step in that direction it will have served a useful purpose, perhaps not only for multimodal analysis but also for the development of 'synesthetic design' (Havercamp 2013).

Secondly, the notion of common qualities has a long history, and that history has, throughout, been connected to issues of perception and cognition. A social semiotic theory of synesthesia, should, on the one hand, be grounded in a solid knowledge of the material qualities of the semiotic resources we use, and the physical and physiological aspects of articulation and interpretation. Many of the common qualities we have proposed clearly connect to the basic human facilities neuroscientists have associated with the limbic brain: emotion (brightness, acuteness, expansion) and attention (regularity, energy), for instance. On the other hand social semiotics must continue to focus on the way changing social practices of meaning making and interpretation create changing semiotic resources. However much the parametric system may differ from the cosmic schemes of the Ancients, or of Feng Shui, it is, like these systems, geared to the values on which the social order is based. The parametric approach to communication has developed for a particular mode of expressing identity in which the meaning potentials of expression media, their potential for creating ever new expressions of individuality, authenticity, brightness, energy, expansiveness and so on, play a key role. But further exploration of the social aspects of synesthesia will have to wait for another occasion.

Figure 2. Common qualities of the parameters of four expression media

| Common quality   | Colour                                  | Graphic shape                                 | Timbre                                       | Texture                                                     |
|------------------|-----------------------------------------|-----------------------------------------------|----------------------------------------------|-------------------------------------------------------------|
| Energy           | Saturation Temperature                  | Weight (Slope + bold)                         | Pitch range Loudness Tension                 | Weight Rigidity                                             |
| Brightness       | Value Luminosity Luminescence           | ?                                             | Pitch level Frontality                       | ?                                                           |
| Roughness        | Modulation (application of colour)      | Irregularity (of contours and visual texture) | Roughness Breathiness Blending               | Relief (roughness)                                          |
| Regularity       | Modulation (distribution of parameters) | Regularity (distribution of parameters)       | Regularity (of parameters in sustained tone) | Relief (distribution of indentations and protuberations)    |
| Differentiation  | Differentiation                         | Differentiation (of shapes)                   | Orchestration Blending                       | Relief (differentiation of indentations and protuberations) |
| Connection       | (merging of transitions?)               | Connection                                    | Legato/staccato                              | (merging of transitions?)                                   |
| Expansion        | ?                                       | Expansion                                     | Aperture                                     | Density                                                     |
| Acuteness        | ?                                       | Angularity                                    | Staccato Friction Tension                    | (sharp protuberations?)                                     |
|                  | Purity                                  |                                               |                                              |                                                             |
|                  | Lustre                                  |                                               |                                              |                                                             |
|                  |                                         | Orientation                                   |                                              |                                                             |
|                  |                                         |                                               | Vibrato                                      |                                                             |
|                  |                                         |                                               |                                              | Liquidity                                                   |
|                  |                                         |                                               |                                              | Viscosity                                                   |

## References

Aleman, A./Rutten, G.J./Sitskoorn, M.M./Dautzenberg, H.G./Ramsey, N., F. 2001: Activation of Striate Cortex in the Absence of Visual Stimulation: An fMRI Study of Synesthesia. In Neuroreport 12(13): 2827-30. Aristotle 2008: De anima . New York: Cosimo Classics.

Campen, C. van 2008: The Hidden Sense - Synaesthesia in Art and Science . Cambridge, Mass: MIT Press.

Cytowic, R.E. 1989: Synaesthesia: a union of the senses . New York: Springer Verlag.

- Cytowic, R.E. 1997: 'Synaesthesia: phenomenology and neuropsychology. In S. Baron-Cohen/J.E. Harrison (eds.), Synaesthesia - Classic and Contemporary Readings . Oxford: Blackwell, 17-42.
- Djonov, E./Van Leeuwen, T. 2013: The semiotics of texture - from tactile to visual. In Visual Communication 10(3): 541-564.
- Harrison, J.E./Baron-Cohen, S. 1997: Synesthesia: an introduction. In S. Baron-Cohen/J., Harrison (eds.), Synaesthesia - Classic and Contemporary Readings . Oxford: Blackwell, 3-16.
- Haverkamp, M. (2013): Synesthetic Design . Berlin: Birkhäuser.
- Herder, J.G. von 2002 [1772]: Treatise on the Origin of Language, Philosophical Writings .  Cambridge: Cambridge University Press.
- Hornbostel, E.M. von 1931: Über Geruchtshelligkeit. Pfl  ügers Archiv für die Gesamte Physiologie 227: 517-538.
- Johannessen, C.M. 2010: Forensic Analysis of Graphic Trademarks - A Multimodal Social Semiotic Approach . PhD Thesis, University of Southern Denmark.
- Jakobson, R./Halle, M. 1971: Fundamentals of Language . The Hague: Mouton.
- Kress, G./Van Leeuwen, T. 2002: Colour as a semiotic mode - notes for a grammar of colour. In Visual Communication 1(3): 343-369.
- Kress, G./Van Leeuwen, T. 2001: Multimodal Discourse - The Modes and Media of Contemporary Communication . London: Arnold.
- Kress, G./Van Leeuwen, T. 2006: Reading Images - The Grammar of Visual Design . 2 nd  Edition. London: Routledge.
- Lakoff, G./Johnson, M. 1980: Metaphors We Live By . Chicago: University of Chicago Press.
- Lomax, A. 1968: Folk Song Style and Culture . Brunswick, NJ: Transaction Books.
- Machin, D./Van Leeuwen, T. 2008: Branding the Self. In C. Caldas-Coulthard/R. Iedema (eds.), Identity Trouble: Critical Discourse and Contested Identities . London: Palgrave Macmillan, 43-58.
- Marks, L.E. 1997: On colored-hearing synaesthesia: cross-modal translations of sensory dimensions. In S. Baron-Cohen/J.E. Harrison (eds.), Synaesthesia - Classic and Contemporary Readings . Oxford: Blackwell, 49-98.
- Schafer, R.M. 1977: The Tuning of the World. New York and Toronto: Knopf  .
- Van Leeuwen, T. 1999: Speech, Music, Sound . London: Palgrave Macmillan.
- Van Leeuwen, T. 2005: Introducing Social Semiotics . London: Routledge.
- Van Leeuwen, T. 2006: Towards a semiotics of typography. Information Design Journal 14(2): 139-155.
- Van Leeuwen, T. 2008: Parametric Systems: The Case of Voice Quality. In C. Jewitt (ed.), The Routledge Handbook of Multimodal Analysis . London: Routledge, 68-78.
- Van Leeuwen, T. 2010: The semiotics of decoration. In K.L. O'Halloran/B.A. Smith (eds.), Multimodal Studies - Exploring Issues and Domains . London: Routledge, 115-131.
- Van Leeuwen, T. 2011: The Language of Colour. London: Routledge.
- Van Leeuwen, T. 2015: Looking Good; Aesthetics, Multimodality and Literacy Studies. In J. Rowsell/K. Pahl (eds.), The Routledge Handbook of Literacy Studies . London: Routledge, 426-435.