7

## Multimodality, Synaesthesia and Intersemiotic Translation

This thematic issue brings together articles that address the challenging complexity which increasingly characterize contemporary communication across various domains. From perspectives belonging to multimodality, synaesthesic research and intersemiotic translation, the six articles of this thematic issue address complex communication practices related to education, corporate communication, media, culture, etc. Apart from specifi  c purposes, each of these articles has a main goal: to stretch the multimodal boundaries of research work in order to be able to account for the changing landscape of contemporary communication.

The fi  rst article by Ilaria Moschini, entitled The 'Face with Tears of Joy' Emoji . A Socio-Semiotic and Multimodal Insight into a Japan-America Mash-Up , has only one word in focus, namely the Oxford Dictionaries Word of the Year (2015): the 'Face with Tears of Joy' emoji . Moschini's article is focused on this pictograph decoding its many semantic and semiotic layers, with a special focus on the context of cultures out of which it originates. According to Moschini, this 'pictographic word' is to be seen as a marker of the mashing up of Japanese and American cultures in the discursive practices of geek communities. The author adopts a social semiotic multimodal approach in order to explore the history of this 'pictographic word' from language to language, from culture to culture, from niche discursive communities to the global scenario, through the lenses of 'transduction' (Kress 1997). The cultural aspect of intersemiotic translation is thus highlighted in this article by diachronically discussing transpacifi  c intersemiotic translations in the context of the global postmodern computer culture where aesthetic and intellectual traditions stimulate each other across national boundaries.

The next paper by Maria Grazia Sindoni, entitled 'The Semantics of Migration.' Translation as Transduction: Remaking Meanings across Modes ,  approaches also a new transduction phenomenon. This time, the novelty is related to complex communication strategies employed when informing about a contemporary crisis, namely to the media rhetoric of labelling people on the move according to explicit and implicit agendas of encouraging or discouraging sympathy. Exploring multimodal data from the British digital press reporting on the 'European migrant crisis' in 2015, the author explains how pictorial materials (e.g. pictures and diagrams) can amplify, reduce or even contradict what is argued in the verbal text. The selected online columns are commented qualitatively from a multimodal critical discourse framework of analysis. The transduction strategies employed by contemporary media outlets are discussed by considering the possible ideological shifts that can be experienced by audiences across cultural borders.

Tuomo Hiippala's  article, Individual  and  collaborative  semiotic  work  in  document  design , shows how multimodal analyses may be complemented with insights gained by using ethnographic methods. The author examines agency, transduction and transformation issues in the context of organizational communication. By analyzing an annual report published by Finnair and interviewing its designers, the author clarifi  es the design process and its consequences for the document in question. The impact of the involvement of multiple professionals (concept planners, copywriters, graphic designers and other specialists) upon the semiotic work is discussed in terms of the agency that is continuously negotiated. The author contrasts these fi  ndings from the interviews with a detailed description of the report's multimodal structure by working with a stratifi  ed defi  nition of a semiotic mode.

* Carmen Daniela Maier Department of Business Communication Aarhus BSS, Aarhus University Jens Chr. Skous Vej 4 DK-8000 Aarhus C cdm@bcom.au.dk

In the next article, Semiotic shortcuts: The graphical abstract strategies of engineering students , Carmen Sancho Guinda draws attention to the convenience of including graphicacy (Trumbo 2006) and graphical abstracts design in higher education curricula. According to the author, science and technology students will sooner or later have to cope with the consequences of globalisation and with the visuality inherent in their fi  elds when communicating academically or professionally with both expert and lay people. By analyzing aeronautical engineering students' graphic abstracts in terms of medium, format, type of representation, guidance devices employed, and instances of authorial subjectivity, the author demonstrates the broad potential of this genre.

Judith Leah Cross' article entitled Enigma: Aspects of Multimodal Inter-Semiotic Translation addresses several aspects of multimodal inter-semiotic translation (MIST). According to the author, MIST is an extremely multifaceted process because it can entail analysing several modes while making cultural assumptions, positing hypotheses, selecting and sequencing elements, and simultaneously positioning the translation across languages, times and spaces. The article explores and evaluates the integrity of combined translations of individual semiotic systems implied by the new multimodal recontextualization of the same stories from books to fi  lms. In her present analytical work, the author selects specifi  c texts from the written biography, Alan Turing: The Enigma (Hodges 1983), in order to compare their re-contextualisation, interpretation and translation into a multimodal text, namely the movie The Imitation Game (2014). The focus of this article is on showing to what degree the meanings of the texts have been accurately and signifi  cantly translated into image, action, music and speech.

The last article of this thematic issue opens a discussion that goes beyond multimodality research as we know it today and proposes a social semiotic theory of synaesthesia. In his article, A social semiotic theory of synesthesia? - A discussion paper , Van Leeuwen draws our attention to the increasing multimodality of contemporary communication and to the need of acquiring a more solid knowledge both of the semiotic resources, and of the physical and physiological aspects of their articulation and interpretation. Van Leeuwen has sketched before (2005) how the integration of semiotic modes (other than the text-image relation) can be explored, but in this article, he takes a step further by addressing in detail the integration of 'production media' (Kress/Van Leeuwen 2001: 66). After presenting briefl  y a series of ideas about synæsthesia in philosophy and the arts, he shows how the descriptive frameworks multimodality researchers have developed for semiotic modes can help identify some of the 'common qualities' that have eluded the neuroscientists. Van Leeuwen reviews the parameters of the individual expression media and then reframes the issue of synesthetic correspondences from a semiotic point of view and explores what correspondences there may exist between the parameters of different expression media and hence between the meanings that can be expressed by them. In order to do this, he systematically compares the common qualities of the parameters of colour, graphic shape (including typography), timbre and texture. His exploration is meant to underscore their semiotic importance in the contemporary multimodal communication of identity and to introduce a social semiotic theory of synaesthesia that could help us interpret the changing semiotic landscape in more nuanced ways.

This thematic issue intends to contribute to a more nuanced understanding of the fast growing complexity of contemporary communication by approaching issues beyond language and by introducing some of the manifold research questions and approaches fostered by their emergence. Maybe this issue will stimulate future discussions about how it is necessary to stretch the theoretical and empirical boundaries of communication research in general in order to approach new contemporary phenomena and their complexity.

## References

Kress, Gunther 1997: Before Writing: Rethinking the Paths to Literacy . London/New York: Routledge.

- Kress, Gunther/Van Leeuwen, Theo 2001: Multimodal Discourse. The Modes and Media of Contemporary Communication . London: Arnold.
- 'Oxford Dictionaries Word of the Year', Oxford Dictionaries Blog , November 16 2015 [online]. http://blog.oxforddictionaries.com/2015/11/word-of-the-year-2015-emoji
- Trumbo, Jean 2006: Making Science Visible: Visual Literacy in Science Communication. In Pauwels, Luc (ed.), Visual Cultures of Science. Rethinking Representational Practices in Knowledge Building and Science Communication. Hanover, NH: Dartmouth College Press, 266-283.

Van Leeuwen, Theo 2005: Introducing Social Semiotics . London/New York: Routledge.