11

## The 'Face with Tears of Joy' Emoji . A Socio-Semiotic and Multimodal Insight into a Japan-America Mash-Up

## Abstract

The blog site of the Oxford Dictionaries features a post dated November 16 2015, which announces that, 'for the fi  rst time ever', their 'Word of the Year' is not a word, but a pictograph: the 'Face with Tears of Joy' emoji . The term emoji , which is a loanword from Japanese, identifi  es 'a small digital image or icon used to express an idea or emotion in electronic communication' (OED 2015).

The  sign  was  chosen  since  it  is  the  item  that  'best  refl  ected  the  ethos,  mood,  and  preoccupations  of  2015'. Indeed, the Oxford Dictionaries' President, Caspar Grathwohl declared that emojis are 'an increasingly rich form of communication that transcends linguistic borders' and refl  ects the 'playfulness and intimacy' of global digital culture.

Adopting a socio-semiotic multimodal approach, the present paper aims at decoding the many semantic and semiotic layers of the 2015 'Word of the Year', with a special focus on the context of cultures out of which it originates. More in detail, the author will focus on the concept of translation as 'transduction', that is the movement of meaning across sign systems (Kress 1997), in order to map the history of this 'pictographic word' from language to language, from culture to culture, from niche discursive communities to the global scenario. Indeed, the author maintains that this 'pictographic word' is to be seen as a marker of the mashing up of Japanese and American cultures in the discursive practices of geek communities, now gone mainstream thanks to the spreading of digital discourse.

## Keywords

Geek Culture - Emojis - Emoticons Kaomoji - Manga - Multimodality - Social Semiotics - Transduction

'I don't consider [my drawings] pictures I think of them as a kind of hieroglyphics … I'm writing a story with a unique type of symbol' Osamu Tezuka

## 1. Introduction

The blog site of the Oxford Dictionaries features a post dated November 16 2015 which announces that, 'for the fi  rst time ever', their 'Word of the Year' is not a word, but a pictograph: the 'Face with Tears of Joy' emoji (see Figure 1).

Figure 1 . The Face with Tears of Joy Emoji

<!-- image -->

*   Ilaria Moschini Department of Languages, Literatures and Intercultural Studies University of Florence Via S. Reparata, 93-95 - 50129 Firenze Italy

ilaria.moschini@unifi  .it Hillary Clinton ©

@HillaryClinton

Tell us in 3 emojis or less.

The term ' emoji' is a loanword from Japanese (the plural of which can be both emoji or emojis ), a portmanteau that blends ' e 'picture' and moji 'letter, character' and identifi  es 'a small digital image or icon used to express an idea or emotion in electronic communication' (OED 2015).

The word was chosen since it is the item that 'best refl  ected the ethos, mood, and preoccupations of 2015'. Indeed, the Oxford Dictionaries' President, Caspar Grathwohl declared that emojis are 'an increasingly rich form of communication, that transcends linguistic borders' and refl  ects the 'playfulness and intimacy' of global digital culture, which is 'visually driven, emotionally expressive, and obsessively immediate' (Steinmetz 2015).

According to data from the Oxford Dictionaries Corpus and Oxford Dictionaries' digital partner, the mobile technology company SwiftKey (2015), the 'Face with Tears of Joy' was the most used emoji in 2015 ('20% of all the emojis used in the UK in 2015, and 17% of those in the US'), while the  word emoji itself has faced a similar increase, since 'its usage more than tripled in 2015 over the previous year' . In addition to that, in March 2015, Instagram declared that digital language has 'evolved such that nearly half of comments and captions on [its platform] contain emoji characters' (Davis/Edberg 2015).

Emojis thus appear to be a widespread method of communication that can potentially cross language barriers and, even, enrich written language since, as the linguist Ben Zimmer affi  rms, emojis can 'function as a new form of punctuation' (Robb 2014).

Moreover, the acquisition by the Library of Congress (Allen 2013) of the intersemiotic 'emojifi  ed' translation of Herman Melville's classic, Moby Dick (Benenson 2010) may also suggest that emojis can be used, like pictograms, to vehicle ideas not only emotions. This conceptual/ narrative usage of emoji characters has been underlined also by the electoral tweet sent by Senator Hillary Clinton in August 2015, where she exhorted her followers to express 'in 3 emojis or less' their feelings about student loan debts (see Figure 2).

Figure 2 . Hillary Clinton's Electoral Tweet

<!-- image -->

The present paper aims at decoding the many semantic and semiotic layers of the 'Face with Tears of Joy', with a special critical focus on the cultural contexts out of which it originates. More in detail, it will focus on the concept of translation as multimodal 'transduction' ,  that is the movement of meaning across sign systems (Kress 1997) in order to socio-semiotically 'map the history of the resource ' of emojis (van Leeuwen 2004: 3) and its 'translation' from language to language, from culture to culture, from niche discursive communities to the global digital scenario. It is a theoretical standpoint that can be contextualized in the emerging interest in Translation Studies towards multimodality, especially with reference to texts that present a multi-coded structure.

As regards its structure, the paper will start from an outline of the theoretical framework adopted; it will then concentrate on the fi  rst set of emojis, those 'pictographic words' created by the Japanese telecommunication planner Shigetaka Kurita and on their main linguistic/semiotic and cultural infl  uences.

y Follow The text will proceed with the analysis of the globalization process of emojis, from their encoding in the Unicode system to their visual hybridization with the 'smiley' symbol. Finally, it will focus on the transpacifi  c intertextual chain of semiosis that pragmatically and historically links emojis to emoticons and kaomoji , that are common markers of facial expressions created with keyboard characters by niche sub-cultural communities, such as early computer users and fandoms.

## 2. Intersemiotic Translation and Multimodal Studies

The English term 'translation' derives from the Old French verb ' translater ' and/or directly from the Latin ' translatus ', the past principle of the verb ' transferre ', that means 'to carry over, to bring over' (OED 2015). The etymology of the term reveals the foundational idea around which the activity of translating is conceptualized in Western Countries (Tymoczko 2005a), that is to 'carry something across', this 'something' being the meaning which is transferred across cultural, national and linguistic borders.

The effect of such movements from a code to another code, from a country to another country, from a culture to another culture usually implies a wide and varied spectrum of changes and alterations in the semantic and semiotic sphere. This is especially true in the contemporary scenario that features the spreading of digitally mediated texts, which are constantly re-codifi  ed and re-signifi  ed giving rise to linguistic, semiotic and cultural hybrids.

As a consequence of the massive technological developments and of the related social process of globalization, interests in Translation Studies have been increasingly moving towards the third category of translation theorized by Roman Jakobson, namely 'intersemiotic translation' (Tymoczko, 2005b: 1090). Indeed, in his seminal paper 'On linguistic aspects of translation' (Jakobson 2004 [1959]: 139), the Russian-American structuralist identifi  ed three types of translation: in the context of the same linguistic code, from language to language and from a signifying system to another.

More in detail, Jakobson's taxonomy featured the fi  rst category as an 'intralingual traslation', that is 'an interpretation of verbal signs by means of other signs of the same language' (e.g. a process of rewording a specialized jargon into common language). His second category, the 'interlingual translation' is a 'proper translation' and refers to the activity of interpreting 'verbal signs by means of some other language'. His fi  nal category, 'intersemiotic translation', or 'transmutation' was defi  ned as the 'interpretation of verbal signs by means of signs of non-verbal sign systems'. A defi  nition that draws on semiotics, that is the general science of communication through signs and sign systems, of which language is but one (Munday 2012).

The above-mentioned link between Translation Studies and semiotics was favored by Jakobson's use of Peircean semiotics, which is based on the assumption that all signs can be translated into other signs (Stecconi 2010). Such initial connection can probably explain why Peircean semiotics fi  gures prominently in research on intersemiotic translation (Gorlé  e 1994, Torop 2000), while other approaches like, for instance, those emerging from the functional and socio-semiotic tradition such as Multimodality do not appear.

However, the situation has changed in the last decade since Translation scholars have dedicated their attention to texts that feature a multisemiotic composition (e.g. Taylor 2004, Gambier 2006, Snell-Hornby 2006, Dejardins 2008, O'Sullivan/Jeffcote 2013, Borodo 2015) and have advocated for theories and frameworks that would help in the practice of translating artifacts, the design of which features the structural combination of different semiotic modes (Kress/van Leeuwen 2001).

Understanding how people make meaning through various semiotic resources and the relationship of these practices with self-representation or with the representation of others is a major focus of Multimodal Studies, especially in its social semiotic formulation (Jewitt 2009). Indeed, a socio-semiotic approach to multimodality postulates a common communication trajectory that involves three interconnected steps: what people want to signify, the relationship between what X 4-0

people want to signify and which ways of communication or signs are more suitable in given socio-cultural contexts (Kress/Jewitt 2003).

The production of signs, that is a 'combination of meaning and form' (Kress 1997: 6), is 'motivated' (Hodge/Kress 1993) and people's moves from mode to mode and from media to media are driven by interests and by 'what is possible to express and represent, readily, easily with a mode, given its materiality and given the cultural social history of the mode' (Kress/Jewitt 2003: 14). 'Mode', which is a key term in Multimodal Studies, refers to the set of resources that people in a given culture can use to communicate, while the word 'media' refers to the channel of communication (Kress 1997: 7).

This socio-semiotic and multimodal approach to intersemiotic translation will be used in the present analysis, as previously said, to historically outline the chain of the subsequent resemiotizations (Kress 2000, Iedema 2001) of the 'Face with Tears of Joy' starting from the creation of the fi  rst set of emojis .

## 3. The cultural and technological background of Japanese emoji

The Japanese telecommunication planner Shigetaka Kurita is credited with inventing emojis in 1999, when he was involved in the launch of a mobile Internet platform called 'i-mode' (Blagdon 2013).

At the time, Kurita was working at NTT Docomo, one of the largest mobile services in Japan as a member of the 'i-mode' team, the aim of which was to provide Internet service on a type of mobile phones, the so-called 'featured phones', which had small LCD screens that could only 'contain' up to 48 letters. This monochrome site of display would have highly limited the exchange of information, and this is the reason why Kurita started to plan the creation of alphanumeric images that would take the emoticons (that had already gained popularity on the Internet) to a strictly iconic level.

Kurita stated that the American AT&T was already offering information for cell phone users but 'everything was shown by text, even the weather forecast', even if 'Japanese weather forecasts have always included pictures or symbols to describe the weather' (Nakano 2015). Kurita's familiarity with this way of producing information led him to envision the possibility to create text messages using images rather than words.

The fi  rst set of emojis designed by him was made of 180 items that comprised not only smiley faces representing emotions, but also objects and symbols such as light bulbs or bombs (see Figure 3). 'I was working with the sense of creating a new alphabet - he declared - It was an attempt to create texts rather than a sense of making pictures' (Nakano 2015).

Figure 3 . Some of the fi  rst emojis designed by Shigetaka Kurita, © NTT Docomo

<!-- image -->

For inspiration, Kurita turned primarily to 'manga, kanjii characters and street signs' (Negishi 2014). As regards manga , that are Japanese comics, the inspiration Kurita derived from them was related primarily to their unique use of symbolic representation of emotions . Indeed, as Wallested (2013: 5) explains:

Manga have a large diversity of metaphorical fi  gure symbols called keiyu , that are not considered as words or representational pictures, but act as symbolic adjectives or adverbs to events depicted. Keiyu consist of manga symbols ( manpu ) and effect symbols ( kouka ). [These] symbols are applied to characters or subjects as representational indicators denoting their physical ( butsuriteki ) states and/or as metaphorical indicators connoting their psychological ( shinriteki ) states (Natsume/Takekuma, 1995).

Many of the metaphorical fi  gure symbols mentioned in the quotation require an effort to be decoded, since their conceptual underpinnings are deeply rooted in Japanese cultural and visual conventions ,  like  the  'gigantic  sweat  drop  conveying embarrassment or nervousness', that is shown in Figure 4 (Cohn 2010: 7).

Figure 4. Visual Morpheme 'Giant Sweat Drop', source Morphology of Japanese Visual Language - retrieved at http://www.visuallanguagelab.com/A/jvlmorphology.html

<!-- image -->

As Wallestad explains, 'one of the most used keiyu-manpu is the drop ( suiteki )', which can represent 'both physical and psychological states' (2013: 6). Indeed, such a symbol can be applied to different parts of the face, with its placement infl  uencing the meaning, and be associated with 'sweat, tears, saliva, nasal discharge, or water. [When] applied to the eyes, the drop(s) ( suiteki ) become tears ( namida ) that can express sadness or overwhelming joy' (Wallestad 2013: 6).

The use of keiyu-manpu is a cultural and aesthetic tradition that, according to the French historian Jean-Marie Bouissou (2011 [2010]: 127), can be traced back to ancient woodblock printing techniques, where movements were suggested by the eyes and the mouths of the represented subjects and to the tradition of the Kabuki theatre, where a wide range of emotions was rendered by the grotesque facial distortions of the actors.

The 'Face with Tears of Joy' emoji appears to be consistent with such a visual vocabulary, especially with the symbolic representation of drops as ' namida ' , that are here used to express a profound and unrestrained state of contentment.

As regards the infl  uence of kanji , the logographic Chinese characters adopted in Japan, Kurita declared that he 'took from kanji the ability to express abstract ideas' in a single character (Blagdon 2013). As a matter of fact, ideograms derive from pictographs - that are pictorial representations for words and phrases - which imply a symbolic conceptualization of ideas. For what concerns their shape, they all feature a squared form and are usually made up of two elements: the ' hen ', which defi  nes the semantic area of the kanji , functioning thus as a sort of key to the general meaning of the character, and a 'body' that specifi  es its actual meaning (Taylor/Taylor 1995).

Japanese traditional writing system seems to have deeply infl  uenced the structure of the basic unit of manga, where each element is not a neutral embellishment but, rather, a means of expression (Bouissou 2011 [2010]: 119). Indeed, as Hirofumi Katsuno and Christine Yano explain (quoting Natsume/Takekuma, 1995), 'a single frame of manga consists of not only simply drawings and words, but a complex visual grammar of subject, object, word balloon, movement, background keiyu (fi  gure symbol) and on'yu (sound symbol)' (2002: 213).

F9F5

~

F9CC

F6B5

F996

Together with the conceptual representation of objects and ideas, Kurita's original emojis appear to have inherited from kanji characters the vertical orientation and their squared form . Moreover, like their predecessors kaomoji (that are Japanese emoticons), emojis have drawn upon the visual language of manga where the eyes are considered 'the locus of facial expressivity' (Katsuno/Yano 2002: 214).

The standardization of manga visual language is commonly attributed to Osamu Tezuka (Cohn 2010: 4), who is defi  ned as the 'godfather of modern manga' and who had been greatly infl  uenced by the drawings of Walt Disney animated movies (Schodt 1983, 1996). As a matter of fact, the salience of the eyes, which has become a distinctive feature of manga visual style , is an element that Tezuka derived from Disney and, more precisely, from the movie Bambi , where huge round eyes were used to emphasize the emotive expressions of the faces (Pellitteri 2008: 198). This element of manga visual design is a marker of the different and subsequent economic and cross-cultural encounters between America and Japan that have occurred since the Nineteenth century (Schodt 1994).

Going back to the creation of emojis, the modal affordance Kurita could use to design them allowed him to 'play' only with a 12-by-12-pixel grid to translate images into a set of characters able to be easily displayed on mobile screens. The result was a collection of very simple signs: the original smiling face, for instance, featured up-side down Vs to indicate the eyes and a rectangle that served as the mouth (see the fi  rst emoji on the left side of the fi  rst column in Figure 3).

Apart from the simplicity of the design due to the above mentioned technological restraints, if we compare Kurita's 'happy face' with the 'Face with Tears of Joy' emoji, we can easily see that the form is different, since the 'Face with Tears of Joy' is a yellow rounded sign .

In order to explain such a difference and to decode the socio-cultural and pragmatic meanings associated to it, it is necessary to trace out other steps in the transduction across languages and across cultures of the face sign that has become a pervasive component of contemporary global digital culture, starting from its inclusion in the Unicode encoding system.

## 4. Going global: from Japan to the Unicode

In 1999 emojis began to be available on Japanese mobile phones and became immediately popular, although they created problems related to the fact that their codifi  cation had not been standardized. Indeed, since each mobile phone carrier had developed different (even if partially overlapping) sets of characters, each mobile vendor used their own text encoding extensions.

Figure 5. Mark Davis' Keynote Speech at the 38 th  Unicode Conference (a snapshot), ©Unicode - retrieved at https://www.youtube.com/watch?v=-n9ONNeACyw

<!-- image -->

As Unicode President and co-founder Mark Davis (2014) explained in his keynote speech at the 38 th  International and Unicode Conference, which was dedicated to 'the past, the present and the future of emojis', there were codes that were associated to different images, but there were also almost identical images that were identifi  ed by different codes, such as the penguins in Figure 5. The result was that characters from different platforms could not always be displayed and emojis

=

Code

1 U+1F600

2 U+1F601

3 U+1F602

4 U+1F603

5 U+1F604

Brow.

Chart Apple Goog.

One Twtr. Wind. GMail DCM KDDI

missing

CRED

missing

missing SB Name missing GRINNING FACE

GRINNING FACE WITH SMILING BYES

FACE WITH TEARS OF JOY

were replaced by the symbol of the question mark that was meant to signify the 'untranslatability' of the message. The situation worsened when non-Japanese mobile companies began to support exchange with the original carriers (Davis/Edberg 2015).

It  was Google that submitted an offi  cial request to the Unicode Consortium to have emojis standardized since, in 2006, it had decided to enter the Japanese mobile market (Nasser/ Benenson et al., 2013). The interest in Japanese mobile market was due to the fact that it was more developed if compared to its American counterpart (Baron 2008: 128). However, it was only with the release of Apple's mobile operating system iOS 5 in late 2011, that emojis 'made their real international debut' (Blangdon 2013).

Technically, nowadays the term 'emoji' refers to a single set of Unicode characters that is associated to an image (Davis/Edberg 2015). Unicode is an international computer industry standard for encoding and displaying most of the world's writing systems that allows one operating system to recognize text from another (Unicode Consortium 2015b). The Unicode represents the foundation for software internationalization and standardization and, as Joseph D. Becker one of the co-founders - affi  rmed (1998 [1988]: 1), it was envisioned as 'a new, worldwide ASCII' that is the American Standard Code for Information Interchange. Indeed, prior to the creation of Unicode, much of the computer world relied on the character encoding for English text, which was the 7-bit ASCII. Since the people of the world, 'need to be able to communicate and to compute in their own native language, other than English', with the spreading of computers there was the rising of the necessity of an 'international/ multilingual text encoding standard that [was] reliable as ASCII, but that cover[ed] all the scripts of the world' (Becker 1998 [1988]: 2).

At semiotic level, the creation and the standardization of emojis deal with the transduction of the message (e.g. the 'Face with Tears of Joy') across two different coding systems: the internal code defi  ned as hexadecimal (e.g. 1F602) and the glyph that is its graphic representation (Unicode Consortium 2015a). While the fi  rst has been fully standardized by the Unicode encoding system, which means that for each conceptual image a unique identifi  cation number is featured, the graphical form of such a code varies with the implementation of emojis across different platforms. Indeed, as one can see in Figure 6, the representation of the Unicode character 1F602, named the 'Face with Tears of Joy' features a range of visual interpretations.

Figure 6. Unicode Emoji Chart, (a selection) ©Unicode http://unicode.org/emoji/charts/full-emoji-list.html

<!-- image -->

The version of the 1F602 glyph chosen by the Oxford Dictionaries to indicate the 'word of the year' is the one designed by Apple , widely recognized as a de-facto standard (Nasser/Benenson et al. 2013). It is a visual representation that, as we shall see in the next paragraphs, intertextually links emojis to emoticons and to the famous symbol of the 'smiley' face.

## 5. The 'Smiley' and its optimistic commercial ethos

The history of the 'smiley' face is 'a narrative that encompasses big business, popular culture and anti-establishment sentiment […] and a series of bitter copyright issues' (Piercy 2013). However, it is largely accepted that the original version of the sign (see Figure 7) was created in 1963

by Harvey Ross Ball, an American freelance designer who was commissioned by the Worcester Mutual Insurance Company to create a new logo in order to 'raise morale among the employees of an insurance company after a series of diffi  cult mergers and acquisitions' (Stamp 2013).

The image was reproduced on button badges for their employees to wear and to distribute to clients to illustrate the new 'service with a smile' company ethos (Piercy 2013). The structural connection between the original 'smiley' face and pin-back buttons as sites of display was meant to reinforce the illocutionary force of the message, since badge pins - a US patent dated 1896 - are natively advertising media that were originally used for political slogans.

Figure 7. The original smiley face as designed by Harvey Ball © The World Smiley Foundation

<!-- image -->

Political buttons have been used in the United States since 1789, when George Washington's supporters wore buttons imprinted with a slogan during the fi  rst presidential inauguration, while the fi  rst campaign buttons were used in 1824 (Badgeomania 2014). However, it is only at the beginning of the Twentieth Century, that US companies started to exploit the communicative potential of these patented pin-buttons, which functioned as billboards displaying persuasive messages in the form of slogans or pictures.

Going back to the 'smiley' button, it gained popularity at the beginning of the 1970s when, combined with the saying 'Have a Happy Day', achieved its iconic status and became the 'ubiquitous icon of post-Vietnam America' (Piercy 2013). Through a chain of subsequent resemiotizations (Kress 2000, Iedema 2001) the image of the smiley face has featured a range of material realizations from pin-buttons to t-shirts, from pillows and posters to ecstasy pills according to the variation of the affordance of modes, of their materiality and of their grounding social, cultural and aesthetic practices. Contextually, its meaning has changed, varying from the original optimistic message of the Worcester Insurance Company, to become a commercialized logo, to an ironic fashion statement, to a symbol of rave culture. However, the 'smiley' appears to primarily embody the spirit of American capitalism and, more precisely, the optimistic commercial ethos we mentioned above (Stamp 2013). And it is with this connotation that we fi  nd it bloodstained and critiqued on the cover of the fi  rst number of the cult comic book series Watchmen , published by DC Comics (Moore et al. 1986-1987).

·WATCHME

Figure 8. Cover of the fi  rst number of Alan Moore's Watchmen , 1986  © DC Comics

<!-- image -->

As regards its physical appearance, the 'smiley' features a rounded form, which was probably functional to its original material realization as a pin button and that recalls the conceptual symbolic representation of a human face. The absence of perspective and the frontal angle 'demand' the viewers to  enter  into  a  friendly  relationship  (Kress/van  Leeuwen  2006  [1996]:  145). The friendship of the invitation is visually marked by the expression of the abstract face, where the smiling mouth (especially in Ball's original design) represents the most salient element in terms of proportions. Another marker of happiness is the color, a bright fully saturated yellow that reinforces the ' positive ' message carried by the mouth . Indeed, as regards the psychological and cultural meaning of this color, van Leeuwen - quoting Faber Birren (1961: 143) - affi  rms ''modern Americans' see yellow as 'cheerful, inspiring, vital, celestial' and relate it to 'high spirit' and 'health'' (2011: 56).

Such a semiotic confi  guration creates a visual form of direct address that asks the viewer to enter into a relation of social affi  nity, thus implicitly demanding to share the same attitude and, probably, also to adhere to the same set of values . Indeed, it is a form of visual reciprocity which is functional to the strengthening of 'mutual' relationship and affi  nity between the viewers and the represented subjects that Kress and van Leeuwen (2006 [1996]: 118), along with Belting (1990: 57), date back to the devotional use of holy images in thirteen-century monasteries.

As regards Apple's design of the emoji smiling faces (see Figure 6), their rounded shape and their color recall Ball's smiley, and also the shades (that can be clearly seen in Figure 1) suggest a corporeity that reminds of the pin-button, as if the most relevant meaning were materially fi  xed in the original pin badge. At the same time, salience appears to be given to both the mouth and the eyes, thus embodying, in this latest digital realization of the smiley, the mashing-up of two different and culturally connoted loci of facial expressivity . It is a combination that can be potentially related to the 'convergence' of discursive communities belonging to different cultural areas, such as US computer scientist user groups, Japanese otaku and sh ō jo subcultures, American Sci-Fi and manga/anime fandoms in the realm of global digital language, in the wake of subsequent transpacifi  c exchanges . It is an intertextual chain of semiosis that - as outlined in the next paragraph - starts with emoticons and ends up in the creation of emojis.

## 6. Emoticons, Kaomoji and Emojis

Despite the resemblance of the word 'emoji' to the English term 'emoticon' (composed by the words 'emotion' and 'icon'), which - according to the Oxford Dictionaries Blog has favored its memorability - emojis are different from emoticons since the latter indicate facial expressions that are created using keyboard characters, while emojis are - as we have seen - pictorial signs. In addition, the cultural origin of the two items of digitally mediated communication is different. As a matter of fact, while emoticons stemmed out of the community of US computer scientist user groups , the creation of kaomoji - that are Japanese emoticons and the 'ancestors' of Kurita's emojis -can be traced back to youth culture, in particular to teenage female subcultures ( sh ō jo) and to obsessive fans of manga , otaku (Azuma 2010 [2001]).

The fi  rst use of an emoticon in a computer message was reported by professor Scott E. Fahlman, who was - as he affi  rms - the fi  rst to use a colon followed by a hyphen and a parenthesis to represent a smiley face in 1982 (Krohn 2004). He posted the graphic representation of a facial expression in a message to an online electronic bulletin board during a discussion 'about the limits of online humor' and about the necessity 'to denote comments meant to be taken lightly' (Lovering 2007). In his message - the transcription of which can be read below 1 - he pointed out the necessity to give computer users a way to mark 'jokes'.

19-Sep-82 11:44  Scott E  Fahlman :-) From: Scott E  Fahlman <Fahlman at Cmu-20c> I propose that the following character sequence for joke markers: :-) Read it sideways.  Actually, it is probably more economical to mark things that are NOT jokes, given current trends.  For this, use :-(

Emoticons, thus, were created with the explicit function of being 'humor indicators' , which are fundamental - as the New Hackers Dictionary affi  rms - in on-line forums where the 'lack of verbal and visual cues can cause what were intended to be humorous, sarcastic, ironic, or otherwise non-100%-serious comments to be badly misinterpreted' (Raymond 1993).

The reference to humor is also culturally connoted since humor and, more precisely, irony which involves playing with form - is a defi  ning feature of hacker subculture that 'love[s] pranks, clever programming tricks, toys and games' (Isaacson 2014: 201). Indeed, hackers celebrate values such as creativity, fun, informality, personal engagement, liberty, gaming and code mastery. Another structural feature of computer culture - which is one of the four pillars of geek culture (Konzack 2014: 59) - is the love of science fi  ction and fantasy and the partaking in the related fandoms, as Stewart Brand described in his Rolling Stone article dated 1972, 'one of the fi  rst and still one of the most quoted descriptions of the Bay area computer scene' (Turner 2006: 116).

It  seems that emoticons themselves can be traced back to the discursive practices of sci-fi and fantasy fandoms ; indeed, according to the astrophysics professor and science fi  ction author, Gregory Benford (1996), 'most of the Net's 'emoticons' […] had appeared in fanzines by the 1950s'. Such fandoms were mostly male communities (Jenkins 2013 [1992]) who used typed-onpaper emoticons like Fahlman's digital ones as visual clues of ironic adherence or disagreement.

In Japan, before the advent of commercial Internet that occurred in 1993, emoticons were familiar to the community of academic users who were connected to American and European universities and research institutes via a non-commercial network, JUNET (Japanese Unix Network). On the other hand, general public users, who were connected through commercial networks, started to elaborate their own form of expressing emotions using keyboards and drawing on pre-existing narrative forms. As a matter of fact, many of otaku 's designed face marks (literally, kaomoji ) feature elements of manga visual style (Wallestad 2013: 8). More in details, they were developed in the sub-community of 'early and heavy users of personal computers' that were typically 'young males with narrowly focused interests, such as computers or comics' and that were referred to as otaku , the colloquial for ' geek ' (Katsuno/Yano 2002: 201). In this initial stage, kaomoji were inserted at the end of the name, as if they were part of the signature. With the advent of the commercial use of the Internet in Japan, kaomoji started to spread beyond the boundaries of otaku subculture. However, as Katsuno and Yano highlight (2002: 211), the true popularization of kaomoji 'rested in the hands of the most popular pagers in Japan and female high-school students'. Some scholars (Sugimoto/Levin 2000) have connected the Japanese female youth use of kaomoji in text messages on pagers and, later on, on mobiles to the emergence of new literary practices and to the rise of 'kitten writing' in which women use kawaii (cute) culture (Kinsella 1995) to create new vernacular forms of language (Hjorth 2009: 68).

1 For the complete retrieval of the entire discussion thread see http://www.cs.cmu.edu/~sef/Orig-Smiley.htm (December 2015).

The contrast between American and Japanese face marks is mainly expressed by the direction of the symbol , since the emoticons are read at perpendicular angles with the words - i.e. : ) -, while kaomoji are read in line with words, like (^\_^); and by the main locus of expressivity, that in America is the mouth (hence, smiley), while in Japan is the eyes (Katsuno/ Yano 2002: 206). We may also argue that, as far as the pragmatic use of such symbols is concerned, originally the two elements of computer-mediated communication (CMC) features different purposes: as a matter of fact, the main aim of Fahlman's emoticons was to provide the meta-communicative frame according to which messages have to be decoded; while kaomoji appear to natively present a visual, conceptual and diegetic function.

Indeed, studies in CMC suggest that - also in contemporary digital discourse - emoticons are used primarily not as markers of emotion (Baron 2000, Crystal 2001, Walther/D'Addario 2004), but as indicators of the pragmatic illocutionary force of the utterances they are co-textually related to (Markman/ Ō shima 2007; Dresner/Herring 2010). It is the same function envisioned by Fahlman when he proposed to use emoticons as ' irony markers' .

On the other hand, kaomoji - being rooted in manga visual language and in kanji writing appear to be more performative and expressive than emoticons since they can function as both ' punctuating devices and potentially units of larger meaning ' (Markman/ Ō shima 2007: 14). As Katsuno and Yano (2007) explain, kaomoji feature a quite complex structure and may incorporate words, movements and onomatopoeic sounds like single frames of manga. Such a complexity allows kaomoji to stand on their own and to express meanings working better than the verbal code.

The fi  rst  set  of  emojis directly stems out of the discursive tradition of Japanese emoticons, since, as we have seen, Shigetara Kurita developed them in order to facilitate the otherwise complex creation of kaomoji symbols on keyboards and thus overcome the limits imposed by the modal affordances of mobile phones (Adami/Kress 2010).

Their use as narrative instruments is evident in many examples such as the already quoted Benenson's 'emojifi  ed' version of Moby Dick or the emoji translation of President Obama's State of the Union Address 2015 published by The Guardian-US (2015). However, the analyses of Chiusaroli (2015) Nasser, Benenson et al. (2013) and Stark and Crawford (2015) show that, when emojis are used in global digital discourse (that is not in the specifi  c cultural context out of which they originated), they are able to convey common and 'universal' meanings, while their full semantic and pragmatic realization is highly dependent on the presence of a verbal co-text / context .

## 7. Conclusions

The aim of the present article was the decoding of the semantic and semiotic layers of the 'Face with Tears of Joy' emoji , which was chosen by the Oxford Dictionaries as the 'Word of the Year' 2015 . In order to map the history of the sign, a socio-semiotic perspective has been adopted and the concept of translation as 'transduction' has been used to trace the movement of meaning across sign systems, with a special critical focus on the cultural contexts out of which they originated.

The author maintains that this 'pictographic word' is to be seen as a marker of the merging of Japanese and American cultures in the discursive practices of geek communities. In particular, she argues that such a sign can be defi  ned a Japan-America mash-up since it presents traces of the two different cultural traditions as well as of the subsequent waves of transpacifi  c exchanges.

Indeed, the 'Face with Tears of Joy' that was selected by the Oxford Dictionaries is Apple's proprietary design (which is the de-facto standard of contemporary digital discourse) and presents the fusion of manga visual style and Japanese kaomoji with the tradition of pin-button 1960s smiley faces celebrating the optimistic commercial ethos of US capitalism. The intertwining of the two cultural traditions has occurred primarily thanks to the exchanges of the discursive practices of computer culture communities , which are deeply intertwined with the practices of geek communities such as sci-fi  , fantasy and comics fandom, now become popular with the spreading of digital discourse .

As a matter of fact, the fl  ow of cultural and subcultural material between America and Japan has created a global postmodern culture where aesthetic, artistic, and intellectual traditions infl  uence each other across national boundaries (Murakami 2005, Kelts 2006, Tatsumi 2006). In particular, such transcultural processes have given rise to a 'pop-cultural fanaticism' that has now become mainstream because of a key generation of people 'raised on a diet' of transpacifi  c cultural products (Rivera 2008: 137).

It appears thus that the Oxford Dictionaries' word of the year here analyzed being - as it is - an example of such an intertextual magma, could be considered a potential component of the 'database fantasyscape', the virtual 'repository' theorized by Brian Ruh (2014) that, merging Azuma's database concept (2007, 2010 [2001]) and Napier's idea of fantasyscape (2007), aims to describe the fl  ow of anime and manga in a transmedia and transnational context. Such an interpretation would conceptualize emojis, like anime and manga characters, as just another global postmodern 'tile' people may gather in order to create their own DIY narrations.

In addition, the transduction of Kurita's emojis to the Unicode encoding system and the creation of the new global standard for the Japanese signs suggest a process of 'universalization' which is deeply imbued with American cultural values , a feature that appears to be distinctive of digital discourse (Moschini 2013).

## Acknowledgements

I would like to thank Dr Gianluca di Fratta for his precious supervision on Japanese culture. Many thanks also go to the reviewer who invited me to read Brian Ruh's work on the database fantasyscape.

## References

(All the digital sources were last accessed in December 2015)

- Adami, Elisabetta/Kress, Gunther 2010: The Social Semiotics of Convergent Mobile Devices: New Forms of Composition and the Transformation of Habitus. In Kress Gunther (ed.), Multimodality. A Social Semiotic Approach to Contemporary Communication . London & New York: Routledge, 184-197.
- Allen, Erin 2013: A Whale of an Acquisition. In Library of Congress Blog, February 22 [online]. http://blogs.loc.gov/ loc/2013/02/a-whale-of-an-acquisition
- Azuma, Hiroki 2007: The Animalization of Otaku Culture. In Mechademia 2, 175-187.
- Azuma, Hiroki 2010: Generazione Otaku. Uno studio della postmodernità. Milano: Jaca Book. [ D ō butsuka suru Posutomodan. Otaku kara mita Nihon shakai . T ō ky ō : K ō dansha 2001].
- Badgeomania 2014: How the Button Was Born , February 6 [online]. https://www.facebook.com/notes/badgeomania/ how-the-button-was-born-us-patent-564356-pinback-button-badge-history/541935432568735?fref=nf
- Baron, Naomi S. 2000: Alphabet to Email : How Written English Evolved and Where It's Heading . London/New York: Routledge.
- Baron, Naomi S. 2008: Always On: Language in an Online and Mobile World . Oxford & New York: OUP.
- Becker, Joseph D. 1998 [1988]: Unicode 88 .  Mountain View, CA: Unicode Consortium Press [online]. http://www. unicode.org/history/unicode88.pdf

Belting, Hans 1990: The Image and its Public in the Middle Ages . New Rochelle, NY: Aristide D. Caratzay.

Benenson, Fred (ed) 2010: Emoji Dick. New York: Harper-Collins Publishers.

- Benford, Gregory 1996: alt.fans. The Internet is Recapitulating Science Fiction Fandom. In Reason, January, 1 [online]. https://reason.com/archives/1996/01/01/altfans
- Birren, Faber 1961: Color Psychology and Color Therapy . Secaucus, NJ: University Books.
- Blagdon, Jenn 2013: How Emoji Conquered the World. The Story of the Smiley Face from the Man Who Invented it. In The Verge , March 4 [online]. http://www.theverge.com/2013/3/4/3966140/how-emoji-conquered-the world
- Borodo, Michal 2015: Multimodality, Translation and Comics. In Perspectives. Studies in Translatology 23 (1), 22-41.
- Bouissou, Jean-Marie 2011 [2010]: Il Manga. Storia e Universi del Fumetto Giapponese , Latina, Tunué Editori dell'immaginario [ Manga. Histoire et univers de la bande dessinée japonaise , Arles, Édition Philippe Picquier, 2010].
- Brand, Stewart 1972: Spacewar: Fanatic Life and Symbolic Death among the Computer Bums. In Rolling Stone , December 7.
- http://digitizingamerica.shanti.virginia.edu/sites/shanti.virginia.edu.digitizingamerica/fi  les/SPACEWAR%20-%20 by%20Stewart%20Brand.pdf
- Chiusaroli, Francesca 2015: La scrittura in Emoji tra dizionario e traduzione. In Bosco, Cristina/Zanzotto, Fabio M./ Tonelli, Sara (eds.), Proceedings of the Second Italian Conference on Computational Linguistics - Trento, Dicembre . Torino: Accademia University Press, 88-93.
- Cohn, Neil 2010: Japanese Visual Language: The Structure of Manga. In Johnson-Woods, Toni (ed.), Manga: An Anthology of Global and Cultural Perspectives. New York: Continuum Books, 187-203.
- Crystal, David 2001: Language and the Internet . Cambridge, UK: Cambridge University Press.
- Davis, Mark 2014: Unicode Emoji - Past , 38th Internationalization and Unicode Conference Keynote Speech [online]. https://www.youtube.com/watch?v=-n9ONNeACyw
- Davis, Mark/Edberg, Peter 2015: Unicode Emoji Unicode® Technical Report #51 [online]. http://www.unicode.org/ reports/tr51/index.html
- Desjardins, Renée 2008: Inter-Semiotic Translation and Cultural Representation within the Space of the Multi-Modal Text. In TranscUlturAl 1 (1), 48-58.
- Dresner, Eli/Herring, Susan C. 2012: Emoticons and Illocutionary Force. In Dana Riesenfel/Scarafi  le Giovanni (eds.), Philosophical Dialogue: Writings in Honor of Marcelo Dascal . London: College Publication, 59-70.
- Gambier, Yves 2006: Multimodality and Audiovisual Translation. In Carroll, Mary/Gerzymisch-Arbogast, Heidrun/ Nauert, Sandra (eds.), MuTra 2006 - Audiovisual Translation Scenarios: Conference Proceedings .  Copenhagen: MuTra [online]. http://euroconferences.info/proceedings/2006\_Proceedings/2006\_Gambier\_Yves.pdf
- Gorlé  e, Dinda L. 1994: Semiotics and the Problem of Translation: With Special Reference to the Semiotics of Charles S. Peirce (Approaches to Translation Studies) . Amsterdam/Atlanta (GA): Rodopi.
- Hjorth, Larissa 2009: The Art of Being Novel: Rethinking Cartographies of Personalization. In Cubitt Sean/Paul Thomas (eds.), Re: Live - Third International Conference on the Histories of Media Art, Science and Technology Proceedings. Melbourne: The University of Melbourne Press, 65-72 [online]. http://www.mediaarthistory.org/relive/ ReLive09Proceedings.pdf
- Hodge, Robert/Kress, Gunther 1993: Language as Ideology . London/New York: Routledge.
- Iedema, Rick A.M. 2001: Resemiotization. In Semiotica 137, 1(4), 23-39.
- Isaacson, Walter 2014: The Innovators. How a Group of Hackers, Geniuses and Geeks Created the Digital Revolution . New York: Simon & Schuster.
- Jakobson, Roman 2004 [1959]: On Linguistic Aspects of Translation. In Venuti, Lawrence (ed.), The Translation Studies Reader . London/New York: Routledge, 138-43.
- Jenkins, Henry 2013 [1992]: Textual Poachers: Television Fans and Participatory Culture (Studies in Culture and Communication) . New York: Routledge.
- Jewitt, Carey 2009: An Introduction to Multimodality. In Jewitt, Carey (ed.), The Routledge Handbook of Multimodal Analysis . London: Routledge, 11-27.
- Katsuno, Hirofumi/Yano, Christine R. 2002: Face to Face: On-line Subjectivity in Contemporary Japan. In Asian Studies Review 26 (2), 205-231.
- Kelts, Roland 2006: Japanamerica. How Japanese Pop Culture Has Invaded the US. New York: Palgrave MacMillan.
- Kinsella, Sharon 1995: Cuties in Japan. In Skov, Lise/Moeran, Brian (eds.), Women, Media and Consumption in Japan. London: Curzon Press, 220-254.
- Konzack, Lars 2014: The Origins of Geek Culture: Perspectives on a Parallel Intellectual Milieu. In Bowman, Sarah L. (ed.), Wyrd Con Companion Book . Costa Mesa: WyrdCon 4, 52-59.
- Kress, Gunther 1997: Before Writing: Rethinking the Paths to Literacy . London/New York: Routledge.

- Kress, Gunther 2000: Text as the Punctuation of Semiosis: Pulling at Some Threads. In Meinhof, Ulrike H./Smith, Jonathan M. (eds.), Intertextuality and the Media: from Genre to Everyday Life. Manchester: Manchester UP, 132-154.
- Kress, Gunther/Jewitt, Carey 2003: Introduction. In Kress, Gunther/Jewitt, Carey (eds.), Multimodal Literacy. New Literacies and Digital Epistemologies . New York: Peter Lang, 1-18.
- Kress, Gunther/van Leeuwen, Theo 2001: Multimodal Discourse: The Modes and Media of Contemporary Communication . London: Hodder Arnold.
- Kress, Gunther/van Leeuwen, Theo 2006 [1996]: Reading Images. The Grammar of Visual Design . London/New York: Routledge.
- Krohn, Franklin 2004: A Generational Approach to Using Emoticons as Non-Verbal Communication . In Journal of Technical Writing and Communication , 43, 321-328.
- Lovering, Daniel 2007: Digital 'Smiley Face' Turns 25 :-). In Live Science ,  September 18 [online]. http://www.livescience.com/4604-digital-smiley-face-turns-25.html
- Markman, Kris M./ Ō shima, Sae 2007: Pragmatic play? Some Possible Functions of English Emoticons and Japanese Kaomoji in Computer-Mediated Discourse. Paper presented at the Association of Internet Researchers Annual Conference 8.0: Let' s Play!. Vancouver, BC, Canada, October.
- Moore, Alan/Gibbons, Dave/Higgins, John 1986-1987: Watchmen . Burbank (CA): DC Comics.
- Moschini, Ilaria 2013: Liberty Icons: Linguistic and Multimodal Notes on the Cultural Roots of Digital Technologies. In LEA - Lingue e Letterature d'Oriente e d'Occidente , 2, 537-552 [online]. http://dx.doi.org/10.13128/LEA-1824484x-1384
- Munday, Jeremy 2012: Introducing  Translation  Studies.  Theories  and Applications .  London/New York: Routledge [Kindle Version].
- Murakami, Takashi 2005: Little Boy: The Arts of Japan's Exploding Subculture . New Haven (CT): Yale UP.
- Nakano, Mamiko 2015: Why and How I Created Emoji , Interview with Shigetaka Kurita. In Ignition [online]. http:// ignition.co/105
- Napier, Susan J. 2007: From Impressionism to Anime: Japan as Fantasy and Fan Cult in the Mind of the West . New York: Palgrave Macmillian.
- Nasser, Ramsey/Benenson, Fred/Salditch, Zoë/Wortham, Jenna/Weber, Lindsey (moderator) 2013 : 'I Have No Words'. Emoji and the New Visual Vernacular . Panel Discussion at the Emoji Art & Design Show, New York, December 14 [online]. http://www.emojishow.com/video
- Natsume, Fusanosuke/ Takekuma, Kentar ō (1995). Manga to iu 'kigou'. In M. Inoue (ed), Bessatsu Takarajima EX, Manga no yomikata . T ō ky ō : Takarajimasha, Inc., 73-124.
- Negishi, Mayumi 2014: Meet Shigetaka Kurita, the Father of Emoji . In The Wall Street Journal/Japan , March 26 [online]. http://blogs.wsj.com/japanrealtime/2014/03/26/meet-shigetaka-kurita-the-father-of-emoji/
- OED 2015 Emoji entry [online]. http://www.oxforddictionaries.com/defi  nition/english/emoji
- O'Sullivan, Carol/Jeffcote, Caterina 2013 (eds.): Translating Multimodalities. In Journal of Specialized Translation , 20, Special Issue.
- 'Oxford Dictionaries Word of the Year', Oxford Dictionaries Blog , November 16 2015 [online]. http://blog.oxforddictionaries.com/2015/11/word-of-the-year-2015-emoji
- Pellitteri, Marco 2008: Il Drago e la Saetta. Modelli, strategie e identità dell'immaginario giapponese . Latina: Tunué Editori dell'Immaginario.
- Piercy, Joseph 2013: Symbols: A Universal Language . London: Michael O'Mara Books [Kindle Version].
- Raymond, Eric S. (ed) 1993: The New Hacker's Dictionary . Cambridge (MA): MIT Press [online]. http://catb.org/~esr/ jargon/html/index.html
- Rivera, Renato 2008: The Popularization of Geek Culture, and the Marginalization of Otaku Culture. In Journal of Kyoto Seika University 34, 122-140.
- Robb, Alice 2014: How Using Emoji Makes Us Less Emotional. And What Linguists Say it Means if Your Smiley Face Has a Nose. In New Republic ,  July 7 [online]. https://newrepublic.com/article/118562/emoticons-effect-way-wecommunicate-linguists-study-effects
- Ruh, Brian 2014: Conceptualizing Anime and the Database Fantasyscape. In Mechademia 9, 164-175.
- Schodt, Frederik 1983: Manga! Manga! The World of Japanese Comics . New York: Kodansha USA Inc.
- Schodt, Frederik 1994: America and the Four Japans. Friend, Foe, Model, Mirror . Berkeley, CA: Stone Bridge Press.
- Schodt, Frederik L. 1996: Dreamland Japan: Writings on Modern Manga . Berkeley, CA: Stone Bridge Press.

- Snell-Hornby, Mary 2006: The Turns of Translation Studies: New Paradigms Or Shifting Viewpoints? Amsterdam/ Philadelphia: John Benjamins.
- Stamp, Jimmy 2013: Who Really Invented the Smiley Face?. In The Smithsonian Blog , March 13 [online]. http://www. smithsonianmag.com/arts-culture/who-really-invented-the-smiley-face-2058483/?no-ist
- Stark, Luke/Crawford, Kate 2015: The Conservatism of Emoji : Work, Affect, and Communication. In Social Media + Society July-December 1 (2), 1-11.
- Stecconi, Ubaldo 2010: Translation Studies. In Gambier, Yves/ van Doorslaer, Luc (eds.), The Handbook of Translation Studies. Amsterdam: John Benjamins, vol. 1, 314-319.
- Steinmetz,  Katy  2015:  Oxford's  2015  Word  of  the  Year  Is  This  Emoji. Time ,  November  16  [online].  http://time. com/4114886/oxford-word-of-the-year-2015-emoji
- Sugimoto, Taku/Levin James A. 2000: Multiple Literacies and Multimedia: A Comparison of Japanese and American Uses of the Internet. In Hawisher Gail E./Selfe Cynthia L. (eds.), Global Literacies and the World Wide Web . London: Routledge, 133-153.
- Tatsumi, Takayuki 2006: Full Metal Apache. Transactions Between Cyberpunk Japan and Avant-Pop America . Durham and London: Duke UP.
- Taylor, Christopher 2004: Multimodal Text Analysis and Subtitling. In Ventola, Eija/Cassily, Charles/Martin Kaltenbacher (eds.), Perspectives on Multimodality . Amsterdam/Philadelphia: John Benjamins, 153-172.
- Taylor,  Insup/Taylor,  Maurice  Martin  1995: Writing  and  Literacy  in  Chinese,  Korean,  and  Japanese .  Amsterdam/ Philadelphia: John Benjamins.
- The Guardian - US, 2015, January 20: State of the Union … in Emojis [online]. http://www.theguardian.com/us-news/ ng-interactive/2015/jan/20/-sp-state-of-the-union-2015-address-obama-emoji
- Torop, Peeter 2000: Intersemiosis and Intersemiotic Translation. In European Journal for Semiotic Studies 12(1), 71100.
- Turner, Fred 2006: From Counterculture to Cyberculture. Stewart Brand, the Whole Earth Network and the Rise of Digital Utopianism . Chicago (IL): University of Chicago Press.
- Tymoczko, Maria 2005a: Enlarging Translation Theory: Integrating Non-Western Thought about Translation. In Hermans, Thomas (ed.), Translating Others . Manchester: St. Jerome, 13-32.
- Tymoczko, Maria 2005b: Trajectories of Research in Translation Studies. In Journal des Traducteurs / Meta: Translators' Journal 50 (4), 1082-1097.
- Unicode Consortium 2015a: Full Emoji Data [online]. http://unicode.org/emoji/charts/full-emoji-list.html
- Unicode Consortium 2015b: What is Unicode? [online]. http://www.unicode.org/standard/WhatIsUnicode.html
- van Leeuwen, Theo 2011: The Language of Colour. An Introduction . London/New York: Routledge.
- van Leeuwen, Theo 2004: Introducing Social Semiotics . London/New York: Routledge.
- Wallestad, Thomas J. 2013: Developing the Visual Language of Comics: the Interactive Potential of Japan's Contribution. In Expressive Culture 7, 3-13.
- Walther, Joseph P./D'Addario, Kyle B. 2004: The Impacts of Emoticons on Message Interpretation in Computer-Mediated Communication. In Social Science Computer Review 19, 324-347.