## Individual and Collaborative Semiotic Work in Document Design

## Abstract

This  article  examines  the  concepts  of  agency,  transformation  and  transduction  in  the  context  of  document  design. These concepts have been previously used to describe communicative actions and sign-making among individuals: whereas agency focuses on the individual's capabilities  as  a  sign-maker,  transformation  and  transduction  describe how individuals transform meanings within one mode of communication or from one mode to another. Organizational communication, however, is rarely an individual effort, particularly in corporate settings: producing multimodal documents that communicate on behalf of entire organizations, such as annual reports, constitutes a collaborative effort involving a variety of specialists, such as concept planners, copywriters and graphic designers.

In  the  age  of  increasing  specialization,  this  kind  of  collaborative  semiotic  work  raises  questions  about  agency, transduction  and  transformation.  In  this  context,  the  concepts  of  agency  and  transmodality,  which  emphasize  the individual, appear to have reduced explanatory power. This leads to the central question of this article, that is, how can the collaborative design process be captured and how does it affect the multimodal structure of annual reports? By analyzing an annual report published by Finnair and interviewing its designers, this article aims to illuminate the design process and its consequences to the document in question.

## Keywords

multimodality, document design, corporate communications, annual reports, agency, transduction, transformation

## 1. The need to interrogate the design process

When it comes to analyzing documents and other multimodal artefacts, researchers are often content to examine the concrete outcome of the design process, that is, how written language, photographs, illustrations and other modes of communication co-operate and interact on the selected material canvas. Understanding these kinds of semiotically-charged organizations and their ways of doing communicative work on different materialities is a core issue within the fi  eld of multimodal research (Bateman 2011: 20). Moreover, given the wealth of questions about multimodality that remain unanswered, the decision to focus on the concrete artefact at hand is understandable, as tracing its entire life-cycle would add considerably to the workload. Consequently, the design process often remains shrouded to the analyst. As Kress (2014: 144) observes:

'In looking at a website, I have access only to the product of the design. Any descriptions or analyses about specifi  cations of the design, about initial conceptions, or the design process are, necessarily, hypotheses.'

Yet multimodal researchers are often keen to form hypotheses about the motivations behind a particular design. This obviously presents a situation in which additional 'data is needed to complement, to 'fi  ll out', data which one theory by itself cannot produce' (Kress 2011: 240). In short, our hypotheses about the design process need to be tested, and given the shortcomings of multimodal theories in this area, other fi  elds of study may need to be brought in to support the analysis.

* Tuomo Hiippala Centre for Applied Language Studies University of Jyväskylä Finland tuomo.hiippala@iki.fi

45

Previous research has, in fact, suggested that investigations targeting the design process should be pulled much closer towards the heart of multimodal research. Bateman (2008), for instance, argues that an improved understanding of the design process is ultimately necessary for a realistic appraisal of multimodal documents. He argues that neglecting the design process 'opens up the danger of over-interpretation because more design freedom, and hence more controllable resources for making meaning, are assumed than are actually available' (Bateman 2008: 18). As Kress (2011) points out, ethnographic methods come across as a natural candidate for exploring how multimodal artefacts are designed (see also Rowsell 2013).

To explore how multimodal analyses may be complemented with insights gained using ethnographic methods, this article focuses on a specifi  c genre: the corporate annual report. Annual reports, which provide information about the fi  nancial and operational performance of a corporation, are strictly regulated in terms of their content. Financial information, for instance, cannot be used to position the corporation in relation to its competitors. Consequently, corporations use non-fi  nancial content to distinguish themselves from their competitors: the genre of annual report does so multimodally by using photographs, diagrams and other forms of visual expression alongside written language (de Groot 2008: 18-19). These multimodal features, together with the constraints affecting their content, make the annual report a prime target for a study combining multimodal analysis and ethnography.

Corporations frequently outsource the design of annual reports to external communications agencies (Stanton/Stanton 2002). From the perspective of multimodal analysis, outsourcing the design process raises several questions: How do concept planners, copywriters, graphic designers and other specialists contribute to the annual report? How are their respective contributions negotiated during the design process? What drives the decision to invest in a specifi  c semiotic mode? The involvement of multiple professionals is particularly interesting in terms of agency, transduction and transformation, which are concepts that multimodal researchers have used to examine an individual's capability to draw on different semiotic modes and to transform meanings between them (Mavers 2011, Newfi  eld 2014, Archer/Breuer 2015). As such, these concepts relate directly to the aforementioned issues of design freedom and to the control over resources available for making meaning.

To identify areas of multimodal research in which the risk of over-interpreting the outcome of the design process is particularly high, I subject the concepts of agency, transduction and transformation to a critical examination in a context that involves multiple individuals performing highly specialized roles. To do so, I complement the multimodal analysis of the annual report with insights about the design process gained by interviewing the professionals responsible for creating the report.

The article itself is structured as follows: I begin by considering the notion of agency in the context of document design, while simultaneously paying attention to the kinds of semiotic work undertaken by the participating individuals, and by examining the constraints that limit their capability to engage in such actions. I then continue with an exposition of the methods and data, before proceeding to the analysis, which contrasts the fi  ndings from the interview with the results of the multimodal analysis. Finally, I conclude the article with a discussion of the results and their implications.

## 2. What constrains document design?

Agency, in short, refers to the capacity to act, which arises from the sociocultural setting of an individual (Hearn 2001: 112). Within multimodal research, the concept of agency has been mainly given consideration within the stream of research known as Mediated Discourse Analysis (hereafter MDA; see e.g. Norris/Jones 2005). Explicating the central tenets behind MDA, Jones/Norris (2005, 169) argue that within many approaches to linguistics and social sciences, agency is often discussed in terms of autonomous individuals, who are completely in control of their actions.

MDA, in contrast, presupposes that agency is never unrestricted, but always subject to constraints arising from socio-cultural setting and social practice. In this aspect, MDA largely agrees with the anthropological defi  nition provided by Hearn (2001), but also extends it by proposing that agency is always distributed among human actors. To put it simply, agency is constantly negotiated between individuals (Jones/Norris 2005: 170).

To narrow down the broad concepts of social setting and practice in order to enable a move towards document design, a useful parallel may be drawn here between MDA and the Genre and Multimodality model (hereafter GeM; Bateman 2008), a framework developed for the description of multimodal documents. The GeM model defi  nes multiple sources of constraints that affect document design, and therefore, all agencies involved in the design process. These include canvas constraints, which relate to how some canvas, such as a printed page, can be manipulated to carry various types of documents, while production and consumption constraints describe how the production technologies and the planned use of the document infl  uence its form and structure. Finally, the available semiotic modes and their established patterns of use for achieving specifi  c communicative goals present a major constraint to any multimodal document (Bateman 2008: 18).

I will provide detailed examples of these constraints later: what needs to be attended to at this point are the available modes of expression. Theories of multimodality work with the assumption that all communication is multimodal, that is, communication involves multiple modes of communication that interact and co-operate with each other. These modes can make and convey meanings, and for this reason, they are often ascribed the epithet semiotic. Not surprisingly, given its prominent position among theories of multimodality, the defi  nition of what constitutes mode continues to be debated in the fi  eld. For current purposes, I will draw on the defi  nition proposed by Bateman (2011, 2014a), due to its compatibility with the GeM model. In short, Bateman's defi  nition acknowledges that the underlying materiality determines the kinds of expressive resources available to the semiotic mode, while also emphasizing that the modes must be structured in a way that supports their interpretation in a given context.

To make sense of how canvas, production and consumption constraints affect multimodal documents, the GeM model bundles these constraints together with the semiotic modes into a notion of multimodal genre. For this reason, the GeM model can be used to describe the multimodality of the annual report and the constraints that affect the agencies involved in designing the report. As a multimodal genre, the annual report exhibits patterned forms of expression, which have been set out extensively by de Groot (2008). These genre patterns are fl  exible enough to admit the kind of variation naturally encountered among the wealth of annual reports in circulation, but they must not be broken in order to meet the audience's genre expectations. In this way, the genre sets constraints to the agencies: unless they adhere to the genre constraints that affect the use of semiotic modes, the annual report may fail to meet its communicative goals.

This is also the context in which the concept of agency becomes contested: document design is often a joint effort, which is likely to involve various professionals ranging from concept planners to copywriters, art directors and many others, and each professional brings their own expertise to the table (Hiippala 2015b: 32-33). To put it simply, the annual report is not the product of a solitary, omnipotent and autonomous agent, but the outcome of the collaboration of multiple (and potentially confl  icting) agencies, which all wield infl  uence over the document in question. For this reason, I argue that the design process deserves far more attention in multimodal research than it has received so far: as pointed out above, without a proper appraisal of this process, there is a considerable risk of over-interpreting the selections made in the document.

Initial work in exploring the relationship between agency and multimodality was pursued by Kress et al. (2001), who distinguished between designing representations and messages. They proposed that designing representations involves identifying the semiotic modes suitable for the task at hand, whereas designing a message involves combining these selections into larger wholes, for instance, into a multimodal document such as the annual report in a way that corresponds to the expectations directed towards the genre:

'With design, the agency of the individual as designer/sign-maker has become central: interest is that which fi  nds its realization in the design, modulated of course by an awareness of the audience and the constraints which its expectations bring. Agency is present in the transformative aspect of sign-making and in the designing of the message for communication.' (Kress et al. 2001: 7).

As sign-makers, the specialists involved in the design process, such as copywriters, project managers and art directors may differ considerably in terms of agency, that is, in their capability to exploit the available semiotic modes. In concrete terms, copywriters may be particularly familiar with the linguistic register of corporate communications within a specifi  c fi  eld of business, whereas project managers know precisely what needs to be done to deliver the report in time and in accordance to the regulations governing annual reports. Art directors, in turn, are likely to possess a particularly good eye for establishing a unifi  ed visual appearance for the annual report and excel in organizing the content into a coherent whole. These specializations, however, also limit the individual agencies participating in the design process: their differing skill sets constrain their capability to draw on different semiotic modes - a project manager is unlikely to be able to stand in for the graphic designer, who in contrast should not be expected to take up writing copy (cf. Hiippala 2015b: 57-58).

The degree of specialization in  contemporary document design also raises questions about the transformative aspects of meaning-making to which Kress et al. (2001: 7) allude in the quote above. Transmodal remaking of meaning, or transduction, as it is traditionally understood in social semiotic approaches to multimodality, refers to the individual's capability to transform meanings from one semiotic mode to another (Newfi  eld 2014). Mavers (2011: 106) provides a succinct defi  nition of transduction as an 'agentive act of shifting semiotic material across modes.' The concept has been frequently used to describe individual sign-makers, particularly in educational contexts, either as participants in classroom interaction and as producers of multimodal artefacts (see e.g. Archer/Breuer 2015).

To draw on a simple example, students may choose to manipulate different materials and to write, draw or do both in response to an assignment, thus expressing the intended meanings in more than one semiotic mode. These kinds of transductions are completely natural, but the concepts remain analytically valid only as long as a specifi  c individual can be held responsible for the agentive act of making meaning. As Jones/Norris (2005, 170) point out, 'agency seems much easier to attribute to individuals when considering moment-by-moment actions.' In the analysis of organizational communication and document design, however, these concepts rapidly lose their explanatory power, as transductions are negotiated between multiple agencies over longer periods of time.

A similar problem emerges with the concept of transformation, which describes rearrangements within a specifi  c semiotic mode, as opposed to carrying over meanings from one semiotic mode to another as a part of a transduction (Bezemer/Kress 2008: 175-176). Observing that transformations can occur both within individual semiotic modes and their combinations, Bezemer/Kress (2008: 188-189) illustrate their point by using the layout of a page from an elementary school biology textbook. They contrast the original version with their own redesign, which adopts the principles of information value zones proposed in Kress/van Leeuwen (2006). This transformation, however, is not motivated by communicative needs that drive the formation of genres, but guided by the exceedingly abstract concept of information value zones, which has drawn considerable criticism in research on information design and multimodality (Waller 2012, Thomas 2014).

That being said, the example discussed by Bezemer/Kress (2008) provides a skewed picture of agency, because any non-hypothetical transformation within a professionally-produced document would be constrained by both genre expectations and the agencies participating in the design process. These constraints would likely work against the abrupt switch between a two- and a three-column layout and leaving empty space around the lower diagram to conform to the proposed information value zones (Bezemer/Kress 2008: 189). In contrast, valid transformations motivated by communicative needs in professional communication could be more accurately described as moves in the space of possibilities offered by the genre (Bateman 2008: 224). As I will show in the subsequent analysis, these moves are indeed jointly negotiated by the participating agencies (cf. Norris/Jones 2005).

Overall, the example discussed above emphasizes why considering professional document design as an individual effort warrants caution. A more appropriate appraisal of the design process might entail considering how multiple agencies negotiate their contribution towards a common communicative goal, while operating within constraints set by the genre selected to do the communicative work - in this case, an annual report. In the following sections I propose one possible way of pursuing research in this area, beginning with a presentation of the methods and data, and followed shortly by the analysis of an annual report.

## 3. Methods and data

In order to investigate how individuals participating in the design of an annual report negotiate their contributions in terms of agency, transformations and transductions, I adopted a mixed method approach combining multimodal and ethnographic analysis. This involved two steps, which are described below.

First,  I  organized  a  semi-structured  interview  with  a  project  manager  (Master's  degree  in French; four years of work experience) and a graphic designer (Bachelor's degree in graphic design;  15  years  of  work  experience)  working  for  Miltton,  a  Helsinki-based  communications agency, which was hired to create the annual report for Finnair - the national fl  ag carrier of Finland. The report for the year 2014, which constituted the main data for the study, was thoroughly examined in preparation for the interview. Based on the preliminary examination, I presented several guiding questions to the interviewees, which are provided in Table 1, in order to lay the groundwork for discussing specifi  c parts of the annual report together. I recorded the interview, which lasted for one hour and fi  ve minutes. The recording was later transcribed by a third party.

1. What is your educational background?

2. How many years of work experience you have in the field of professional communication?

3. Do you re-use parts of the annual report from the previous year for designing a new report?

4. How many people participate in creating the annual report? What are their roles?

5. How much content does the client supply?

6. Does the client have influence over the structure and organization of the content in the annual report?
7. Does the client have its own image bank?
8. Is the report intended to be read on screen or on paper?

Table 1. Guiding questions presented to the interviewees

Second, I analyzed a part of the annual report - a total of six pages comprising of non-fi  nancial information, which was discussed with Miltton's employees - using the GeM model, in order to provide a rough sketch of the multimodal structure of this sequence (Bateman 2008: 115-129). In this case, applying the GeM model involved breaking each page into units, such as headers, paragraphs, photographs and diagrams, while also representing their logical organization, that is, identifying which elements belong together. The resulting description was stored into a fi  le in XML markup language, using the annotation schema provided by the GeM model. I then visualized this description using computational tools developed for working with GeM-annotated multimodal corpora (Hiippala 2015a).

Finally, the fi  ndings from the semi-structured interview were considered together with the description of the annual report's multimodal structure. In the following sections, I report on these fi  ndings, extending the description to other analytical layers of the GeM model as necessary.

## 4. Analysis

The analysis begins with a description of planning and producing an annual report, before shifting the attention to the sequence discussed with Miltton's employees. I then consider this sequence in greater detail, characterizing the semiotic modes and their structural confi  guration, while also paying specifi  c attention to issues of agency. Next, I target a part of the six-page sequence to examine the notions of transduction and transformation, before moving on to the concluding remarks.

## 4.1.  Collaborative planning and production

Miltton's employees characterized the production of an annual report as a hectic process. The content is received in quick bursts and rushed into the annual report as the fi  scal year comes an end and reporting begins. Yet the annual report consists of much more than just numbers: as noted above, fi  nancial information cannot be used to position a corporation in relation to its competitors. This establishes non-fi  nancial information as a ground for competition, and corporations regularly benchmark their annual reports against those of their competitors (cf. de Groot 2008: 81-82). The annual report may thus be considered a form of strategic communication, which is carefully planned according to the current and anticipated operational situation of the company (Erickson et al. 2011).

Outsourcing strategic communications naturally requires close cooperation between the client and the customer. During the planning stage, the client and the communications agency meet face-to-face to discuss the forthcoming report. The communications agency proposes a plan for its layout and visual appearance, and upon the client's approval, this plan is put into effect by designing multiple templates for individual page layouts, into which the content may be fed as it becomes available. Naturally, the client and the communications agency stay in touch throughout the design process to discuss its progress.

Finnair's annual report is designed with both digital and print media in mind right from the outset. The report is published only as a Portable Document Format (PDF) fi  le, which may be realized using different media, that is, either rendered on screen or printed out on paper. This decision is motivated by how the intended audiences are expected to consume the report: the digital format enables the general audience to effi  ciently access the entire annual report for a comprehensive description of the company, its operations and performance, whereas analysts and shareholders may print out parts of the report that contain the information essential for their professional interests. Designing the report for multiple media also has a concrete effect on the form of the annual report: many screens favour a landscape orientation, which the printed page also supports, but does not necessarily prefer for large-sized, bound documents, such as an annual report with 165 pages. However, because the annual report is not intended to be printed out in its entirety, the decision to use a landscape orientation appears to be tolerated by the genre constraints, which are determined by the ways of consuming annual reports.

Before proceeding to discuss the multimodal characteristics of the annual report, the agencies responsible for creating the content selected for the report need to be identifi  ed. The interview

10

TT

revealed that the main content is created at Finnair by two teams: one team produces non-fi  nancial information, while the other is responsible for fi  nancial information. The non-fi  nancial information supplied to Miltton consists mainly of written text, but information graphics that have been used in previous annual reports are also provided for reuse. Photographs are made available through Finnair's image bank - a standard tool for establishing and maintaining a unifi  ed visual image for a corporation (cf. Machin 2004).

The communications agency then proceeds to

1. organize and paginate the content using the layout templates previously agreed upon with the client;
2. to highlight the content that Finnair requests to be emphasized;
3. to create illustrations, diagrams and tables to accompany the written content.

In the following sections, I consider these tasks from the perspectives of agency, transduction and transformation, by using the GeM model as an analytical framework to explore the annual report.

To establish a point of departure for this discussion, the communications agency is essentially responsible for shaping the content provided by Finnair into an artefact that corresponds to the defi  nition of the annual report as a multimodal genre. What defi  nes this genre is naturally negotiated within a community of users and subject to internal and external infl  uences, as developments in other genres may exert pressure on the annual report. De Groot (2008: 81-82) identifi  es several infl  uences, such as obligatory content demanded by the law, competitors' reporting strategies and the corporation's own reporting traditions.

This constitutes a crucial issue, because at any given moment, the notion of genre is largely responsible for determining the available semiotic modes and their preferred confi  gurations (Hiippala 2015b: 37). As I will show below, genre expectations wield considerable infl  uence on the design of an annual report. Moreover, these expectations do not only correspond to those that are assumed for the intended audiences, but also include those held by the agencies involved in the design process.

Figure 1. The layout structure of pages 7-12 in Finnair's annual report. Graphic elements, such as photographs, maps and illustrations are marked using colour in the tree diagram

<!-- image -->

In preparation for a closer analysis of the annual report, Figure 1 shows the general organization of the sequence spanning a total of six pages discussed with Miltton's employees. The pages in this sequence are represented as thumbnails in the upper part of Figure 1. Below the thumbnails, a tree diagram represents the hierarchical organization of the content, as defi  ned using the GeM model's layout layer, which is hereafter referred to as the layout structure (Bateman 2008: 115129).

The topmost parent node of the layout structure represents the group of six pages, while its six direct descendants stand for each individual page in the sequence. The children of these individual nodes, in turn, represent the layout units on each page. The layout units are defi  ned by the GeM model and include, for instance, text paragraphs, photographs, headers, captions and page numbers. Alternatively, the units in the diagram may stand for composite nodes that join pieces of content together: several composite nodes on pages 10-12 have been expanded in the lower part of Figure 1 as an example.

In addition to describing how the content is organized, Figure 1 represents the subject matter of the sequence. The sequence deals with two topics, describing (1) Finnair's strategy and value creation and (2) megatrends affecting the airline. These topics are marked using the corresponding numbers in Figure 1, and the extent of each topic in the sequence is represented using dashed lines between the layout units. What deserves particular attention here is the overlap between the two topics: whereas the fi  rst topic runs from page 7 to page 11, the second topic is introduced already on page 10 before continuing on page 12. I now examine this overlap in greater detail, focusing particularly on issues of agency.

## 4.2.  Agencies and their infl  uence

The fi  rst step in performing a more detailed analysis of the sequence presented in Figure 1 involves identifying the deployed semiotic modes. To avoid establishing under-differentiating dichotomies such as 'language' and 'image', I characterize each page using abstractions geared specifi  cally towards describing the multimodality of page-based documents. One proposal for this kind of abstraction is put forward by Bateman (2009), who introduces the semiotic mode of textfl  ow to describe multimodal documents organized around written language.

As with unfolding written language, the organizing principle underlying text-fl  ow is linearity, but this does not rule out non-linguistic contributions: text-fl  ow may be (and often is) interrupted or accompanied by other semiotic modes, such as photographs, diagrams and illustrations, to name but a few examples. Moreover, despite its apparent visual simplicity, text-fl  ow can naturally draw on various discourse structures to support its expression, which may be uncovered using more fi  ne-grained linguistic analyses, as de Groot (2008) has convincingly shown. In short, the immense meaning potential of text-fl  ow is based on its capacity to draw on written language, which may be further augmented by embedding other semiotic modes into its linear organization as a part of a document.

Continuing with the example in Figure 1, the fi  rst topic focusing on strategy and value creation is realized using text-fl  ow on pages 7-11. Various types of multimodal discourse structures, such as cohesion and coherence, may be found supporting the underlying linear organization as the text fl  ows across the pages (Bateman 2014b, Hiippala 2016). These discourse structures may be assumed to hold across the two topics, which are visualized using the dashed lines in Figure 1, although they are not expanded in the diagram.

What is worth observing here is that not a single page in this sequence consists purely of paragraphed text-fl  ow, that is, text-fl  ow is always accompanied by some other semiotic mode. Whereas page 7 includes a map and highlighted text, page 8 features another highlight and an illustration, and a photograph occupies two-thirds of page 9. Pages 10 to 12, in turn, feature a diagram, a table and additional illustrations. Despite the contributions from other semiotic modes, which may be roughly characterized as cartography, illustration, photography and diagrams, the over-

*

all organizing principle of these pages remains linear and governed by text-fl  ow. This argument is supported by the observation that text-fl  ow establishes discourse relations across the entire sequence of pages: no other semiotic mode deployed in the annual report does the same.

Despite having an organization that relies on the continuity of written language, why does the sequence in Figure 1 avoid pages containing only paragraphed text-fl  ow? The interview revealed that both the project manager and graphic designer appear to hold strong preconceptions about the reception of annual reports, particularly in relation to the 'attention span' of the report's intended audience. This is a recurring argument, as several media have been accused of impairing the cognitive abilities of contemporary audiences by shortening their attention span (Newman 2010). Over time, the culprits have included television, cinema, video games, and most recently, the internet.

Newman (2010: 582-583) argues that media professionals are particularly keen on circulating discourses about attention spans, which results in a feedback loop between the professionals and the artefacts they design. This is also the case for the annual report: the representatives of the communications agency do not expect pages consisting exclusively of paragraphed written text to be able to hold the readers' attention: they assume that photographs and illustrations are required to 'liven up' the page and to entertain the reader.

Within the GeM framework, this preconception about the readers' attention span may be considered an assumed consumption constraint, which is subsequently projected on the genre structure by the agencies involved (cf. Bateman 2008: 18). Pages 10-12, on which the two topics identifi  ed above overlap as indicated by the dashed lines in Figure 1, exemplify how this constraint affects the annual report. It may be argued that the organizing principle behind text-fl  ow - linearity - and the discourse structures that guide its interpretation demand support in the form of topical continuity across pages.

As Martin (1994) has shown, the operation of these discourse structures relies on the continuity of written language. Yet the assumed consumption constraint works against providing the necessary continuity by denying the inclusion of entire pages to dedicated text-fl  ow in the artefact, because pure text-fl  ow is assumed to overburden the reader. Consequently, as each page is required to deploy some other semiotic mode besides paragraphed written text, the continuity of text-fl  ow must be negotiated within this constraint, which governs the placement of content across pages.

Figure 2. Visualizing the layout constraint that prevents topical continuity. By not allowing pages consisting of pure text-fl  ow, the constraint creates an overlap between two topics discussed in the annual report (cf. Figure 1).

<!-- image -->

In terms of agency, this constraint affects primarily the graphic designer, who needs to negotiate the organization of text-fl  ow across the pages of the annual report. Figure 2 visualizes this con- Shift in economic and Changes in consumer preference, expectations, purchasing patterns and demographics straint, which operates on the layout structure, in effect: a simple solution to ensure the continuity of topics across the content would require replacing the diagram on page 10 with the paragraphed text-fl  ow on page 11. Such a solution would prevent the overlap between the two topics discussed in the sequence, but would require the designer to choose an alternative transformation to emphasize the content selected from the accompanying instance of text-fl  ow. Figure 1 indicates that several different alternatives are indeed available: page 8, for instance, uses a highlight without illustrations, which would likely fi  t on a page consisting entirely of text-fl  ow. Generally, this illustrates how the graphic designer is forced to negotiate these choices under pressure from other agencies, which can occasionally enforce design choices that actually work against supporting the communicative goals defi  ned for the annual report, that is, maintaining topical continuity across the artefact. Natural disasters, To sum up, individual agencies operate within limits set by other agencies in professional document design. Moreover, design choices are negotiated within the communicative goals set for the annual report as a multimodal genre. What is essential for multimodal analysts to understand is this: although the choices made in the semiotic modes appear to be highly coordinated, this does not mean the choices refl  ect the communicative intentions of a single agency. In contrast, it is entirely natural for well-formed multimodal artefacts to exhibit cohesion across the content and coherence across their structure, as multiple agencies will strive towards these goals as well. This is precisely why the studied phenomenon must be circumscribed carefully, separating the production of the artefact from the analyst's interpretation of the fi  nal product. With this distinction in mind, in the following discussion I proceed to consider transformations and transductions in the annual report.

## 4.3.  Transductions and transformations

In the previous section, I argued that text-fl  ow is the dominant semiotic mode in the six-page sequence. Next, I expand this discussion by attending more closely to the second topic identifi  ed in this sequence, that is, megatrends affecting Finnair and the airline industry (see Figure 1). More specifi  cally, I focus on transductions and transformations of meaning - in this order - while also considering the constraints imposed by various agencies on these processes.

Figure 3. The diagram on page 10 of the annual report.

<!-- image -->

Urbanisation As pointed out above, the 'megatrends' topic is discussed on pages 10 and 12 of the annual report. Within these two pages, there is one clear candidate for an example of a transduction - the diagram on page 10. This diagram, which I will now subject to a closer examination, is reproduced in Figure 3. The fi  rst step is to establish its context of occurrence, as the diagram is preceded by a header and four paragraphs. This text introduces the topic by specifying external factors that infl  uence the airline business, which are also presented in the diagram below. I take this diagram as an example of a transduction from text-fl  ow to the diagrammatic mode.

To provide a brief recap of the diagram, the megatrends - 'shift in economic and political focus towards Asia', 'urbanisation', 'technological development' and 'responsibility' - are positioned on the outer edge. The inner circle, in turn, features examples of the external factors infl  uencing the airline business, which are represented by combinations of written language and two-dimensional, computer-drawn illustrations. The diagram's centre is occupied by arrows that point towards another illustration of an aircraft, which represents the airline industry.

Figure 3 may also be used to indicate the major differences between diagrammatic mode and text-fl  ow, which support the argument that this diagram is the result of a transduction between these semiotic modes. This is evident in the shift that occurs in the multimodal structure of the annual report. Whereas text-fl  ow may be accompanied by other semiotic modes, as the entire sixpage sequence exemplifi  es by including a map, a photograph, a diagram and several illustrations alongside text-fl  ow within just six pages, the diagrammatic mode integrates verbal and visual inputs more tightly into organizations that can convey a multitude of different text-image relations, and begins to make use of the layout space (for research in this area, see Engelhardt 2002).

In other words, here the diagrammatic mode draws on written language, two-dimensional illustrations and line drawings (arrows), and composes them into a meaningful organization in the layout space (cf. also Bateman 2014b: 158-159). What this shows is that the diagrammatic mode has access to an equally wide range of semiotic resources as text-fl  ow: the difference, then, emerges in how the modes organize the available resources to support their contextual interpretation. This is the responsibility of a specifi  c component: discourse semantics.

Because the discourse semantic stratum, which Bateman (2011) conceptualizes as an essential component of a full-blown semiotic mode, is crucial for identifying the transduction from textfl  ow to diagrammatic mode, the stratum deserves closer attention. As a part of a semiotic mode, the discourse semantic stratum helps to form and resolve hypotheses about discourse structure by providing a set of candidate interpretations. Such discourse structures may be described, for instance, by defi  ning rhetorical relations between clauses and larger parts of text, as well as written language and graphic elements (see e.g. Taboada/Habel 2013).

To draw on an example from Figure 3, the illustrations in the inner circle are likely to be interpreted as elaborating the written labels due to their position in the layout. Although some illustrations (such as the fuel pump) can stand in for the written labels ('fuel price'), but generally, the written labels are needed to determine the factors that affect the airline business. To reiterate, their positioning signals that they are meant to be interpreted together: in this way, the discourse semantics of diagrammatic mode instruct the reader to resolve the text-image relations holding between the labels and the illustrations. Contrastingly, the positioning of the labels (megatrends) on the outer circle is not meaningful, that is, the examples in the inner circle (e.g. 'political environment') are not intended to exemplify the megatrend placed closest to them on the outer circle (e.g. 'technological development').

Finally, the interpretation of the arrows in the diagram is constrained by their position and direction, which suggest a circumstantial relation between the illustrated labels in the inner circle and the illustration in the middle of the diagram. A simple thought experiment may help to illustrate how discourse semantics set up candidate interpretations in this case: if the arrows were positioned between the labelled illustrations, pointing from one illustration to another, the arrows would be likely interpreted as indicating some kind of causal relation between the external factors represented using the illustrated labels.

To sum up, the discourse semantics of diagrammatic mode are fundamentally different than those of paragraphed text-fl  ow, particularly for text-image relations, which can exploit the two-dimensional layout space to make additional meanings. Text-fl  ow, in contrast, is largely organized around the principle of linearity. In other words, the differences in discourse semantics suggest that the shift from text-fl  ow to the diagrammatic mode is indeed a transduction, as opposed to a mere transformation within the semiotic mode of text-fl  ow. At the same time, the transduction into a diagram is clearly something the genre can easily accommodate, as the annual report features various types of diagrams. Handling such transductions is also demanded from the reader, who needs to apply the appropriate discourse semantic interpretations to make sense of the page. For the entire page, they are provided by text-fl  ow; for the diagram, by the diagrammatic mode.

Finally, it is necessary to consider what motivated the transduction on page 10. According to the communications agency, the client requested the transduction by providing a sketch of the diagram, which the graphic designer then adapted to the visual identity of the annual report. Although the interview revealed that the client provides most of the content in written language, a request for a transduction should not be considered extraordinary as a part of a collaborative design process. Most importantly, what should be noted here is that the transduction does not arise from the actions of a single agency, but multiple agencies collaborating with each other. Moreover, in terms of the topic structure, the diagram appears to serve a specifi  c purpose, which may explain the client's request. By summarizing the megatrends, the diagram prepares the reader for the more detailed description of the same topic on page 12, which is realized using text-fl  ow.

As pointed out above, page 12 discusses the topic of megatrends in greater detail (see Figure 2). The detailed description is realized using paragraphed text-fl  ow, which is accompanied by a summary of the topic in a table. The table is organized into three columns, which list the (1) megatrends, (2) their impact on the airline industry, and (3) Finnair's response. Whereas the fi  rst column features headings only, the second and third columns include cells with bulleted lists that outline the challenges facing the airline industry and Finnair's response to these challenges. In addition, two decorative illustrations showing a branch of a tree with fl  owers and a skyline of a city are featured on page 12. As explicated above in connection with the diagram in Figure 3, text-fl  ow is able to integrate a variety of graphical semiotic modes - such as two-dimensional illustrations - into its expression on the level of a page. However, the mere presence of illustrations should not be considered an example of a transduction or a transformation, as they cannot be traced back to specifi  c content realized using the mode of text-fl  ow.

Identifying actual transformations within the semiotic mode of text-fl  ow, however, requires a closer examination of its properties. I argue that this may be achieved by drawing on the work of Twyman, who presents a model for describing how various 'modes of symbolization' are confi  gured in page-based documents (1979: 120). Of particular interest is the category of 'linear-interrupted' text, which Bateman (2009: 61) acknowledges as bearing close resemblance to his defi  nition of text-fl  ow as a semiotic mode.

Twyman (1979: 121) observes that very few instances of written text are truly linear: fl  owing text is often organized into lines, which are interrupted at specifi  c points to produce paragraphs. From linear-interrupted text, Twyman's categories move progressively towards non-linear confi  gurations of written language, which ultimately begin to exploit the available layout space in its entirety. Two confi  gurations within this continuum, lists and tables, warrant further attention, as both are featured on page 12 of the report. Whereas lists are rather self-explanatory, tables may be either numerical or verbal (Twyman 1979: 125). Because lists and tables draw primarily on written language to make meanings, I argue that these confi  gurations represent alternatives available within the semiotic mode of text-fl  ow. Although tables gradually abandon linear organization for a spatial one, their semiotic potential is largely founded on the discourse structures provided by written language.

In the case of the table on page 12 of the annual report, these structures may be made explicit using Rhetorical Structure Theory (RST). RST is an established theory of discourse structure, which proposes a set of rhetorical relations that are taken to hold between discourse elements (Taboada/Mann 2006). Whereas 'classical' RST is typically used to draw relations between linguistic units, the GeM model extends RST to cover discourse relations that hold between different semiotic modes, such as instances of text-fl  ow, photographs and many more (Bateman 2008: 146-149).

In principle, RST assigns discourse elements into two categories, nuclei and satellites, which stand for the primary and ancillary information in a given rhetorical relation (Taboada/Mann 2006: 426-427). The application of RST is best illustrated using an example: Table 2 describes the relation holding between two items in the table on page 12. Here the relation holding between the nucleus and the satellite is assumed to be that of Solutionho, whose criteria are given at the bottom of the table (for relation defi  nitions, see Bateman 2008: 149).

Traffic between Asia and

Europe grows

SOLUTIONHOOD

Renewal of the new generation long-haul

fl eet from 2015 onwards

| Satellite                                                                            | Nucleus                                                                                                |
|--------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|
| Constraints on the satellite: a question, request, problem, or other expressed need. | Constraints on the nucleus: a situation or method supporting full or partial satisfaction of the need. |

Table 2. A SOLUTIONHOOD relation holding between two items in the table

A variety of rhetorical relations, such as SOLUTIONHOOD in Table 2, can occur in both tables and paragraphed written text (Taboada 2005). The difference arises in how larger groups of relations are structured: unlike paragraphed text, tables do not structure the relations into recursive organizations. A recursive organization, which involves embedding another rhetorical relation into the satellite of a relation positioned 'above', requires maintaining coherence across the entire passage (Hiippala 2015b: 137-139). Tables, however, are largely exempted from this requirement, because they do not need to knit together multiple rhetorical relations to establish a coherent unit of discourse. Their spatial organization takes care of this need.

That being said, if linear-interrupted text, lists and tables are taken to represent different confi  gurations of text-fl  ow as a semiotic mode, it may be proposed that the shift from paragraphed written text to a table is a transformation rather than a transduction from one semiotic mode to another. To conclude, the combination of a table and a bulleted list represents just one alternative among several confi  gurations available for text-fl  ow. Teasing out the possible confi  gurations will undoubtedly require extensive empirical research, which is nevertheless ultimately necessary for identifying any transformations within a semiotic mode, as I argue in the following section, which presents the concluding remarks.

## 5. Conclusion

In this article, I have explored the concepts of agency, transduction and transformations in professional document design, focusing on a specifi  c multimodal genre: the corporate annual report (de Groot 2008). The aforementioned concepts have been frequently used to describe the individual's capability to draw on and manipulate multiple semiotic modes, particularly in educational settings (see e.g. Archer/Breuer 2015). To examine the validity of these concepts in settings in which multiple agencies interact, I interviewed professionals responsible for producing a corporate annual report, and contrasted these fi  ndings with a description of the report's multimodal structure.

In terms of agency, I showed that the process of designing an annual report is a collaborative effort involving multiple agencies, which affect the concrete outcome - the multimodal artefact - in different ways. In particular, assumptions concerning the consumption of the annual report play a signifi  cant role in determining its multimodal structure. These consumption constraints included, for instance, layout choices supporting the intended medium of the artefact, that is, whether the annual report is read on screen or printed out on paper. In addition, the interview revealed that the professionals avoid pages consisting solely of written language, because such pages are assumed to be unable to hold the reader's attention. This view bore close resemblance to discourses of reduced 'attention span' presumably caused by new media, which have been suggested to be particularly popular among communication professionals (Newman 2010).

For transductions and transformations, I emphasized the need to determine and characterize the semiotic modes active in the artefact. This is necessary for identifying any transduction or transformation between/within semiotic modes. In other words, without a suffi  cient understanding of the semiotic modes, the distinction between a transduction - a shift from one mode to another and a transformation - a reshaping of meaning within a single semiotic mode - remains elusive. By drawing on the defi  nition of a semiotic mode proposed by Bateman (2011, 2014a), I proposed that distinguishing between semiotic modes is best pursued by considering their discourse semantics, that is, how they combine written language, graphic elements and other resources for making meaning into structures that support their contextual interpretation. I illustrated this approach by contrasting the diagrammatic mode with text-fl  ow - a foundational semiotic mode organized around linear-interrupted written text (Hiippala 2016).

Despite the wealth of insights gained from the interview with communication professionals, the semi-structured interview failed to provide an in-depth view of what drives the selection of a particular semiotic mode. This constitutes a limitation of the current study, which could be overcome by directly observing the design process as a part of the team at the communications agency (cf. Kress 2011). Such an approach could also provide a more detailed view of the agencies involved, their hierarchical relations and role in the design process. Methodologically, the fi  ndings suggest that the combination of multimodal and ethnographic methods holds considerable potential for improving our understanding of how documents are produced. In terms of theory, it appears that working with a stratifi  ed defi  nition of a semiotic mode - that is, a defi  nition that accounts for the materiality, resources and their interpretation - is benefi  cial to any investigation targeting agency, transduction and transformation.

## 6. References

- Archer, Arlene/Breuer, Esther (eds.) 2015: Multimodality in Writing: The State of the Art in Theory, Methodology and Pedagogy. Leiden: Brill.
- Bateman, John A. 2008: Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. London: Palgrave Macmillan.
- Bateman, John A. 2009: Discourse across semiotic modes. In Renkema, Jan (ed.), Discourse, of course: An overview of research in discourse studies. Amsterdam: Benjamins, 55-66.
- Bateman, John A. 2011: The Decomposability of Semiotic Modes. In O'Halloran, Kay L./Smith, Bradley A. (eds.), Multimodal Studies: Multiple Approaches and Domains. London: Routledge, 17-38.
- Bateman, John A. 2014a: The Constitutive Role of Semiotic Modes for the Theory and Practice of Multimodal Analysis. In DeCoursey, Christina (ed.), Language Arts in Asia 2: English and Chinese through Literature, Drama and Popular Culture. Newcastle: Cambridge Scholars Publishing, 8-33.
- Bateman, John A. 2014b: Multimodal coherence research and its applications. In Gruber, Helmut/Redeker, Gisela (eds.), The Pragmatics of Discourse Coherence: Theories and Applications. Amsterdam: Benjamins, 145-177.

- Bezemer, Jeff/Kress, Gunther 2008: Writing in multimodal texts: A social semiotic account of designs for learning. In Written Communication 25(2), 166-195.
- de Groot, Elizabeth B. 2008: English annual reports in Europe: a study on the identifi  cation and reception of genre characteristics in multimodal annual reports originating in the Netherlands and in the United Kingdom. Ph.D. thesis, Radboud University Nijmegen/LOT, Nijmegen and Utrecht.
- Engelhardt, Yuri 2002: The Language of Graphics: A Framework for the Analysis of Syntax and Meaning in Maps, Charts and Diagrams. Ph.D. thesis, Institute for Logic, Language and Computation, University of Amsterdam.
- Erickson, Sheri L./Weber, Marsha/Segovia, Joann 2011: Using communication theory to analyze corporate reporting strategies. In International Journal of Business Communication 48(2), 207-223.
- Hearn, Laura M. 2001: Language and agency. In Annual Review of Anthropology 30, 109-137.
- Hiippala, Tuomo 2015a: gem-tools: Tools for working with multimodal corpora annotated using the Genre and Multimodality model. doi:10.5281/zenodo.33775 (accessed 15 April 2016).
- Hiippala, Tuomo 2015b: The Structure of Multimodal Documents: An Empirical Approach. New York and London: Routledge.
- Hiippala, Tuomo 2016: Aspects of multimodality in higher education monographs. In Archer, Arlene/Breuer, Esther (eds.), Multimodality in Higher Education. Leiden: Brill, 53-78.
- Jones, Rodney H./Norris, Sigrid 2005: Introducing agency. In Norris, Sigrid and Jones, Rodney H. (eds.), Discourse in Action: Introducing Mediated Discourse Analysis. London: Routledge, 169-171.
- Kress, Gunther 2011: Partnerships in research: multimodality and ethnography. In Qualitative Research 11(3), 239-260.
- Kress, Gunther 2014: Design: The Rhetorical Work of Shaping the Semiotic Work. In Archer, Arlene/Newfi  eld, Denise (eds.), Multimodal Approaches to Research and Pedagogy: Recognition, Resources, and Pedagogy. New York and London: Routledge, 131-152.
- Kress, Gunther/Jewitt, Carey/Ogborn, Jon/Tsatsarelis, Charalampos 2001: Multimodal Teaching and Learning: The Rhetorics of the Science Classroom. London and New York: Continuum.
- Kress, Gunther/van Leeuwen, Theo 2006: Reading Images: The Grammar of Visual Design. Second edition. London: Routledge.
- Machin, David 2004: Building the World's Visual Language: The Increasing Global Importance of Image Banks in Corporate Media. In Visual Communication 3(3), 316-336.
- Martin, James R. 1994: Modelling big texts: A systemic functional approach to multi-genericity. In Network 21, 29-52.
- Mavers, Diane 2011: Children's Drawing and Writing: The Remarkable in the Unremarkable. London and New York: Routledge.
- Newfi  eld, Denise 2014: Transformation, transduction and the transmodal moment. In Jewitt, Carey (ed.), The Routledge Handbook of Multimodal Analysis. Second edition. London and New York: Routledge, 100-113.
- Newman, Michael Z. 2010: New media, young audiences and discourses of attention: from Sesame Street to 'snack culture'. In Media, Culture & Society 32(4), 581-596.
- Norris, Sigrid/Jones, Rodney H. 2005: Discourse in Action: Introducing Mediated Discourse Analysis. London: Routledge.
- Rowsell, Jennifer 2013: Working with Multimodality: Rethinking Literacy in a Digital Age. New York and London: Routledge.
- Stanton, Patricia/Stanton, John 2002: Corporate annual reports: research perspectives used. In Accounting, Auditing & Accountability Journal 15(4), 478-500.
- Taboada, Maite 2005: Discourse markers as signals (or not) of rhetorical relations. In Journal of Pragmatics 38(4), 567-592.
- Taboada, Maite/Habel, Christopher 2013: Rhetorical relations in multimodal documents. In Discourse Studies 15(1), 65-89.
- Taboada, Maite/Mann, William C. 2006: Rhetorical Structure Theory: looking back and moving ahead. In Discourse Studies 8(3), 423-459.
- Thomas, Martin 2014: Evidence and circularity in multimodal discourse analysis. In Visual Communication 13(2), 163-189.
- Twyman, Michael 1979: A Schema for the Study of Graphic Language. In Kolers, Paul A./Wrolstad, Merald E./Bouma, Herman (eds.), Processing of Visible Language, Volume 1. New York: Plenum Press, 117-150.
- Waller, Robert H. W. 2012: Graphic literacies for a digital age: The survival of layout. In The Information Society 28(4), 236-252.