<!-- image -->

<!-- image -->

## Anthony Baldry* & Paul J. Thibault*

## Applications of multimodal concordances

## Abstract

Multimodal corpus linguistics has so far been a theoretical rather than an applicative discipline. This paper sketches out proposals that attempt to bridge between these two perspectives. It does so with particular reference to the development of the conceptual and  software  tools  required  to  create  and  concordance  multimodal  corpora  from the  applicative  standpoint  and  as  such  is  designed  to  underpin  the  study  of  texts  at universities in foreign-language teaching and testing cycles. One branch of this work relates to multimedia language tests which, as illustrated in Section 2 , use concordancing techniques to analyze multimodal texts in relation to students' understanding of oral and written forms of discourse in English. Another branch is the exploration of multimodal tests concerned with the explicit assessment of students' knowledge of the principles and/or models of textual organization of multimodal texts. The two types of test are not mutually exclusive. A third branch of research thus relates to the development of hybrid tests which, for example, combine a capacity to analyze multimodal texts with an assessment of students'  language  skills,  such  as  fl  uency  in  speaking  and  writing in  English  or  which  ascertain  the  multimodal  literacy  competencies  of  students  and the  differing  orientations  to  meaning-making  styles  that  individuals  manifest.  The paper  considers  these  different  applicative  perspectives  by  describing  the  different categories of concordance achievable with the MCA online concordancer ( Section 2 ) and by defi  ning their relevance to multimodal discourse analysis (Section 3). It  also illustrates the use of meaning-oriented multimodal concordances in the creation and implementation of multimodal tests (Sections 4) .  It  concludes by suggesting that the re-interpretation of the nature and functions of concordances is long overdue and that the exploration of new types of concordance is salutary for linguistics and semiotics in general.

## 1. Introduction

Studies relating to tagged multimodal corpora are, with very few exceptions (e.g. Baldry/Thibault 2001, 2005, 2006b, forthcoming), conspicu-

* Anthony Baldry Dipartimento di Storia e Comparazione degli Ordinamenti Giuridici e Politici Università degli Studi di Messina IT-98122 Messina anthony.baldry@gmail.com

* Paul J. Thibault Faculty of Humanities University of Agder Serviceboks 422 N-4604 Kristiansand pauljthibault@gmail.com

Hermes - Journal of Language and Communication Studies no 41-2008

<!-- image -->

11

<!-- image -->

<!-- image -->

<!-- image -->

ous by their absence. This is not surprising given that a multimodal perspective in corpus linguistics entails a much wider and more complex vision of meaning-making than the language-only perspective. Nevertheless, collections of digitalized multimodal texts (fi  lms, websites or printed  materials)  can  now  be  searched  by  adopting  concordancing techniques developed specifi  cally to this end (see Section 2 ).

Our joint work in the last ten years demonstrates the signifi  cance of  multimodal  corpora  in  gaining  insights  into  the  meaning-making processes found in multimodal texts, including insights into the role language has to play in these processes (Baldry/Thibault 2001, 2005, 2006b: 168; Baldry 2004, 2007). This research has related mainly to the development of multimodal corpus linguistics as a theoretical rather than as an applicative discipline. In the discussion below, we shall sketch some proposals that attempt to bridge between these two perspectives. Our main focus is on the development of the conceptual and software tools required to create and concordance multimodal corpora from the applicative standpoint (Baldry 2005a; Taylor Torsello/Baldry 2005; Ackerley/Coccetta in press). This research is designed to underpin the study of texts at universities in foreign-language teaching and testing cycles. One aspect of this work relates to multimedia language tests . As illustrated in Section 2 , these typically use digital technologies (concordancing techniques included) to analyze multimodal texts in relation to students' understanding of oral and written forms of discourse in English. A different branch of this applicative research is the exploration of multimodal tests concerned with the assessment of students' understanding of models of multimodal text analysis developed for both printed and digital media (Baldry/Thibault 2006a, 2006b; O'Halloran 2004; Kress/van Leeuwen 2006 [1996]). Unlike multimedia language tests, multimodal tests provide explicit assessment of students' knowledge of the principles and/or models of textual organization of multimodal texts. They are, thus, not necessarily concerned with foreign-language learning and could, for example, relate to modules in communication studies concerned with the general organizational principles of printed and digital media (Baldry in press a).

The two types of test are not mutually exclusive and make it possible to envisage the development of hybrid tests within text-based foreign  language  learning  combining  a  capacity  to  analyze  multimodal texts with an assessment of students' language skills, such as fl  uency in speaking and writing in English. Such tests may also be developed to ascertain the multimodal literacy competencies of students and the differing orientations to meaning-making styles that individuals manifest.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

Accordingly, the paper provides a description of the different categories of concordance achievable with the MCA online concordancer ( Section 2 ), an illustration of their applications in multimodal discourse analysis ( Section 3 ) and a characterization of the use of meaning-oriented multimodal concordances in the creation and implementation of multimodal tests which includes a brief outline of the characteristics of a hybrid test that strengthens the adoption of a multimodal perspective in text-based studies (Section 4) . The paper concludes (Section 5) by suggesting that the re-interpretation of the nature and functions of concordances is long overdue and that the exploration of new types of concordance is salutary for linguistics and semiotics in general.

## 2. Concordances in relation to language tests based on multimodal corpora

This section describes the role played by multimodal corpora and, more specifi  cally,  by  various  types  of  concordances in the development of multimedia language tests . Table 1 sketches out the nature and functions of four types of concordance. Of these, the language-only, form-oriented type has received most attention in research and classroom teaching. It is instantly recognizable thanks to its single central column delimited by automatically-created alignments, bold type, colour and/or gaps. The rows of individual concordances combine to create a semi-tabular format created automatically by the concordancing software. This allows the linguist to perceive recurrences in wordings and relate them to co-texts given in the individual rows on either side of the central column (Sinclair 1991). Apart from greatly assisting the 'phraseological approach' (Hunston 2006: 55) to language studies, the main reason for using this type of concordance is the widespread availability of lemmabased concordancers and the fact that tags , i.e. metatextual descriptions associated to specifi  c forms, are either unnecessary or already embedded in texts thanks to automatic tagging systems such as CLAWS (Condron et al. 2000). This approach to concordancing is attractive since it minimizes the work of unearthing examples and presents them in tidy patterns. Nevertheless, it leaves the question of interpretation in terms of meaning-making processes unanswered.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

| MULTIMODAL   | Type 3. Like Type 1 concordances, this type relates to forms used as text-making resources in text corpora but, in keeping with the resource integration principle (Baldry/Thibault 2006a: 18-19), focuses on typical combinations of, and relations between , forms used at the same point of time in specific texts. For reasons of space, the example shown is a fragment of a single concordance which records the presence of written, oral, vocal, visual, vectoral, spatial, and temporal resources and, in part, begins to suggest their function in relation to mini-genres (Baldry/Thibault 2006a: 42) such as slogans and logos in a TV advert. : visual, vectoral and spatial resources   | Type 3. Like Type 1 concordances, this type relates to forms used as text-making resources in text corpora but, in keeping with the resource integration principle (Baldry/Thibault 2006a: 18-19), focuses on typical combinations of, and relations between , forms used at the same point of time in specific texts. For reasons of space, the example shown is a fragment of a single concordance which records the presence of written, oral, vocal, visual, vectoral, spatial, and temporal resources and, in part, begins to suggest their function in relation to mini-genres (Baldry/Thibault 2006a: 42) such as slogans and logos in a TV advert. : visual, vectoral and spatial resources   | logo slogan: oral: wording: freedom to choose slogan: oral: voice: male voice slogan: written: wording: freedom to imagine slogan complex: written + oral overlap: exactly synchronized logo + slogan overlap : exactly synchronized Type 4. Like Types 2 and 3 , this type of concordance uses descriptive labels to search a multimodal corpus. focuses on meaning that derives from a combination diverse semiotic resources. See Table 3 for an illustration of this type of concordance and, for more complete examples , Baldry/Thibault 2001: 94-6; see also Baldry 2005a; 2006: 118-9; Baldry/Thibault 2006b).                                        |
|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| MONOMODAL    | Type 1 . This type of concordance relates to forms as text-making resources retrieved from text corpora and specifically to the textual uses of one specific resource (e.g. a concordance of punctuation marks, symbols, lines, colours or ambient sounds). In practice, almost all concordances of this type are lemma-based studies of language and consist of single lines of wordings arranged in a tabular form; hence the reference in this paper to this type as language-only, form-oriented concordances . The examples below of forget are taken from the British National Corpus (BNC).                                                                                                    | FR0 3687 Sheldukher… don't forget the Cell…'; FU6 1895 I never forget a face --; (He looks into the SPY's face)… not that I know yours that is. FYA 859 It's it's four hundredths of don't forget that times is of. G0T 167 Some mornings he would rush away and forget to leave anything. G2W257 Don't forget to say you saw it first in Ski survey, February/March issue 1991! G63 472 But er they would maybe lay it by for two or three weeks and then forget for two or three weeks, and then when it came to the end of whole lot of weeks                                                                                                                                                      | the quarter they were a behind, you know, they couldn't pay it then. H7A 2505 " If you tell me to forget it I will, Mr Feather. Type 2. As with Type 1 concordances, this type relates to forms produced by one resource but explicitly extends the range of a concordance to meaning by providing information about the functions various forms have in specific contexts. To date, all such concordances appear to have dealt with language, taking various forms, including the two-line concordances exemplified in Table 2, where the first line contains actual wordings in one or more texts, the second, the functions carried out by these wordings. |
| MONOMODAL    | FORM-ORIENTED MEANING-ORIENTED                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | FORM-ORIENTED MEANING-ORIENTED                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | FORM-ORIENTED MEANING-ORIENTED                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |

Table 1. What types of concordance exist?

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

## 2.1.   New types of concordance and concordancing: principles and potential applications

In this respect, Table 1 presents two distinct though related perspectives, one relating to the degree to which the copresence of meaning-making resources  is  taken  into  account,  the  second  relating  to  the meanings that these codeployments have in specifi  c (con)texts. Despite the greater work involved, in particular the spadework of manual tagging, building analyses of meaning into concordances is a signifi  cant step for both linguistic  and  semiotic  studies  (Baldry/Thibault  2001,  2006b;  Baldry 2005a, 2005b; Taylor Torsello 2001; Taylor Torsello/Baldry 2005) which includes many undertakings, such as the development of multimodal transcriptions (Baldry/Thibault 2006a), multimodal concordances (Baldry 2005a: Chap 1; Baldry 2005b; Baldry/Taylor 2004; Baldry/Thibault 2006b), multimodal tagging procedures (Baldry 2005a: 1527) and the construction of a new generation of online con  cor  dancers such as MCA (Baldry 2005a, Baldry/Beltrami 2005) and based on relational databases, in the case of MCA, Microsoft Sequel Server .

Table 1 defi  nes four types of concordance which can be implemented with MCA (Baldry 2005a) and ultimately encourages a re-interpretation and evolution of traditional form-oriented language-only concordances in a way that greatly assists their applicative use in various forms of student assessment and/or text-based studies of fi  lms. Meaning-making cannot be achieved by a single meaning-making resource in isolation (see Section 3 ). Form and meaning cannot exist separately (Baldry/ Thibault 2006b: 175-7). Hence the need to transcend traditional formoriented  language-only concordancing  by  introducing  other  types  of concordance into daily teaching practice, all of which -more from an applicative standpoint than a theoretical one -helps achieve a balance between form and meaning. A detailed illustration of the way in which meaning-oriented concordances achieve this is given in Tables 2 and 3.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

| different clause-level transitivity configurations consisting -form is deployed. The concordance shown here, which clause-level configurations involving the process lexeme semantic models of human experience (Davidse                                  | the race of his life Non-affected: Range: Cognate: Expand process                                                                                                                     | into another world. Non-affected: Range: Circumstance: Extent into trouble by doing too much, by being over-zealous Range: Circumstance: Extent                                  | into trouble. Range: Circumstance: Extent                                               | along the unlit landing in desperate need of the lavatory. Circumstance: Spatial Trajectory (towards goal)      | for president of the top body in world motor sport? Non-affected: Range: Circumstance: Desired Goal it didnt affect the   | thru to Coton, but Coton could nt hold it anyway so IMO `;goal'; trajectory Circumstance: Spatial Trajectory (towards goal)   | I had seen no staff in the cinema for hours. like a well-oiled machine, instead of like a slightly out-of-control summer camp. Circumstance: Manner: Comparison the Drake Fellowship; this was a scheme set up by the Prince in the wake of the inner-city riots, to give young people from those areas adventure training.   | Affected: Medium fast and I learnt later that bathing could be perilous. Circumstance: Manner                        | at about £13 billion. Range: Circumstance: Extent                                               | a deficit of £1,500 million. Affected: Medium                            |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------|
| in the functions of words, in this case relation to the way in which the running sample, shows a range of experiential to and cut across the transitive and ergative 280-302).                                                                            | Running Process: pseudo-effective                                                                                                                                                     | was running Process: transitive: pseudo-effective: accidental Might be running Process: transitive: pseudo-effective: accidental:                                                | goal achieving is running Process: transitive: pseudo-effective: accidental:            | goal-achieving was running Process: transitive: middle: controlled: deliberate: motion along spatial trajectory | towards achievable goal: goal-directed was …running Process: transitive: middle: controlled: deliberate: goal-directed    | was running Process: transitive: middle: uncontrolled: accidental: motion along spatial towards goal was running              | was running Process: ergative: middle was running                                                                                                                                                                                                                                                                             | Process: ergative: effective was running Process: ergative: middle was running.                                      | Process: ergative: middle is running                                                            | Process: ergative: middle is running Process: ergative: pseudo-effective |
| This type of concordance often highlights contrasts of process(es), participant(s) and circumstances in is a reworking of a concordance taken from a BNC running and the ways in which its meanings relate 1999: 25-53; Halliday/Matthiessen 2004 [1985]: | AHC 212 But having talked his reluctant owner and trainer into running, Davies proceeded to kid and cajole their unpredictable 11-year-old into Actor: animate / Affected: Medium AR7 | 1550 We had arrived in Barbados only 12 hours earlier, and instead of dead streets and dog walkers, I Actor: animate / Affected: Medium KRP 546 So, of course you Actor: animate | CRB 3531 But it is already clear that the programme Actor: inaminate / Affected: Medium | ACW1518 Then the griping pains started, and soon he Actor: animate                                              | This man Actor: animate J1E 578 He did give a little shove to a defender who was                                          | shielding Wallace from the ball as it Actor: inanimate A6C 596 Though the film JY6 2266 It                                    | Affected: Medium: inanimate A7H 1273 So he wrote to George Thurstan, an ex-marine who                                                                                                                                                                                                                                         | Agent: Instigator H9X 603 The river Affected: Medium GW3 1785 The little boy had gone over to his granny because his | nose Affected: Medium K9W 158 `;To put our industry in context, the balance of payments deficit | Affected: Medium EC3 1154 It is evident that the public sector Setting   |

Table  2. A  detailed  illustration  of  a  monomodal  meaning-oriented  ( Type  2 ) concordance

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

Table 3. A detailed illustration of a multimodal meaning-oriented ( Type 4 ) concordance

<!-- image -->

<!-- image -->

( Suntitlino frames I HAVE TO WORK

<!-- image -->

ANSWER ALL YOUR QUESTI

TELL ME WHY

WH

I'VE GOT A JOB

7 Tert traoments I D

IRmEtEEs

? Concordancino

<!-- image -->

Figure 1. Stages in the reconstruction of meaning units in a fi  lm

<!-- image -->

Figure  1 gives  a  classroom  demonstration  of  the  circumstances  that have led to the redevelopment of the form-oriented, language-only concordance in such a way as to increase its relevance to text studies that embrace a wide variety of genres. It illustrates three crucial moments in a lesson recorded in January 2004, part of a corpus of fi  lmed lessons. The students were following a 24-hour B2 level fi  lm-based course that prepared  them  for  exams  leading  to  the  international  certifi  cation  of their profi  ciency in English. They were asked by the teacher to fi  ll in a handout summarizing the events in a particular phase in the fi  lm (for phase see Baldry/Thibault 2006a: Chap. 4). As they did so, they were instructed by the teacher to consult each other, in English, with a view to reconstructing what they had understood so far about the fi  lm on the basis of a single viewing. Part of this classroom test required the students to provide appropriate subtitles for a sequence of frames ( Figure 1.1 ). To help them in this reconstruction, the students were prompted by a series of text fragments ( Figure 1.2 ) and given access to computers in order to carry out keyword searches using MCA ( Figure 1.3 ). This allowed them to establish how individual words, such as work and job , were used in the fi  lm script and gave them a chance to view and hear the specifi  c fi  lm sequence ( utterance, subphase or phase ) in which these words occurred. Concordancing exercises were thus part of an overarching activity that exercised various skills -reading, writing, listening, speaking and refl  ection on the functions of language in context -in an integrated way. Throughout this activity, concordancing was based on the principle of comparing what might have been said with what was actually said . Thus, activities included watching phases in the fi  lm with the soundtrack suppressed and using the observation of resources such as movement, gaze and gesture to make hypotheses about the actual words used in specifi  c utterances which were then verifi  ed or refuted by keyword concordancing (Baldry et al. 2005).

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

Careful analysis of the fi  lmed corpus helped identify a number of diffi  culties arising with this approach to classroom concordancing. Space permits us to mention only one of these in passing, namely the need to explore the properties of texts on a comparative basis. There was, in other words, a need to use concordances in such a way as to promote the principle,  mentioned  above,  of  highlighting  the  relationship  between the potential and the actual . Figure 2 maps out and summarizes three steps in the reinterpretation and extension of form-oriented languageonly concordances incorporated in the latest version of MCA ( Release 3.6 ) that do just this. The extensions built into MCA enhance the comparative basis of classroom concordancing. The corpus in Figure 2 is derived from posters displayed in an Italian supermarket chain. They reincarnate foodstuffs, in particular fruit and vegetables, as people, animals and inanimate objects. The posters, one of which is reproduced in Figure 2 (top-right), are multimodal texts made up of visual and verbal clusters (Baldry/Thibault 2006a: 31). The verbal clusters take the form of captions placed at the bottom of the posters. A soundtrack in various languages has been added to the corpus. In keeping with traditional concordancing, searches still focus on wordings but are performed in terms of three new types of concordance, the media-indexed concordance, the tabulated concordance and the captioned concordance associated, respectively, with MCA 's Search , Pivot and Captioning buttons displayed in the left-hand column of the example. They are, thus, extensions to lemma-based concordancing designed to take into account the characteristics of multimodal texts and, above all, analytical comparisons of texts (Baldry in press b).

The process of re-interpreting the nature and function of concordances may be characterized in relation to traditional views of co-texts (Sinclair 1991) as generating extended co-texts (Baldry in press b). From another standpoint, however, this research may be taken as reconstruing concordances as subtexts of and points of access to specifi  c parts of digitalized texts (fi  lms, audio recordings, photos, printed media etc.). Extension 1 described in Figure 2 exemplifi  es this. The same process is further evidenced by MCA 's subtitling procedures (see Figure 2 , Extension 3 ) where concordances are construed as co-contextualizing sub- RESEARCH

<!-- image -->

<!-- image -->

Project Definitinn

Play Media

Grammar Selection

Media Indexing

Task definition

Sequence Analysis

Search

Pivot

Captioning

10

15

20

25

30

35

A ballerina or carrots?

A wood or broccoli?

Sea horses or chili peppers

Columns or cheese?

Dragonflies or peas?

Tennis balis or grapefruit?

Seals or aubergines?

ot air ballons or onions?

Ballerina o corole?

A ballerina or cirrots?

Extension 1. The media-indexed or multimedia concordance is an extension to lemma-based

<!-- image -->

19

24

29

34

a captioned corpus

Baliarina o carote!

Mosco o broccolit

Cavallucci a peperonoin

Colonne a farmaggio?

Ubelule o pistili?

Palle da tennis o pompelmi?

Foche o melanzane?

Ballerina o carote?

concordancing

<!-- image -->

texts of the texts to which they relate (Baldry 2005a: 97; see also Taylor 2004). Overall, the approach adopted allows a comparison, across languages and across modalities, of oral and written forms of wordings, all of which is particularly useful, for example, in the early stages of foreign language learning. The new types of form-oriented, language-only concordances produced by the extensions described in Figure 2 may be called extended concordances.

Figure 2. What kinds of extensions can be made to traditional lemma-based concordancing?

<!-- image -->

## 2.2.  Defi  ning the multimodal concordance in relation to multimodal tests.

Potentially, the use of these extended concordances goes beyond wordings. For  example, captioned  concordances can  be  used  to  produce Type 2 monomodal concordances -see Table 1 -by directly analyz- ing wordings in the soundtrack in relation to the experiential meanings (Halliday 1994 [1985]). Similarly, they also can instantiate multimodal concordances ( Types 3 and 4 in Table 1 ) consisting of two-line selections from the tagged corpus. For teaching and learning purposes, this is good enough to highlight the changing pattern of resource integrations ( Type 3 concordances) in specifi  c texts in the corpus illustrated in Figure 2 and, of course, in other corpora. Though not shown in Figure 2 ,  one example is the relationship of vertical vectors (fi  rst line in the concordance) to horizontal vectors (second line in the concordance). Equally, though again not illustrated in Figure 2 for reasons of space, captioned concordances can highlight relations existing between textual subunits in multimodal texts ( Type 4 concordances in Table 1 ) for example to establish the mutual positioning of the two main clusters in the corpus illustrated in Figure 2 : the depicted object cluster (represented in the fi  rst line in the concordance) and the slogo (i.e. slogan + logo) cluster (represented in the second line in the concordance). This shows that in this corpus, somewhat against expectations, the mutual position varies and, by varying, imposes different reading paths on readers as they attempt to 'decipher' each of the posters.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

A fuller description of the extensions described above and their applications goes beyond the purposes of this paper (see Baldry/Thibault forthcoming). Further description of the Type 4 concordance is instead given below in Section 4 .

To summarize this Section : once automatic tagging is put aside there is no reason why corpora and concordancers should not be constructed in such a way as to highlight typical textual patterns recurrent in specifi  c  texts that are not achievable with traditional concordancing (see Coccetta, this volume). This insight has led to new forms of lexis-oriented concordancing. These begin to move down the multimodal cline of  concordance types indicated in Table 1 away from monomodality and towards multimodality by stimulating students' awareness of the resource integration principle (Baldry/Thibault 2006a), and down the meaning-oriented cline towards the interpretation of words as used in specifi  c multimodal texts. The testing procedures implied in Figures 1 and 2 are formative in that they relate to classroom activities. We could, however, easily devise an end-of-course, summative test that ascertains students' individual skills in understanding fi  lms based, for example, on the appropriate captioning of a sequence of frames as illustrated in Figure 2.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

## 3. Multimodal Discourse Analysis in the Environment of a Multimodal Concordance

The extensions to lemma-based concordancing referred to in Figure 2 can also be connected to multimodal discourse analysis in interesting and fruitful ways. In this Section, we shall focus on some aspects of the captioned  corpus  of  advertisements  from  the Esselunga supermarket chain in northern Italy to develop this point. Consider, for example, the poster captioned Ballerina o carote? , as shown in Figure 2 . As a statement of fi  rst principles, the linguistic items Ballerina o carote? , Bosco o broccoli? , and so on, have no meaning except in relation to the visual images that they are integrated with. The picture of the carrots-cum-ballerina is a visual metaphor that also indexes in a somewhat playful way the phenomenon of 'concealment' which has been studied by Gestalt psychologists of visual perception (Gottschaldt 1926; Arnheim, 1999). Concealment refers to the potential for a visual image to be seen in two quite different ways: is it a vase or a human face?, and so on.

The verbal caption asks the reader to choose between two confl  icting interpretations of the image. The caption specifi  es two distinct domains of activity ('ballerina' vs. 'carote') and asks the reader to make a choice. The use of the Italian conjunction 'o' ['or' in English] may suggest that the relationship between the two semantic domains is a symmetrical one in which the two conjuncts on either side of the conjunction are accorded an equal value weighting. As we shall see below, the Italian conjunction 'or' in the Esselunga texts does not have this meaning. The symmetrical relationship is implied in Martin's (1992: 205-6) systemic interpretation of the English conjunction 'or' as having the general logico-semantic meaning of 'alternation'. This is the case, for instance, in the following examples featuring the use of the conjunction 'or' from the BNC (British National Corpus) :

A0T 690 Thinking is a capacity to refer at will, whose nature is made more mysterious if we try to chop it up into mental episodes with particular epistemic content or causal power.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

A5F 92 Brodkey's  relations  with  his  adoptive  mother  Doris, a  Jewish  Clytemnestra  to  his  haunted  Orestes,  involve  a  bizarre  attempt  at  incestuous  symbiosis  ('The  feeling  of  obliteration or castration or whatever it was was unsettling as hell') and a deathbed reconciliation with a nervous breakdown as a postlude.

ACJ 968 There can therefore be little doubt that, in general, sexual assaults constitute a substantially more serious form of harm than mere property offences --; which is not to deny that some forms of theft or destruction of property can be more serious than some minor sexual assaults.

ADE 1089 This is obviously true whether the bereavement is caused by death or separation or change of circumstances.

ADW 1373 Servants were people who worked for the government or for a company: in Kufra 68 per cent, in Ajdabiya 52 per cent of men were government servants, about one third of them conscripts.

In the above examples, the conjuncts linked by 'or' are construed as being in a relationship of alternation. There is no implication that one conjunct is preferred or valued more highly than the other. Generally speaking, the semantic domains linked by 'or' in these examples stand in  a  relationship  more  like  'complementarity'  instead  of  'contrast'. Thus, thinking can be chopped up into mental episodes on the basis of 'epistemic content' or 'causal power'; as such the two terms are equally valid criteria for chopping up mental episodes. Similarly, 'death', 'separation' and 'change of circumstances', which are linked by the conjunction 'or', are seen in this example (ADE 1089) as equally possible causes of bereavement. In this and other examples cited above, the terms linked by 'or' are construed in their texts as all belonging to some common semantic class on the basis of some criterion or criteria which they are deemed to share.

However, in the Esselunga advertisements  it  is  the  semantic contrast between the two lexicalized domains which is salient. Moreover, this contrast must be seen not in terms of the verbal caption on its own, but in terms of its relation to the disjunctive meanings evoked by the visual image and hence to the overall multimodal discourse context that helps to establish the meaning of 'or' in these examples. This further highlights the importance of the ways in which meaning is created in the process of integrating language with non-linguistic aspects of the world in which language is used. Multimodal corpus linguistics therefore seeks to develop criteria for showing how this fundamental aspect of contextualization is built into corpus design and tagging practices. Crupi (2006) makes some pertinent proposals in this regard in relation to the conjunctions but , yet , and still in the perspective of the sign-based linguistic framework of the Columbia School, though without developing this framework in the directions of multimodal discourse analysis, as here. Instead of being alternatives which are placed on an equal footing, the conjunction 'or' signals an asymmetrical relationship between the linked items -the conjuncts -X and Y. In each case, the reader is being guided to choose the preferred alternative Y, which corresponds to the food item in all of the instances featured in Esselunga 's advertising campaign ( see below ). The discourse context can be schematized as an activity sequence, as in Figure 3 .

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

Figure 3. Schematisation of discourse-level activity sequence in relation to the Esselunga poster

<!-- image -->

The point  of  this  schematization  is  to  draw  attention  to  the  ways  in which the discourse context is created through the integration of diverse  activities  that  readers  must  perform  and  the  texts  (e.g.  caption + image) that are embedded in and participate in these activities. The starting point for this observation is the self-evident fact that the verbal caption and the picture are not two separate and independent domains the 'verbal' and the 'visual' - which can be studied and analyzed sepa- rately with the aid of their own specialized tools of analysis. It makes little sense to study the verbal caption on its own as a separate 'linguistic' domain because the caption exists and has the possible meanings it has only by virtue of its integration with the picture. A quick visual scan reveals the self-evident character of this observation: caption and image are juxtaposed as two parts of the one overall visual-spatial image. They are not separate 'verbal' and 'visual' domains to start with.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

In its own modest way, the example shows that words in their varied manifestations, in the activities we call speech and writing, do not constitute  autonomous  and  independent  domains.  Instead,  words  are defi  ned and have their meanings by virtue of the activities they are integrated with. Multimodality and multimodal corpus linguistics must explicitly recognize this fact and make it a central informing principle of its description, analysis and theorization of all forms of verbal and other semiotic behaviour. In the present case, the caption functions to help us make communicational sense of the picture by subtly suggesting a preferred choice from the two alternatives that are linked by 'or'. By the same token, we can only make communicational sense of the words in the caption by integrating them with and relating them to the picture. Multimodal corpus linguistics must accordingly be able to shed light on the varied ways in which we make communicational sense of texts like the Esselunga example under consideration. Moreover, it must do so on the basis of the everyday practices in which we perform the kinds of integrating (contextualizing) operations referred to above.

The genre in question is that of advertising . Advertising texts work to grab your attention and motivate you to buy goods-and-services that have been transformed into commodities by the discourse of advertising (Pateman 1983; Thibault 1990: 115-118; Beaugrande 1997: 4-5). Food items such as carrots are essential goods which, in the text, are transformed into commodities according to a market-driven ideology of consumerism. The 'ballerina or carrots' text seeks to motivate consumers to buy carrots not ballerinas at the Esselunga supermarket chain in  northern  Italy. The  given  food  item  -  carrots  -  is  therefore  transformed into a commodity potentially to be consumed by Esselunga 's customers. Why then are carrots compared to ballerinas in the playful use of the psychological tests of visual perception and the resulting visual metaphor both of which are evident in the image? In this respect, the text invites the reader to indulge in a little semiotic play in which the reader/consumer can take pleasure in appreciating the joke and its aesthetics at the same time that the decision to buy the commodity is linked to the consumer's sense of satisfaction at being able to participate in the playful and sophisticated tendencies of the text and the complicity that this participation entails with the intended meanings of its producers.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

Consider, for example, the different effects that would be produced by the caption 'buy our carrots!' juxtaposed with a picture of a fresh, juicy  looking  carrot  ripped  straight  out  of  the  garden.  In  the  present case, the overall context of situation of the text consists of managing and orienting the ways in which the activities and purposes of potential customers of Esselunga encounter food commodities in the supermarket. The 'ballerina or carrots' text semiotically mediates such encounters by offering consumers a sense of the aesthetic and intellectual pleasures to be gained for their part in playing the game afforded by the text at the same time that their decision-making is oriented to the purchase of food products such as carrots at Esselunga .

It is in this context that we can better understand the meaning of the conjunction 'or'. The verbal caption ballerina o carote? is the Predicate of an ellipted interrogative clause which functions to guide the reader/ consumer to make a choice. The expanded and more cumbersome full clause - actually two clauses - would be as follows: E' una ballerina o (sono) carote? ['Is it a ballerina or are they carrots?' + interrogative intonation]. On one level, the choice is between two readings of the picture and the sense of playfulness that this is likely to invoke in readers whether they are aware of the underlying psychological discourse from the fi  eld of visual perception or not. On another level, the choice is between a would-be decision to buy or not to buy the commodity that is focused on. We have already noted that, on the fi  rst reading, the choice is between the two disjunctive domains that are lexicalized by the terms 'ballerina' and 'carote'. This reading highlights the ambivalent status of the visual image and intertextually invokes the psychological discourse of 'concealment' that is much discussed in Gestalt psychology (Gottschaldt 1926; Arnheim 1999), or at any rate the ability to see the visual pun and hence to connect the visual patterns to two distinct areas of meaning and social practice that are intersected in the image. It is as if the reader is participating in or being invited to participate in a psychological test of visual perception.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

The reduction of the full clause complex, as expanded above, to the nominal group complex ballerina o carote? , and its framing as an interrogative, puts the discourse focus on these items and, of course, on the putative choice between them by virtue of this strategy of the deletion and reduction of lexicogrammatical items being weighted towards the items in end-position in their respective clauses. Moreover, the choice of  the  shorter,  simpler  lexicogrammatical  pattern  rather  than  the  full clause complex in this and the other examples in the corpus in Figure 2 also favours the focus on the lexicalized items in the predicate position.

Rather than treating carrots as something already known to the reader, which would require their positioning as Given information in the clausal Subject (e.g. in the clause our carrots (Given) are freshly handpicked from the garden (New)), the discourse strategy which the lexicogrammatical selection enacts treats the choice between the two disjunctive lexicalized domains as the New information in the discourse context. This is reinforced by the framing of the utterance as interrogative: the reader is thus being guided to make a choice in the fi  rst instance between two interpretations of the visual image and, by implication, to respond to the question posed by the linguistic choices in the caption.

At this point, we can return to the meaning of the conjunction 'or'. In all of the examples cited in Figure 2 , the playful item occurs in fi  rst position (henceforth conjunct X) and the food item - typically a fruit or vegetable - occurs after the conjunction in second position (henceforth conjunct Y). The more salient end-focus is, in all cases, on conjunct Y the food commodity - which the advertisements in this series are seeking to motivate customers to buy. The overall multimodal textual relationship may be modelled as in Figure 4 .

Figure 4. Modelling conjunctive relations in multimodal texts

<!-- image -->

The very small corpus featured in Figure 2 shows that the conjunction 'or' does not link alternatives that are equally entertained from the point of view of the overall discourse goals and strategies of the advertising genre. Instead, 'or' signals in each case that conjunct Y is the preferred alternative and that the discourse strategies chosen seek to orient readers-cum-consumers to this alternative and to the courses of action (e.g. the purchasing of carrots at Esselunga ) that follow from opting for this alternative. Along the way, readers are invited to indulge themselves in a little pleasure-giving game, as mentioned above, but they know that Esselunga is in the business of selling carrots, peas, broccoli, cheese, and so on, rather than ballerinas, dragonfl  ies, forests and Grecian columns. After  all,  the  advertising  discourse  seeks  to  stimulate  in  individuals a need for commodities and to recruit individuals to the goals and activities that will lead to the purchase of such commodities. At the  same  time,  consumers  are  also  co-opted  to  the  fantasy  world  of free choice and pleasure-seeking - both intellectual and physical - that these activities appear to offer in exchange for the individual's willingness to become a consumer of both commodities and the gratifi  cation fantasies they are seen to provide. Essential goods such as food items, as the examples show, are by no means exempt from these processes of commodifi  cation.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

The evidence provided by this small-scale multimodal corpus of Esselunga advertisements suggests that the meaning of the conjunction 'or' is specifi  ed by its integration to the overall communicative purposes of the multimodal discourse context and that this meaning cannot be separated from or treated in isolation from this context. According to the analysis proposed above, or links discourse items that are not equal or symmetrical. Instead, the reader is being asked to evaluate the items that occur after or (conjunct Y) as the preferred alterative . The meaning of or in this context can therefore be glossed as: [EXTERNAL ADDITIVE: ALTERNATIVE: VALUE WEIGHTED: PREFERRED].

The  analysis  and  interpretation  developed  above  does  not  claim to exhaust the possible meanings of 'or'. Instead, the purpose of this discussion is to show how multimodal corpus linguistics can help to shed light on the practices whereby individuals experience and understand  language. The  advertising  text  that  was  focused  on  here  plays on and manipulates a fundamental aspect of language behaviour: language is a means for learning about and investigating the nonlinguistic world around us. In the present case, the reader is positioned as one who would ask and/or respond to a question about some aspect of the situation that the text helps him or her to activate - in the fi  rst instance, the visual image; in the second, his or her choices as a consumer when shopping in Esselunga . In activating this aspect of language, the reader is nudged towards a consideration of carrots and other food items as potential commodities to be acquired.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

From a narrowly linguistic perspective, the lexicogrammatical or cotextual environment of the captions is quite sparse. The captions are not surrounded by extended verbal co-text in relation to which their meanings may be further specifi  ed. However, the discussion above shows that further dimensions of analysis that push beyond the immediate cotext or the lexicogrammatical selections per se can be adduced in our efforts to develop multimodal corpora. We can therefore shed light on the meanings of linguistic and other items by taking as a fundamental principle the view that 'language' and its defi  nition is never separable from a whole range of other phenomena with which language is integrated in and through meaning-making activities. As argued in Section 4 , and in Table 4 in particular, a number of interrelated scales of multimodal textual organization which interact with each other and with the goals of the producers and consumers of texts can be postulated as analytical and theoretical constructs which can aid us in the investigation of the meaning potential of particular items in particular kinds of environments. Thus, as shown above, we can shift the scope and reach of multimodal corpus linguistics to a number of scalar levels of organization such as the ones that we drew on in this Section. In this way, specifi  c  choices in different semiotic resource systems can be connected to their co-deployment in particular multimodal texts and to the largescale genre structures and activity-types in which they are embedded, as follows:

1.  DISCOURSE  GENRE: Advertising  as  Social/Communicative  Practice: Organizing the activities and priorities of individuals in terms of the ideology of consumerism;
2. THE MULTIMODAL TEXT: the composite text consisting of [caption + image] semiotically  mediates  the  encounter  between  individuals  and the goals of Esselunga to maximize sales;
3.  THE  SEMIOTIC  RESOURCES  that  are  selected  and  strategically  co-

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

deployed to enact and realize these activities and textual mediations: e.g. visual metaphor + use of 'or' to signal value-weighted preferences towards commodities and concomitant courses of action in relation to lexicalized external domains of social activity (e.g. ballerina, carrots) that  are  indexed  and  the  playful  invoking  of  multiple  contexts  of interpretation.

The examples featuring 'or' taken from the BNC (see above) were not linked to perceptually recoverable information that is relevant to the  interpretation  of  the  semantic  relationship  between  the  conjuncts so linked. The relationship between conjuncts is symmetrical and no conjunct is tied to a value weighting or preference. Moreover, there is no perceptual cueing of a preferred ranking of items through the multimodal integration of verbal text and visual image. A Search Parameter may accordingly be proposed based on the following selection of features, which characterise the meaning of 'or' in this sample, as follows: [or: Alternation: symmetrical: complementarity; - value weighting; perceptually cued].

On the other hand, the conjunction 'o' in the Esselunga corpus can be described in terms of the following feature selection: [or: Alternation: asymmetrical: contrast; + value weighting; + perceptually cued: visual image].

In this case, the Search Parameter specifi  es that such uses of 'or' implicate a value weighting of one conjunct as the preferred item and that the meaning of the conjuncts linked by 'or' is also perceptually cued by the multimodal integration of verbal caption and visual image.

This  raises  the  further  enticing  possibility  that  multimodal  corpus linguistics, in the longer run, will be able to provide powerful computational tools for modelling probabilistic, syntagmatic multimodal corpus data within the framework of a systemic (paradigmatic) approach to multimodal discourse analysis and its role in the development of multimodal corpus linguistics (see Tucker 2006: 90).

The two Search Parameters proposed above for the two sets of examples - from the BNC and the Esselunga corpora, respectively - suggest that textual syntagms can be modelled in the framework of a systemic approach that incorporates feature specifi  cations that cut across different semiotic modalities such as the verbal and the visual in the Esselunga corpus. It therefore becomes possible to say that the meaning of a linguistic item may be dependent on syntagmatic and paradigmatic environments that cannot be described in linguistic semantic terms alone. The relevant Search Parameters can therefore be devised so that multimodal corpora like those based on MCA are constructed with multimodal feature specifi  cations as an integral part of the tagging process and the grammar descriptions based on this.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

## 4. Concordances in relation to multimodal tests based on multimodal corpora

Having illustrated how multimodal discourse analysis can be enlightened by the use of multimodal concordancing, we can now move on to the discussion in this section of multimodal tests, i.e. classroom tests concerned with an explicit demonstration of awareness of the typical organization of multimodal texts (Baldry/Thibault 2006a). The development of a multimodal test, and in particular the hybrid test mentioned in Section 1 , leads us to focus on the 4 th type of concordance illustrated in Table 1 .

When, in the light of what we have stated in the previous Section, we take a second look at Table 1 , and compare it to Table 4 , we begin to see that its apparently orderly arrangement of concordances in terms of two intersecting clines, one relating to the relative copresence of meaning-making resources, the other relating to the degree to which meaning-making in specifi  c (con)texts is considered, is far from exhaustive. In actual fact, Table 1 conceals the presence of other, only partially represented clines. One is the scalar cline. This is alluded to in the previous Section and in the example given in the Type 4 concordance ( Table 3 ), concerned specifi  cally with meaning processes in multimodal texts. This includes, for example, the examination of the hierarchy of textual  (sub)units  (Baldry/Thibault  2006a:  144)  and  the  cohesive  devices (Baldry/Thibault  2006a:  179-80)  that  link  them  to  each  other  in  cocontextualizing patterns, all the way up from basic meaning-making resources (language, movement, gaze, gesture, colour and so on) through subclusters, clusters, superclusters (and similar units in fi  lms such as visual transitivity frames, subphases, phases and macrophases) to complete texts, and indeed beyond this to the study of intertextual relationships between texts (Baldry/Thibault 2006b: 177-180). Associated with this view of the nature and functions of concordancing is yet a fourth cline alluded to in Table 1 , the genre cline, which relates to the relationship between a text as single instance and a text as type , i.e. a member of a genre or subgenre (Baldry 2004, 2005a, 2005b, 2007). In this respect, the scalar cline intersects with the genre cline. Given that the relationship between instance and type is the essence of all types of concordancing,  meaning-oriented  concordances  are  particularly  valuable when they show how all four clines typically interact in texts, and, of course, within genres. Table 4 exemplifi  es this by expanding the Type 4 concordance described in the bottom right-hand corner of Table 1 into 4 subtypes.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

| TEXT 82:   | MACROPHASE 1: PRESENT   |
|------------|-------------------------|
|            | MACROPHASE 2: PRESENT   |
|            | MACROPHASE 3: ABSENT    |
|            | MACROPHASE X: PRESENT   |
| TEXT 83:   | MACROPHASE 1: ABSENT    |
|            | MACROPHASE 2: PRESENT   |
|            | MACROPHASE 3: PRESENT   |
|            | MACROPHASE X: PRESENT   |
| TEXT 84:   | MACROPHASE 1: PRESENT   |
|            | MACROPHASE 2: PRESENT   |
|            | MACROPHASE 3: ABSENT    |
|            | MACROPHASE X: PRESENT   |

| TEXT 82:   | MACRO- PHASE 1: PRESENT   | PHASES 1: PRESENT    | PHASE 1 > PHASE 2> VTFs 3 VTFS 0   |
|------------|---------------------------|----------------------|------------------------------------|
| TEXT 83:   | MACRO- PHASE 1: PRESENT   | PHASES 1, 2: PRESENT | PHASE 1 > PHASE 2> VTFs 2 VTFs 4   |
| TEXT 84:   | MACRO- PHASE 1: ABSENT    | PHASES 0 PRESENT     | PHASE 1 > PHASE 2> VTFs 0 VTFs 0   |
| TEXT 85:   | MACRO- PHASE 1: PRESENT   | PHASE 1 PRESENT      | PHASE 1 > PHASE 2> VTFs 1 VTFs 0   |

<!-- image -->

Table 4. What types of multimodal meaning-oriented concordance exist? Subtypes of Type 4 condordances

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

We may mention in passing that this does not exhaust the list of possible concordance types and that no attempt is made here either to provide such a list or to provide further illustration of the subtypes of multimodal  meaning-oriented  concordances.  Nevertheless, Table  4 helps us redefi  ne the nature and functions of concordances and encourages us to pose questions as to whether concordances can help us carry out the work of analyzing the scalar structure of individual multimodal texts (Baldry/Thibault 2006a) and if so to what extent. It also encourages us to pose questions as to whether concordances can help characterize the existence of subgenres within genres or sets of texts that enact and exemplify fusions between genres.

The answers to these questions lie in the way we look on concordances and on concordancing and, specifi  cally, how we reinterpret the function(s) of concordancing in relation to the analysis of the meaningmaking processes in multimodal texts. We may look on sets of multimodal concordances as records of human activity arranged so that similarities and differences are highlighted within and across texts. That is, by rearranging activities so that their patterned nature is made prominent,  multimodal  concordances  reconstruct processes and stages in texts involving temporal and/or causal sequencing and patterns of relationships involving mergings between different activities. Multimodal concordances thus identify similarities and differences across patterns of behaviour both in hierarchical terms (e.g. within the same class of activity and within the same text) and in relational terms (e.g. across classes of activity and across texts that record these classes of activity). The focus on activities, in particular their concurrent nature in many texts  and  their  co-deployment  of  meaning-making  resources,  has  already been suggested in the Type 4 concordance illustrated in Table 3 .

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

There is now only a short step to be taken as regards enacting a test that uses the meaning-oriented forms of concordancing to be used in conjunction with assessment of students' awareness of the multimodal organization of texts. Table 5 gives a checklist of questions used as the basis for such a test which are 'valid' for many multimodal genres. Specifi  cally, by virtue of their reference to time-related events, questions 8, 9, 10, 11b, 12, 15, 18 and 19b (the  ones in italics) could, in most circumstances, only be posed in relation to dynamic multimodal texts, such as fi  lm texts, whose meaning-making is by defi  nition dependent on the passage of time, while the majority of the others can also be put forward in relation to many printed texts. More often than not, they all take on a special signifi  cance in relation to dynamic multimodal texts such as fi  lms and websites. In this respect, it may be claimed that, as hinted above, one of the main advantages of using multimodal concordances in relation to multimodal tests is precisely that they assist in the work of encouraging the use of fi  lm texts in language learning.

There are various ways in which the questions in Table 5 can be used to test students' awareness of the multimodal organization of texts. One prototype is mentioned in a previous paper (Baldry 2006: 118-9) where comparative searches were made in relation to the transitivity system in fi  lms involving hiding that involved a tagging system similar to the Type 2 concordance mentioned in Table 1 .  Another is to require students to create a small corpus of websites (usually restricted to the home page and other pages directly linked to the home page) that is tagged in relation to the checklist in Table 5 . Space does not allow us to describe the detailed mechanics of the construction of an MCA corpus (see Baldry 2005a). However, the corpora thus created are typically concerned with the exploration of Internet genres and subgenres such as virtual tours and  virtual  museums,  online  newspapers,  presentations  of  professional associations and many other information websites (Baldry/Thibault 2006a: 104). Such explorations can be presented as part of a PowerPoint presentation that both illustrates and compares the characteristics of the different websites. At one level, this is a test of language ability that specifi  cally requires students to separate the written content of their PowerPoint presentation from their oral presentation and interpretation of the texts they are analysing (Baldry in press a). At another level, multimodal concordances , derived from students' mini-corpora, provide the basis for tabulated comparisons in the PowerPoint presentation of the various textual units in websites selected for comparison. In essence, these two stages respectively incarnate the multimodal test and the hybrid test described in Section 1 .

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

|   Answer the following questions in relation to the prescribed text(s) | Answer the following questions in relation to the prescribed text(s)                                                                                                                   |
|------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|                                                                      1 | Who/what are the participants in the text? Who/what are the main participants?                                                                                                         |
|                                                                      2 | Are they human, animal or inanimate?                                                                                                                                                   |
|                                                                      3 | Which participants are active? Which participants are passive?                                                                                                                         |
|                                                                      4 | What relationships seem to exist between the participants? How do they interact?                                                                                                       |
|                                                                      5 | What kinds of activities and events are associated to them? What temporal scales are involved?                                                                                         |
|                                                                      6 | What actions do the participants undertake? For example, do they look, move, talk and indicate they're thinking?                                                                       |
|                                                                      7 | Who/what does what to whom/what?                                                                                                                                                       |
|                                                                      8 | What can the viewer learn from the participants' changes in facial expression, their gesture and the way that they look at each other?                                                 |
|                                                                      9 | What changes occur to the participants' body state: do they bend over, hide, lie down, cover up, protect themselves, become deformed?                                                  |
|                                                                     10 | What changes appear to occur as regards the mental, emotional and behavioural aspects of the participants? For example, do they start laughing, crying or appear to become angry?      |
|                                                                     11 | Is a story told? If so, what are its characteristics? Assuming you have seen the first part , how do you think the story is likely to end?                                             |
|                                                                     12 | Is the story told by one or more narrators? Are they off-screen or on- screen?                                                                                                         |
|                                                                     13 | What attitudinal stances does the text suggest? How does the text evaluate the actions and events that are represented?                                                                |
|                                                                     14 | What events and actions change the user's attitude towards the participants as the text unfolds?                                                                                       |
|                                                                     15 | What resources or combinations of resources does the text use to indicate a particular value stance?                                                                                   |
|                                                                     16 | Does the text make you think of other texts? Which texts? Why? How do they link us to the text's wider context of culture?                                                             |
|                                                                     17 | Identify the meaning-making resources that you think are most salient or important in the text? Why did you choose these particular resources? What is their significance in the text? |
|                                                                     18 | What features - e.g. patterns of movement, action, colour, sound - connect one part of the text to other parts? What changes take place in these patternings?                          |
|                                                                     19 | What is the significance of colour? Do particular colours stand out? What changes in colour as the text unfolds indicate special meanings?                                             |
|                                                                     20 | What processes of recontextualisation are apparent in the text?                                                                                                                        |

Table 5. A checklist of questions applicable to multimodal texts

<!-- image -->

<!-- image -->

<!-- image -->

## 5. Conclusion

The research reported above is breaking new ground not just by promoting new forms of multimodal concordancing. Rather, within applicative frameworks, it is attempting to integrate meaning-oriented multimodal concordancing with more traditional forms of concordancing. However, if progress is to be made, the further development of, for example, multimodal tests, would appear to be linked to the simplifi  cation of the procedures involved.

On a wider, theory-oriented front, the design and development of research into multimodal corpus linguistics has been infl  uenced by the suggestions  of  a  growing  number  of  researchers  concerned  with  the shortcomings of traditional concordancing, including, for example, attempts  to  associate  probabilistic  approaches  with form-oriented  language-only concordances (see Tucker 2006: 90-4). Building a versatile online concordancer catering for many types of concordancing is significant in this respect. As MCA evolves, it attempts to maintain the possibilities  afforded  by form-oriented language-only concordance tools essentially concerned with the concordancing of words, their parts and combinations, such as Micro-OCP (Hockey/Martin 1987), MonoConcPro (Barlow 1999) , SARA (Aston/Burnard 1998), TACTweb (Rockwell et al. 1997) or WordSmith Tools (Scott 2001). But it also attempts to embrace the insights achieved with language-based meaning-oriented concordancers , such as Systemics , (O'Halloran/Judd 2002).

However, multimodal concordancing and multimodal tests can go beyond this. Hence, the paper has explained how multimodal concordancing will increasingly play a central role in extending the range of digital texts embraced by university text-based studies relating, for example, to languages and linguistics. Just as form-oriented, languageonly concordancing is a powerful tool in the analysis of lexicogrammatical  structures  in  texts  and  genres,  similarly  multimodal concordancing creates many possibilities for the analysis of the meaning-making units in, for example, fi  lm texts. In the future, viable multimodal tests will infl  uence the overall nature of course design. The increase in the paradigmatic inventory of possible 'text analysis' questions ( Table 5 ) that can be asked in teaching and testing modules is in itself a powerful demonstration of the capacity for multimodal text studies to meet soci- ety's new demands for a better understanding of the nature and evolution of texts in an age characterized by mergings of existing genres and the emergence of new ones.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

## References

## Software

- BNC (British National Corpus). http://www.natcorp.ox.ac.uk/
- CLAWS ,  the  Constituent Likelihood Automatic Word-tagging System, UCREL, University of Lancaster, 1987. http://www.comp.lancs.ac.uk/computing/research/ucrel/ history.html
- MCA. http://mca.unipv.it
- Microsoft  SQL  Server, Microsoft  Corporation,  1993. See www.microsoft.com/sql/default.asp. See  also www.microsoft.com/sql/evaluation/anniversary/SQLRoadmap. doc : 'The Data Management Landscape: Microsoft SQL Server 10 Years Later'.
- Micro-OCP, Oxford University Computing Services [Distributor: Oxford University Press], 1987. http://www.oucs.ox.ac.uk/internal/annrep/annrep87-88.html
- MonoConcPro, Athelstan  Publications,  Houston,  1999. http://www.athel.com/mono. html
- SARA, SGML-Aware Retrieval Application, Humanities Computing Unit, Oxford University Computing Services, 1994. http://www.hcu.ox.ac.uk/BNC/using/papers/burnard96a.htm
- Systemics 1.0., O'Halloran, Kay L./Judd, K. 2002. [CD-ROM]. Singapore: Singapore University Press. http://courses.nus.edu.sg/course/ellkoh/Overview.html
- TACTweb Bradley, Rockwell, John and Geoffrey 1997. http://tactweb.humanities.mcmaster.ca/
- WordSmith  Tools ,  Scott,  Mike  1996.  Oxford:  Oxford  University  Press,  version  1.0, 1996 to v. 3.0, 1999, & v. 4.0 forthcoming. http://www.lexically.net/wordsmith/

## Books and articles

- Ackerley, Katherine/Coccetta, Francesca in press: Multimodality in an online English course. In Baldry, Anthony/Montagna, Elena (eds.) in press, Interdisciplinary Perspectives on Multimodality: Theory and practice. Proceedings of the Third International Conference on Multimodality. Campobasso: Palladino.
- Arnheim, Rudolf 1999: On the Integration of Gestalt Theory. Gestalt Theory - An international multidisciplinary journal 21, 3, 178-180.
- Aston, Guy/Burnard, Lou 1998: The BNC handbook: exploring the British National Corpus with SARA. Edinburgh: Edinburgh University Press.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

- Baldry, Anthony 2000: English in a visual society: comparative and historical dimensions in multimediality and multimodality. In Baldry, Anthony (ed.), Multimodality and multimediality in the distance learning age. Campobasso: Palladino, 41-89.
- Baldry, Anthony 2004: Phase and transition, type and instance: patterns in media texts as seen through a multimodal concordancer. In O'Halloran, Kay (ed.), Multimodal Discourse Analysis . London and New York: Continuum, 83-108.
- Baldry, Anthony 2005a: A Multimodal Approach to Text Studies in English. The role of MCA in multimodal concordancing and multimodal corpus linguistics . Campobasso: Palladino.
- Baldry, Anthony 2005b: Children's use and awareness of genre: A case study of the evolution  from  multimodal  transcription  to  multimodal  concordancing  based  on system networks. In Taylor, Christopher/Taylor Torsello, Carol/Gotti, Maurizio (a cura di), Centri Linguistici: approcci, progetti e strumenti per l'apprendimento e la valutazione. Atti del 3° convegno nazionale AICLU . Trieste: Dipartimento di Scienze del Linguaggio, dell'Interpretazione e della Traduzione, Università degli Studi di Trieste, 423-441.
- Baldry, Anthony 2006: Promoting comparative multimodal concordancing: its role in language education, teacher training, subtitling and minority language learning. In Vasta, Nicoletta (ed.), Forms of promotion: texts, contexts and cultures. Bologna: Pàtron, 97-124.
- Baldry, Anthony 2007: The role of multimodal concordancers in multimodal corpus linguistics. In Royce, Terry/Bowcher, Wendy (eds.), New Directions in the Analysis of Multimodal Discourse . New Jersey: Lawrence Erlbaum Associates, 173-193.
- Baldry, Anthony in press a: What is Multimodality for? Syllabus Construction in English Text Studies for Communication Sciences. In Solly, Martin (ed.), Researching Language in Education: verbal / visual narrative texts. Bern: Peter Lang.
- Baldry, Anthony in press b: Turning to Multimodal Corpus Research for Answers to a Language-course Management Crisis. In Taylor Torsello, Carol et al. (eds.), Corpora for University Language Teachers. Bern: Peter Lang.
- Baldry, Anthony/Beltrami,  Michele  2005:  The  MCA  Project:  concepts  and  tools  in multimodal corpus linguistics. In Carlsson, Maj Asplund/Løvland, Anne/Malmgren, Gun (eds.), Multimodality:  Text,  culture  and  use.  Proceedings  of  the  Second  International Conference on Multimodality. Kristiansand: Agder University College/ Norwegian Academic Press, 79-108.
- Baldry, Anthony/Caldirola, Elena/Grunther, Patti 2005: Knowing and learning about what to expect: The role of multimodal concordancing in explorations of fi  lm texts in  foreign  language  short  courses.  In  Taylor,  Christopher/Taylor  Torsello,  Carol/ Gotti, Maurizio (a cura di), Centri Linguistici: approcci, progetti e strumenti per l'apprendimento  e  la  valutazione.  Atti  del  3°  convegno  nazionale  AICLU. Trieste:  Università  degli  Studi  di  Trieste,  Dipartimento  di  Scienze  del  Linguaggio, dell'Interpretazione e della Traduzione, 547-563.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

- Baldry, Anthony/Montagna, Elena in press: Interdisciplinary Perspectives on Multimodality: Theory and practice. Proceedings of the Third International Conference on Multimodality. Campobasso: Palladino.
- Baldry, Anthony/Taylor,  Christopher  2004:  Multimodal  concordancing  and  subtitles with MCA. In Partington, Alan/Morley, John/Haarman, Louann (eds.), Corpora and Discourse . Bern: Peter Lang, 57-70.
- Baldry, Anthony/Thibault, Paul. J. 2001: Towards multimodal corpora. In Aston, Guy/ Burnard, Lou (eds.), Corpora in the Description and Teaching of English . Bologna: CLUEB, 87-102.
- Baldry, Anthony/Thibault, Paul. J. 2005: Multimodal transcription and text analysis. In Hasan, Ruqaiya/Matthiessen, Christian/Webster, Jonathan J. (eds.), Continuing Discourse on Language. London and New York: Equinox, 174-176.
- Baldry, Anthony/Thibault, Paul. J. 2006a: Multimodal Transcription and Text Analysis . A multimedia toolkit and coursebook . London and New York: Equinox.
- Baldry, Anthony/Thibault, Paul. J. 2006b: Multimodal corpus linguistics. In Thompson, Geoff/Hunston, Susan (eds.), System and Corpus: Exploring Connections . London and New York: Equinox, 164-183
- Baldry, Anthony/Thibault, Paul. J. forthcoming: Multimodal Corpus linguistics . London and New York: Routledge.
- Barlow, Michael 1999: MonoConc 1.5 and ParaConc. International Journal of Corpus Linguistics 4, 1. Amsterdam: John Benjamins Publishing, 319-327.
- Beaugrande, Robert de 1997: New Foundations for a Science of Text and Discourse: Cognition, communication, and the freedom of access to knowledge and society . Norwood, NJ: Ablex.
- Coccetta, Francesca (this volume): First Steps towards Multimodal Functional Concordancing.
- Condron, Frances/Fraser, Michael/Sutherland, Stuart 2000: CTI Textual Studies guide to digital resource for the humanities . Oxford: Humanities Computing Unit University of Oxford.
- Crupi, Charlene D. 2006: Structuring cues of conjunctive yet , but , and still : a monosemic approach. In Davis, Joseph/Gorup, Radmilla J./Stern, Nancy (eds.), Advances in Functional Linguistics: Columbia School beyond its origins . Amsterdam/Philadelphia: John Benjamins, 263-281.
- Davidse, Kristin 1999: Categories of Experiential Grammar . Nottingham, UK, Department of English and Media Studies, Nottingham Trent University: Monographs in Systemic Linguistics No. 11.
- Gottschaldt, Kurt 1926: Über den Einfl  uß der Erfahrung auf die Wahrnehmung von Figuren. Psychologie Forschung 8, 261-317.
- Gregory, Michael 2002: Phasal analysis within communication linguistics: two contrastive discourses. In Fries, Peter/Cummings, Michael/Lockwood, David/Spruiell,

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

- William (eds.), Relations and Functions within and around Language . London and New York: Continuum, 316-345.
- Kress, Gunther/van Leeuwen, Theo 2006 [1996]: Reading Images: The grammar of visual design . London: and New York: Routledge.
- Halliday, Michael 1994 [1985]: An Introduction to Functional Grammar . 2 nd edition. London and Melbourne: Arnold.
- Halliday, Michael/Matthiessen, Christian 2004 [1985]: An Introduction to Functional Grammar . 3 rd  edition, London: Arnold.
- Hockey, Susan/Martin, Jeremy 1987: The Oxford Concordance Program Version 2. In Literary and Linguistic Computing 2, 125-131.
- Hoey, Michael 2001: Textual interaction: An introduction to written discourse analysis. London and New York: Routledge.
- Hunston, Susan 2006: Phraseology and system: a contribution to the debate. In Thompson, Geoff/Hunston, Susan (eds.), System and Corpus: Exploring Connections . London and New York: Equinox, 55-80.
- Martin, James R. 1992: English Text: System and structure . Philadelphia/Amsterdam: John Benjamins.
- O'Halloran, Kay (ed.) 2004: Multimodal Discourse Analysis , London and New York: Continuum.
- Pateman, Trevor  1983:  How  is  understanding  an  advertisement  possible?  In  Davis, Howard/Walton, Paul (eds.), Language, Image, Media .  Oxford:  Basil  Blackwell, 187-204.
- Rockwell, Geoffrey/Passmore, Graham/Bradley, John 1997: TACTweb: The Intersection of Text-Analysis and Hypertext, Educational Computing Research vol. 17, no. 3, 217-230.
- Royce, Terry/Bowcher, Wendy (eds.) 2007: New Directions in the Analysis of Multimodal Discourse . New Jersey: Lawrence Erlbaum Associates.
- Scott, Mike 2001: Comparing corpora and identifying key words, collocations, and frequency distributions through the WordSmith Tools suite of computer programs. In Ghadessy, Mohsen/Henry, Alex/Roseberry, Robert. L. (eds.), Small corpus studies and ELT: theory and practice . Amsterdam: John Benjamins, 47-67.
- Sinclair, John 1991: Corpus, Concordance, Collocation . Oxford and Singapore: Oxford University Press.
- Taylor, Christopher J. 2004: Multimodal text analysis and subtitling. In Ventola, Eija/ Charles, Cassily/Kaltenbacher, Martin (eds.), Perspectives on Multimodality . Amsterdam and Philadelphia: Benjamins, 153-172.
- Taylor Torsello, Carol 2001: Tagging systemically . Paper presented at ISFC XXVIII, Carleton University: Ottawa.
- Taylor Torsello, Carol/Baldry, Anthony 2005: SFL in text-based, web-enhanced language study. In Hasan, Ruqaiya/Matthiessen, Christian/Webster, Jonathan J. (eds.),

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

Continuing Discourse on Language. A functional perspective vol. 1. London and New York: Equinox, 311-342.

- Thibault, Paul J. 1990: Questions of genre and intertextuality in some Australian television advertisements. In Favretti, Rema Rossini (ed.), The Televised Text . Bologna: Pàtron, 89-131.
- Thibault, Paul J. 2000: The multimodal transcription of a television advertisement: theory and practice. In Baldry, Anthony (ed.), Multimodality and Multimediality in the Distance Learning Age . Campobasso: Palladino, 311-385.
- Thibault, Paul J. 2005: What kind of minded being has language: Anticipatory dynamics, arguability, and agency in a normatively and recursively self-transforming learning system, Part 2. In Linguistics and the Human Sciences 1, 3, 355-401.
- Tucker, Gordon 2006: Systemic in corporation : on the relationship between corpus and systemic functional grammar. In Thompson, Geoff/Hunston, Susan (eds.), System and Corpus: Exploring Connections . London and New York: Equinox, 81-102.

<!-- image -->

<!-- image -->

CSA Sociological Abstracts offers a world of relevant, comprehensive, and timely

bibliographical coverage. Thousands of easily-searchable abstracts enhance discovery of

full-text articles in thousands of key journals from 35 countries, books, and conference

ProQuest

Start here.

papers plus citations of dissertations and other media. This continuously-growing collection

includes thousands of records that are updated monthly, with backfiles to 1952-plus

scholar profiles, browsable indexes, and a thesaurus through the SA Illumina® interface.

So whatever your quest, start here with CSA Sociological Abstracts.

42

<!-- image -->

## To access international literature as diverse as the study of sociology, start here.

CSA Sociological Abstracts offers a world of relevant, comprehensive, and timely bibliographical coverage. Thousands of easily-searchable abstracts enhance discovery of full-text articles in thousands of key journals from 35 countries, books, and conference papers plus citations of dissertations and other media. This continuously-growing collection includes thousands of records that are updated monthly, with backfiles to 1952-plus scholar profiles, browsable indexes, and a thesaurus through the CSA Illumina ® interface.

So whatever your quest, start here with CSA Sociological Abstracts.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

The CSA Sociological Abstracts Discovery Prize.

Tell us how CSA Sociological Abstracts has advanced teaching and learning at your institution, and you may win the CSA Sociological Abstracts Discovery Prize.

Visit: info.csa.com/sociologicaldiscovery

## CSA Sociological Abstracts

For a free trial, contact pqsales@il.proquest.com or log onto www.proquest.com/go/add today.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->