<!-- image -->

<!-- image -->

## Carmen Daniela Maier* & Anthony Baldry*

## Transcription and Analysis of Multimodal Texts

The use of new technologies and innovative combinations of semiotic resources in the various fi  elds of contemporary communication has brought multimodality into focus as an approach that can provide the tools  for  investigating  complex  texts.  Multimodal  research  advances knowledge of semiotic resources such as language, images, sound, etc. and the ways in which their complex interplay creates meaning in a variety of texts that use different media. This thematic section of Hermes: Journal of Language and Communication Studies includes fi  ve articles which systematically use multimodal approaches to face up to the challenges posed by the transcription and analysis of these complex texts.

The section opens with Anthony Baldry and Paul Thibault's article, Applications  of  Multimodal  Concordances ,  which  continues  and  extends  their  long-standing  work  in  developing  new  approaches  to  the transcription and analysis of visual texts and interactive digital sites insofar as it sketches out proposals that attempt to bridge between theoretical  and  applicative  perspectives  in  multimodal  corpus  linguistics. It does so with particular reference to the development of the conceptual and software tools required to create and concordance multimodal corpora from the applicative standpoint and as such is designed to underpin the study of texts at universities in foreign-language teaching and testing cycles. One branch of this work relates to multimedia language tests which use concordancing techniques to analyze multimodal texts in relation to students' understanding of oral and written forms of discourse in English. Another branch is the exploration of multimodal tests concerned with the explicit assessment of students' knowledge of the principles and/or models of the textual organization of multimodal

* Carmen Daniela Maier Aarhus School of Business, University of Aarhus Fuglesangs Allé 4 DK-8210 Aarhus V cdm@asb.dk

*  Anthony Baldry Dipartimento di Storia e Comparazione degli Ordinamenti Giuridici e Politici Università degli Studi di Messina IT-98122 Messina anthony.baldry@gmail.com

Hermes - Journal of Language and Communication Studies no 41-2008

<!-- image -->

7

<!-- image -->

<!-- image -->

<!-- image -->

texts. As the two types of test are not mutually exclusive, a third branch of research thus relates to the development of hybrid tests which, for example, combine a capacity to analyze multimodal texts with an assessment of students' language skills, such as fl  uency in speaking and writing in English or which ascertain the multimodal literacy competencies  of  students  and  the  differing  orientations  to  meaning-making styles that individuals manifest. The paper considers these different applicative perspectives by describing the different categories of concordance achievable with the MCA online concordancer, by defi  ning their relevance to multimodal discourse analysis and by illustrating the use of meaning-oriented multimodal concordances in the creation and implementation of multimodal tests .  It  concludes by suggesting that the re-interpretation  of  the  nature  and  functions  of  concordances  is  long overdue and that the exploration of new types of concordance is salutary for linguistics and semiotics in general.

In  the  second  article, First  Steps  towards  Multimodal  Functional Concordancing , Francesca Coccetta describes a pilot project set up at the University of Padua which uses the MCA online concordancer to make a small corpus of fi  lm texts accessible to language learners. The project allows students to investigate how the same language function is enacted by a set of different language forms while at the same time providing access to the multimodal context in which they are produced. The article illustrates some of the benefi  ts that this approach brings to language learning and gives examples of teaching materials based on the fi  lm corpus which are designed to promote language learners' communicative competence.

The third article, Multimodal Communication of Specialized Knowledge across Hypertext Innovation and Generic Tradition , highlights the importance of detecting the kind of multimodal meaning-making structures that can be established when communicating specialized knowledge in a hypertext environment.

In this article, which also draws on a multimodal theoretical framework, Carmen Daniela Maier explores knowledge communication in interactive educational texts. The analytical focus is fi  rst on how specialized  knowledge  is  multimodally  constructed  inside  the  generic framework of traditional lessons through different types of interactive exchanges and across several semiotic modes. The analysis also dis- cusses how the linear reading path imposed by the generic structure of traditional  lessons  is  disrupted  by  hypertext's  meaning-making  pathways. The article concludes that the stable generic structure of lessons combined with the openness of hyperlinks can be and, to some extent, is being exploited in websites to enhance the process of progressively acquiring, producing and exchanging specialized knowledge across several semiotic modes.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

In  the  fourth  article ,  Engaging  with  children' s  graphic  ensembles of an archaeological site: A multi-modal social semiotic approach to learning , Sophia Diamantopoulou uses social semiotics and multimodality  as  tools  to  introduce  a  different  perspective  when  it  comes  to building a descriptive and an interpretative framework for the analysis of children's production, which is taken to be representative of their learning. The article presents two examples from children's multimodal production as an opportunity to engage critically with what learning in an archaeological site means. The insight into the children's work is based on the assumption that learning can be multimodally mediated through a particular pedagogy and made accessible to us through the material realisation of children's production across multiple modes. The paper explores the implications of this position for generating knowledge about children's learning. Furthermore, it also discusses the importance of anchoring multimodality, as an analytical perspective, onto a theory with its own descriptive and interpretative language - in this case social semiotics and discourse theory.

In the last article, Multimodal Literature 'Moves' Us: Dynamic Movement and Embodiment in VAS: An Opera in Flatland , Alison Gibbons uses cognitive fi  ndings in order to analyze the expressive synthesis between word and image in multimodal fi  ction. Focusing on the 'imagetext novel' VAS: An Opera in Flatland , written by Steve Tomasula and designed by Stephen Farrell (2002), this article uses cognitive-poetic analysis to explore multimodal printed literature. The examination of the visual elements is supported by theories from visual perception and multimodal research based on cognitive and perceptual methodology which is,  in  turn,  strengthened  by  refl  ection  on  recent  fi  ndings  from neuroscience. The article thus presents a fresh approach to multimodality, which in keeping with the work of other multimodal researchers, considers all modes of meaning-making to be on an equal but collabo- rative footing but which also considers the neurological and embodied aspects of a multimodal literary experience.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->