The Process of Creation: A Novel Methodology for Analyzing Multimodal Data

Hard cut. Moving on a road past palm trees in Jamaica. Close up of posters, one that says One Love and has a picture of Bob Marley. Film of Jamaican man and little girl, who at first looks at the camera but then turns away. Voiceover: So along with a new mom, came siblings, cousins, aunts, uncles, and friends. Family climbing up a waterfall, one little white girl, Melinda is at the top of the line. Voiceover: "We had fun in Jamaica, we had fun in New York". Panning Tree line, fruit, chicken coop, chickens under the coop, someone swimming in the water. Bob Marley music: I wanna love ya. Voiceover: But the fun didn't last. Bob Marley music: And treat you right. Voiceover: When I was eight years old. Bob Marley music: I wanna love ya. Voiceover: Beverly and Nikki moved to Florida. The above text is the transcript of a clip from Jewmaican, a short documentary film produced at Reel Works Teen Filmmaking. It is a film about a young woman whose mother committed suicide when she was an infant. She was raised in part by her Jamaican nanny who, "ultimately became her mom" (Reel Works, n.d.). The film explores how she reconciles her experiences with her biological, Jewish family and her adopted, Jamaican family. We came across Jewmaican through our research on how young people learn to make digital art about the stories of their lives through their participation in youth media arts organizations (Halverson & Gibbons, 2010; Halverson, Lowenhaupt, Gibbons, & Bass, 2009). In our work, we have been interested in understanding both how the artistic production process happens and what the products represent. Specifically, we are interested in how filmmaking supports young people in exploring and representing complex issues of identity through the construction of multimodal artifacts. As the above transcript hopefully makes apparent, we found traditional methods of engaging with discourse--the creation of a flat text-based transcript and discourse analysis--unsatisfactory for capturing films as products of identity. Our dissatisfaction led us to a fundamental methodological question: How can we analyze youth produced films as multimodal artifacts of youth engagement with identity? This article represents our nearly two-year journey developing a successful and reproducible method for analyzing multimodal data. Our work is grounded in multimodality as a theoretical lens to inform the co-construction of an analytic methodology and an analytic tool for working with multimodal, video-based data. We draw specifically on Gunter Kress' (2000, 2003; Kress & van Leeuwen, 2006) construction of multimodality coupled with Jay Lemke's (1998, 2002, 2007) understanding of the combinatorial nature of modes in meaning making to frame our analysis of youth produced videos as multimodal texts. Using research that explores how youth-produced videos provide evidence of identity exploration and representation in action, we outline the development of a multimodal methodology inspired by our desire to understand the products of youth filmmaking. We then describe the co-evolution of this analytic methodology and video analysis software tool that focuses on the multimodal nature of video data, particularly ways of looking at the interaction within and across modes. Finally, we offer other examples of how our model for multimodal data collection and analysis can extend current educational research practices that use video data to understand meaning making in action. While our focus is on analyzing youth films as representations of identity, it is important to note that the films themselves are the final products of many months of workshops, shooting, editing, and mentoring. In other writing, we describe the results of our instrumental case studies (Stake, 2000) with four youth media arts organizations (YMAOs) across the United States (Halverson, in press; Halverson et al. …