On the multimodality of English [ADV and ADV] construction:
A collostructional approach
Daiya Kuryu
Keio Research Institute at SFC, Japandaiya.kuryu@gmail.com
Keywords: multimodal constructions, collostructional analysis, gesture lexicon
In the study of multimodal constructions, it remains an important methodological issue that theentrenchment of multimodal constructions depends not only on the frequency with which certain
gestures co-occur with particular verbal constructions, but also on how salient and typical the gestures
are for the constructions (Hoffmann 2017). As several researchers have proposed applyingcollostructional analysis to address this issue (e.g., Hoffmann 2017; Schoonjans 2017; Zima 2017), thisstudy employs such analysis to the actual investigation of multimodal constructions. What makes thisanalysis possible is the notion of the gesture lexicon (Kipp 2004), where its lexical entries, or gesturelemmas are “taken as prototypes of recurring gesture patterns where certain formational features remain
constant over instances” (Kipp et al. 2007: 4). With reference to previous studies on recurrent gestures(Ladewig 2014), a total of 62 gesture lemmas were identified in this study, and some of them showedsignificantly high collostruction strength to the constructions investigated. Thus, the results provide
compelling evidence that multimodal constructions are entrenched in our mind.
Collected from the TED Corpus Search Engine, an online corpus system that searches transcriptsof over 4,800 TED Talks (Hasebe 2015), the data used in this study form a multimodal corpus with 407
speakers performing 1,092 gestures in total. Following Kipp’s NOVACO scheme (Kipp 2004), allgestures were coded in ELAN and assigned gesture lemmas. The construction under investigation isEnglish [ADV and ADV] construction, instances of which compose the reduplicative adverbialconstructions (over and over [N = 160], again and again [N = 107], and on and on [N = 48]) and the
oppositive adverbial constructions (back and forth [N = 110], up and down [N = 92], and in and out [N =59]). They form a constructional network (Figure 1) through formal or semantic analogy.
Regarding the reduplicative adverbial constructions, each construction exhibits a similar tendency
that the gesture lemma PROGRESS (Figure 2) has the highest collostruction strength. Of even greaterinterest is the fact that the reduplicative adverbial constructions as a whole show much higher
collostuctional strength to the gesture lemma (Table 1), indicating that language users have lexicallyschematic multimodal constructions. Conversely, the oppositive adverbial constructions display different
dispositions, each of which favors particular gesture lemmas. Moreover, Table 2 demonstrates that theoppositive adverbial constructions holistically exhibit high collostruction strength to the repetitive
gestures that involve opposite movements regardless of the directions (Figures 3–6). This finding raises
the possibility of multimodal constructions that are both lexically and kinesically schematic in nature.
References
Hasebe, Yoichiro. 2015. Design and implementation of an online corpus of presentation transcripts ofTED Talks. Procedia - Social and Behavioral Sciences 198. 174–182.
Hoffmann, Thomas. 2017. Multimodal constructs – multimodal constructions? the role of constructionsin the working memory. Linguistics Vanguard 3(s1). 1–10.
Kipp, Michael, Michael Neff & Irene Albrecht. 2007. An annotation scheme for conversational
gestures: How to economically capture timing and form. Language Resources and Evaluation
41. 325–339.
Kipp, Michael. 2004. Gesture generation by imitation: From human behavior to computer characteranimation. Bocca Raton, Florida: Dissertation.com.
Ladewig, Silva H. 2014. Recurrent Gestures. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H.
Ladewig, David McNeill & Jana Bressem (eds.), Body – language – communication: An
international handbook on multimodality in human interaction. Handbooks of linguistics andcommunication science (38.2.), 1558–1574. Berlin & Boston: De Gruyter Mouton.
Schoonjans, Steven. 2017. Multimodal construction grammar issues are construction grammar issues.
Linguistics Vanguard 3(s1). 1–8.
Zima, Elisabeth. 2017. Multimodal constructional resemblance. The case of English circular motionconstructions. In Ruiz de Mendoza Ibáñez, Francisco José, Luzondo Alba Oyón & Pérez Paula
Sobrino (eds.), Constructing families of constructions: Analytical perspectives and theoreticalchallenges, 301–337. Amsterdam: John Benjamins Publishing Company.

Tables & Figures
Table 1. Gesture lemmas attracted to RACs
Cogestures (N = 294)

Frequency

P Fisher exact

Frequency

in the corpus(Intermodality Strength)

39.58

147

1.86E-36

I.To-Fro (45)ATTRACTED
I.Progress (107)

Table 2. Gesture lemmas attracted to OACs

ExpectedCogestures (N = 238)

Expected

Frequency

P Fisher exact

Frequency

in the corpus(Intermodality Strength)

ATTRACTED
12.42

57

3.18E-21

I.Wiping-Window (15)

5.65

21

2.22E-05

I.Up-Down (35)

9.37

43

2.40E-17

Beat (81)

60.04

223

5.14E-04

I.Small-To-Fro, 2H (38)

12.21

56

3.17E-14

I.Clockwork (4)1.35

52.04E-02

I.Small-Up-Down (17)4.14193.96E-10

I.Small-To-Fro, 1H (16)

4.79

22

3.53E-07

I.In-Out (8)

2.4

11

4.09E-04

Figure 1. The constructional network

Figure 2. PROGRESS

Figure 3. TO-FRO

Figure 4. SMALL-TO-FRO, 2H

Figure 5. UP-DOWN

Figure 6. IN-OUT