Super Synthesis Pros., orWhy CHI PLAY Needs Research Synthesis
Katie Seaborn

seaborn.k.aa@m.titech.ac.jp
Tokyo Institute of TechnologyTokyo, JapanABSTRACT
Games user research is a-booming—or maybe a-goomba-ing—witha boundless parade of papers popping up from every nook and pipe.
We may need a super power—or super method—from another world.
I outline three motivations for jump-starting research synthesis ingames user research. I argue that: research synthesis will validate
this field of study and enrich primary research (meta-scholarship);we must level up both primary and secondary research (education);
and we should reflect this epistemological stance in community
structures and adopt established tools and protocols (standardization). I offer power-ups to get the toads rolling.

CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI); • General and reference → Surveys and overviews;• Applied computing → Computer games.

KEYWORDS
research synthesis, literature reviews, systematic reviews, games
user research, player experience
ACM Reference Format:
Katie Seaborn. 2023. Super Synthesis Pros., or Why CHI PLAY Needs Research Synthesis. In Companion Proceedings of the Annual Symposium on
Computer-Human Interaction in Play (CHI PLAY Companion ’23), October
10–13, 2023, Stratford, ON, Canada. ACM, New York, NY, USA, 3 pages.
https://doi.org/10.1145/3573382.3616051

1

INTRODUCTION TO THE MUSHROOMSYNTHDOME

Super Mario is a transmedia phenomenon, with titular character
Mario going on to star in movies, television series, comics, andcartoons. The appeal spans cultures and generations, offshoots and(re)incarnations, a plethora of merchandise, and even a theme parkin Japan. Similarly, games user research is a transdisciplinary areaof study: anchored to a wealth of philosophical bases, drawing froma variety of methodological traditions, and contributing to a range
of knowledge sectors spanning computer science, psychology, andthe humanities. Like its fountainhead, human-computer interaction(HCI), games user research is entering a phase characterized by aPermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
CHI PLAY Companion ’23, October 10–13, 2023, Stratford, ON, Canada© 2023 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-0029-3/23/10.
https://doi.org/10.1145/3573382.3616051

235

long-running conference, recognition within and beyond the field,
and an increasing number of papers being published each year [16].
I argue that now is the time for a transformation in community
positionality and practice. In short, games user research needs research synthesis, and pronto. Like Mario, a simple plumber finding
himself in a strange new world, we can approach this moment withcuriosity and bravery, even if there may be growing pains and nota few missing princesses along the way.
Research synthesis, like the Mushroom Kingdom, treads groundboth familiar and uncanny. An umbrella term, it refers to metamethods that aim to capture and synthesize the literature on aparticular topic [12]. The purpose is to bring together primaryresearch outcomes in a meaningful way and ideally in a format forcommunication within and beyond the field. Here, I motivate theuptake of research synthesis work within games user research. As abase of human knowledge, our field requires research synthesis andmay benefit from a customized approach–its own "Koopa Troopa"of "survey" soldiers, heralding from other domains but acclimatized
to this "level" of study. We must "hammer" this initiative into our"world" in a community-oriented and community-driven way. I
sound the call for a power-up.

2THREE MOTIVATIONS TO POWER-UPRESEARCH SYNTHESIS IN GAMES USERRESEARCH
2.1 Meta-Scholarship: Beware of Pixels andPantomimes
Mario’s distinctive look—a mustachioed ball of red, white, andblue—has become iconic. Yet, it emerged from the era’s technicalshortcomings: limited pixels and colour sets, restrictions on animation. Even after technical advances and many iterations on thefranchise, the creators have stayed the minimalist course, adding
novel aspects, such as the game-changing wall-jumping ability,with prudence. This formula of restraint is a time-tested success inthe case of Mario. The impact of the series and character on gamingin general cannot be understated.
Novelty is a driver of much research. Calls for novelty and innovation pepper the author instructions at CHI PLAY and adjacentvenues. But new ideas are not necessarily good for knowledge generation and advancing practice [3]. Publication biases [11], replication crises [18], p-hacking [10], and preprint mayhem [1] aredisastrous side-effects, and games user research will not be exempt.
At the same time, the status quo of what games user research "is"can become entrenched [5]: Goomba after Goomba after Goomba,
soaking up Mario’s attention. We need to zoom out and take in thewhole level. Like well-honed character design and level-crafting,

CHI PLAY Companion ’23, October 10–13, 2023, Stratford, ON, Canada

Seaborn

research synthesis can guide the choice of studies, balance novelty
against established tracks, and allow those outside of the field tograsp the fundamentals.

2.3

Power-ups:• Time Limit: Games user research is at a critical juncture: ayoung but maturing field. We do not want to find ourselvesin a "paperdemic" [6], unable to justify our worth to thelarger research community. The timer is counting down, andwe have many levels to traverse. We must takenow.
• Stomp Novelty: Let us embrace novelty but not to the extent that we shift the focus away from better understanding known player phenomena and replicating results. Even
Mario has to slow down at points, to creep over quicksandor crawl along rooftops. Slow science [19] has a place here.
• World Select Menu: We need a resource hub to encouragecommunity-centrism and discourage reinventing the wheel.
CHI PLAY can also offer a publication track similar to HRI’s
Short Contributions1 .

2.2

Games user research, like HCI [16], is characterized by unique"worlds." We should adopt standardized tools–but also consider
whether and how they need to be transformed. We hail from avariety of backgrounds and have a range of epistemological stances.
Indeed, I chose "research synthesis" over other terms because thisdiversity should be embraced.
Power-ups:• Phenomenon Registration: Phenomena of interest, subjectsof study, hypotheses, and measured constructs should beregistered at a central repository. Primary research should belinked to each registration on publication. Examples include
validated measures, such as the PXI [2] and its "chibi" version[9], as well as others frequently used in the field (refer toBrühlmann and Mekler [4]), accounts of experiences withspecific games (compare Tetris to League of Legends), andplayer behaviours, such as cheating [15] or (too afraid tocite) violence.
• Reporting Structures: Data are meaningless without framing.
CHI PLAY advanced the notion of citing games, but theseguidelines mysteriously disappeared after 20207 and needrefining, such as for series. Gerling and Birk [7] advocatedfor standardizing how games as research data are described.
Metadata and schema may also be used for automated synthesis and need standards to be adapted for the particularities
of games user research.
• Primary Reporting Tools: We can adapt reporting frameworks
and schema, such as SPIDER [13] or ROSES [8], decided inworkshops to instill community norms and encourage theiruse by publishers in official templates.
• Secondary Reporting Tools: We need to identify a selectionof reporting frameworks, such as PRISMA [14], and adapt
these to games user research using a similar approach as forprimary reporting tools.

Education: Becoming Super Synthesis Pros.

We have warped into a strange new world. But we need not be
fearful. Like Mario, we have a suite of power-ups—methods, protocols, tools, and templates—to assist us in becoming super synthesispro(fessional)s. For this, we need to consider two broad perspectives: primary research and secondary research synthesis. Somewill wear both hats; others, one. Still, we must work in sync toenable knowledge-building. Even so, an informal survey of the HCI
community (n=48) indicated that people desire more consistency
and rigour in how research synthesis is done2 . Primary research is
not merely coins to collect, nor is secondary research as forbidding
as Bowser and his minions. We all participate in an ecology ofresearch praxis, and as players, we owe it to each other to learn
about and value our place in this world.
Power-ups:• Tutorial Level: We can world-hop between the initial guides
offered in medicine and health (e.g., Tawfik et al. [20] may
be especially relevant, since it is focused on simulation data)and other media, such as YouTube3 .
• Sandbox: We can learn by doing: tracing the players before
us by replicating existing syntheses (e.g., how "WEIRD" and"uncanny" [17] is CHI PLAY research ...?) or playing with
open data sets4 , like the longitudinal PowerWash Simulator
data set5 and the PXI (Player Experience Inventory) Benchmark Data6 .

1 https://humanrobotinteraction.org/2022/short-contributions/

2 Results @ HCI Systematic Review-lution: https://ajkt.github.io/sysreviewlution/informalsurvey2023.html
3 For example, the guides by Research Shorts (e.g., https://youtu.be/WUErib-fXV0) andDr. Amina Yonis (e.g., https://youtu.be/TLvF80WIXX8).
4 To take random yet awesome example of curation: https://github.com/
leomaurodesenv/game-datasets
5 https://psyarxiv.com/kyn7g
6 https://playerexperienceinventory.org/bdata

Standardization: Synth-o-Ware, from Warp
Pipes and Wing Caps to Hammers andHurly Gloves

3CONCLUSIONS AND NEXT STOMPSMario often pops a Super Mushroom when starting a new adventure.
I offer this set of mushrooms motivations for the community tomunch on. Will we hit the block and power-up? Or will we barrel
headlong into a shell-full troop of unknown number? Researchsynthesis need not be as elusive as a princess in another castle.

ACKNOWLEDGMENTS
Cappies off to fellow systematic review-lutionary, Dr. Katja Rogers,
for inspiration and sundry. I also thank the chairs and the external
peer reviewers for jumping on board and augmenting the decisionmaking process on this paper. Nintendo-esque icons were sourced
from pngwing (non-commercial use) and fall under fair use.

7 https://chiplay.acm.org/2020/guidelines/236

Super Synthesis Pros.

CHI PLAY Companion ’23, October 10–13, 2023, Stratford, ON, Canada

REFERENCES[1] 2020. Rise of the preprints. Nature Cancer 1, 11 (Nov. 2020), 1025–1026. https://doi.org/10.1038/s43018-020-00151-y Number: 11 Publisher: Nature PublishingGroup.
[2] Vero Vanden Abeele, Katta Spiel, Lennart Nacke, Daniel Johnson, and Kathrin
Gerling. 2020. Development and validation of the player experience inventory:
A scale to measure player experiences at the level of functional and psychosocial
consequences. International Journal of Human-Computer Studies 135 (2020),102370. https://doi.org/10.1016/j.ijhcs.2019.102370[3] Mara Balestrini, Yvonne Rogers, and Paul Marshall. 2015. Civically Engaged
HCI: Tensions between Novelty and Social Impact. In Proceedings of the 2015
British HCI Conference (Lincoln, Lincolnshire, United Kingdom) (British HCI’15). Association for Computing Machinery, New York, NY, USA, 35–36. https://doi.org/10.1145/2783446.2783590[4] Florian Brühlmann and Elisa D Mekler. 2018. Surveys in games user research.
Games user research (2018), 141–162.
[5] Johan S. G. Chu and James A. Evans. 2021. Slowed canonical progress in large
fields of science. Proceedings of the National Academy of Sciences 118, 41 (Oct.
2021). https://doi.org/10.1073/pnas.2021636118[6] Ricardo Jorge Dinis-Oliveira. 2020. COVID-19 research: Pandemic versus “paperdemic”, integrity, values and risks of the “speed science”. Forensic sciences
research 5, 2 (2020), 174–187. https://doi.org/10.1080/20961790.2020.1767754[7] Kathrin Gerling and Max V Birk. 2022. Reflections on Rigor and Reproducibility:
Moving Toward a Community Standard for the Description of Artifacts in Experimental Games Research. In Extended Abstracts of the 2022 Annual Symposium on
Computer-Human Interaction in Play. 266–267. https://doi.org/10.1145/3505270.
3558360[8] Neal R. Haddaway, Biljana Macura, Paul Whaley, and Andrew S. Pullin. 2018.
ROSES RepOrting standards for Systematic Evidence Syntheses: pro forma, flowdiagram and descriptive summary of the plan and conduct of environmental
systematic reviews and systematic maps. Environmental Evidence 7, 1 (March2018). https://doi.org/10.1186/s13750-018-0121-7[9] Aqeel Haider, Casper Harteveld, Daniel Johnson, Max V Birk, Regan L Mandryk,Magy Seif El-Nasr, Lennart E Nacke, Kathrin Gerling, and Vero Vanden Abeele.
2022. miniPXI: Development and validation of an eleven-item measure of theplayer experience inventory. Proceedings of the ACM on Human-Computer Interaction 6, CHI PLAY (2022), 1–26. https://doi.org/10.1145/3549507[10] Megan L. Head, Luke Holman, Rob Lanfear, Andrew T. Kahn, and Michael D.
Jennions. 2015. The extent and consequences of p-hacking in science. PLOS Biology 13, 3 (March 2015), e1002106. https://doi.org/10.1371/journal.pbio.1002106

237

Publisher: Public Library of Science.
[11] Kasper Hornbæk. 2015. We Must Be More Wrong in HCI Research. Interactions
22, 6 (oct 2015), 20–21. https://doi.org/10.1145/2833093[12] Marina Krnic Martinic, Dawid Pieper, Angelina Glatt, and Livia Puljak. 2019.
Definition of a systematic review used in overviews of systematic reviews, metaepidemiological studies and textbooks. BMC medical research methodology 19(2019), 1–12.
[13] Abigail M Methley, Stephen Campbell, Carolyn Chew-Graham, Rosalind McNally,and Sudeh Cheraghi-Sohi. 2014. PICO, PICOS and SPIDER: A comparison study ofspecificity and sensitivity in three search tools for qualitative systematic reviews.
BMC health services research 14, 1 (2014), 1–10. https://doi.org/10.1186/s12913014-0579-0[14] David Moher, Alessandro Liberati, Jennifer Tetzlaff, Douglas G Altman, and thePRISMA Group*. 2009. Preferred Reporting Items for Systematic Reviews andMeta-Analyses: The PRISMA statement. Annals of internal medicine 151, 4 (2009),
264–269. https://doi.org/10.7326/0003-4819-151-4-200908180-00135[15] Cale J Passmore, Mathew K Miller, Jun Liu, Cody J Phillips, and Regan L Mandryk.
2020. A cheating mood: The emotional and psychological benefits of cheatingin single-player games. In Proceedings of the Annual Symposium on ComputerHuman Interaction in Play. 58–70. https://doi.org/10.1145/3410404.3414252[16] Katja Rogers and Katie Seaborn. 2023. The Systematic Review-lution: A Manifesto
to Promote Rigour and Inclusivity in Research Synthesis. In Extended Abstractsof the 2023 CHI Conference on Human Factors in Computing Systems. ACM, NewYork, NY, USA, 1–11. https://doi.org/10.1145/3544549.3582733[17] Katie Seaborn, Giulia Barbareschi, and Shruti Chandra. 2023. Not Only WEIRD
but “Uncanny”? A Systematic Review of Diversity in Human–Robot Interaction
Research. International Journal of Social Robotics (2023), 1–30. https://link.
springer.com/article/10.1007/s12369-023-00968-4[18] Patrick E. Shrout and Joseph L. Rodgers. 2018. Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis. Annual
Review of Psychology 69, 1 (2018), 487–510. https://doi.org/10.1146/annurevpsych-122216-011845[19] Isabelle Stengers. 2018. Another Science is Possible: A Manifesto for Slow Science.
John Wiley & Sons. Google-Books-ID: oxJSDwAAQBAJ.
[20] Gehad Mohamed Tawfik, Kadek Agus Surya Dila, Muawia Yousif Fadlelmola
Mohamed, Dao Ngoc Hien Tam, Nguyen Dang Kien, Ali Mahmoud Ahmed, andNguyen Tien Huy. 2019. A step by step guide for conducting a systematic review
and meta-analysis with simulation data. Tropical medicine and health 47, 1 (2019),
1–9. https://doi.org/10.1186/s41182-019-0165-6