WORKSHOP REPORT

Building Community: Report on the 2nd
International and Interdisciplinary Perspectives
on Children & Recommender Systems (KidRec)at IDC 2018
Jerry Alan Fails
Dept. of Computer Science
Boise State University, Boise, ID, USA
jerryfails@boisestate.edu
Maria Soledad Pera
People and Information Research Team
Dept. of Computer Science
Boise State University, Boise, ID, USA
solepera@boisestate.edu

Natalia Kucirkova
UCL Institute of Education
London, UK
n.kucirkova@ucl.ac.uk

Abstract
We discuss the outcomes of the 2nd Workshop of the International and Interdisciplinary
Perspectives on Children & Recommender Systems (KidRec 2018), co-located with the 2018
ACM Interaction Design and Children (IDC) Conference, which took place June 19-22 in
Trondheim, Norway. The goal for this workshop was to explore research and industry efforts
focused directly on or facilitating the algorithmic recommendation process for children. The
diversity of attendees–including K-12 teachers, educators, researchers in various areas ofcomputer science, and industry participants–made it possible to continue to build community
around this important topic and further discuss and outline the salient research questions
and the next steps to promote more research in this area.

1

Introduction

Building on the open challenges presented at the first edition of the KidRec workshop [7],Fails, Pera, Kucirkova and Garzotto organized the “2nd International and Interdisciplinary
Perspectives on Children & Recommender Systems (KidRec)”, as part of the ACM Interaction Design and Children Conference (IDC) in Trondheim, Norway, June 2018.
IDC is a conference that is specifically focused on the target audience of interest for
KidRec: children. Further, IDC welcomes attendees who are experts in various areas ofinterest in computer science and it is also frequently attended by educators and social science
researchers. Due to the multidisciplinary nature of the work often presented at IDC and the

ACM SIGIR Forum

138

Vol. 52 No. 2 December 2018

diverse audience participating at this conference, we felt that IDC was the ideal venue tonurture dialog around child-centered recommender systems.
The workshop brought together an interdisciplinary group of researchers in education,
child-development and computer science as well as childrens designers interested in issues related to recommender systems designed for children. The workshop participants discussed existing platforms and apps that incorporate recommendation-based technology, in addition to
ethics, techniques, privacy and security issues implicated in design solutions of educationallyrelated recommender systems.

2

Workshop Description

As outlined in the workshop proposal [3], we aimed to organize a highly-interactive workshop where accepted contributions were meant to start the conversation on open issues andchallenges in the area of recommendations to and for children.

2.1

Audience of Interest & Format

Since workshop objectives included sharing research and projects that reach beyond classic
recommendation techniques and discussing the many child-related challenges of recommendersystems; we approached researchers and developers with diverse expertise: Technology forchildren; Human computer Interaction; Information Retrieval; Assessment; Ethics, Educators, Literacy Development, and e-commerce.
By reaching out to members from academia and industry, we anticipated continuing tofoster a multidisciplinary perspective that pinpoints challenges and aspects to consider when
designing, developing, and deploying recommendation technology for which children are themajor stakeholders.

2.2

Accepted Contributions

Six papers were selected to be presented during the workshop [1, 2, 4, 5, 6, 8]. Based onthese accepted contribution, we planned two interactive panel sessions during the day. The
goal for these sessions was for presenters to highlight key elements of their work and initiate
discussion among the attendees. A summary of the two interactive panels follow the abstracts
for accepted contributions.

2.2.1

Abstracts

Alternative approach to Recommender Systems for children: positioning thechild at the start, not the end of the chain - by N. Kucirkova.
This paper presents a design concept for recommender systems embedded in a story-telling
context that uses tangibles to support childrens motivation to read. It is argued that thecurrent model of recommendation systems which follows either extrinsic motivation or commercial interests cannot be pursued to support childrens intrinsic motivation to read. An
alternative model that dynamically recommends content based on the childs response is presented in the form of a prototype called Digistoryland.

ACM SIGIR Forum

139

Vol. 52 No. 2 December 2018

Teaching an Alien: Children Recommending What and How to Learn - by M.
Landoni, E. Murgia, F. Gramuglio and G. Manfredi.
In this paper, we describe how, inspired by fiction design, primary school children are helping
us to create a teachable 3D Tutor, with the appearance and personality of a friendly Alien.
In line with existing literature, we assume that children enjoy learning by teaching and, bymaking the overall process more fun, the level of childrens engagement and their motivation
towards it will increase. In our study, children will act as recommenders, as they are asked toprepare the lessons, i.e., decide and select the material to teach and the way it is presented
to the 3D Alien. Their goal is to make sure s/he will have a successful learning experience.
We will test the validity of these recommendations by measuring the level of engagement inchildren of the same age group when offered these lessons.
Recommending Texts to Children with an Expert in the Loop - by M. S. Pera, K.
Wright and M. Ekstrand.
In this position paper we discuss a number of open problems we believe the community should
address in order to enhance the recommendation task for children. We specifically outline
algorithmic and evaluation limitations when it comes to recommending reading materials forchildren in the classroom setting. Furthermore, we focus on the need to involve an expert aspart of the recommendation process to better serve the population under study.
Designing a Personally Meaningful ABCmouse.com: Challenges and Questionsin an EdTech Recommendation System - by T. Horiuchi, M. Rothschild, R. Barrera
and S. Gururajan.
Designers and curriculum specialists at Age of Learning, Inc. are enhancing the current
content strategy for flagship product ABCmouse.com. With an activity base of over nine
thousand activities and growing, specialists from the fields of user experience, research, engineering, and analytics have come together to tackle the pragmatic questions around design,
automation, metric, and experience, as well as broader philosophical/pedagogical questions
around agency, learning, and user profiling.
Insights From Pediatric Speech Language Pathologists: Considering Instructional Goals In Recommendations For Children with Disabilities - by Y. Du.
One of the most controversial debates today is whether popular technology like mobile devices
is negatively impacting communication and social interaction for modern youth, including
those with disabilities. My research aims to investigate the design challenges and opportunities when mobile applications are implemented by speech language pathologists (SLPs) who
work with children with disabilities. In this paper, I examine empirical research on how SLPs
critique and evaluate mobile apps. I suggest ways these reviews might be used by others to
guide the design of app recommendation systems that embed instructional goals for children
related to communication and social interaction skills..
The Ethics of Recommending Based on Data Collected from Children - by J. Fails.
Data collection, mining, even hoarding, has become a common practice in todays digital age.
This data is collected to enable targeted recommendations and personalized advertising. Thisdata is being collected about children as well as adults. The collection and voluntary sharing
of information that enables effective recommendations raises several ethical questions suchas: what information is appropriate to gather regarding children, what is the age of consent

ACM SIGIR Forum

140

Vol. 52 No. 2 December 2018

for sharing information, who owns the data, who decides what information can be forgotten
and when, and what who influences the kinds of recommendations that are made. These andother questions are presented in this position paper for discussion at the workshop.

2.2.2

First Interactive Panel: Limitations and Resources

The first interactive panel provided a forum for attendees to (i) discuss the limitations thatarise when suggestions are used within a classroom setting and (ii) share resources. Examplesof the latter include the WizeNoze search engine and the LITERATE news engine, which
offer curated and/or labeled content for teachers as an initial attempt to suggest material
suitable to the educational context in respond to a query.
• Natalia Kucirkova [5] started the day by discussing how personalization could be strategically embedded into the recommendation process to motivate children to read.
• Emiliana Murgia and Monica Landoni [6] shed a light on the influence recommendations
can have on learning engagement when it comes to primary school students.
• Sole Pera [8] brought up important points to be considered when suggesting texts
to children in the classroom setting, including background knowledge and readability
levels. She also discussed ways to represent and capture complex information needs andrequirements including when teachers are looking for information sources for a class anda recommender system presents them with candidate suggestions that are later to beshared with their students.

2.2.3

Second Interactive Panel: Process and Ethics

The discussion following the second session instead led to higher level concerns and directions
to follow when planning research and deployment of technologies that will directly impact
children.
• Takeshi Houruchi and Ray Barrera [4] presented the ABCmouse.com platform, which
offers educational content for children up-to 8 years old. They specifically discussed theinherent need for multidisciplinary teams when it comes to designing, deploying, andcommercializing educational technology for children.
• Yao Du [1] highlighted the benefits, but also the many limitations, when it comes torecommendations for children in a medical environment.
• Jerry Fails [2] discussed the need to create data sources that can be used to inform
design and evaluate recommendation technologies for children, while balancing ethicalconcerns, which are of special importance for younger audiences.

2.3

Discussion, Challenges, and Next Steps

During the interactive panels, all workshop participants noted challenges and opportunities
that needed to be addressed within the field using sticky notes. All sticky notes were collected
and utilized to perform an affinity diagramming exercise to identify groupings of challenges
and opportunities as well as salient outliers (or singletons). We list some of these challenges
and open questions below.

ACM SIGIR Forum

141

Vol. 52 No. 2 December 2018

• Evaluation and Measures. What is it that we should evaluate when trying to qualify performance of a recommender system for children, and then how do we adequately
measure that? For example, it is easy to measure engagement (e.g., based on clicks), but
is it possible to infer relevance from engagement? Additional questions include: How
do we collect representative quantitative and qualitative data for assessment without
being intrusive to the target subject (i.e., the child)? How to appropriately balance orreconcile observational, behavioral data with self-reported (or even parent-, or teacherreported) survey data? “Observational data” is problematic too, as it is hard to knowwhat or whom you are observing and what impact that should have on the overall
assessment of recommendation technology. For example, even if we had access to samples of purchases of children materials on Amazon, it would still be hard to infer muchfrom that source data given that parents purchasing items does not necessarily meanchildren actually used or favored said items.
• Multidisciplinary Approach. The discussion at the workshop emphasized the importance of interdisciplinary approaches to designing, developing, and evaluating recommender systems for children. With that understanding there are several open questions:
How can the various stakeholders help better understand the interaction and interplay
between systems and children? What are the relative strengths, competencies, andcontributions of the various stakeholders (e.g., children, parents, teachers, practitioners)? How do we involve all of the stakeholders in a meaningful way? Is it by bringing
them at different stages of the design and development process, as a cohesive team in
a participatory design approach, or some other process? In addition, because of theinterdisciplinary approach how do we then train practitioners on the best practices forutilizing recommender systems effectively in complex environments, such as educationalor medical settings? Because of the firm belief that the interdisciplinary nature of teams
is a must in the design of successful technologies focused on recommendation that are
to be later welcomed by the community at large, another important question raised
was how to continue to grow the interdisciplinary composition of this community?
• Balancing Perspectives. Since there is a need for broad representation of disciplines,
how are various perspectives appropriately balanced? For example, how do you balancechild interest versus curriculum? How do recommender systems balance the agency ofthe child, versus adult interest, or “what is best” for the child? And how is “whatis best” determined or properly balanced? Is it (or how much of it is) determined byparents, or educators, and is it influenced by social identity or educational value? Or is it
determined by various kinds of children? For example, do peers that read more become
stronger influencers with regards to what kinds of books are appropriate for their peers?
Can resources like child book reports (that are done by children around the world), beutilized as input to guide recommender systems for children? In addition, how muchshould the various stakeholders know and learn from others? Should practitioners
become researchers, and researchers practitioners, in order to effectively address theopen questions relating to recommender systems for children?
• Shared Terminology and Resources. Given the diverse composition of researchers,
developers, and practitioners in this area, there was a shared understanding on the need
to better match or map out the terminology utilized across the different disciplines.
Open questions include how to match key terminology among the various disciplines andhow to define and present concepts in a cohesive way that allows mutual understanding

ACM SIGIR Forum

142

Vol. 52 No. 2 December 2018

amongst the various stakeholders?
In addition to common terminology, there is a need for shared data resources. The
paucity of appropriate datasets for children was also noted in our previous KidRec
workshop. With regards to this, questions were raised such as: What information do
we share? Where do we share it? Could we setup a task and repository similar to
the TREC data [9] with information regarding children’s recommendations? As noted
above, could this repository allow for submissions of raw data that is structured ina uniform way that includes data such as children’s book reports, children’s usage or
behavioral logs, and beyond?
• Ethics and Regulations. There are legal requirements in various parts of the world
regarding what kind of data can be stored about people, and children in general (e.g.
GDPR & COPPA). Beyond these legal requirements, what are the ethical requirements
the various stakeholders should consider with regards to recommender systems for children? Which stakeholders have what responsibility? Who is responsible for the ethicaldesign and implementation of using these systems? And who has access to (and withwhat granularity) to the data stored about each user? Should there be “heavy-handed”
top-down governmental regulation? Alternately, what are other approaches that canbe applied to ensure privacy and the ethical treatment of children and data?
• Interaction Styles. With the dramatic growth in voice-based interfaces being more
readily available on mobile devices and in homes (e.g. Siri, Alexa, Google Home), theeffectiveness of different modalities (i.e., visual and auditory) need to be considered.
What are the relative benefits and drawbacks of various modalities when applied torecommendation technologies? How should these modalities be appropriately mixed tobest address the needs of children?

3

Next Steps & Future Workshop

The workshop effectively brought together stakeholders from various perspectives. It fostered a rich discussion that concluded in immediate next steps for our community. Oneof these is to begin work on a white paper describing a road map to a TREC-type taskthat elicits contributions from children, schools, and practitioners around the world. Doing so will require clearly articulating topics, arguments, needs, players, relevance scores,
and proposed assessment that can define such a task. Also, there was discussion about the
UN Article 17 regarding the Rights of a Child and how we could work to draft an article
that also addresses children’s rights pertaining to data collected about them with regards torecommender systems (and beyond).
As a first practical step to inform the wider public about the ethical issues in children’s recommender systems, the group is working on a brief entitled “Children and Search/Recommendation
Algorithms: what adults need to know”, which at explains the key benefits and limitations ofchildren’s recommender systems to parents, teachers and designers of technology for children.
We plan to continue the multiple-stakeholder conversations at the next ACM IDC conference, which will take place in Boise, Idaho, in June 2019. In the meantime, additional
information pertaining to KidRec 2018, including a list of accepted papers and the detailed
workshop schedule, can be found at the workshop website: https://kidrec.github.io/;and you can join the conversion by emailing kidrec-group@boisestate.edu.

ACM SIGIR Forum

143

Vol. 52 No. 2 December 2018

4

Acknowledgments

We would like to thank the KidRec 2018 Program Committee members for their prompt
and insightful reviews, which directly impacted the level of the workshop discussions. We
also thank the Organizing Committee for IDC 2018–especially the Workshop Chairs Maarten
Van Mechelen (TU Delft, Netherlands) and Kshitij Sharma (EPFL/UNIL, Switzerland)–forgiving us the opportunity to host this workshop in conjunction with the main IDC 2018conference. We also thank the workshop presenters and attendees, who greatly contributed
to discussion and helped build a foundation for future areas of work and topics of interests
upon which this emerging community can continue to grow.

References[1] Y. Du. Insights from pediatric speech language pathologists: Considering instructional
goals in app recommendations for children with disabilities. In 2nd KidRec Workshop
co-located with ACM IDC 2018. Available at: https://drive.google.com/file/d/
1ni90vrLzd99e7rHZNB4BsT3ULG2Uhv9h/view, 2018.
[2] J. Fails. The ethics of recommending based on data collected from children. In 2nd KidRec
Workshop co-located with ACM IDC 2018. Available at: https://drive.google.com/
file/d/1ai2S3KqmKAt61UjuK2Jp75AnQ7A9TIqV/view, 2018.
[3] J. A. Fails, M. S. Pera, N. Kucirkova, and F. Garzotto. International and interdisciplinary
perspectives on children & recommender systems (kidrec). In Proceedings of the 17th
ACM Conference on Interaction Design and Children, IDC 2018, Trondheim, Norway,
June 19-22, 2018, pages 705–712, 2018.
[4] T. Horiuchi, M. Rothschild, R. Barrera, and S. Gururajan. Designing a personally
meaningful abcmouse.com: Challenges and questions in an edtech recommendation system. In 2nd KidRec Workshop co-located with ACM IDC 2018. Available at: https://drive.google.com/file/d/1PpF0vAi4Osr6HCTDVTFc4on_h445so2d/view, 2018.
[5] N. Kucirkova. Alternative approach to recommender systems for children: Positioning the child at the start, not the end of the chain. In 2nd KidRec Workshop
co-located with ACM IDC 2018. Available at: https://drive.google.com/file/d/
1OdW2ojdr1nlmOg7b9x-wtBnKoFVm-_YU/view, 2018.
[6] M. Landoni, E. Murgia, F. Gramuglio, and G. Manfredi.
Teaching an alien:Children recommending what and how to learn. In 2nd KidRec Workshop colocated with ACM IDC 2018. Available at: https://drive.google.com/file/d/
1A-grfEO1psjVeEiYsmSTyx7Y6J3ZDkM3/view, 2018.
[7] M. S. Pera, J. A. Fails, M. Gelsomini, and F. Garzotto. Building community: Report on
kidrec workshop on children and recommender systems at recsys 2017. SIGIR Forum,
52(1):153–161, 2018.
[8] M. S. Pera, K. Wright, and M. D. Ekstrand. Teaching an alien: Children recommending
what and how to learn. In 2nd KidRec Workshop co-located with ACM IDC 2018. Available at: https://drive.google.com/file/d/1i6N7_waF0a9CM5e-aUIAdfgvxZgU55VD/
view, 2018.
[9] TREC. Text Retrieval Conference Website, 2018. https://trec.nist.gov/data.html.

ACM SIGIR Forum

144

Vol. 52 No. 2 December 2018