<figure><img src="image_1.png" /></figure>

HAL
open science

# Digital Peer-Tutoring: Early Results from a Field Evaluation of a UX at Work Learning Format in SMEs

Torkil Clemmensen, Jacob Nørbjerg

► To cite this version:

Torkil Clemmensen, Jacob Nørbjerg. Digital Peer-Tutoring: Early Results from a Field Evaluation of a UX at Work Learning Format in SMEs. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.52-58, 10.1007/978-3-030-46540-7_6. hal-03188824

HAL Id: hal-03188824

https://inria.hal.science/hal-03188824

Submitted on 2 Apr 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Digital Peer-Tutoring: Early results from a field evaluation of a UX at work learning format in SMEs

Torkil Clemmensen[0000-0002-0934-2336] and Jacob Nørbjerg[0000-0002-9557-276X]

Copenhagen Business School

tc.digi@cbs.dk, jno.digi@cbs.dk

**Abstract.**Digital Peer-Tutoring is a new learning format that enables production workers in Small to Medium Sized Enterprises (SMEs) to co-design their interaction with assistive technologies such as collaborative robots. The video-based learning format is based on design thinking and helps shop floor workers create and document solutions to robot interaction problems, and share their how-to knowledge with their colleagues. Early field evaluation results indicated that workers benefit from the Digital Peer-tutoring learning format and produced how-to videos for their colleagues. Furthermore, the Digital Peer-tutoring learning format was also found useful by the company management and ownership as means of documentation and customer communication. Thus, the learning format can also support SMEs on their path to digitalization

**Keywords:**collaborative robots, assistive technologies, UX at work

## 1 Introduction

In this paper we propose a new learning format 'Digital Peer-tutoring' as a means to design and share solutions to worker-technology interaction problems in small to medium sized enterprises¹ (SMEs).

Peer tutoring has long been suggested as a way to help students deal with design problems [13]. Design, understood here as design thinking [4], is an iterative process consisting of generative and evaluative stages, which eventually converge on a solution to the design problem. Design thinking is typically applied to solve non-routine, wicked problems in an organization, when there is a need for novel how-to knowledge. To engage in creation and sharing of new how-to knowledge requires hands-on experience, which is where peer tutoring becomes very helpful. The new learning format Digital Peer-Tutoring aims to help workers interacting with collaborative robots on the shop floor use digital media to engage in teaching and learning with colleagues about their user experiences. We ask the questions: *Can a Digital Peer-Tutoring learning format enable shop floor workers to design positive UXs for them*

¹ The category of micro, small and medium-sized enterprises (SMEs) consists of enterprises which employ fewer than 250 persons with an annual turnover not exceeding 50 million euro, and/or an annual balance sheet total not exceeding 43 million euro.
selves and their colleagues? What kind of ethical stance is implied by the use of Digital Peer-Tutoring?

The paper reports from the initial part of a research project aiming to develop a Digital Peer-Tutoring learning format for shop floor workers in SMEs. The project aims to develop capabilities among shop floor workers to design and document, with short videos, solutions to operational and collaboration issues related to collaborative robots.

The research is situated within the KomDigital regional development project that brings together 18 of the Copenhagen Capital Region's companies, unions, employer associations, and educational institutions. The project aims to improve digital competencies in a broad sense among the employers and employees in SMEs thereby enabling the companies to adopt and implement digital technologies. The target companies come from all sectors, including construction and building, small scale production, product development, and finance, and the technologies include data mining and analysis, collaborative robots and other forms of production automation, AI based financial advice, and more.

KomDigital achieves its goals through new digital learning formats, which can be made available to target companies and organizations. The formats are tailored to the working conditions and needs of companies and employees, so that both employees, managers, companies and organizations can use new digital technologies to expand and grow.

# 2 Related work

## 2.1 Digitalization in SMEs

SMEs depend on their workers' knowledge and innovative capabilities to create new ways of working with technology, and they generally lack the capability and capacity for comprehensive digital transformation [6,9]. Collaborative robots that work alongside a human worker can be integrated into the production without radical reconfiguration or automation of established workflows. A human worker can program a collaborative robot to perform tasks such as lift, pick and place, move, or otherwise process physical objects [5,12,14]. Thus, worker designed interaction with collaborative robots and other assistive technologies is a useful first step towards digitalization in an SME.

## 2.2 Peer tutoring

Peer tutoring [7,13] overlaps somewhat with other notions of providing informal technical help between colleagues, such as over-the-shoulder-learning [17], over-the-shoulder-guidance in tertiary education [2], peer-assisted learning [8] and peer teaching [15] in the medical domain, and over-the-shoulder appropriation [1] and peer interaction [10] in software development.

In this paper we build primarily on the approach put forward by Twidale [17] in that we aim to support the provision of informal technical help between colleagues.
Similar to Schleyer et al.[13] we acknowledge the role of peer tutors at various levels
towards developing problem solving skills among colleagues. Specifically, we intro-
duce a new role of digital competence facilitator, a ‘Digital Coach’, as explained be-
low.

## 2.3 Digital Peer-Tutoring

What distinguishes ‘digital peer-tutoring’ from traditional peer-tutoring is that the concept builds entirely on the use of video. The idea is that workers learn from creating and redesigning videos while sketching [11] as part of applying design thinking to design their own and their colleagues’ work flow and interactions with collaborative robots. Ørgreen et al. [11] suggested to link sketching techniques and creativereflection processes to video productions, and we extent this proposal to cover linking all parts of design thinking (problem definition and user needs finding, sketching, prototyping hypotheses, evaluation) to workers’ video production. Secondly, we propose that video-based reasoning, instead of paper or verbal exchange, empower workers to explore and take ownership of their work. Vistisen et al.[18] proposed to support ethical userstances during the design process of products and services, andproposes using animation-based sketching as a design method. We follow that line of thought, though we are less interested in professional designers, and more interested in workers’ own production (and consumption) of videos-as-digital-peer-tutoring.

# 3 Case setting: A collaborative robot in specialized glass manufacturing

After learning from initial talks with three different SMEs in Denmark, we agreed with the ABC company to adapt and evaluate the digital peer-tutoring learning format in one of their production facilities. The ABC company is a European SME specializing in glass processing. The company produces individual pieces and small batches with special specifications as well as entire series of several thousand units.

About a year prior to our visit, the ABC company purchased and installed a 100,000€ collaborative robot in order to explore if and how it could be used in their production. At the time of our visit, the robot was used only during the final polishing steps of one large scale order, and it was idle much of the time. Workers and management agreed, however, that the robot could be used for other purposes as well, and thus enable the company to accept more large batch orders, but no initiatives had been implemented for several months due to lack of time to experiment with the robot. Furthermore, the initial design decision had been a stationary installation, that is, the robot could not be moved to other positions on the floor where it could interact with other machines or workers.

The initial design decisions seemed to be related to a limited initial understanding of the robot's capability and a lack of strategic intent. In any case, it was clear that there was an unexplored potential (and risks) for enhancing the factory's capacity
while empowering workers and help them design their own user experiences with the robot.

## 4 Method: Action design research with SMEs

Our approach to building new digital competences in SMEs is inspired by action design research (ADR). ADR argues that IT artifacts are ‘ensembles’ formed by the organizational context during development and use. Research in this tradition interweaves constructing the IT artifact, intervention in the organization, and evaluating outcomes [16].

We visited the company 6 times over a six-week period during the spring 2019. The purpose of our first visit was to develop insights into the company, the motivation for purchasing the robot, and challenges with its current as well as potential future use of the robot. We observed the robot’s current (very limited) use, interviewed and discussed with robot vendors, managers and shop-floor workers, and observed work and demonstrations of the robot.

The digital peer-tutoring learning format (see section 5) was implemented in four sessions over the next four visits, followed by a final evaluation on the sixth visit. We documented all observations, interviews, and learning sessions with video and audio recordings and photos, and we collected the videos produced by the workers.

The learning format was evaluated after each session and at a final one-day meeting with participation from all key stakeholders.

## 5 The digital peer-tutoring learning format

<table><caption>**Table 1.** Overview of training sessions.</caption><thead><tr><td>Sessions</td><td>Themes</td><td>Topics</td><td>Worker-created-how-to-videos</td></tr></thead><tbody><tr><td>1</td><td>The problem</td><td>Personas<br>Interaction<br>Collaboration with tech.</td><td>1. A persona<br>2. An interaction problem<br>3. A collaboration problem</td></tr><tr><td>2</td><td>Solution sketch</td><td>How to sketch a solution<br>Interaction<br>Collaboration</td><td>Three design ideas for<br>4. Interaction<br>5. Collaboration</td></tr><tr><td>3</td><td>Design prototype</td><td>Interaction and collaboration prototypes</td><td>6. Elaborate one design idea into a prototype</td></tr><tr><td>4</td><td>Evaluate prototype</td><td>How to evaluate/test prototypes</td><td>7. Test the prototype with a colleague</td></tr></tbody></table>

The digital peer-tutoring learning format consisted of an ensemble of instruction-videos, quizzes, example-solution-video, and worker-created-how-to-videos. Together with the case company production site, we designed and implemented four training sessions with selected shop-floor workers (Table 1).
We developed short (3-5 minutes) instruction videos for each session that explained the theme, introduced techniques that the participants could use to investigate problems and describe solutions, and concluded with an exercise where the participants should develop a short video (3-5 minutes). We also produced short example videos with our 'answers' to the video assignment for each session.

All video material – including instruction material was recorded with standard smartphone hardware and software, and published without editing, in order to promote a 'simple-yet'sufficient' attitude towards to video production.

For each session, a ‘digital competence facilitator’ (student assistant) travelled to the factory and discussed the material with the participants, and helped them produce their own ‘employee-videos’, which were subsequently uploaded to a shared (secure) site for later download and knowledge sharing within the company.

# 6 Field evaluation results

The evaluation of the ‘Digital Peer-Tutoring’ learning format consisted of weekly evaluations after each of the four sessions, and a final evaluation with participation from all key stakeholders. Here, we report about the initial results from the final evaluation; a one-day meeting in the location of the factory of the case company. The participants in the evaluation were all those present at the upstart meeting 6 weeks before. They were: company managers (Company manager J and Company manager K), learning format users (Worker Br, Worker H, Worker Bi), corporate learning consultant (corporate learning consultant F), educational institution teacher(s) (Teacher J, Teacher T), pilot project manager(s) (Teacher T, Teacher J), pilot project documentarist (Documentarist F), and a digital competence facilitator (Digital competence facilitator S).

The initial results from the final evaluation reveal both short- and long-term benefits and challenges of Digital Peer-Tutoring.

## 6.1 Short term benefits

The workers liked the learning format and found it useful: "...worker-video on iPad [could beuseful]...", [Worker Br]. This confirms previous findings on the usefulness of video [11], and extends it to the shop floor workers.

However, the workers found that the instruction videos were too long and complicated. "[They should be cut down to a list of four points]" [Worker Br]. Too long videos can be an expression of an 'apathetic ethical stance', a stance that reduces the worker-user to be a mean of input for the intended final design[18].

On the other hand, the workers expressed that they could use video to both think about a problem, sketch different solutions, and evaluate their use: "Sketches ... I had read up on it, go and think about it...." [Worker Br], and "the workers should be able to pause the video ..." [Worker Bi]. Thus, there were indications that the format helped workers explore new technologies from an emphatic ethical user – that is, from their own – perspective [18]. Company manager K supported this: "We, as a business must
spend more time on [workers' use of video to innovate]." The management perspective adds a new layer to understand short term benefits of video-sketching and ethical design, and thus center our focus on the multi-layered essence of user experiences at work.

## 6.2 Long-term benefits

The stakeholders also commented on the long-term benefits of the learning format:

1. The format could be used to tackle issues in the manufacturing, as "help videos" [Worker Bi], and a "'Company database of videos that could be accessed even years after production'" [Company manager J],

2. New employees could be introduced to the job: "A new one that is totally novice [could use worker-created-how-to-videos]" [Company manager J],

3. Help dyslexic employees who could watch how to do things, rather than read,

4. Supplier courses could be made memorable by "record[ing] what the supplier shows on the shop floor" [Corporate learning consultant F], and "Cut out what is not useful [from the supplier teaching]" [Company manager K]

5. Starting new ways to produce, for example "recording the results from the compa-ny's informal and formal experiments on the shop floor" [Company manager J, Company manager K], and "recording order-specific ideas for how-to, so next time this order comes in, the video shows what to do" [Worker Bi], and "The video can be used to "squeeze" a good idea out of an experienced employee who will have to think a little about the idea" [Teacher T].

6. Finally, the stakeholder group discussed that the learning format could also be used to produce videos for customers for marketing purposes and quality documen-tation.

These benefits allude to a diversity of user experiences in work situations, and per-haps tell us that the ethical stances taken by workers-as-designers-of-their-own-work may be confounded by management's strategic interest in how-to knowledge.

## 7 Discussion and conclusion

We conclude that our proposed Digital Peer-Tutoring learning format enabled shop floor workers to design positive UXs for themselves and their colleagues, and beyond ways that we expected. The participating shop floor workers stated in various ways that they liked the Digital Peer-tutoring how-to videos and found them useful. This corresponds to the claim made by Twidale (2005)[17] that it is possible to use peer tutoring to give informal technical help between colleagues, and with Ørngreen et al. (2017) [11] who suggest to link various sketching techniques and creative reflection processes to video productions. The videos helped workers create ideas about robot use, identify problems not formulated before, sketch alternatives, test solutions, and demonstrate them to colleagues.
Company owners, management, and workers had unexpected ideas about how to use the peer-tutoring videos within and outside the company, in for example internal quality control and customer communication. Thus, similar to the point made about peer tutoring [13], we should acknowledge the role of Digital Peer-Tutoring in developing problem solving skills at various organizational levels.

Based on the categories proposed in [18], we furthermore observe that the ethical stance built into the ‘Digital Peer-Tutoring’ learning format can be characterized as ‘apathetic’, when too long and complex instructional videos, intended to teach workers’ design thinking and enabling their own video-production, tend to make workers give up. However, the learning format also showed to be ‘empathetic’, as workers produced their own videos and evaluated solutions together, effectively co-designing work procedures.

We developed the Digital Peer-Tutoring learning format to improve workers' capability to create and share solutions to human-robot collaboration challenges in SMEs. Thereby we also answer the call for research into how SMEs can adopt and implement new technologies that build upon and enhance worker capabilities, skills, and knowledge[3,6].

## 8 References

[1] Sebastian Draxler and Gunnar Stevens. 2011. Supporting the collaborative appropriation of an open software ecosystem. *Comput. Support. Coop. Work* 20, 4–5 (2011), 403–448.

[2] Angela C Hague and Ian D Benest. 1996. Towards over-the-shoulder guidance following a traditional learning metaphor. *Comput. Educ.* 26, 1–3 (1996), 61–70.

[3] Lea Hannola, Alexander Richter, Shahper Richter, and Alexander Stocker. 2018. Empowering production workers with digitally facilitated knowledge processes—a conceptual framework. *Int. J. Prod. Res.* (2018). DOI:https://doi.org/10.1080/00207543.2018.1445877

[4] Jon Kolko. 2010. Abductive Thinking and Sensemaking: The Drivers of Design Synthesis. *Des. Issues* 26, 1 (January 2010), 15–28. DOI:https://doi.org/10.1162/desi.2010.26.1.15

[5] Danica Kragic, Joakim Gustafson, Hakan Karaoguz, Patric Jensfelt, and Robert Krug. 2017. *Interactive, Collaborative Robots: Challenges and Opportunities*. Retrieved May 19, 2019 from https://www.ijcai.org/proceedings/2018/0003.pdf

[6] Thomas Ludwig, Christoph Kotthaus, Martin Stein, Volkmar Pipek, and Volker Wulf. 2018. Revive Old Discussions! Socio-technical Challenges for Small and Medium Enterprises within Industry 4.0. In *Proceedings of 16th European Conference on Computer-Supported Cooperative Work*. DOI:https://doi.org/10.18420/ecscw2018_15

[7] D J Magin and A E Churches. 1995. Peer tutoring in engineering design: A case study. *Stud. High. Educ.* 20, 1 (1995), 73–85.

[8] Johanna Martinez, Christina Harris, Cathy Jalali, Judy Tung, and Robert Meyer. 2015. Using peer-assisted learning to teach and evaluate residents’ musculoskeletal skills. *Med. Educ. Online* 20, 1 (2015), 27255.
[9] Henrik Mucha, Sebastian Büttner, and Carsten Röcker. 2016. Application Areas for Human-Centered Assistive Systems. In *Human-Computer Interaction–Perspectives on Industry 4.0. Workshop at i-KNOW 2016 Graz, Austria, Oct 2016*.

[10] Emerson Murphy-Hill, Gail C Murphy, and Joanna McGrenere. 2015. How Do Users Discover New Tools in Software Development and Beyond? *Comput. Support. Coop. Work* 24, 5 (2015), 389–422.

[11] Rikke Ørngreen, Birgitte Henningsen, Peter Gundersen, and Heidi Hautopp. 2017. The Learning Potential of Video Sketching. In *Proceedings of the 16th European Conference on elearning ISCAP Porto, Portugal 26-27 October 2017, 422–430.*

[12] Natalia Sanchez-Tamayo and Juan P Wachs. 2018. Collaborative Robots in Surgical Research: a Low-Cost Adaptation. In *Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction*, 231–232.

[13] G K Schleyer, G S Langdon, and S James. 2005. Peer tutoring in conceptual design. *Eur. J. Eng. Educ.* 30, 2 (2005), 245–254.

[14] Ruth Schulz. 2017. Collaborative robots learning spatial language for picking and placing objects on a table. In *Proceedings of the 5th International Conference on Human Agent Interaction*, 329–333.

[15] Jacinta Secomb. 2008. A systematic review of peer teaching and learning in clinical education. *J. Clin. Nurs.* 17, 6 (2008), 703–716.

[16] Sein, Henfridsson, Purao, Rossi, and Lindgren. 2011. Action Design Research. *MIS Q.* (2011). DOI:https://doi.org/10.2307/23043488

[17] Michael B Twidale. 2005. Over the shoulder learning: supporting brief informal learning. *Comput. Support. Coop. Work* 14, 6 (2005), 505–547.

[18] Peter Vistisen, Thessa Jensen, and Søren Bolvig Poulsen. 2016. Animating the ethical demand: Exploring user dispositions in industry innovation cases through animation-based sketching. *ACM SIGCAS Comput. Soc.* 45, 3 (2016), 318–325.