2021 10th International Congress on Advanced Applied Informatics (IIAI-AAI)

Developing a Lecture Video Recording System
Using Augmented Reality
Yuma Ito, Masato Kikuchi, Tadachika Ozono, Toramatsu Shintani
Department of Computer Science, Graduate School of Engineering
Nagoya Institute of Technology
Gokiso-cho, Showa-ku, Nagoya, Aichi, 466-8555 Japan
E-mail: {higashi,kikuchi,ozono,tora}@ozlab.org
Abstract—Assistive technology is a prerequisite for making ahigh-quality lecture video. It is therefore imperative to edit thelecture video after recording. In this study, we aim to reducethe cumbersome task of lecture video editing by developinga system that enables the addition of visual effects in thevideo while recording. In particular, we use augmented reality(AR) technology to digitize and display in real-time lecturematerials, assistant agents, and other recording contents usedby the lecturer. Our system realizes such a mechanism as alecture recording environment. In addition, our system basedon AR technology can support the work of the lecturer, which
is difﬁcult to do by oneself while conducting the lecture, using
the information of the lecturer’s position and the progress of thelecture. We evaluated the system functionality and performance,and veriﬁed the system’s correct behavior. If the burden of
making lecture videos can be reduced, the lecturer will be able
to devote more time to improving the quality of lecture contents,
which is expected to contribute to the improvement of lectures.
Index Terms—Augmented Reality, Semi-autonomous Agent,
Making Video Effect, Lecture Video Making Support

reality (AR) space. Our system provides the lecturer such a
lecture video recording environment in the AR space. Our
system using digital lecture contents can adapt to various
lecture formats envisaged by lecturers, even when recording a
lecture without the need for video editing. Another advantage
of using AR technology, which is a feature of our system,enables our system to handle three-dimensional information
based on depth data. For instance, it includes the plane
information and position of the lecturer in the lecture space.
Our system uses this information to conﬁgure the displayof AR Slide, to control lecture assistant agent, and system
functions for lecture support. Especially in designing sucha system, we need to reduce unwanted editing work afterrecording due to system errors or delays.
The rest of the paper is organized as follows. Section II
presents related works. Section III describes how to support thecreation of lecture videos using AR technology. Sections IV
and V describe the implementation and experiments. Finally,
we discuss our system considering the results in Section VI
and conclude this paper in Section VII.

I. I NTRODUCTION
In some cases, lecture videos are used as pre-distributed
materials in on-demand classes and ﬂipped classrooms [1].
Lecturers produce lecture videos in various ways. The lecture
video that this study aims to produce shows the lecturer’s
facial expressions and motions, is easy to watch, and has visual
effects to a certain extent. Lecture videos are preferred to be
in a format that shows the lecturer [2].
Making lecture videos consists of recording and editing.
After the recording, lecturers usually edit videos to add richeffects, however, editing is time-consuming. We aim to realizea system that can automatically add appropriate effects to avideo while recording it. This means that lecturers need notedit after recording to create high-quality lecture videos. To
reduce editing work after recording, this system must operate
exactly as intended by the user while recording a long lecture
video.
Our approach provides support for video making duringlecture recording, thereby reducing the burden after recording.
Speciﬁcally, our system uses augmented reality (AR) technology to create a lecture video with added visual effects
in real-time in accordance with the progress of the lecture.
Our system digitizes and displays lecture materials, lectureassistant agents, and other lecture recording contents. The
lecturer uses these tools to conduct lectures in augmented

978-1-6654-2420-2/21/$31.00 ©2021 IEEE
DOI 10.1109/IIAI-AAI53430.2021.00011

II. R ELATED W ORK
A. Video Effect Production Support
Some researchers have focused on adding visual effects
to lecture and presentation videos using AR and 2DCG
technology to have a high effect in terms of expression andentertainment. In particular, the addition of visual effects using
AR technology is superior in that it enables three-dimensional
visual expression and reduces complicated video editing work.
In a previous study, we developed a video editing system
for ﬂipped classrooms using AR technologies [3]. We designed
the agents playing the role of a virtual student in AR space,called AR puppets, which interact with the lecturer and help
students learn. We used AR puppets and created lecture videos
together to add visual effects to the lecture videos. Saquib et al.
[4] developed a system that displays materials such as ﬁgures
and graphs in videos during a live presentation, with which thepresenter can interact in real time with gestures and postures.
Relevant to our system is to display the digital materials andthe presenter as if they were in real space. It allows theaudience to watch it as a single video content without having
to pay attention to both materials and the presenter.

65

(a) one AR Slide(b) real lecture material (whiteboard)(c) multi AR Slides

Fig. 1. Example of displayed lecture material by our system.

III. L ECTURE V IDEO M AKING S UPPORT USING AR
Many lecturers create lecture videos using standard presentation tools such as Microsoft PowerPointi and Keynoteii .
These tools make it easy to create a narrated presentationor presentation with an overlaid lecturer’s capture. Our study
focuses on features of AR technology, which realize added
graphical expressions of presentation using three-dimensional
spatial information to video during lectures without any editing
work. Our system uses digital lecture contents for lectures,such as lecture materials and agents. This can ﬂexibly adapt
to various lecture formats by manipulating and adjusting thelecture recording contents (Fig. 1).

TECHNOLOGY

Our system realizes a collaborative presentation by a lecturer and lecture assistant agent in the AR space. We assume
that the lecturers record lecture videos by themselves, and thelength of typical videos is longer than 30 minutes. This section
describes in detail how we support the creation of lecture
videos using AR technology.
A. Design Lecture Material Object
Lecturer can use two types of lecture materials in oursystem: AR Slide and real lecture materials. AR Slide is alecture material displayed as a digital object in the AR space
using AR technology. AR Slide can be used in image formats
such as PowerPoint, textbooks, and reference images, as wellas in video format, taking into account the lecture format
assumed by each lecturer. The real lecture materials in thispaper refer to the materials that the lecturer writes directly onreal objects such as blackboards and whiteboards. The lecturer
usually gives a lecture using this AR Slide.
By using AR Slide, there is no need for a monitor or
projector to display the lecture materials, nor is there a needto edit the video later, and it is possible to display thematerials following the progress of the lecture. In addition,
it can be displayed clearly. A digitally displayed AR slide
can be displayed stably without being affected by the actual
recording environment. Moreover, AR technology has people
occlusion, which grasps the front-back relationship between
AR objects and people and renders AR contents appropriately.
This technology makes it easy to realize representations thatlook like chroma key compositing of AR Slide and the lecturer.
The type of lecture videos that different lecturers want tomake varies. This may also vary depending on the ﬁeld andcontent of the lecture. It would be a burden for the lecturer
to prepare for the lecture video that the lecturer assumes inadvance or in the middle of the recording. Our system enables
the lecturer to freely manipulate and adjust the AR Slide, andcorresponds to various lecture formats that change the display
style of AR Slide. The layout of the materials is limited with amonitor or projector screen. AR technology makes it possible

B. Human-like Virtual Agent
Human-like virtual agents, which have a form and communication similar to humans, have been researched and used invarious situations, as language instructors [5] and a museum
guide [6]. Particularly in a presentation situation, the presenter
makes a collaborative presentation with a virtual agent [7], andprojects an avatar that reﬂects the presenter’s behavior on theslides [8]. Mumm et al. [9] investigated how the presence of
virtual agents can affect the motivation of participants toward
given tasks. It is inferred that abstracted humanoid agents
may have a positive effect on human motivation. We notably
focus on visual effects of virtual agents without the need for
manpower.
In this study, we use a human-like virtual agent as thelecture assistant agent, who provides the graphical effect asvideo content and lecture assistance for making the lecture
videos more comprehensible. In a previous study, we exploredways to control a virtual agent in an AR space [10]. We
considered whether the performer could control the virtual
agent himself or herself without disturbing the progress of
performance. We extend this work further and design threedimensional control of lecture assistant agent. Our system
controls lecture assistant agent semi-autonomously using theinformation in the lecture space, such as the lecturer’s position,lecture materials, and lecture progress.
i https://www.microsoft.com/en-us/microsoft-365/powerpoint
ii https://www.apple.com/jp/keynote/66to place the lecture materials anywhere in AR space. The
lecturer can freely adjust the size and aspect ratio of AR Slide.
There are mainly three display styles of AR Slide. Fig. 1
denotes the actual display styles of AR Slide by our system.
(a) is the most basic display style, showing a single slide. (b)is the case when the lecturer uses real lecture material suchas a blackboard and whiteboard. The AR slide is moved tothe upper left corner, and the lecturer gives a lecture using
a real lecture material. (c) is a style in which multiple slides
are displayed in the video. Fig. 1c shows the main slide andprevious and next slides are displayed. In another case, thelecturer can proceed with the lecture displaying the left slide
ﬁxedly on a particular slide (this function is called a Pin
Function). The lecturer can switch these styles appropriately
during recording, allowing the lecturer to ﬂexibly change
depending on the content of the lecture.

Fig. 2. Visual Effects by lecture assistant agent. The agent pop-up AR objects
in a text format chronologically above its head.

B. Lecture Assistant AgentOur study realizes an agent provides supplementary explanations for lectures. Lecture assistant agent plays two roles.
One is to provide a visual effect to relieve the monotony ofthe lecture video. The other is the role of assisting the lecture.
We are familiar with the format in news and commercials,
where the main speaker and assistant speaker are separated
and give a presentation together. Because the lecturer records
a lecture video by himself or herself, the lecture assistant agent
should be controlled semi-autonomously. Proceeding with thelecture itself becomes a burden to the lecturer. Therefore, itis necessary to avoid that the control of the lecture assistantagent becomes obstructive to the progress of the lecture. In
our system, lecture assistant agent semi-autonomously controls
its movements using comprehensive information about lecture
progress, lecture materials, the position of the lecturer andpointer, and so on.
We assumed that there would be two types of supplementary explanations needed in a lecture: explanations about thelecture content and about the lecture progress. Supplementary
explanations about the lecture content require appropriate
content and timing for the progress of the lecture. This isbecause meaningless explanations can distract students fromunderstanding lecture video content. Because it is difﬁcult for
the agent to provide such explanations with semi-autonomous
control, this paper mainly focuses on supplementary explanations about the lecture progress. Fig. 2 denotes lecture assistantagent that prompts the viewer to pay attention when the lecture
progress transitions, such as switching slides or using a pointer.

Fig. 3. Recording Content Switching Function. It is automatically controlledcamera using the lecturer’s position information. Above: Close-up display oflecture materials. Below: Close-up display of the lecturer.

the camera, which requires human work, and to make it asautomatic as possible. An example of the recording contents
switching function of this system is shown in Fig. 3. Thisfunction provides a close-up view of the lecture materials
when the lecturer exits the camera screen (The above Fig.
3). In addition, our system controls the camera based on thelecturer’s position and displays a close-up of the lecturer (Fig.
3 below).
Our system creates lecture videos with added visual effects
in real time to reduce the burden of video editing. However,editing video after recording is necessary to some extent, suchas cut editing for retake parts. Therefore, we propose a system
to record the retake point of the video using the information of
the lecture’s progress and to assist in editing the video after

C. Functions using Lecturer’s Recording Information
Our system handles two types of information as Lecturer’s
Recording Information: the lecturer’s position and the progressof the lecture. On TV news, we can see that the cameracrews switch between the material, the newscaster, and theguest according to the situation. It is possible that adjusting
recording contents according to the progress of the lecture
will make it easier for viewers to see and understand thecontents. However, it is desirable to avoid direct control of67

Fig. 4. Video cut editing assistance function after lecture recording. Two
methods to save the retake point are used. Left: Insert a visual effect into thevideo. Right: Save as a text log along with the operation record.

Fig. 6. Adjusting function of AR Slide.

environment. In second step, each device is assigned to arecording device or a remote operation device.
The lecturer prepares for the lecture recording in third step.
Speciﬁcally, the lecturer sets up lecturer’s position tracking and
AR Slide. The lecturer always holds a remote operation device
while recording because of the speciﬁcations of this system.
Therefore, our system estimates the lecturer’s position basedon the position coordinates of the remote operation device.
Therefore, it is necessary to acquire the position coordinatesof the remote operation device in the AR space created by therecording device. This mechanism is implemented by a remote
operation device sharing its coordinates with a recording
device through self-localization estimation in AR technology.
However, the coordinate information provided by the selflocalization estimation is expressed in relative coordinatesfrom the origin of the AR space constructed by oneself. Hence,
it is essential to match the origin and axis (direction) of the
AR space that each device constructs. Our system uses theoverall spatial mapping information, which is referred to as
the AR map, and shares it among other devices to align theorigin and axis of the AR space. Subsequently, it estimates
the position of the remote operation device and lecturer by
self-localization estimation.
When setting up the AR slide, the system displays thelecture materials prepared in advance as AR Slide. In addition,
the lecturer can adjust the position and size of the AR slide(Fig. 6). Our system uses the plane detection of AR technology
to render the lecture materials as if they were pasted on a
vertical surface in the recording space.
After ﬁnishing preparation for recording, the lecturer cangive a lecture by manipulating the AR slide. The lecturer
will use the system functions according to the content andprogress of the lecture. In the previous section, we introduced
some system functions. The other function of our system is todisplay a pointer on the AR slide (Fig. 7).

Fig. 5. Hardware setup.

recording. Fig. 4 denotes two recording methods: one is torecord by inserting a visual effect into the video, and the otheris to record by saving as text log along with the operation log.
IV. L ECTURE V IDEO R ECORDING S YSTEM
Our system, developed as iOS application with Unity, uses
two iOS devices. Fig. 5 denotes the hardware conﬁgurationof our system. These devices are called recording device
and remote operation device, respectively. Recording Device
creates lecture videos with added visual effects. Because thelecturer is approximately several meters away from the recording device, it is impossible to directly control the recording
device during the lecture. Therefore, the recording device, thatis, the recording system, was remotely controlled via a remote
operation device. We used iPhone 8 running on iOS14.4 as the
remote operation device and iPad Pro 2nd generation runningon iOS14.4 as the recording device.
A. System Flow
We describe the procedure for making a lecture video with
our system. The procedure consists of four steps mentioned
as follows: 1) launch the application, 2) register the recording
device and remote operation device, 3) prepare for recording:
track lecturer’s position and put AR slide with recording
device, and 4) record a lecture: conduct a lecture with a remote
operation device.
In ﬁrst step, the lecturer starts this application on each of thetwo iOS devices. After some time, the network is automaticallyset up between the two devices to create a communication

B. System Architecture
Our system consists of a recording device, a remote operation device, and a database (Fig. 8). The remote operation
device has a communication control module that is responsible

68The material is saved in the database as image or video format.
The lecturer can freely change and use the materials from a
web browser by referring to them from the database without
the need to import them into the device in advance.
Recording device mainly comprises recording information
management subsystem, lecture assistant agent, and AR slide.
The recording information management subsystem handles
the lecture progress and lecturer’s position information asrecording information. Since the lecturer uses AR Slide andsystem functions to conduct a lecture, the progress of thelecture is directly related to the system’s operation. Therefore,
the system determines the lecture progress based on theoperation information.
Lecture assistant agent is controlled semi-autonomously.
Recording information management subsystem commands
the agent to perform lecture assistance considering lecture
progress. The system controls the gaze and posture of thelecture assistant agent in three dimensions, depending on theposition of the lecturer and AR Slide. AR Slide is displayedby projecting material data onto a ﬂat AR image object. AR
Slide has a layered structure of multiple AR objects. The
system controls the AR Slide, such as page manipulation andchanging display style, by a combination of showing andhiding these objects and adjusting their position and size.
Finally, the system makes AR video as a lecture video in realtime by combining digital data of AR contents and camera
images using AR technology.

Fig. 7. Pointer function. The remote operation device displays a 2D image ofa material synchronized with an AR slide that exists in the AR space of therecording device. A pointer is displayed in relation to the touched position of
the material displayed on the remote operation device.

V. E VALUATIONIn this section, we evaluate the functionality and performance of our system, as a recording system using AR technology. We conducted experiments to demonstrate that our system
can add appropriate visual effects in real time in accordancewith the lecturer’s operation. Speciﬁcally, we measured theiraccuracy and process speed. We evaluated that the system
can operate exactly as the user intended while recording a
lecture video. The system should perform appropriate editing
in real time for the lecturer’s operations. In other words, thesystem can select appropriate commands for the lecturer’s
operations. Therefore, we measured the elapsed time as the
response speed of the system, which is from the issue ofcommand by the remote operation device to the completionof that system function. We selected four functions, which
are required to execute in real time. We measured 100 times
for each function and calculated the average and maximum
values. Table I shows that the accuracy is 100% and responsetime is low. The average and maximum response time for the
Pin Function is 74 ms and 302 ms, respectively, which is theslowest among the measured functions. The results indicatethat our system has sufﬁcient real-time performance for adding
visual effects during lecture recording.
Our system allows the use of the information in therecording space, such as the lecturer’s standing position. As
explained in Section IV, the accuracy of the lecturer’s position
recognized by the system depends on the position tracking

Fig. 8. System Architecture.

for the communication function between devices. The lecturer
remotely controls the recording device via the communication
control module based on the input from the remote operation
device. This module also updates the UI of the remote operation device by receiving the system information from therecording device, such as lecture information and results ofoperation commands, so that the lecturer can directly receive
the system feedback.
The database holds two types of data: AR map and lecturematerial. The system needs to share an AR map with eachdevice to track the lecturer’s position. Speciﬁcally, one device
sends the AR map to the other device via the database. The
lecture materials for AR slide are also stored in the database.

69

accuracy of the remote operation device. Therefore, we measured the accuracy of the position tracking of remote operation
device. We describe the measurement procedure. First, thesystem places a target object randomly in the AR space of therecording device. Second, the user overlaps remote operation
device on the target object. Finally, the system measures the
distance between the position of the target object and thetracked position of the remote operation device by the system.
Because there might be some hand tremors and misrecognition, the coordinates by position tracking are calculated
as a ﬁve-second moving average value. We conducted theexperiments in two cases. The results are shown in Table II.
Case 1 involves measuring in various spaces within a shortperiod after setting up tracking. Case 2 involved setting up
tracking only once and measuring it in the same space for 30min.

We assume that a lecturer records long lecture videos
by using our system, so the system is required to execute
appropriate commands for the lecturer’s operations in realtime in order to reduce time-consuming editing work after
long recording. Because of the speciﬁcation of AR technology,
the extraction of spatial feature points is always executed aslong as the system is running. Therefore, it is assumed that theaccuracy will decrease as the system is working for a long time
because of the speciﬁcation of position tracking. However, we
did not observe any loss in accuracy (Case 2). The variance
was smaller than that observed in Case 1. The results indicatedthat the newly acquired spatial feature points were processed
appropriately without conﬂicting with the old. We conclude
that the tracking implementation of our system is practical for
long-term lecture recording.

TABLE IR ESPONSE TIME AND ACCURACY

We proposed a lecture video recording system providing AR
space for giving lectures including AR Slide, Lecture Assistant
Agent, and so on. Our experimental results demonstrated thatthe system can add visual effects to lecture videos on recording
without unwanted editing work caused by system errors ordelays. Our system can help lecturers create lecture videos
with visual effects to help students. However, we need to
conduct usability tests of our system in the near future.

Page Operation
Display Format Change
Pointer
Pin Function

max
[ms]
189
162
177
302

mean
[ms]
61
53
69
74

VII. C ONCLUSION

accuracy
[%]
100
100
100
100

ACKNOWLEDGMENT

TABLE IIP OSITION ERROR OF RECOGNIZED R EMOTE O PERATION D EVICE

Case 1Case 2

count

min
[cm]

max
[cm]

mean
[cm]

variance

100
54

2.5
3

35.5
35.6

16
12.7

60.7
44.7

This work was supported in part by JSPS KAKENHI Grant
Numbers 19K12097,19K12266.
R EFERENCES
Response[1] H. Okumoto, M. Yoshida, K. Umemura, Y. Ichikawa,Collector: A Video Learning System for Flipped Classrooms 2018
5th International Conference on Advanced Informatics: Concept Theory
and Applications (ICAICTA), pp.176-181, 2018.
[2] P. J. Guo, J. Kim, R. Rubin, How Video Production Affects Student
Engagement: An Empirical Study of MOOC Videos ACM conference
on Learning at Scale, pp.41-50, 2014.
[3] H. Kataoka, T. Ozono, T. Shintani, Realizing a Video Editing System
for Flipped Classrooms with AR Puppets IEICE Technical Report,
Vol.119, No.317, AI2019-32, pp.13-18, 2019.
[4] N. Saquib, R. H. Kazi, L.-Y. Wei, W. Li, Interactive Body-Driven
Graphics for Augmented Video Performance CHI 2019, ACM, No.622,
12p, 2019.
[5] M. Macedonia, I. Groher, F. Roithmayr, Intelligent virtual agents as
language trainers facilitate multilingualism Frontiers in Psychology,
14;5:295, 5p, eCollection 2014.
[6] N. Tcchasarntikul, P. Ratsamee, J. Orlosky, T. Mashita, Y. Uranishi, K.
Kiyokawa, H. Takemura, The Effect of the Presence of an Embodied
Agent in an AR Guiding System Transactions of the Virtual Reality
Society of Japan, Vol.25, No.1, pp.68-77，2020.
[7] H. Trinh, L. Ring, T. Bickmore, DynamicDuo: Co-presenting with
CHI ’15: Proceedings of the 33rd Annual ACM
Virtual Agents
Conference on Human Factors in Computing Systems, pp.1739-1748,
2015.
[8] F. Matulic, L. Engein, C. Träger, R. Dachselt, Embodied Interactions
for Novel Immersive Presentational Experiences Conference: the 2016
CHI Conference Extended Abstracts, pp.1713-1720, 2016.
[9] J. Mumm, B. Mutlu, Designing motivational agents: The role of praise,social comparison, and embodiment in computer feedback! Computersin Human Behavior Volume 27, Issue 5, pp.1643-1650, 2011.
[10] H. Kataoka, T. Ozono, T. Shintani, Developing an AR Pop-up PictureBook and its Effect Editor Based on Teaching Motions Information
Engineering Express, International Institute of Applied Informatics,
vol.7, no.1, pp.1-10, 2021.

VI. D ISCUSSION
Our system provides an environment for recording lecture
videos in AR space. As shown in Table I, to reduce the burdenof video editing, the system adds visual effects to the video
in real time. Our experimental results show that the system
can add appropriate visual effects for the lecturer’s operations;
therefore, the system does not cause unwanted editing work
after recording due to system errors or delays. Our system
can help lecturers create lecture videos because the system
can add visual effects to lecture videos on recording without
time-consuming editing work. Furthermore, the system is also
expected to reduce the additional work, such as retaking andvideo editing after recording due to the addition of visual
effects beyond the lecturer’s expectations.
The feature of our system is the acquisition of spatial
information using AR technology, and the utilization of thesame to create visual effects and functions. The accuracy of
the acquired spatial information affects the functionality ofthe entire system. In Case 1, the average error of the position
tracking of the remote operation device was 16cm. From asystem perspective, tracking accuracy is sufﬁcient, even to acertain degree, rather than a strict accuracy. Therefore, theperformance of the system is practical enough.

70