IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

Developing a Course on Human-Centered Machine Learning-Based
Product Design
Eui-Chul Jung
Seoul National University, South Korea
Meile Le
Hanyang University, South Korea

61

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

Abstract
Interpreting and incorporating machine learning technology from a human perspective helps
define the role of product designers in the era of artificial intelligence. With this background,
this study developed a 7-week design course about machine learning-based product design.
Subsequently, in Fall 2023, a class with seven undergraduate students from different majors
was held on the subject of “home camera.” The design process consisted of (a) user researchand need definition based on machine learning understanding and (b) machine learning-based
interaction design development using a coding platform. The course was designed to explore
the role of designers who implement human-centered design planning and prototypes
considering machine learning tools. In addition, the course provided a basis for suggesting thedirection of design education methods given emerging AI technology. The essence of design
education is not merely about efficiency; rather, it is about nurturing designers’ ability to thinkcritically and use tools to cultivate innovation.
Keywords: design education, designer role, human-centered design, machine learning, ML
product design course, product design62

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

With the rapid advancement of technology, machine learning (ML) has become an
indispensable tool for enhancing and optimizing user experience (UX) design of product use.
ML applications are pervasive in our daily lives, whether through apps or online services. They
perform various tasks, such as detecting and filtering spam, sorting or curating media feeds,
predicting estimated driving times, converting speech to text, and auto-correcting typing errors.
These applications significantly enhance the quality of interaction between users andtechnology, demonstrating the substantial potential of ML to improve interactive experiences.
Despite the advancements in technology, the market frequently encounters failures with MLbased products. Dove and colleagues (2017) criticized that developers and designers often use
ML technology more as a marketing gimmick rather than a tool to address real user needs, apractice aptly described as ‘putting lipstick on a pig.’ Furthermore, the implementation of ML
does not always enhance UX. Errors in prediction and opaque decision-making processes canlead to user frustration. For instance, overly personalized recommendation systems may
confine users within ‘filter bubbles’ (Areeb et al., 2023), limiting their exposure to new anddiverse content. Automated customer service systems often fall short of providing satisfactory
solutions for complex or non-standard issues. The opaque ‘black box’ nature of these systems
can erode user trust. More critically, ML systems have been shown to amplify social
inequalities and biases (Holstein et al., 2019), exacerbating issues like gender bias andcontradictions (Ahn et al., 2022). Such challenges include privacy violations, data biases, andbroader issues of inequity, all of which pose hurdles in the UX design of ML-based products.
To address these challenges, product designers must move beyond the traditional foci on
usability, utility, and interaction aesthetics towards more comprehensive design strategies.
There is a pressing need to incorporate courses in design education that blend human-centered
design thinking with ML to truly meet user needs. Significant research and curriculum
development have already focused on the technical aspects of ML, such as those documented
by Sulmont and colleagues (2019) and Munir and team (2022). However, as ML products
increasingly encounter market bottlenecks and issues, more researchers are recognizing thatthe development of ML products should center more on human-centered design, not justtechnological innovation. Many studies, including those by Yang and colleagues (2021) and
Riedl (2019), have analyzed the challenges ML products face from the perspectives of social
ethics and user behavior psychology. Yet, from the standpoint of design education, the researchand practice on how to educate future designers to comprehensively utilize ML technology inthe design process while considering user needs are still very limited.
Therefore, this study proposed a human-centered product design planning and prototype
creation course based on machine learning. The purpose of the course is to cultivate designers
and developers who are more attentive to fulfilling actual user needs and enhancing UX whenutilizing ML technologies. In this study, a mixed-methods approach was utilized to develop,
implement, and evaluate the course. Our methodology focused on Course Development,
starting with a comprehensive review of the existing courses, industry requirements, and thelatest academic research. This initial analysis helped identify the key gaps and needs in theapplication of ML in UX design to ensure that the course incorporated the latest technological63

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

advancements to address real-world challenges UX designers face. Based on the findings fromthe initial needs analysis, the course was structured as a seven-week program, segmented into
four distinct modules: Problem Definition and Planning, Model Construction, Model
Evaluation, and Implementation. Each module combined lectures with practical sessions,emphasizing real-world projects where students applied ML to tackle user-centric problems.
Evaluation methods included student surveys and interviews to assess their understanding andthe practical application of their skills. Feedback from industry stakeholders and long-term
tracking of alumni will further inform the course’s effectiveness and effects on professional
practices.
Through a combination of theoretical learning and practical application, the course aims toenable students to effectively apply ML technologies in design practices while criticallyassessing and integrating these technologies to ensure that design solutions genuinely meetuser expectations and needs.
Human-Considered Machine Learning Design Approaches
A lot of research has delved into how to use tools such as Midjourney in the discovery ordevelopment stage, which is the divergent stage of the design process that inspires the designconcept (Chiou et al., 2023; Turchi et al., 2023; Zhang et al., 2023). However, due to the
purpose-oriented nature of product design and the strong connection to human behavior,relying on interesting or unexpected accidental elements of the results generated by machinelearning tools is insufficient. Particularly in the field of interaction design, which has intricate
connections to human physiology and behavior, solutions are required that go beyond what
image-generating tools can typically offer (Tholander & Jonsson, 2023). To use the machinelearning tool more actively in the product design process, it is necessary to consider the
curriculum for creation that can understand human needs and behaviors and solve problems.
Machine learning research often ignores human factors, such as usability, intuition, effort, andhuman learning, and focuses only on the efficiency of algorithms. However, humanintervention is important for real-life applications. It is important to keep in mind that humanvalues, goals, and social structures always play an important role in collecting training data,
coordinating algorithms, and integrating machine learning into real-world systems. Humancentered machine learning is not a single approach but encompasses a wide diversity ofproblems, methods, technologies, and theories (Gillies et al., 2016).
However, many UX designers currently lack a deep understanding of artificial intelligence,
which hampers their ability to effectively contribute to interdisciplinary teams during ML
product development. UX designers may not fully grasp the dependent relationship of ML withdata and ground truth. Some designers and commentators even treat ML as something magical(Dove et al., 2017). Researchers have found that practitioners without a background in AI facechallenges in engaging with data and AI. Product professionals often struggle to understandwhat AI can do and find it difficult to translate business problems into data science problems(Piorkowski et al., 2021). This lack of designer involvement often leads ML products toemphasize technology at the expense of understanding user needs.

64

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

For ML product development, Stanford d. School’s “Designing Machine Learning” course uses
dimensions of idealism vs. realism and creativity vs. descriptiveness to categorize UX design
into four stages: the Idealistic and Creative Crafting stage, the Idealistic and Descriptive
Research stage, the Realistic and Creative Mapping stage, and the Realistic and Descriptive
Monitoring stage. This framework highlights the unique challenges in machine learning-based
UX product design compared to traditional UX design. In the Research stage, designers face
the challenge of bridging the gap between theoretical research and practical application toeffectively translate theoretical research into practical technologies or products. The Craftingstage deals with challenges in realizing innovative ideas as concrete products, particularly
when technical or engineering support is insufficient. The Mapping stage involves addressing
the opaque nature of machine learning systems and enhancing system transparency andinterpretability to ensure that users can understand and trust these technological solutions. In
the Monitoring stage, the challenge lies in overseeing the actual deployment of products andresponding to potential system failures or performance declines to ensure continuity of the UX
and system reliability.
Moreover, the design of intelligent systems also involves managing the dynamic relationshipbetween user control and system automation. The challenge of integrating “intelligent”
technology into people’s lives or determining when automation is needed and when people
desire a sense of control can be distilled into the debate between “do it for me” and “do it
myself” automation (Koulu, 2020; Shneiderman & Maes, 1997). In machine learning (ML)systems, the relationship between user control and automation is often dynamic, influenced byuser capabilities, their needs, and the capabilities of the ML system itself. Finding the optimal
dynamic balance in a UX setting presents a significant challenge for designers (Berberian etal., 2012; Parasuraman et al., 2000). Designers must navigate these complexities to create
experiences that effectively balance automation and user control, ensuring that systems
enhance users’ lives without overwhelming or alienating them.
Another major challenge faced by UX designers in the field of Machine Learning (ML) is its
ethical implications (Dove et al., 2017). As machine learning technology increasingly affects
many aspects of everyday life, UX designers must navigate the complex ethical landscape toensure that these systems are designed responsibly. Ethical considerations in UX design affectnot only user trust but also the broader social effects of the technology. This requires designers
to address issues like data privacy, bias in algorithmic decision-making, and the transparencyof ML systems during the UX design process.
Several initiatives and tools have been developed to address the challenges faced in designingmachine learning systems centered on human needs and ethical values. People + AI Research(PAIR), an initiative in Google, is an approach to designing machine learning focused onhumans (Google, 2019). According to Google, the aim of PAIR is “to explore the human sideof AI by doing fundamental research, building tools, creating design frameworks, and working
with various communities.” This guidebook provides a series of articles outlining
considerations for product development in AI: User Needs Defining Success, Data Collection
Evaluation, Mental Models, Explainability Trust, Feedback Control, and Errors Graceful65

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

Failure. Yildirim and colleagues’ (2023) research revealed that practitioners use the guidebook
not only to address AI’s design challenges but also to improve education, cross-functional
communication, and internal resources.
IDEO’s AI Ethics Cards help guide an ethically responsible, culturally considerate, andhumanistic approach to designing with data (Sampson & Chapman, 2019). The card set
consists of four major design principles and ten activities intended to be used in teams working
on the development of data-driven products and services. They help designers maintain ahuman-centered focus during the service or product development process.
Mathewson (2019) conducted an anthropocentric approach to interactive machine learning
design as follows: (a) Define the hypothesis and state the investigated question of interest, (b)Loop in humans and define your values and principles, (c) Define the goal, (d) Define the data,(e) Build a model, (f) Evaluate the model, (g) Analyze trade-offs, and (h) Re-evaluate the modeland iterate. Von Wangenheim and von Wangenheim (2021) explained the approaches in termsof the (a) needs identification and characterization of the context, (b) idea creation andspecification of the intelligent system, (c) requirements analysis of the ML model, (d) datapreparation, (e) model training and evaluation, (f) prediction, (g) model export, (h) model
deployment, and (i) SW system test.
These frameworks and tools, as outlined in Table 1, offer a structured approach for integrating
ethical considerations, user needs, and human factors into AI and ML system design, guiding
curriculum development.

66

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

Table 1
Design Process for Human-Centered Machine Learning from Literature
Reference

Stanfordd.School(2018)
People + AI
Research(2019)

IDEO’s AI
Ethics Cards(2019)

Product Design Process for Human-Centered Machine Learning
Problem definition Model
Model evaluation Implement
and planning
construction
Researching
Crafting
Monitoring
Mapping
User Needs
Defining Success
Mental Models

Principles: Don’t
presume thedesirability of AI
Respect privacy
and the collective
goodMathewson, (a) Define theK. W. (2019) hypothesis Statethe investigatedquestion of interest(b) Loop in humans
Define your values
and principles(c) Define the goal
von(a)Needs
Wangenheim identification and& voncharacterization of
Wangenheim the context(2021)(b) Idea creationand specification of
the intelligent
system(c) Requirements
analysis of the ML
model

Data Collection
Evaluation

Principles: Data
is not truth(d) Define thedata(e) Build model(d) Data
preparation(e) Model
training

67

Explainability
Trust
Feedback Control
Errors Graceful
FailurePrinciples:
Unintended
consequences ofAI areopportunities fordesign(f) Evaluate
model(g) Analyze tradeoffs(h) Re-evaluate
and iterate(e) Model
evaluation(f) Prediction,(g) Model
export(h) Model
deployment(i) SWsystem test

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

Core of Machine Learning-Based Product Design
Course Structure
Drawing from the studies outlined in Table 1, the product design process for human-centeredmachine learning can be segmented into four key stages: problem definition and planning,model construction, evaluation and implement. In the problem definition and planning stage,
the following steps are included: (a) defining user needs, (b) generating ideas, and (c) creating
an interaction flow to concretize ideas. The model construction stage encompasses (a) defining
the necessary data, (b) collecting and preprocessing data, and (c) model training. The model
evaluation stage involves assessing the model’s performance through a confusion matrix and
exploring methods to address errors. The following two steps, emphasizing multimodal
interaction and implementing interaction using Arduino, have been added to improve theability to design and implement a product from a human interaction perspective. Multimodal
interaction is a key element of user-centered machine learning design that addresses diverse
user groups, natural interaction, information richness, and flexibility in responses. The course
follows the flow as depicted in Figure 1.
Figure 1
Product Design Course Structure Based on Design Process Sample

Phase 1: Definition of User Needs
The first phase of the course focused on human-centered machine learning solutions, and theemphasis was placed on equipping students with the necessary skills and insights to approachmachine learning (ML) from a user-centric perspective. To effectively utilize machine learning
to address real-world problems and create value, it is crucial to start from the user’s perspective.
Instead of asking whether machine learning can be used, designers should frame questions interms of human-centered machine-learning solutions. When designing user-centered machinelearning solutions, it is essential to first explore machine learning concepts and elements froma UX perspective, consider machine learning design within the context of UX, understand theprocess for structuring machine learning projects with UX in mind, and explore when artificial
intelligence (machine learning) can yield effective results and when it might not be the best fit.
The methodology for defining user needs involves three stages:

68

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

1. Listing Existing Evidence: Collect existing data, research findings, and information
related to the issues and needs that users have encountered. This establishes afoundation for comprehending user requirements.
2. Detailed Description of User Needs: Utilize the gathered evidence to create acomprehensive and detailed description of user needs. This involves understanding what
users desire, where they face difficulties and the specific problems they aim to solve.
3. Assessing the Potential Suitability of AI Solutions: Based on the existing evidence andthe detailed description of user needs, evaluate whether machine learning-based
solutions have the potential to effectively meet these needs.
In this phase, when students are tasked with defining user needs in the context of machinelearning projects, the focus shifts towards practical application and engagement with real-world
problems through a user-centric lens. Understanding a user-centric standpoint and user needs
is fundamental to creating successful machine-learning solutions that genuinely address user
problems and enhance their experiences.
Phase 2: Multimodal Interaction Idea Creation
The second phase involved the practical application of combining machine learning withmultimodal interaction to enhance UX and facilitate seamless interaction between physical anddigital environments. Karray and colleagues (2008) argued that the interaction between humanmachine systems fundamentally occurs through the exchange of information via various input
and output methods between computers and humans. Hinckley and colleagues (2014) further
elucidated the nature of these exchanges, highlighting that inputs to a system constitute theinformation relayed by the user to the machine. Conversely, system outputs are essentially the
feedback provided to the user, which aids in navigating and accomplishing tasks. This dynamic
exchange effectively bridges the gap between the internal operations of the system and thetangible, physical world, thereby facilitating a seamless interaction that spans the digital andphysical realms. The multimodal human-machine interaction model is depicted in Figure 2.
Figure 2
Multimodal Human-Machine Interaction Model69

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

Humans naturally engage in multimodal interactions with the world, utilizing various sensory
channels to perceive and respond to other people and the external environment. Several studies(Bolarinwa et al., 2019; Mathewson, 2019; Xiao et al., 2003) have demonstrated that
implementing multimodal interaction systems enhances flexibility and facilitates the exchange
of information, allowing for natural and realistic interactions between intelligent systems andusers. The combination of machine learning and multimodal interaction serves to improve the
UX and enables more effective handling of interactions between physical and digital
environments.
Therefore, in this phase, students were asked to critically examine the dynamics between data
input and output in the context of machine learning tools guided by a human-centeredperspective. This exploration was pivotal, as it required students to consider how users interact
with machine learning systems and how these systems, in turn, respond to and guide user actions.
The emphasis on a human-centered approach encouraged students to prioritize user needs,
preferences, and behaviors in the design and development of machine learning applications.
Phase 3: Developing an Interaction Flow
In the third phase, students were asked to develop interaction flows for machine learning-based
interactive products or systems. This phase emphasized the significance of understanding anddesigning products to enhance the dynamic relationship between humans and machine learning
systems. Ghim (2021) highlighted that interactive products, unlike static products, possess two
unique characteristics: human-machine interaction and temporal sequencing. For instance,
users provide input to the product through physical actions and interpret the product’s operation
through mental processes.
Machine learning-based interactive products detect input through sensors, which can come fromusers or environmental changes, and respond through actuators or output components. The
exchange of these actions and communications occurs sequentially over time to achieve theproduct’s intended goals. Based on these two aspects, a framework can be established to help
understand and design machine learning-based interactive products/systems, as shown in Figure 3.
Figure 3
Human-Machine Interaction Flow70

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

In machine learning interactions, the role of humans in the relationship between humans andmachines varies depending on user characteristics, user needs, and task requirements. It isessential to consider the points at which the roles of humans and machine learning interchange
based on the user’s usage context. As shown in Figure 4, when users have a high level of task
performance capability, a strong interest in the task, and engage in creative work or highly
sensitive domains, machine learning interactions should be designed to enhance users’
capabilities and creativity. In such cases, the machine learning system should play a supportive
rather than a controlling role. On the contrary, when users have low task performancecapability, minimal interest in the task, and engage in repetitive tasks or work in low-sensitive
domains, machine learning interactions should be designed to simplify and automate tasks. The
system should require minimal user input and decision-making, focusing on efficiency andreducing the cognitive load for the user. The machine learning system should take on a moreproactive role, providing assistance, guidance, and automation to make the user’s tasks easierand more manageable.
Figure 4
Changing Relationship Between Human and Machine Learning Based on User
Characteristics and Task Features

Considering the fluid dynamics between humans and machine learning, as well as the intricate
aspects of human-machine interaction and temporal sequencing, students were assigned thechallenge of crafting interaction flows. These flows had to be tailored to meet the uniquerequirements and characteristics of both the users and their projects. This task emphasized theimportance of understanding not just the technological capabilities of machine learning
systems but also the human context in which these interactions occur.
Phase 4: Defining the Necessary Data and Data Preprocessing
Phase 4 emphasized the practical aspects of data handling for machine learning projects,
specifically the collection, labeling, and preprocessing of data to meet user requirements. The
process started with identifying the necessary dataset and then pinpointing the essential featuresand labels within this dataset that align with user needs. To enable machine-learning products
to make accurate predictions, their underlying machine-learning models had to be trained based71

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

on patterns and correlations within the data. This data, known as training data, could encompass
a diverse range, including images, videos, text, and audio. Students could utilize existing data
sources or collect new data specifically for training their system.
The quality of the training data, including how they were sourced, collected, and labeled,played a pivotal role in determining the output of the system. The quality, relevance, andaccuracy of this data, rather than just quantity, were paramount. Properly labeled and wellstructured data ensure that the machine learning model can learn effectively, leading to reliable
and functional outputs. In addition to data collection, data preprocessing was an integral aspectof this phase. It involved cleaning the data, handling missing values, and standardizing or
normalizing it for consistency. Preprocessing may also include transforming variables to makethem more suitable for analysis. The goal of this step was to refine the data into a format thatis conducive to training a high-performing machine learning model.
Through these tasks, students learned the critical role of high-quality, well-labeled, andproperly processed data in building effective machine-learning systems. This phase not only
enhanced their technical skills in handling data but also deepened their understanding of thefoundational principles that govern the success of machine learning products, emphasizing theimportance of meticulous preparation and analysis of data to meet user requirements.
Phase 5: Model Training
In Phase 5, students used a coding platform called Naver Entry to process the collected data
and build machine-learning models based on that data. Naver Entry is an educational platform
designed to foster computational thinking and provide hands-on experience with artificial
intelligence features through a block-based interface. One of the key advantages of Naver Entry
is its user-friendly interface, which is particularly accessible for design students with minimal
programming background. Because of its ease of use, students could focus more on theapplication and implications of machine learning in their field rather than on the complexities
of coding. Another key reason for choosing Naver Entry for model training in this phase wasits adeptness in handling multimodal data, including images, audio, and text. This flexibility is
crucial for a comprehensive learning experience and aligns well with the diverse nature of datatypically encountered in real-world scenarios. It met the course requirements and provided
students with practical experience in model training. Naver Entry allows users to train themodels based on the following:a. Image Classification: Train a model that can classify images uploaded or captured via
a webcam.
b. Text Classification: Train a model capable of classifying text that you either write
directly or upload as a file.
c. Voice Classification: Train a model that can classify voices recorded from a
microphone or uploaded as audio files.
d. Numeric Classification: Train a model that classifies numeric data in tables into various
classes based on the nearest neighbors (K-nearest neighbors) for each data point.

72

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

Phase 6: Assessing the Model’s Performance
In the “Evaluation” phase, students delved into the critical process of assessing the performanceof their machine learning models through the lens of a confusion matrix, a pivotal tool inunderstanding the nuances of model accuracy. The confusion matrix (Figure 5), also referred
to as an error matrix, is a structured representation that showcases how well a classification
model predicts outcomes across a set of test data, distinguishing between true positives (TP),true negatives (TN), false positives (FP), and false negatives (FN).
Figure 5
Confusion Matrix

Note. True positive (TP): Observation predicted as positive is positive. False positive (FP): Observation predicted
as positive is negative. True negative (TN): Observation predicted as negative is negative. False negative (FN):
Observation predicted as negative is positive.

The effectiveness of a model is gauged by the magnitudes of TP and TN indicating correct
predictions and the minimization of FP and FN signifying errors. The decision to either focus
on reducing FP and FN or on maximizing TP and TN plays a significant role in shaping theuser experience (UX), as noted by Davis and Goadrich (2006). This decision is not arbitrary
but hinges on the specific goals of the machine learning application, the nature of the data, andthe intended outcomes of the model.
When designing, the evaluation metrics vary based on the machine learning’s purpose, datacharacteristics, and model objectives. For example, even for the same recommendation system,
the important evaluation metrics can differ depending on the application. In product(advertisement) recommendations, high precision may be crucial because it increases the
proportion of recommended products that are of interest to the user. Precision is importantwhen leading to purchase actions. On the other hand, in music recommendations, high recall
can help users explore and discover various types of music. Accordingly, music platforms
should focus on increasing recall to assist users in finding diverse music and prevent churn.
In machine learning predictions, providing appropriate feedback to users when errors (FP/FN)
occur is essential for maintaining user trust in the AI product. Feedback methods include
notifying users of errors, explaining why those errors occurred, apologizing to users, soliciting73

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

feedback from users to improve and adjust system performance based on user feedback, andproviding options for AI intervention for more accurate results when necessary.
In this study, students were first asked to assess what should be emphasized in this task. For
instance, in safety-related issues, one should focus on reducing the likelihood of FN (false
negatives). In other words, extra attention should be paid to cases where a problem genuinely
occurred but was not predicted by the machine learning model. With a match rate of 20-50%(an area with a high likelihood of false negatives occurring), they needed to consider how toprovide feedback to users. They were required to outline how to request feedback from users
and how to provide feedback to users in each situation, as shown in Figure 6.
Figure 6
Assessing the Model’s Performance and Exploring Methods to Address Errors

In each situation, students had to outline specific approaches for engaging with users, ensuringthat the feedback loop is not only informative but also fosters trust and reliability in the machinelearning application. By focusing on reducing false negatives or false positives and enhancing
the dialogue between users and the system, students learned to create more resilient,
responsive, and user-centric machine-learning solutions.
Phase 7: Implement Interaction Using Arduino
In Phase 7, the application of machine learning extended beyond theoretical concepts, merging
with the physical world and tangible products. The key objective in this phase was to bringmachine learning into real-life applications by employing Arduino. This phase reemphasized74

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

the significance of multimodal interaction, focusing on the dynamic interplay between user
input and system output, and integrated this with practical work using Arduino.
Students were tasked with exploring and prototyping various methods of user sensing andfeedback. They used their machine-learning models in conjunction with Arduino to develop
intelligent system prototypes. This process involved designing and building Arduino circuits,
gaining a deep understanding of how inputs and outputs operate within the Arduino ecosystem,
and coding these interactions to function as intended. This phase allowed students to test andvalidate their designs against user needs and requirements. It provided an opportunity to assess
whether the prototypes meet the intended purposes and to understand the practical role ofmachine learning in actual product development.
By culminating the learning process with the development of a functioning prototype, students
gained valuable insights into the practical applications of machine learning, understanding its
role and potential in enhancing user interactions and experiences in the realm of physicalproducts, thereby bridging the gap between theoretical knowledge and practical application.
Course Implementation and Evaluation
In the Fall Semester of 2023, a course was conducted with seven undergraduate students under
the theme of ‘home camera.’ The group included four design majors, one education major, onelife sciences major, and one marine engineering major. Except for the marine engineering
student, who had some basic programming experience, the other students had no priorexperience in coding. The students carried out various projects, including smart cameras for
efficient study, Smile Boxes that help users maintain a positive mindset, pet-related products,
and security-related products. The researchers aimed to assess the effectiveness of the course
proposal by examining the student outcomes, focusing particularly on students who
demonstrated the most fidelity to the machine learning-based product design process.
Student Project: Smart Home Camera for Pet Companionship
Many people consider their pets as part of their family. These individuals require more thansimple observation; they need delicate care services. They want to provide appropriate support
to their pets when they are alone, including observation, health monitoring, communication,
provision of treats, and guiding their pets to specific areas.
One student defined the following design goals based on her experience of raising a pet andmarket research:a. Real-Time Monitoring: Design that allows users to monitor their pets’ status andactivities in real-time and respond effectively.
b. Behavior Analysis: Functional design that uses machine learning to analyze petbehavior and notify the user.

75

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

c. Enhanced Interaction: Design that enables bidirectional interaction between users andtheir pets through a smartphone app.
d. User-Friendly: Design that considers elements to minimize pet stress.
She ensured that machine learning solutions can be helpful in providing personalized services
to users and aligning with the potential suitability of user needs for machine learning solutions.
Based on the use needs, she created multimodal interaction ideas for providing an enhanced‘pet care service’ experience Using Machine Learning:a. Utilize machine learning to provide personalized services to users (e.g., tracking pet
activity).
b. Process audio/video data based on learned patterns to send notifications only under
specific conditions.
c. Detect and respond to user-defined events using machine learning (e.g., entering
specific areas, behaviors like scratching food bowls).
d. Improve remote interaction between users and their pets (e.g., operating a speaker after
tracking the pet).
Accordingly, the interaction flow, shown in Figure 7, was created. The interaction flow design
should consider both the user’s and the pet’s actions to create a comprehensive UX. Since pets
can also be users, the actions of pets are included in the flow.
Figure 7
Interaction Flow of Smart Home Camera for Pet Companionship76

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

Then, she defined the data needed to train the machine-learning model: User and pet basic
information data and user profiles, pet species, age, health status, behavior patterns, preferencedata, and audio/video data. Pet behavior data: Data related to a pet’s entry into specific areasor specific behaviors. User feedback and interaction data: Data on interactions between users
and pets, user feedback, and service usage records.
This student’s work considered various user needs, including real-time monitoring, behavior
analysis, enhanced interaction, and user-friendliness. The effectiveness of the project wasevaluated based on the integration of machine learning features that aligned with the user needs
and the comprehensiveness of the interaction design. The project was noted for its systematic
approach and deep consideration of both user and pet perspectives within the interaction flow.
The effectiveness of the student projects was assessed by examining how well the integrationof machine learning features met actual user needs and the depth of the interaction design. Thisparticular project was recognized for its systematic approach and thorough integration of bothhuman and animal perspectives within the interaction flow.
Course Evaluation
During the 7-week class held in the Fall Semester of 2023 with seven students, meaningfuladvancements in understanding user needs for machine learning-based product design wereobserved. The students learned to reflect these needs consistently throughout the design
process. Additionally, they gained insights into the design methods that utilize machinelearning technologies appropriate for their selected project topics with the aid of “Entry,” ablock coding tool, which simplified the implementation of machine learning programs.
According to student evaluations, the class was beneficial as it provided them with
opportunities to handle various AI-powered programs and apply machine learning to
unfamiliar areas. Design students appreciated the way the course bridged the gap between
creative design and technical implementation, allowing them to explore new dimensions in
product design. Students from other majors valued the human-centered approach and AI
understanding, noting that the problem-solving mindset gained from the course would benefit
their future work by enabling innovative AI integration in their fields. However, they notedsome challenges with remote delivery, including difficulties in performing practical exercises
due to the limitations of virtual instruction. From the instructor’s perspective, the class faced
certain limitations. Primarily, the focus was on supervised learning techniques due to time
limitations and the restrictions of the coding tools used, which narrowed the scope of AI
applications. Additionally, the collected data often did not adequately reflect the users’ actual
situations and needs, leading to a deficiency in students’ understanding of the data. Moreover,time constraints prevented the inclusion of product appearance and interface design in the
curriculum.
Future iterations of the course should incorporate a broader array of machine learningtechnologies, including unsupervised learning techniques, such as clustering models andreinforcement learning. This expansion would enable students to apply their design skills in77

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

more diverse real-world contexts. Additionally, the curriculum should be enhanced with more
extensive data utilization, ensuring students can extract and model meaningful information
effectively. These improvements will not only deepen students’ understanding of machinelearning applications in design but also increase their ability to use machine learning to address
real user needs.
Conclusion
The rapid advancement of machine learning (ML) technologies brings with it a myriad ofopportunities and challenges, particularly in the realm of UX design. While ML has thepotential to significantly enhance interactions between users and technology, its practicalimplementation often falls short due to technological limitations, a general failure to meet realuser needs effectively, and ethical concerns. This paper underscores the existing gaps in design
education and practice, advocating for a shift towards more human-centered methodologies indeveloping ML-based products.
The newly proposed Machine Learning-Based Human-Centered Product Design Planning andPrototyping Course, structured as a seven-week program with four distinct modules—problem
definition and planning, model construction, model evaluation, and implementation—aims toredress these educational shortcomings. This course weaves together a comprehensive review
of existing courses, current industry demands, and recent academic research to meticulously
equip future designers with the necessary skills to effectively utilize ML technologies while
prioritizing user needs.
Feedback from the initial course, which included professional evaluations of student projects
and student surveys and interviews, indicated that the curriculum could successfully bridge thegap between theoretical knowledge and practical application. Participants showed marked
improvements in their ability to integrate ML technologies into their design practices,demonstrating a heightened sensitivity to user-centered design principles.
The continuous evolution of machine learning (ML) technologies will undoubtedly present
new challenges and opportunities. Therefore, ongoing research, curriculum updates, and robust
feedback mechanisms are crucial to ensure that educational offerings remain relevant andeffective. It is necessary to continue to refine the modules proposed in the machine learning
design education method. Furthermore, the findings from this course can serve as afoundational model for future developments in design education, particularly in integrating
emerging technologies and user-focused design. Subsequent research should try to refine thisapproach into a comprehensive design education methodology—one that can foresee andunderstand the ensuing changes in consciousness, societal norms, cultural shifts, economictransitions, and the broader life evolutions they instigate.
The essence of design education is not merely about efficiency; rather, it is about nurturing
designers’ ability to think critically and use tools to cultivate innovation. Historically,
photography was primarily approached from a technical perspective, aiming to capture subjects

78

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

effectively. However, it soon evolved into a revered art form. The transformative power of newtechnology on human life and culture is evident, particularly when considering how modern
photography practices significantly influence our society and daily lives. As designers
increasingly incorporate ML tools into their toolkit, it becomes crucial to reflect on their
evolving role. These tools should be harnessed not just for efficiency but also for their ability
to foster human-centric innovation, potentially leading to shifts in human consciousness.

Acknowledgements
This work was supported by the Ministry of Education of the Republic of Korea and the
National Research Foundation of Korea (NRF-2021S1A5A2A01070316).

79

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

References
Ahn, J., Kim, J., & Sung, Y. (2022). The effect of gender stereotypes on artificial intelligence
recommendations. Journal of Business Research, 141, 50–59.
https://doi.org/10.1016/j.jbusres.2021.12.007
Areeb, Q. M., Nadeem, M., Sohail, S. S., Imam, R., Doctor, F., Himeur, Y., ... & Amira, A.
(2023). Filter bubbles in recommender systems: Fact or fallacy—A systematic
review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery,
13(f), e1512. https://doi.org/10.1002/widm.1512
Berberian, B., Sarrazin, J. C., Le Blaye, P., & Haggard, P. (2012). Automation technology
and sense of control: a window on human agency. PloS one, 7(c), e34075.
https://doi.org/10.1371/journal.pone.0034075
Bolarinwa, J., Eimontaite, I., Dogramadzi, S., Mitchell, T., & Caleb-Solly, P. (2019, June).
The use of different feedback modalities and verbal collaboration in tele-robotic
assistance. In 2019 IEEE International Symposium on Robotic and Sensors
Environments (ROSE) (pp. 1–8). IEEE. https://doi.org/10.1109/ROSE.2019.8790412
Chiou, L. Y., Hung, P. K., Liang, R. H., & Wang, C. T. (2023, July). Designing with AI: An
Exploration of Co-Ideation with Image Generators. In Proceedings of the 2023 ACM
Designing Interactive Systems Conference (pp. 1941–1954).
https://doi.org/10.1145/3563657.3596001
Chmielewski, D., Hu, K. (2023). Disney creates task force to explore AI and cut costs.
Reuters. Retrieved from https://www.reuters.com/technology/disney-creates-taskforce-explore-ai-cut-costs-sources-2023-08-08/
Davis, J., & Goadrich, M. (2006, June). The relationship between Precision-Recall and ROC
curves. In Proceedings of the 23rd international conference on Machine learning (pp.
233–240). https://doi.org/10.1145/1143844.1143874
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017, May). UX design innovation:
Challenges for working with machine learning as a design material. In Proceedings ofthe 2017 chi conference on human factors in computing systems (pp. 278–288).
https://doi.org/10.1145/3025453.3025739
Ghim, Y. G. (2021). Allocated Flow Diagramming: A Structured Process and Methods for
Teaching Interactive Product Prototyping in Industrial Design. Archives of Design
Research, 34(d), 7–21. https://doi.org/10.15187/adr.2021.11.34.4.7
Gillies, M., Fiebrink, R., Tanaka, A., Garcia, J., Bevilacqua, F., Heloir, A., ... & Caramiaux,
B. (2016, May). Human-centred machine learning. In Proceedings of the 2016 CHI
conference extended abstracts on human factors in computing systems (pp. 3558–
3565). https://doi.org/10.1145/2851581.2856492
Google (2019). People + AI Guidebook. Retrieved fromhttps://pair.withgoogle.com/guidebook
Hinckley, K., Jacob, R. J., Ware, C., Wobbrock, J. O., & Wigdor, D. (2014). Input/Output
Devices and Interaction Techniques. Chapman and Hall.
Holstein, K., Wortman Vaughan, J., Daumé III, H., Dudik, M., & Wallach, H. (2019, May).
Improving fairness in machine learning systems: What do industry practitioners
need?. In Proceedings of the 2019 CHI conference on human factors in computing
systems (pp. 1–16). https://doi.org/10.1145/3290605.3300830

80

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

Karray, F., Alemzadeh, M., Abou Saleh, J., & Arab, M. N. (2008). Human-computerinteraction: Overview on state of the art. International journal on smart sensing andintelligent systems, 1(a), 137–159. https://doi.org/10.21307/ijssis-2017-283
Koulu, R. (2020). Human control over automation: EU policy and AI ethics. Eur. J. LegalStud., 12, 9.
Mathewson, K. W. (2019). A human-centered approach to interactive machine learning.
arXiv preprint arXiv:1905.06289. https://doi.org/10.48550/arXiv.1905.06289
Marq’s Blog (2023). Artificial Intelligence Design Tool Statistics & Trends in 2023.
Retrieved from https://www.marq.com/blog/artificial-intelligence-design-toolstatistics-trends-in-2023
Munir, H., Vogel, B., & Jacobsson, A. (2022). Artificial intelligence and machine learning
approaches in digital education: A systematic revision. Information, 13(d), 203.
https://doi.org/10.3390/info13040203
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model for types and levels ofhuman interaction with automation. IEEE Transactions on systems, man, andcybernetics-Part A: Systems and Humans, 30(c), 286–297.
https://doi.org/10.1109/3468.844354
Piorkowski, D., Park, S., Wang, A. Y., Wang, D., Muller, M., & Portnoy, F. (2021). How ai
developers overcome communication challenges in a multidisciplinary team: A casestudy. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), 1–25.
https://doi.org/10.1145/3449205
Riedl, M. O. (2019). Human‐centered artificial intelligence and machine learning. Humanbehavior and emerging technologies, 1(a), 33–36. https://doi.org/10.1002/hbe2.117
Roettgers, J. (2023). Ikea’s generative AI furniture designs are trippy, retro, and in-spiring,
FAST COMPANY. Retrieved from https://www.fastcompany.com/90871133/ikeagenerative-ai-furniture-design
Sampson, O., Chapman, M. (2019). AI NEEDS AN ETHICAL COMPASS. THIS TOOL CAN
HELP. IDEO. Retrieved from https://www.ideo.com/journal/ai-needs-an-ethicalcompass-this-tool-can-help
Shneiderman, B., & Maes, P. (1997). Direct manipulation vs. interface agents. interactions,
4(f), 42–61. https://doi.org/10.1145/267505.267514
Sulmont, E., Patitsas, E., & Cooperstock, J. R. (2019). What is hard about teaching machinelearning to non-majors? Insights from classifying instructors’ learning goals. ACM
Transactions on Computing Education (TOCE), 19(d), 1–16.
https://doi.org/10.1145/3336124
Tholander, J., & Jonsson, M. (2023, July). Design ideation with ai-sketching, thinking andtalking with Generative Machine Learning Models. In Proceedings of the 2023 ACM
Designing Interactive Systems Conference (pp. 1930–1940).
https://doi.org/10.1145/3563657.3596014
Turchi, T., Carta, S., Ambrosini, L., & Malizia, A. (2023, May). Human-AI co-creation:
evaluating the impact of large-scale text-to-image generative models on the creative
process. In International Symposium on End User Development (pp. 35–51). Cham:
Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-34433-6_3

81

IAFOR Journal of Education: Technology in Education

Volume 12 – Issue 2 – 2024

von Wangenheim, C. G., & von Wangenheim, A. (2021). Overview on a human-centricinteractive ML process for teaching ML in K-12. Working Paper
WP_GQS_01_2021_v10, GQS/INCoD/UFSC, Florianópolis, Brasil.
Xiao, B., Lunsford, R., Coulston, R., Wesson, M., & Oviatt, S. (2003, November). Modelingmultimodal integration patterns and performance in seniors: Toward adaptive
processing of individual differences. In Proceedings of the 5th international
conference on Multimodal interfaces (pp. 265–272).
https://doi.org/10.1145/958432.958480
Yang, S. J., Ogata, H., Matsui, T., & Chen, N. S. (2021). Human-centered artificial
intelligence in education: Seeing the invisible through the visible. Computers andEducation: Artificial Intelligence, 2, 100008.
https://doi.org/10.1016/j.caeai.2021.100008
Yildirim, N., Pushkarna, M., Goyal, N., Wattenberg, M., & Viégas, F. (2023, April).
Investigating how practitioners use human-ai guidelines: A case study on the people+
ai guidebook. In Proceedings of the 2023 CHI Conference on Human Factors in
Computing Systems (pp. 1–13). https://doi.org/10.1145/3544548.3580900
Zhang, C., Wang, W., Pangaro, P., Martelaro, N., & Byrne, D. (2023, June). Generative
Image AI Using Design Sketches as input: Opportunities and Challenges. In
Proceedings of the 15th Conference on Creativity and Cognition (pp. 254–261).
https://doi.org/10.1145/3591196.3596820
Corresponding author: Eui-Chui Jung
Email: jech@snu.ac.kr

82