<figure><img src="image_1.png" /></figure>

HAL
open science

# Intelligence as a Design Material - The Use of AI in Products and Services

Tilman Dingler

► To cite this version:

Tilman Dingler. Intelligence as a Design Material - The Use of AI in Products and Services. 19th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2023, York, United Kingdom. pp.238-245, 10.1007/978-3-031-61698-3_23. hal-05305343

HAL Id: hal-05305343

https://inria.hal.science/hal-05305343v1

Submitted on 9 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Intelligence as a Design Material - The Use of AI in Products and Services

Tilman Dingler(✉) 5

Delft University of Technology, Delft, The Netherlands

t.dingler@tudelft.nl

**Abstract.** Artificial Intelligence (AI) is increasingly weaving its way into our daily lives. Designers, researchers, and entrepreneurs alike flock towards its applicability as a design material and building block for intelligent products and services. While AI tools are becoming indispensable for users to organise their physical and digital lives, questions of dependence and agency need to be addressed, especially in areas where technologies shape our decision-making. By discussing available frameworks and the role of human bias, this article dives into how to escape the risk of manipulation by offering human control. AI needs to be seen as a design material that can help us create more sensitive products and be used to enhance users' innate capabilities rather than patronize them; a design material with the potential to enhance, rather than diminish, human skills and experiences.

**Keywords:** intelligent products and services · bias-aware systems

## 1 The Duality of AI: Guiding Choices and Challenging Autonomy

In an age where artificial intelligence has become an integral part of our daily routines, the very agency of our decisions is now inextricably intertwined with Artificial Intelligence (AI) tools that guide and shape our choices. Recommendation algorithms, for example, are used by popular streaming platforms like Netflix and music streaming services like Spotify. These algorithms analyze vast amounts of data, including users' viewing or listening history, preferences, and demographic information, to curate personalized content recommendations for users. The platforms employ AI to suggest movies and playlists that align with users' previous viewing or listening habits and preferences. These recommendations subsequently make suggestions for new content and thus influence users' choices of what to consume next. In this way, AI algorithms not only enhance the user experience but also have a profound impact on what people consume, starting with entertainment choices all the way to shaping our cultural experiences.

Such intelligence basically offers the convenience of assisting its users in navigating the vast amounts of offerings and the complexities of our modern lives. Yet, as our reliance on AI for convenience and efficiency deepens, a paradox
emerges: the more we offload tasks and decision-making to these digital aides, the more we may find ourselves in a position of vulnerability and dependence. This growing interdependence between humans and AI raises critical questions about the potential trade-offs we make while embracing technology, and what aspects of our autonomy and decision-making may be at risk in this symbiotic relationship.

## 2 A Human-Centered AI Design Framework

In 1962, two labs were started in walking distance from each other at Stanford: John McCarthy, who co-authored the document that coined the term "artificial intelligence" (AI) in 1956, created the Stanford Artificial Intelligence Laboratory; his goal was to create technologies that could essentially replace a human. Around the same time, another character opened his lab on the other side of campus: Douglas Engelbart, the inventor of the computer mouse, founded the Augmentation Research Center (ARC) with the goal of intelligence augmentation. Rather than replacing the human, he wanted to extend the human operator's abilities. This approach included thinking about the impact and ethics of these systems and what the role of people is in human-machine collaboration.

McCarthy's view of automation was very much one-dimensional (see Fig. 1):

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** One-dimensional view of human control being opposing labels at the end of a spectrum.</figcaption></figure>

Examples at the edge of each end of this spectrum are a manually operated watering can (high human control) as opposed to a fully automated irrigation system. Or, pen and paper provide the writer with full control over the writing process whereas ChatGPT may automate key aspects of that same process. An auto-complete function may sit somewhere in between.

This one-dimensional trade-off was adopted by the US Society of Automotive Engineering in its six levels of autonomy for self-driving cars [7] starting with level 0 (no driving automation) and ending with level 5 (full driving automation). Another more nuanced, yet still one-dimensional framework came from Parasuraman and colleagues [8] where level 0 offers no assistance (the human must take all decisions and actions) and systems on level 10 decide everything and act autonomously while ignoring the human.

The one-dimensional view of automation (from human control to fully automated) may seem, however, outdated and may be in need of reconsideration: a multi-dimensional framework, proposed by Ben Shneiderman. His Human-Centered AI (HCAI) framework consists of a two-dimensional grid where human control and automation are complementary rather than antagonistic.
<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Shneiderman's Human-Centered AI Framework, a two-dimensional grid that helps map intelligent products along the dimensions of human control and automation.</figcaption></figure>

Figure 2 shows Shneiderman’s HCAI framework and allows product designers to explicitly think about aspects, involving agency (human control), reliability (based on sound software engineering practices), safety, and trust (through independent oversight). By mapping intelligent products along these dimensions, designers can better understand and assess the balance between human involvement and automation in order to create products that enhance user experience while maintaining user agency and autonomy.

An example Shneiderman gives in his 2022 book on “Human-Centered AI” [9] is a medical device that administers painkillers in a hospital. A design that would fall into the upper left quadrant (higher human control, less automation) would allow patients to determine the dosing and frequency of the painkiller administration at the press of a button. However, such a design would come at the risk of overdosing unless technical measures are in place to prevent receptive button presses. This is where the design would subsequently move towards automation. An even safer and more reliable design may use machine learning to adjust the dosage while still allowing the human to trigger the drug release while preventing overdosing.

Examples of *human mastery* without automation or machine intervention can be activities we have become accustomed to enjoy as we get better at them, such as playing an instrument or riding a bike. Automating parts of these experiences could even be seen as subtracting from the experience. Adding an electric motor that automatically recharges using the bike's torque, on the other hand, would again move the design of a bicycle further towards automation.

Examples that are considered both *automated and under human control* include the design of an elevator that moves fully automated to a designated floor as commanded by a human user. Also, a camera takes a photo triggered by a
human operator, yet shutter speed, ISO, and aperture are controlled by the device
and the image may be automatically post-processed by adding a range of filters.

Therefore, when approaching any task, designers must carefully consider
whether to design for complete computer control or prioritize human mastery. The
objective is to create designs that are not only efficient but also well-established,
backed by reliable procedures, safety protocols, and trustworthy oversight. The
goal, however, is to integrate AI with product designs in ways that “amplify, aug-
ment, empower, and enhance people”, according to Shneiderman.

3 AI Technologies are Impacting Every Aspect of Our Lives

AI is already a key ingredient in our everyday products and services: Apple's Siri brings speech recognition into people's phones. Google Translate allows anyone to provide it document in any language and it will return a pretty good translation. A modern car comes with various kinds of intelligent features, starting with cruise control (simple velocity sensor) all the way to AI-powered cameras that recognise overtaking vehicles, lane markings, and barriers. Over the next 10 years, cars will increasingly do things that will protect drivers from their mistakes, i.e., make us safer.

Intelligent products are not limited to software either. Advances in the mate-
rial sciences allow product designers to use thermochemical properties to make
products safer. Muchkin, a product company from California, for example, has
specialized in products for infants and toddlers. Its infant spoon tips turn white
when baby's food is too hot (approximately 43°C and above) thanks to a
patented polymer design (see Fig. 3).

As AI technologies advance, new tooling is available. Analogously, programming languages moved from low-level assembler to high-level coding programming with, for example, object-oriented paradigms. While training and deploying a Machine-Learning model used to be the domain of AI experts, there are increasingly tools available that don't require in-depth programming knowledge to do simple classification tasks. ChatGPT is probably the most prominent example of how an AI tool can be picked up by non-tech-savvy users and used for writing, data analysis, or even programming tasks.

More sophisticated tools to create intelligent products and services will empower new players to innovate and disrupt. And in order to design intelligent products designers now have a whole range of tools at their disposal. It is getting increasingly easier to build products and services with AI/ML thanks to the modularisation, packaging, and usability of AI libraries. Fast.ai, for example, makes deep learning accessible to product builders with very limited programming knowledge. Google provides a whole range of freely available online learning modules<fnref n="1" /> and further publishes guidelines on human-AI collaboration<fnref n="2" />. In most

<footnote n="1">¹ https://ai.google/build/machine-learning/.</footnote>

<footnote n="2">² https://pair.withgoogle.com/.</footnote>
<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Munchkin's patented spoon tips as an example of using smart materials to make physical products safer (for infants).</figcaption></figure>

of these libraries, courses, and guidelines, however, the human augmentation part is often not fully utilised. It falls upon the designer to use these new tools and point them in the right direction using human-centred design methods.

As AI continues to creep into everyday products and the underlying technology evolves, it becomes increasingly clear that AI tools are becoming more accessible and user-friendly, empowering a wider range of designers to harness the potential of artificial intelligence. This democratization of AI tools, however, also ushers in a set of challenges, particularly concerning the ethical implications and potential risks tied to the widespread integration of AI.

## 4 AI and the Risk of Manipulation

As AI algorithms become more sophisticated and powerful, there is a concern about their potential to manipulate user behaviour and shape our choices. Recommendation algorithms are essentially designed to prioritize certain content and promote specific products with the explicit goal of influencing users' decisions. While convenience comes from not having to sift through the vast amount of available information and products, the risk of manipulation also raises important ethical questions about the responsibility of developers and designers to ensure that AI technologies are used transparently.

Media companies and their practices have recently come under scrutiny for the lack of transparency of their algorithms and the hidden harm that may lurk in social media news feeds. Whistleblower Francis, a former product manager at Facebook, exposed internal documents about Facebook's practices to keep users' attention at all costs. Among others, these documents showed the concerning impacts that Facebook's practices in attention-grabbing news delivery had on the youth, in particular. The case of Cambridge Analytica has shown the dangerous effectiveness of micro-targeting on social media to influence public opinion. The
rapid development of such systems and Silicon Valley’s recent mantra of “move fast and break things” warrants a comprehensive research agenda on what people pay attention to and why, how information can be effectively filtered in the vast information space without biasing opinion formation and what makes online users fall for misinformation.

In this regard, we have recently built a strand of research that looks at the interplay of computing systems and users’ cognitive biases, i.e., a set of heuristics learned over time that serve as shortcuts to effectively navigate dense or complex information spaces. These biases play an important role in how information is perceived and processed. Social media and computing systems generally influence that perception through the 1) content they show, 2) the interfaces they display, and 3) the behaviour they exhibit. All these can both utilise and exploit cognitive biases. In our work, we study these biases, how they manifest, and how computing systems exacerbate or mitigate them.

# 5 Bias-Aware Computing

Machine-learning techniques play an essential role in our work to detect and model the occurrence of cognitive biases. We are interested in studying the intricate relationship between computing systems and people’s biases as we aim to prevent users from being manipulated online. Therefore, we coined the term Bias-aware Systems, i.e., computing systems that detect and take into account the presence of cognitive biases in users [1].

We started by devising a method to collect ground truth on biases as they often manifest as implicit assumptions or hidden tendencies. The original test was devised by Sriram and Greenwald in 2009 [10] and used to uncover implicit biases in people’s perception of race, among others. As topics become increasingly and quickly polarised in the news, e.g., discussions around climate change or gun ownership, we needed an effective way to devise new association tests by integrating crowdsourcing into the creation and validation of topic-specific tests. This technique allows researchers to devise new association tests on emerging topics and collect ground truth on people’s tendencies where public opinion may differ from private truths.

In our subsequent work, we investigated the occurrence and manifestations of cognitive biases. A recent study looked into the physiological expression of cognitive biases, i.e., using functional near-infrared spectroscopy (fNIRS), a noninvasive optical imaging technique, to measure changes in oxygenated and deoxygenated hemoglobin concentrations within the brain. Such changes act as an indication of which parts in the brain exhibit activity and we were able to pick up on higher neural activation levels in individuals who were consuming media content that did not match their own opinion on a topic, i.e., were incongruent with their ideological stance. These were thus indicators of the occurrence of cognitive dissonances, which we can subsequently use to build classifiers to determine whether people fall victim to confirmation bias or not when consuming news items.
This ability to measure both the occurrence and impact of cognitive biases provides researchers with a tool for examining how user interfaces and algorithms influence opinion formation. When systems are equipped to detect these biases, it becomes possible to develop strategies to address and alleviate their effects. Consequently, interventions can be devised and tested to assist users in recognizing and countering cognitive errors resulting from their biases, fostering a more critical interaction with computing systems and information. In an era marked by the spread of misinformation, gauging users' attitudes and responses to potentially biased information aids in the creation of more effective information consumption habits, breaking down filter bubbles, and encouraging exposure to a wider range of ideas and viewpoints. Promoting critical thinking and informed decision-making is essential for fostering a diverse and healthy public discourse, with the potential to combat the spread of misinformation and the risk of user manipulation.

We have since organised a range of workshops on this topic at CHI [2, 4, 5] and as a Dagstuhl Seminar [6] in an attempt to create a community of transdisciplinary researchers addressing the risks of exploiting cognitive biases [3]. When addressing these issues, just like with the exploration of AI integrations in services and products more generally, we can adhere to human-centred design methods. Human control is essential while rolling out such technologies and we can resort to a long list of well-trodden HCI methods, including user observations, stakeholder engagement, usability testing, iterative refinement, and continuous evaluation to make sure such technologies are safe, reliable, and trustworthy.

# 6 Conclusion

The use of AI in products and services has become integral to our daily lives, influencing our choices and decision-making. While AI enhances user experience and convenience, it also raises questions about autonomy and dependence. A human-centred AI design framework, proposed by Ben Shneiderman, emphasizes the complementary nature of human control and automation. As AI technologies advance, they become more accessible, empowering designers to harness their potential. However, the widespread integration of AI also brings ethical concerns, such as the risk of manipulation and the importance of transparency. Overall, intelligent products aim to amplify and enhance human capabilities while considering the balance between automation and human control.

# References

1. Boonprakong, N., Chen, X., Davey, C., Tag, B., Dingler, T.: Bias-aware systems: exploring indicators for the occurrences of cognitive biases when facing different opinions. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. CHI 2023, Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3544548.3580917
2. Boonprakong, N., et al.: Workshop on understanding and mitigating cognitive biases in human-AI collaboration. In: Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing, pp. 512–517. CSCW 2023 Companion, Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3584931.3611284

3. Boonprakong, N., Tag, B., Dingler, T.: Designing technologies to support critical thinking in an age of misinformation. IEEE Pervasive Comput. (2023)

4. Dingler, T., Tag, B., Karapanos, E., Kise, K., Dengel, A.: Workshop on detection and design for cognitive biases in people and computing systems. In: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. pp. 1–6. CHI EA 2020, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3334480.3375159

5. Dingler, T., Tag, B., Lorenz-Sreen, P., Vargo, A.W., Knight, S., Lewandowsky, S.: Workshop on technologies to support critical thinking in an age of misinformation. In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. CHI EA 2021, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3411763.3441350

6. Dingler, T., Tag, B., Vargo, A.: Technologies to support critical thinking in an age of misinformation (dagstuhl seminar 22172). In: Dagstuhl Reports, vol. 12. Schloss Dagstuhl-Leibniz-Zentrum für Informatik (2022)

7. International, S.: Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles (sae j3016) (2021)

8. Parasuraman, R., Sheridan, T.B., Wickens, C.D.: A model for types and levels of human interaction with automation. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 30(3), 286–297 (2000)

9. Shneiderman, B.: Human-Centered AI. Oxford University Press, Oxford (2022)

10. Sriram, N., Greenwald, A.G.: The brief implicit association test. Exp. Psychol. 56(4), 283–294 (2009)