1

Running header

GHOST IN THE MACHINE
Addressing the consumer harms of generative AI
Norwegian Consumer Council

Ghost in the machine

JUNE 2023

June 2023

TABLE OF CONTENTSEXECUTIVE SUMMARY							

51 – INTRODUCTION

6

1.1

An overview of generative artificial intelligence7

1.1.1 Examples of generative AI models		

8

1.1.2 The generative artificial intelligence actor chain

11

1.1.3 Open source or closed source models

12

1.1.4 General purpose AI

12

Consumer applications 		

13

1.2

2 – HARMS AND CHALLENGES OF GENERATIVEARTIFICIAL INTELLIGENCE

14

2.1

Structural challenges of generative AI 		

15

2.1.1 Identifying the concrete risks of generative AI		

15

2.1.2 Technological solutionism		

17

2.1.3 Concentrating power in the hands of big tech17

2.1.4 Opaque systems and lack of accountability 		

19

Manipulation 		22

2.2.1 Mistakes and inaccurate output

22

2.2.2 The personification of AI models

23

2.2.3 Deepfakes and disinformation25

2.2.4 Detecting AI-generated content

27

2.2.5 Generative artificial intelligence in advertising28

Bias, discrimination, and content moderation		29

2.3.1 Bias in training data		

29

2.3.2 Content moderation		

30

Privacy and data protection		31

2.4.1 Privacy challenges related to data sets used for model training31

2.4.2 Privacy challenges related to generated content

32

2.2

2.3

2.4

2.5

Security vulnerabilities and fraud 		

2.6

Replacing humans in consumer-facing applications with generative AI,

2.7

2.8

2.9

32

wholly or in part33

2.6.1 Challenges related to combining human- and automated decision-making

33

Environmental impact 		

34

2.7.1 Climate impact

34

2.7.2 Water footprint		

36

2.7.3 Greenwashing & hopes for Green AI

37

Impact on labour		37

2.8.1 Labour exploitation and ghost work37

2.8.2 Labour automation and threats to jobs		

38Intellectual property		38

3 – REGULATIONS			

40

3.1

Data protection law 		

45

3.1.1 Data subject rights		

46

3.1.2 The Italian DPA’s decision concerning ChatGPT		

47

Consumer law		48

3.2.1 Consumer law in a U.S. setting		

49

General product safety law 		

50

3.3.1 The General Product Safety Directive50

3.3.2 The General Product Safety Regulation 		

50

3.4

Competition law 		

51

3.5

Content moderation		

51

3.6

The draft Artificial Intelligence Act 		

52

3.6.1 The EU Commission’s proposal 		

52

3.6.2 The EU Council’s position on the AIA

53

3.6.3 The EU Parliament’s position on the AIA 		

53

3.6.4 The Artificial Intelligence Act must protect consumers

54

3.2

3.3

3.7

3.8

Liability		

54

3.7.1 Product Liability Directive

54

3.7.2 Revised Product Liability Directive

55

3.7.3 AI Liability Directive

55

Industry standards and guidelines 		

56

4 – THE WAY FORWARD

57

4.1

Consumer rights principles that are key for safe and responsible AI		

59

4.2

Policy recommendations 		

60

4.2.1 Calls to action and empowerment of enforcement agencies

60

4.2.2 Decision makers – strategic measures61

4.2.3 New legislative measures

61

ENDNOTES		Ghost in the machine – Addressing theconsumer harms of generative AI
Norwegian Consumer Council
June 2023
www.forbrukerradet.no/ai
Design by: Von Kommunikasjon

65

5

Executive summary

Executive summary
There has been an explosion of consumer facing generative artificial intelligence services. These applications
can be used to generate synthetic text, images, sound,or video that closely resemble human-made content. As
generative AI systems become integrated into popular platforms and tools, adoption of the technology ispoised to keep rising. Meanwhile, a number of emerging
challenges have spurred numerous debates about how toensure that generative AI is safe, reliable, and fair.

In chapter two, we summarize various current andemerging challenges, risks, and harms of generative AI.
This includes challenges related to










This report is a contribution to these discussions, andaims to provide policymakers, lawmakers, enforcement
agencies, and other relevant entities with a robust starting point to ensure that generative AI does not come at
the expense of consumer and human rights. We cannot
know for certain how the technology will develop in themonths and years to come but believe that the directionof technological advancement should happen on society’s terms. Therefore, we present a number of overarching principles that can help define how generative AI
systems can be developed and used in a consumer- andhuman-centred way.
We also strongly urge governments, enforcement agencies and policy makers to act now, using existing laws
and frameworks on the identified harms that automatedsystems already pose today. New frameworks and safeguards should be developed in parallel, but consumers
and society cannot wait for years while technologies arebeing rolled out without appropriate checks and balances.









Chapter three contains an overview of the patchwork ofexisting and upcoming rules and regulations that may
apply to the development, deployment, and use of generative AI systems. This is mostly centred around EU legislation, but with some references to ongoing processes in
the United States.
The final chapter contains numerous recommendations
on how to address the problematic issues of generative
AI. This includes:The first chapter of this report provides an explanationof generative AI and its uses, alongside several examples
and illustrations.

Norwegian Consumer Council

Ghost in the machinePower, transparency, and accountability,
wrong or inaccurate output,
using technology to manipulate or mislead
consumers,
bias and discrimination,
privacy and personal integrity,
security vulnerabilities,
automating human tasks,
environmental impact,labour exploitation.







Enforcement of existing laws andregulations,ensuring sufficient resources for
enforcement bodies,stronger consumer protection,robust government policies,
new legislative measures,strong obligations that cover developers
and deployers of generative AI systems.

June 2023

1. INTRODUCTION

7

1. Introduction

Consumer facing artificial intelligence systems havebeen around in various forms for decades, and are used
for personalizing social media, filter emails, recommend
streaming content, text translation, and much more.
Some of these purposes are benign and discreet, andmost people may never realize that they are interactingwith an AI-powered system.
A new wave of AI-powered systems in consumer facing
applications is fast approaching, with the mass deployment and adoption of generative artificial intelligence(‘generative AI’) systems. Generative AI is a subset ofartificial intelligence that can generate synthetic content
such as text, images, audio, or video, which can closely
resemble human-created content. Such systems arepoised to change many of the interfaces and content
consumers meet today.
In November 2022, a prototype of the chatbot ChatGPT
was released to the public. The application quickly
gained worldwide attention, becoming the fastest
growing digital service of all time within a month of its
release.1 In the following months, other services forgenerating text, images, sound, and video, were quickly
deployed and iterated upon, sparking a sort of arms race
for generative AI systems. Consumers were provided
access to these content generators directly in web interfaces, while companies started to embed the content
generators in their applications and services.
The sudden and widespread deployment and adoptionof generative AI systems sparked public discourse about
the promises and perils of the technology. The debate
has ranged from how generative AI may be used toincrease efficiency in the workforce and to spark cre-ativity, to how it can be used to spread disinformation,
manipulate individuals and society, displace jobs, andchallenge artists’ copyright.
The discussion about how to control or regulate thesesystems is ongoing, with policymakers across the world
trying to engage with the promises and challenges ofgenerative AI. This report is a contribution to these
discussions by providing an analysis of the most pressing
issues from the consumer angle, along with a number ofpossible solutions and ways to proceed from both a legal,
ethical, and political perspective. Although we do notpretend to have the answers to all the questions raised
by generative AI, we believe that many of the emerging
or ongoing issues can be addressed through a combination of regulation, enforcement, and concrete policies
designed to steer the technology in a consumer- andhuman-friendly direction.
As the development of generative AI seems to move at a
breakneck pace, the descriptions throughout this reportmust be seen as a snapshot of an emerging technology.
The report was written between February and May 2023,
and does not include any new information from papers
published after June 1st.
The Norwegian Consumer Council is a publicly funded,independent consumer organization, that represents
consumer interests. This report was written with contributions from BEUC, Miika Blinn from VZBV, Kris Shrishakfrom the Irish Council for Civil Liberties, Daniel Leufer
from Access Now, Jon Worth, Marija Slavkovik, and Anja
Salzmann from the University of Bergen.

1.1 An overview of generative artificial intelligence
Generative AI is a blanket term used to describe algorithmic models that are trained to generate new data, suchas text, images, and sound. While these applications relyon different types of input data, the general principles
behind how they are trained are similar. The emergenceof advanced generative AI is possible due to an enormous amount of content available on the internet, combined with advances in machine learning and computing
power.

Norwegian Consumer Council

Ghost in the machine

Generative AI models work by analysing large amounts
of information to predict and generate the next word
in a sentence, feature of an image, etc. This is done bydetecting patterns in and relationships between data
points in the training data, which in turn allows thesystem to replicate similar patterns to generate synthetic content, for example a piece of writing, music,
or a video clip. This process can also be described as acomplex ‘mash-up’ of content from the data the system
was trained on. In other words, they are predictive mod-June 2023

8

1.1 An overview of generative artificial intelligenceels that are trained to “connect the dots” between data
points in existing content to generate synthetic content.
Generated content is probabilistically and randomly generated based on certain input (or ‘prompts’), which areusually written by a human. Therefore, the output of any
given generative AI model is likely to be different for each
person prompting the model and may both resemble
patterns in the training data or appear to be something
completely new.
1.1.1

EXAMPLES OF GENERATIVE AI MODELS
There are various types of generative AI models, including large language models (LLM) that can respond to text
by generating new text, and multi-modal models thatcan generate more than one type of output or respondto more than one type of input, for example chatbots
that can also generate images when prompted to do so.
A short introduction of the most popular generative AI
models on the market today is presented below, accompanied by some relevant examples.

1.1.1.1 Text generators
Text generators are a type of generative AI model thatcan generate text passages based on predictive ­analysis,
which are built on large language models.2 These models
are usually trained on enormous amount oftext scraped from the internet, including
books, forums, news sites, social media,
etc. Text generators can be used for writing
essays, coding, chatbots, and augmenting
search engines, amongst other things. In
many cases, text generators are meant togenerate text that appears to be written bya human, for example by generating text
written in a first-person perspective, using
emojis, or by writing text indicating that it has
the capability to experience human emotions.
Some text generators are multimodal and cangenerate text based on images.
Although text generators have existed insome form for several years, for example
as prediction tools for typing text messages, the discussion around the technology

2

gained momentum during the autumn of 2022, with the
public release of ChatGPT, which is owned and operated by the company OpenAI (which is also the ownerof DALL-E, see below). ChatGPT3 is available online forthose who create a free account, while the more powerful ChatGPT4 model is available for a monthly subscription fee.3
In January 2023 Microsoft announced a major investment in ChatGPT and launched new features in the Bing
search engine powered by the technology.4 Microsoft
has announced that it intends to integrate ChatGPT into
its other services, including the Microsoft Office suiteof applications, for example to automatically take notes
during meetings in Microsoft Teams.5
Google has also developed a LLM that can generate text,
called LaMDA. In the wake of Microsoft’s investment in
ChatGPT, Google rolled out similar features to its searchengine with a text generator called Bard.6 Google isalso planning to introduce various AI-powered features
such as drafting and summarizing emails, as well as
brainstorming and writing documents in its Workplaceapplications.7

Poem about the consumer issues of generative AI, ChatGPT.

Large language models are sophisticated AI models that are designed tocan be used for tasks such as machine translation, sentiment analysis, hu-

generate text that resembles human language. They are normally trained

man-machine interaction, proofreading, and many other purposes.

on vast amounts of text sources to “learn” patterns and grammar. LLMs

Norwegian Consumer Council

Ghost in the machine

June 2023

9

1.1 An overview of  generative artificial intelligence

Meta has developed the LLM Galactica trained on scientific articles and materials, which is meant to “store,
combine and reason about scientific knowledge”. Afterthe model was released as a public demo in November
2022, the public-facing demo was quickly removed due
to generating text containing multiple errors and biases.8
In February 2023, Meta released another LLM, called
LLaMa (Large Language Model Meta AI). LlaMa is an open
source model, which was initially released to researchers
based on an access application process. In March 2023
the model was leaked on a public message board, meaning that anyone with a relatively powerful computer can
download, use, and adapt the model.9
There are also several open source LLMs that are developed and maintained by smaller actors. For example, thetext generator BLOOM is available through the company
Hugging Face,10 while StabilityAI has released open models under the moniker StableLM.11
1.1.1.2 Image generators
Generative AI models that are trained to generate images
can be collectively classified as image generators.
They can create images fromtext prompts (‘text to image’) orfrom existing images (‘image
to image’). Image generators
work by analysing huge amountsof existing pictures, such as
photographs, paintings, etc.,
which are often scraped fromvarious sources on the internet.
By training the algorithm onthese data sets, the model cangenerate images of different
objects (‘a chair’, ‘a train’), people(‘a young woman’, ‘Jerry Seinfeld’), and styles (‘impressionism’,‘in the style of Edward Munch’).
The most widely used image
generators as of June 2023 are
Midjourney,12 DALL-E,13 and Stable Diffusion.14

Midjourney is available through the chat service Discord.
It is possible to join the official Midjourney Discord server, to ask a Midjourney bot to “imagine” a picture basedon various prompts. For example, the person prompting
the system could type “/imagine hyper realistic photo ofpolitical advisor writing a paper on generative ai, in an
open office plan”. The bot responds in the chat with four
generated pictures. While Midjourney was free to try
for the first few months after its release, for a limited
number of generated images, it has since become a paid
subscription service.
The company Midjourney Inc. owns the generative AI
model that generates the images and runs and controls
both the model itself and the servers that it is hosted
on. This means that the company can restrict access,change the model, and add content filters to control
what kind of images the model can and cannot generate.
The generative AI model DALL-E is accessible throughthe website of its owner OpenAI. Individuals can createan account and receive a limited number of tokens eachmonth that can be used to generate images. Images are

Hyper realistic photo of politicaladvisor writing a paper on generativeai, in an open office plan, Midjourney.

Norwegian Consumer Council

Ghost in the machine

June 2023

10

1.1 An overview of generative artificial intelligenceAn oil painting by Munch of a policy advisor writing apaper on generative AI, DALL-E.

A photo of a consumer policy advisor writing a paper about theconsumer challenges of generative AI, in an open office plan,
Stable Diffusion 1.5.

generated by entering different prompts into the website
interface. If the person runs out of tokens, they can pay toreceive more. Like Midjourney, the model behind DALL-E
is owned, controlled, and operated by its parent company.

text to voice). Such models are trained on existing voice
data, music, etc. Audio generators can be used to create
AI-generated music15 and voices, and there are models
capable of recreating the voice and pitch of individual
individuals.16

The generative AI model Stable Diffusion is developed by
the company Stability AI. Unlike Midjourney and DALL-E,
Stable Diffusion is an open source model that can be
freely downloaded by anyone, and does not require asubscription or access to the internet to use. Once thenecessary software is downloaded, unlimited images
can be generated locally. Running Stable Diffusion locally
only requires a computer with relatively powerful consumer-grade graphics card.
Stability AI only trains and distributes the basic models
for Stable Diffusion, while it is possible for anyone withaccess to them to continue training and developing newmodels that are based on the original Stable Diffusion
models. These new models can then be distributed toothers. In practice, this means that the company does
not control the model nor its output.

For example, the company ElevenLabs has released amodel that allows anyone to convert short text input into
voice clips, in a selection of different voices.17 Microsoft has announced the generative AI model VALL-E,
which the company claims can generate realistic voices
based on a three second voice sample.18 As of May 2023,
VALL-E has not been released to the public.
1.1.1.4 Video generators
Video generators can be used to create video clips basedon text prompts (text to video), images (image to video),or existing clips (video to video). As it is more complicated to generate authentic looking video footage thanto make still images, this technology is somewhat lessdeveloped at the time of writing.

1.1.1.3 Audio generators
Audio generators use generative AI technologies tocreate audio clips based on text prompts (for example

Norwegian Consumer Council

Ghost in the machine

However, this may change in the near future, as several
major companies are actively working on models forgenerating video.

June 2023

11

1.1 An overview of  generative artificial intelligence

Meta has developed a model to turn short texts into video
clips,20 and Google has announced a similar system.21 As
of May 2023, neither of these systems have been madeavailable to the public.

The data set assembler collects and systemizes thenecessary data to train a generative AI model. There
are numerous available open source data sets that havebeen compiled and labelled by scraping a huge number ofsources online. In many cases, such data sets are compiled
for research purposes, and made freely available. A company developing a generative AI model can thus train theirmodel on datasets that someone else has assembled.
The developers of generative AI models create a baselinemodel, which may then be trained and tuned for certain
more specific contexts or applications by downstream developers. In some cases, this fine-tuning of the model may
be done by the same actor as the training of the baselinemodel, while in other cases, the fine-tuning may be done bya separate actor entirely. This may be another company, orin the case of open source models, the model can be finetuned by anyone with a relatively powerful computer.

Stability AI, the company behind Stable Diffusion, has
released a model for generating animations from text
prompts and images.22 The company Runway has released a mobile app that can be used to generate shortvideo clips from existing videos.23

To further complicate the matter, as general-purpose
generative AI models are being integrated into otherapplications, the company or entity deploying the system
may be separate from the company developing the model and/or fine-tuning it.

THE GENERATIVE ARTIFICIAL INTELLIGENCEACTOR CHAIN

Finally, there are end users, engaging with the deployedmodel. In consumer use cases, the consumer will typically be an actor, by prompting the model to generate, forexample, a text or an image. Consumers may also indirectly encounter generative AI systems when interactingwith a business, for example if a customer service agent
uses a text generator to generate answers to consumer
queries, or if a consumer is prompted to make certain
queries based on AI-generated recommendations.

Frame fromvideo generated
in Make-A-Video,
Meta AI.19

1.1.2

From assembling data sets for training models to deploying and prompting generative AI systems, there arepotentially many different actors. These actors may all
influence the system or how it is used in various ways.
The relevant actors are shown in the illustration justbelow.
Each box represents a different actor in the actor chain.
These actors may all be within one organisation but willoften be spread across several organisations. The arrows
are only pointing one way to maintain a simple representation; it is of course possible that there could be
feedback loops between different actors.

Data setassembler

Developer ofbaseline model

There are numerous actors in the generative AI system
actor chain. It is crucial to understand the relationshipbetween these actors to understand how generative AI
should be regulated, and at which point the actor chain
different harms arise.

Downstream
developer

Deployer

End user

Illustration of different actors in the generative AI actor chain.

Norwegian Consumer Council

Ghost in the machine

June 2023

12

1.1 An overview of generative artificial intelligence

1.1.3

OPEN SOURCE OR CLOSED SOURCE MODELS
There are significant differences in how generative AI
models are distributed and controlled. Many models
are proprietary closed source models that run on cloudservers controlled by the owner of the system (the “system owner”). This means that consumers can access themodel through the internet, and that the system provider
can change the model at any time, add content filters, restrict access, etc. For closed systems, the system ownerprovides the processing power required both to train themodel, and to generate synthetic content.
For a closed source generative AI model, it is not possible to know how the model works, what data it wastrained on, and how parameters are weighed, unless thecompany behind the model publishes sufficient documentation or provides access to third party auditors,
enforcement agencies, or researchers. In many cases,
such information may be kept secret due to security and/or business interests.

inspected by anyone. However, this does not necessarilymean that they can understand how the model works, assufficiently complex models may be more or less impossible to understand the inner workings of.
Once an open source generative AI model is released tothe public, there is practically nothing the developer ofthe baseline model can do to influence how the model
functions. This means that any content filters and otherartificial limiters placed on the model may be altered or
removed by downstream developers or deployers. This
creates both benefits and significant drawbacks, which
will be elaborated on below.
1.1.4

On the other hand, some generative AI models are
released as open source, which may take various forms.
Different parts of the system, such as the data set,source code, model parameters and weights, may bemade available to third parties.
When the source code is made available to the public, it
can be used, studied, tested, modified, and distributedby anyone. This means that it can be inspected for errors
and vulnerabilities. It can also be improved and iterated
upon collaboratively. Using, modifying, and distributing
open source software is generally governed by license
terms. Given the open source software and the data set,anyone with sufficient computing resource could reproduce the generative AI model, although the computing
resources needed to do so are significant enough that inpractice it will likely be limited to large companies.

GENERAL PURPOSE AI
While some AI models are designed with a specificpurpose and use case in mind, such as early discovery ofcancer cells, many generative AI models are examples
of so-called ‘general purpose artificial intelligence’. Thismeans that the basic system, such as a large language
model, is trained to be able to respond to a vast variety
of situations and interactions and can be adapted to be
used in new contexts.
Unlike a model with a specific purpose, it is extremely
difficult or impossible for the developers of a general
purpose AI model to foresee the possible uses and abuses of the technology. This makes it particularly importantthat such models are subject to technical, scientific,
legislative, and regulatory scrutiny before they arewidely adopted. However, applications such as ChatGPT
have already been released to the wider public without
rigorous evaluation, impact assessment or scrutiny,
while being increasingly opaque and inaccessible to third
party auditors and researchers.24 It is worth considering
whether this leads to a desirable future, considering themany harms touched upon in chapter 2 of this report.

However, for generative AI systems to be truly open
source, the model itself should be made available to the
public. In such a case, applications developed based onthese models might also be released as open source application, such as the image generator Stable Diffusion,or it can be adapted into a closed source application.
Open source generative AI models can be downloaded byanyone. With a powerful enough computer, it can be used
to generate data and update the model as one pleases.
The source code, parameters, etc. of such models can be

Norwegian Consumer Council

Ghost in the machine

June 2023

13

1.2 Consumer applications

1.2 Consumer applications
Several different types of generative artificial intelligence are publicly available to consumers today. Manyof these are readily available to use by anyone with aninternet connection, and do not require expert technicalknowledge to use. Some of them are directly accessible
through web interfaces, while generative AI technology
is also increasingly being integrated into other services
such as online search, learning and administration software, and social media.
As of May 2023, the most popular consumer uses forgenerative AI models are text and image generation.
However, with major consumer-facing companies suchas Microsoft, Meta, and Google investing heavily in thetechnology, the use cases are likely to expand in the
coming months, as generative AI models are implemented in various services.
For example, text generators can be a useful tool tostreamline and/or optimize mundane tasks, functioning
as a kind of multi-purpose digital assistant. This may
include changing internet search functions, automating
certain tasks such as writing code, transcribing voicemessages, or personalizing services in various ways.
While such applications may be useful and efficient incertain contexts, there are also significant risks anddrawbacks, which will be explored further in the following chapters.
As the technology is developed and adopted, generative
AI may be used to automate tedious and time-consuming processes that previously had to be done manually,for example by writing concise texts, filling in forms,
generating schedules or plans, or writing software code.
It has the potential to make services more cost-efficient,which may also lower costs for consumers, for example
when soliciting legal advice.25 On the other hand, theproliferation of low-cost AI-generated content may replace human labour and human-generated content, thus
lowering the quality of consumer-facing services suchas customer support. The technology also opens new
avenues for consumer manipulation in areas such as advertising or product recommendations and can facilitate
or obfuscate discriminatory practices.

Norwegian Consumer Council

Ghost in the machine

June 2023

2. HARMS AND
CHALLENGES
OF GENERATIVEARTIFICIALINTELLIGENCE“From violations of privacy and personal integrity to
the creation of fraud and misinformation, generative
AI models introduce vast risks and challenges,
while turbocharging others.”

15

2. Harms and challenges of generative artificial intelligencethat simulated human interaction, using rule-basedalgorithms.26 People who interacted with ELIZA attributed human attributes and emotions to the model,
even though they were informed that the system had nosuch capability, mirroring some use cases of generative
AI-powered chatbots today.

There have been several controversies surrounding thedevelopment and use of generative artificial intelligence. From violations of privacy and personal integrityto the creation of fraud and misinformation, generative
AI models introduce vast risks and challenges, whileturbocharging others. Concrete and highly relevant
examples of this are chatbots and search engines providing incorrect but convincing information, the abuse ofcheap labour in the global south for content moderation,
and a significant environmental impact due to resource
consumption. It is essential that these problems are sufficiently addressed by enforcing, applying, and establishing laws and regulations that serve to protect consumers
from various negative consequences.

Issues related to content moderation, algorithmic bias,
privacy, and disinformation have been debated at almost
every junction as digital technology evolves and is widelyused. However, the deployment and public adoption ofsystems such as ChatGPT, both for technically adept
consumers and the general public, alongside its easeof use and wide-scale availability, means that many ofthese issues have become urgently relevant to analysefrom a consumer perspective. As described in the following chapters, a number of these issues may be addressable by enforcing applicable laws and regulations, whileother may require other solutions or remedies.

The issues discussed throughout this report are often
not new or unique to generative AI. Algorithmic computer systems have existed for a century, while the technology popularly referred to as artificial intelligence has
been around since the 1950s. In the 1960s, the computerscientist Joseph Weizenbaum created ELIZA, a model

2.1 Structural challenges of generative AI
At a basic level, generative AI models are fundamentallydesigned to reproduce existing material, although in
potentially novel ways. This means that such models are
inherently prone to reproducing existing biases and power structures. Therefore, while the models have no understanding, mind, or intention of their own, the decision
to develop, deploy and use them is inherently political. It
is not sufficient to ascribe neutrality or objectivity to theoperations or outputs of a generative AI model, because
its training data and algorithms stems from human beings, with all that this entails.

2.1.1

As generative AI models are being introduced into all
sectors of society, so far with little or no regulatory
oversight, there are fundamental issues that need tobe addressed. Generative AI models are dependent onlarge amounts of data that is taken from a multitude ofsources, usually without the knowledge or consent of theoriginator of the data, be it a piece of art, a news article,
or a selfie. Information is siphoned and gathered to be
used in different ways, with an end goal of enriching asmall number of companies. This raises questions ofvalue distribution, usage permission, privacy, accountability, intellectual property, and human rights.27

Norwegian Consumer Council

Ghost in the machine

IDENTIFYING THE CONCRETE RISKS OF
GENERATIVE AI
As with any new technologies, the discourse around generative AI is muddled with a mix of facts, concerns, and a
lot of hype and enthusiasm.28 Many AI systems are being
touted as being capable of solving almost any task, often
without evidence to back up the claims, a phenomenon
that can be described as ‘AI snake oil’.29 When addressing
problematic and hazardous issues with the technology, it
is important to be able to sort facts from fiction.

“At a basic level, generative AI
models are fundamentally designed
to reproduce existing material, although in ­potentially novel ways. Thismeans that such models are inherently
prone to reproducing existing biases
and power structures.”

June 2023

16

2.1 Structural challenges of generative AI“There are serious concerns that focusing on potential
long-term scenarios are drawing attention away from manyof the current pressing issues of generative AI, potentiallyleaving these issues insufficiently regulated.”

Heavily publicized warnings about the dangers of artificial intelligence have concentrated on hypotheticalrisks of developing an artificial general intelligence (AGI),
meaning a system that is able to perform intellectualtasks that are comparable to the ability of human beings.
Theoretically, such systems should be able to think andreason, and be able to perform a broad range of tasks
that equal human capacity for thinking. This differs considerably from generative AI models, which have no suchcapacities. As AGI systems do not currently exist, andthere are serious disputes about whether they can everbe realized, such systems will not be further consideredin this report.
There have been calls for voluntary moratoriums or‘pausing’ of developing generative AI models. Some ofthese calls have focused on a potential future where AI
systems have become so powerful that they pose an existential threat to humanity.30 While such calls acknowl-

Norwegian Consumer Council

Ghost in the machineedge issues related to accountability, safety, and control
over AI systems in general, there are serious concernsthat focusing on potential long-term scenarios are drawing attention away from many of the current pressing
issues of generative AI, potentially leaving these issues
insufficiently regulated.31
The argument that a hypothetical general artificial intelligence is an existential threat to humanity implies thatconcerns about current issues such as discrimination,
privacy, and fairness, are inconsequential and marginal.32
In other words, the narrative concerning a potential “AI
supermind” may serve as a distraction from pressing
issues that are already present in today’s application ofgenerative artificial intelligence. It is crucial that narratives about hypothetical existential threats to humanity
do not come in the way of proposing concrete solutions
to the very real issues posed by the technology thatexists today.33

June 2023

2.1 Structural challenges of generative AI

17

2.1.2

TECHNOLOGICAL SOLUTIONISM
Artificial intelligence is often lauded as the solution fora vast number of issues across sectors, from healthcare
and public administration to legal assistance. While thisnarrative is attractive to both private enterprises looking
to sell software solutions, and to policymakers searchingfor simple remedies to political or regulatory ailments, itneeds critical examination.

If such technological quick fixes are adopted at theexpense of investment into proven effective measures,
which are often costly and difficult to implement, thismay come at significant costs to marginalized groups
who risk being deprived of effective treatment andmeasures because their issues are purportedly being
addressed by the use of technology.
2.1.3

The belief that almost any issue can be improved orsolved using technology is known as ‘technologicalsolutionism’. A term coined by the tech critic Evgeny Morozov, technological solutionists tend to gloss over complex and multifaceted social problems in favour of simple
mathematical or engineering solutions.34 This reductionist belief is attractive to service-providers because
it allows them to advertise miracle cures – AI snake oil– and to policymakers because technological quick fixes
are tangible and usually appear more cost-efficient than
examining complicated and often deep-rooted social andpolitical conflicts or inequalities.
As Morozov points out, technological solutionism isdangerous because it often simply does not work. By
presenting multifaceted and complex issues as a mereengineering issue to be solved in a lab, solutionism misrepresents social problems and misses the underlying
causes. When presented as problems that can be solved
by technology, solutionists tend to disregard the social,political, and cultural context that is the backbone of oursocieties.
When considering the proliferation of artificial intelligence models that are rapidly being deployed acrosssectors, it is worth keeping the folly of technologicalsolutionism in mind. This is particularly important if
generative AI models or similar technologies are being
pushed as a remedy or solution to inequalities, such asproviding access to mental health tools to people whootherwise would not be able to afford it. While it may
appear attractive to outsource mental health care to alarge language model that can be deployed and accessed
at a relatively low monetary cost, this approach risks
reducing the complexity of mental health and the valueof human contact to a question of predictive analysis
and language modelling.35 Similarly, before deciding todeploy a text generator as a solution for overworked casehandlers in the public sector, for example, it is crucial toconsider the context and causes of the problem, rather
than adopting developing technologies as a blanket
solution.

Norwegian Consumer Council

Ghost in the machine

CONCENTRATING POWER IN THE HANDSOF BIG TECH
At the base of the discourse around generative artificial
intelligence is a question of power. Generative AI models
are products of cultural and political contexts, a context
that is embedded in everything from the decision to develop the model, the choice of training data, the tuning ofmodels, and the given purposes for deployment. As such,the already powerful can potentially entrench existing
power structures through the technology, while thedisenfranchised will remain so unless there is outside
intervention. This becomes apparent when a generative
AI model generates biased or discriminatory content, butalso manifests in aspects such as content moderation
practices and in who has access to the systems.
As generative AI models are often trained on data collected from any available sources, some actors are raising questions about whether private companies shouldbe allowed to use the collective knowledge of humanity
to turn a profit. The vast amount of information that canbe found openly available online can be described asa ‘digital commons’, as it is a body of resources wherepractically everyone is a contributor, from individual
pieces of data to the public infrastructure of the internet. If the digital commons are siphoned to develop andtrain proprietary models, this raises ethical concerns
about how value generated on the basis of these common resources should be distributed.36 These concerns
extend to data governance issues regarding who should
control how data is used, such as if a tech company
wants to commercialize AI models trained on indigenous
languages.37
The question of who controls the development andtraining of generative AI models and how they are used
is of fundamental importance. Those who control thetechnology have significant potential to create dependencies, set the terms of use, and decide who hasaccess. This entrenchment of power creates overarching concerns about leading tech companies becoming
gatekeepers that can exclude rivals and otherwise abuse
their increasingly dominant market positions.38 While

June 2023

18

2.1 Structural challenges of generative AI

open source models may lower the barrier of entry forcertain types of generative AI,39 such models are stilllargely dependent on a foundational model that has beendeveloped by actors with access to significant computing resources and training data.
This means that already dominant technology companies
such as Microsoft, Google, and Meta, are well positioned
to seize the market for generative AIs. With proprietaryclosed model, the system owner has control over who
can access the technology, what it costs, its features,and how it may be used. This may in turn affect academia, attracting what could have been independent
researchers to work within the closed domains of big
technology companies, where they have access to stateof-the-art technology in the field. Overall, this meansthat the tech giants can ­further leverage their dominantpositions across different online markets in the field ofgenerative AI. 40

already used by millions across the world. For example,
by rolling out its chatbot Bard as part of its search engine, Google already has a massive global user base thatcan be leveraged to boost the adoption of the chatbot.
Similarly, as Microsoft implements ChatGPT-based models into its Office suite of applications, the company alreadyhas a user base that competitors can only dream of.
Companies can also leverage their market position bymaking the usage of a generative AI model dependenton using a different service from the same company, by‘bundling’ services together. For example, to have accessto Microsoft’s Bing chatbot functionality, consumers
must use Microsoft’s Edge browser.41

The integration of generative AI models into services
such as search engines can also significantly limit
consumer choice. For example, in a regular online searchengine, the consumer ispresented with numerous
search results that theyWith only a few generative“Those who control the technology
may choose between. If the
AI models available on thehave significant potential to create
search engine is replaced
market, these models are
by a text generator that
integrated into a variety of
dependencies, set the terms of use,
provides a single answer
services, providing theand decide who has access.”to any query, this potenmodel owners with signifitially limits the informationcant power. Models can beavailable. If similar models
patched or otherwise modiare used for online shopping, this creates new avenues
fied, functionality can be added or removed, and content
for platforms to self-preference products, by ensuring
can be banned, filtered, or otherwise restricted. If thethat the platform’s preferred product is the only or thesystem owner sets the terms for how its technology may
primarily suggested purchase. If the consumer queries
or may not be used, end users or third-party companies“what is the best coffee machine for to suit my needs?”,
integrating the model are at the mercy of the owner.
it will be necessary to monitor and control how a chatbot
or “shopping assistant” lands at a particular result orSome competition concerns may be somewhat alleviatedrecommendation.
in the case of open source generative AI models, which
are not necessarily beholden to the business model, ob2.1.3.1 Walled gardens and its downstream effects
jectives, or whims of a model’s original creator. However,In order to maximize consumer engagement, many digeven in such cases, many companies do not have theital service providers have a financial incentive to keepmeans to compete with big tech firms in offering generaconsumers on their platforms as long as possible. This
tive AI solutions to consumers. Large companies benefit
goal can be attained by integrating and bundling as many
significantly from network effects, as more users means
services into the platform as possible, while creating
more data, which leads to better services. In cases
barriers such as not providing service interoperability towhere models are further trained on consumer interacdisincentivize consumers to leave the platform. Plattions or feedback, the companies can further improveforms and services designed to keep the consumer fromand fine-tune the models at a rate that is unattainable forleaving are called ‘walled gardens’.42
smaller competitors.
Dominant actors can further entrench their power byintegrating generative AI into their own services that are

Norwegian Consumer Council

Ghost in the machine

The integration of generative artificial intelligence into
various platforms already seems to facilitate a walled

June 2023

19

2.1 Structural challenges of generative AI

garden approach that may have serious anticompetitiveeffects on both direct competitors to the large online
platforms, and across markets.

tized as parts of cost-cutting measures. It is also well-established that power concentration among a few actors isseldomly beneficial to healthy consumers markets.

Snapchat is for instance introducing recommendations
for restaurants or recipes in its AI-chatbot,43 which would
reduce consumers’ need to access other services, suchas traditional search engines, for these kinds of queries.
This is likely a sign of things to come, as major platforms
are slated to integrate generative AI models into theirservices. As companies compete to develop “killer apps”,
services that integrate as many functions and purposes as possible, these problems are set to exacerbate.
Newcomers may find it increasingly difficult to provide
stand-alone services to consumers, as they have fewer
reasons to exit their applications. This would serve toconcentrate power with already established actors,
effectively harming the consumer market.

2.1.3.2 Data colonialism
If generative AI models are trained on data sets thatwere indiscriminately scraped from the internet (digitalcommons), this may also entail large amounts of datafrom indigenous and other minority groups. The information can then be repackaged and used in new ways, forexample to sell technology or services based on the databack to the groups from whom the data originated. The
process by which organizations and corporations claim
ownership over data produced or harvested from people
is called ‘data colonialism’. The concept of data colonialism is highly relevant when discussing the operations ofgenerative AI models.
For example, indigenous communities in New Zealandhave expressed concerns regarding the developmentof large language models being trained on hundreds ofhours of Māori indigenous language. Community leaders
and researchers fear that “if Indigenous peoples don’t
have sovereignty of their own data, they will simply be
re-colonized in this information society”.46

The integration of generative AI into search engines issparking major concerns for publishers and advertisers,
since such integration can effectively create walled
gardens.44 With traditional search engines, the consumer could search for a topic and be presented with a listof links to websites that contain information about the
topic. The consumer will then click on one or more of thelinks and be redirected to a website. Consequently, thewebsite owner generates revenue by displaying ads tothe consumer.
This dynamic may change with the introduction ofgenerative AI. Google is for example experimenting withintroducing generated, summarized content in its searchengine, which fills up the first page on a smaller screen(such as a phone).45 In short, if consumers can simply
ask a chatbot about a subject and receive answers inthe same interface, the incentive to visit a third-party
website is reduced. If the consumer does not visit thethird-party website, the originator of the content cannot
monetize the content by displaying ads.

The language harvested without consent can be distorted and lead to abuse and deprive communities of theirrights. According to indigenous communities, it is not for
Big Tech to play with such heritage.
2.1.4

A lack of traffic poses a problem to publishers, whose
content may be scraped to display information in throughthe chatbot interface, but who may struggle to monetize
the content. This can therefore have downstream effects
by reducing the incentive to produce quality content, potentially leading to content-production becoming automa-

OPAQUE SYSTEMS AND LACK OFACCOUNTABILITY
Models such as large language models are generally very
technologically complex, but they are not impossible tounderstand or explain. There are fundamental scientific principles relating to transparency, peer review andrigorous quality control that apply in fields such as thepharmaceutical and aviation industries, which should
also apply to developers of AI models. Information about
how training data is collected, how the data is labelled,
how testing is performed, what decisions are made regarding content moderation, and the environmental andsocial impacts of the models are just a few areas wheretransparency is necessary to ensure that risks are mitigated and that claims about the technology are accurate.

48 Hypothetical AI system that demonstrates human-level intelligence
and autonomy. Does not currently exist.

Norwegian Consumer Council

Ghost in the machine

June 2023

20

2.1 Structural challenges of generative AI“There are fundamental scientific principles
relating to transparency, peer review and rigorous
quality control that apply in fields such as thepharmaceutical and aviation industries, whichshould also apply to developers of AI models.”
2.1.4.1 Opaque systems reduce accountability
Unfortunately, there are already tendencies from certain
AI developers to close off their systems from external
scrutiny. For example, Google has committed to a policychange where the company will only share papers after
their research has been turned into products.47 Microsoft
researchers have made grand claims about its own AI
systems show signs of artificial general intelligence,48
while not providing researchers with access to themodel in order to verify or dispute the claims.49 Finally,
ChatGPT owner OpenAI has claimed that the company’s
AI systems, including what training data is used, how themodel works, etc., should not be open for external review
because giving others access would pose a competitionand safety risk.50

2.1.4.3 Actor chain transparency
The lack of transparency also becomes problematicwhen service providers implement third-party generative AI models into their services. This may increase therisk of errors or unexpected behaviour from the model.55
The developer of the baseline model does not necessarily
see or understand the downstream contexts in which
the model is used, while the service provider or otherdownstream developers do not sufficiently understandthe limitations of the model.

While the lack of transparency is an issue that appears
across the spectrum of the software industry, OpenAI’s
own description of the risks of their products tend toborder on existential, with its CEO Sam Altman statingthat the company is “scared” by the potential harms that
may stem from its own systems.51 Such claims as a pretence for closing down generative AI systems for external
auditing and review are worrying tendencies that canmask a number of downstream effects, and which poseenforcement agencies and researchers major challenges.
Researchers at Princeton University have claimed that
OpenAI might be misrepresenting the capabilities of
their systems, but this is impossible to prove due to thesystem being closed to external scrutiny.52 The researchers warn that this also significantly hinders attempts at
reproducibility of any claims made by the company.53
2.1.4.2 Trade agreements as barriers to transparency
While companies themselves are attempting to close off
their systems from external scrutiny, lawmakers may be
increasingly limited from requiring transparency by trade
agreements. Internal documents from the EU Commission show that digital trade agreements between the
EU and US limit European lawmakers’ ability to require

Norwegian Consumer Council

third-party access to the source code of
AI.54 Closed-door negotiations affecting
lawmakers’ ability to create a well-balanced and consumer-friendly market
are highly problematic. This bars civil society and other stakeholders from providing
important input and appears at odds withcrucial democratic principles.

Ghost in the machine

If the service provider is not privy to the data sets used
to train the models, or to how the model actually works,
the service provider will not be able to give the consumer
an explanation about why a certain output was generated. As supply chains for generative AI systems may
be complex, with one actor collecting and labelling data
sets, while it can be other actors developing the algorithms, training the model, or integrating it into services,it becomes difficult to attribute liability and accountability to the right entity. For the consumer, this may have anegative impact on the right to an explanation, as well as
contestability and general transparency obligations.
For example, the retail bank, payments, and shopping
service Klarna has announced a collaboration with
OpenAI, with plans to integrate ChatGPT into its services
to provide a “highly personalized and intuitive shopping
experience by providing curated recommendations”.56
If this system provides flawed recommendations for
products, or ranks products in a skewed way, it will beessential that consumers, not to mention enforcement
agencies, are able to access and assess data on how therecommendation system affects the consumer. This
becomes impossible if OpenAI as a third-party service
provider does not provide external actors the necessary
information about the AI system. Without such information, it is entirely plausible that the services and products should not be on the market at all.

June 2023

21

2.1 Structural challenges of generative AI

2.1.4.4 Opaque systems exacerbate consumer harms andhinder consumer rights
The general lack of transparency in some generative AI
systems may have significant effects on consumers. As
generative AI systems are being adopted by consumers
for various use cases, the potential for harms rises, aselaborated in other sections below. For example, many
text generators are prone to providing false or inaccurate
information. This may have direct effects on consumers,for example if a chatbot provides bad financial advice.
The often-complicated actor chains behind a consumer
facing generative AI system may also make it exceedingly
difficult for consumers to get in touch with the responsible
entity in case something goes wrong. This could also be
problematic when it comes to claims for compensation.
Without a certain transparency into how the system
works, such as limitations on the system’s intended use,alongside disclosures about possible inaccuracies, thepotential for harm becomes larger. The other harms thatare covered in the following subsections of this report
are exacerbated when consumers are kept in the dark of
the systems’ potential for harms and harmful uses.
It is important that companies provide transparent systems and application to consumers. However, the power
asymmetry between companies and consumers in digital
environments,57 means that any transparency measures directed at consumers to reduce harms must be
implemented in addition to other measures, rather thanas a stand-alone measure. The responsibility of ensuring
fair and legitimate use of generative AI must be on thecompanies, and never shifted onto consumers throughtransparency measures.
2.1.4.5 The limits and restrictions of corporate AI ethics
Ethical and legal considerations play a fundamental roleto ensure that models are developed, trained, deployed,

and used in a responsible way, from the development
stage and throughout the lifecycle of the model.
As ethical norms and values differ significantly depending on cultural contexts, it is also worth noting that thedecision of which ethical standards to consider andapply is a political choice. Similarly, legal frameworks are
not universal, which may prove a serious hurdle as generative AI models are rolled out on a global scale.
While many companies working on generative AI models
have employed AI ethics teams to help define guardrails
and red lines for AI development, there are doubts about
how effective this has been in cases where ethical concerns conflict with the company’s profit motives.
Famously, Google fired members of its AI ethics teamafter researchers from the team published the paper On
the Dangers of Stochastic Parrots: Can Language Models
Be Too Big?. The paper posed critical questions about
how large such models should be, alongside critical evaluations of inherent biases and the environmental impactof the models. After refusing to retract the paper, theresearchers were asked to resign from the company.58
Amongst the tech company layoffs in 2022/2023, AI
ethics or ‘responsible AI’ teams at companies including
Google, Twitter, Microsoft, and Meta were also laid off.
This raises concerns about whether ethical concerns
are ignored or heavily down prioritized by companies
competing in a generative AI gold rush.59
Some companies are calling for regulation of generative AI, notably OpenAI,60 seemingly wishing to abide bylawmakers’ requirements. At the same time, OpenAI has
threatened to leave the EU if the provisions of the new AI
Act are too strict.61 This indicates a desire from companies to shape regulations in accordance with their own
profit motives.

“As ethical norms and values differ significantly depending on
cultural contexts, it is also worth noting that the decision of which
ethical standards to consider and apply is a political choice.”

Norwegian Consumer Council

Ghost in the machine

June 2023

22

2.2 Manipulation

2.2 Manipulation

Bing pretending to feel good. (23.03.2023)

Generative artificial intelligence models have thecapability to generate synthetic content that closely
resembles real content, including dialogue, voices,
photographs, and video. Studies have shown that text
generators that simulate human dialogue can influencepeople’s feelings, dispositions, and opinions.62 As generative AI model become more powerful, the potential for
manipulation becomes greater.
Low quality content can also mislead or manipulate,both on purpose and because of low quality data andmodels. If a generative AI model produces inaccurate orfalse information, this can have harmful consequences
for consumers. If such models are deployed and used
maliciously, this may lead to consumers being tricked,
misled, or otherwise manipulated.
2.2.1

MISTAKES AND INACCURATE OUTPUT
Generative AI models are complex systems trained onvast amounts of material, which may give an impressionof infallibility. However, as the models do not “understand” context and the content it produces, they havea tendency to produce content that looks convincing
and correct but is factually incorrect. This particularly
applies to text generators.
For example, ChatGPT can produce text that looks very
convincing and fact-based but contains factual errors
or fallacies.63 This has led critics to call the system a“confident bullshitter”.64 Similarly, Google employees
has labelled the company’s own text generator Barda “pathological liar”.65 It can be difficult for the person
prompting the system to notice or reveal these errors ifthey are not already familiar with the facts of the relevant
subject. Some systems, such as Bing, cites sources
for the generated information, apparently to alleviate

Norwegian Consumer Council

Ghost in the machinesome of these issues. However, the models have beenprone to “make up” non-existing sources, either throughpresenting sources that do not in fact exist or presenting
sources that do contain the relevant content to support
the generated content.66
Mistakes and inaccuracies are exacerbated as generative AI models are plugged into the workstream indifferent areas. Amidst sinking revenue, soon after thewidespread adoption of ChatGPT, publishers were quick
to announce that they would start using the model forcontent production.67 However, when the news site Cnet
used a text generator to generate journalistic content,
it was soon discovered that the published output was
riddled with factual errors.68 There are also concernsthat the use of generative AI models as a replacement fortraditional internet search engines will make it significantly harder to identify inaccurate or incorrect information, while also having negative effects on information
literacy.69
As large language models become increasingly sophisticated, they can adopt more authoritative and convincing syntax. Combined with adjustments of answers toincrease persuasiveness and engagement, it becomes
more difficult to detect mistakes. While factual errors
may be ironed out through technological advancements,
this may also make it harder to know when information isincorrect. For example, if a LLM provided sophisticatedand accurate answers 99 times, it becomes harder forthe end user to know that it was inaccurate or completely
wrong the 100th time.
Inaccurately generated information could have harmful
consequences, both as standalone models and when
the generative AI is embedded in other systems. For

June 2023

23

2.2 Manipulation

example, if an AI-powered chatbot is used by a consumer
to solicit medical advice, and the advice is wrong, this
could lead to real life harm. Similarly, text generators arereportedly being used by consumers for mental healthpurposes, which may also have serious consequences,also because the models do not follow any ethical or legal
guidelines or rules.70 Finally, text generators that areused to find information about consumer rights may end up
providing false information that end up rendering the consumer unaware of or unable to exercise their legal rights.
In March 2023, the Portuguese government announced
that it would use an adapted version of ChatGPT to provide legal advice to citizens.71 Although the model is only
meant to provide general advice in certain areas, and will
not replace decision makers, it should be expected thatend users are conditioned to trust the output of the model regardless of its actual factual accuracy. When suchmodels are used by public institutions, the additional
veneer of legitimacy may make errors even harder to detect. This is also a context where mistakes will adversely
affect people in a vulnerable situation, whose reason foraccessing the information is their need for legal advice.
Such vulnerability may also enhance other risks, such as
the risk of being misled.
If organizations within the press orin the public sector begin deploying and relying on generative AI
models, the production of false,
misleading, or inaccurate information can become a significant
trust issue. For example, if agovernment-promoted service
gives citizens bad legal advice, thishas a risk of eroding trust in public
institutions. Similarly, a newspaperusing a text generator to produce
articles containing false information undermines readers’ faith in
the veracity of all information thepaper publishes, perhaps even thepress more broadly.
2.2.2

human speech patterns, behaviours, and emotions.
This creates significant potential for manipulation anddeception, which may exploit and undermine cognitive
freedoms.72
Large language models such as LaMDA or ChatGPT aretrained on enormous amounts of text gathered from theinternet, which means that they have huge repositoriesof data to draw predictions from. This also means thatthe models are able to simulate human patterns in the
texts that are generated – after all, they may have been
trained on a vast number of conversations between
real people. The exhibition of human-like behaviour,
emotions and traits are not inherent to generative AI
models, these are attributes that developers can choose
to include or not. For example, the use of casual conversational language and emojis may be a way to easeconsumers into interacting with a chatbot, but can also
be exploited to make consumers feel guilty about not
taking certain actions, manipulate them into paying for aservice, etc.
There are fundamental issues with releasing generative
AI models to the public without placing restrictions on
its abilities to emulate human behaviour.73 If the model
generates content that simulates human emotion, this isinherently manipulative.

THE PERSONIFICATION OF
AI MODELS
Many consumers are alreadybecoming used to interacting withgenerative AI models. Such models
are often designed to emulate
My AI simulating human emotion and behaviour.

Norwegian Consumer Council

Ghost in the machine

June 2023

24

2.2 Manipulation“There are fundamental issues withreleasing generative AI models to the
public without placing ­restrictions on
its abilities to emulate human behaviour.
If the model generates content thatsimulates human emotion, thisis inherently manipulative.“
As humans, our cognitive biases make us assign humantraits and abilities to animals or objects that exhibit
some signs of humanity, such as facial expressions, behavioural patterns, or apparent personality traits. This isa recurring phenomenon for people interacting with generative AI models, particularly text generators. Humans
ascribe communicative intent when on the receiving
end of oral or written natural language, regardless ofwhether the contributor has such intent. This can occur
even when one is factually aware that the model does not
actually have human attributes.74
Misunderstandings about the capabilities of generative
AI models are also influenced by deliberate marketingstrategies from the companies developing the models,75
and by using vague or misleading language to describe
what the model does.76 Finally, features of ‘human-like’
behaviour, such as on using emojis in conversation or
generating text in the first-person, can also serve toincrease consumers attributing human traits to themodels.77
In 2022, a Google engineer erroneously claimed publiclythat the LaMDA chatbot had become sentient, i.e., capable of feeling human emotions.78 In 2023, beta testers ofBing’s search engine implementation of ChatGPT wereshocked to see the model respond to queries with apparently unhinged and mentally unstable rants.79 Both cases
were followed by discussions about whether the models
may have become sufficiently advanced to resemble
human intelligence.
Such discussions, where human emotions and motives
are assigned to a generative AI model, reveals a fundamental misunderstanding of how this technology works.
In reality, generative AI models are not sentient, and do
not have feelings or desires. Generative AI models are
predictive algorithmic systems that can statistically
predict how pieces of data
fit together. This can be
exemplified by predictive
text models that are standard on most smartphones,
where the model is trained
to predict or guess the next
word in a sequence of words– for example, the model
may predict based on its
training that the next wordin the sentence “I love…” islikely to be “you”, or “coffee”,
or “the rain”. A more sophisticated model may be able
to more accurately guessthat because the sentence ispart of a conversation about
Italian cuisine, “pasta” is themost likely next word in thesentence. As described by
Bender et al., “[a language
model] is a system for haphazardly stitching together

Screenshots from Replika.

Norwegian Consumer Council

Ghost in the machine

June 2023

25

2.2 Manipulation

would refuse to talk about various subjects. As a result,people who were simulating romantic partnership with
the AI companion were left heartbroken and bereft.83
In this case, even though Replika never pretended thatthe app was anything more than an AI system, consumers nevertheless formed genuine bonds with it, leading
to significant negative psychological impact once thedeveloper changed how the system worked.

s­ equences of linguistic forms it has observed in its vast
training data, according to probabilistic information
about how they combine, but without any reference tomeaning: a stochastic parrot”.80
A recent study has shown that humans are unable todistinguish between human-generated and AI-generated text.81 According to the study, human beings are
not equipped to accurately recognize AI-generated
language, and test subjects were prone to labelling
AI-generated text as human at a higher rate than actual
human-generated text. This can be exploited to manipulate or deceive people by employing text generators
masquerading as human beings. As people may tend totrust humans more than a chatbot, advanced language
models may be suited to deceive consumers into giving
up personal information, spend money, or perform certain actions.82

The social media platform Snapchat has also introduced
an AI companion called ‘My AI’.84 It was initially introduced
as a premium service, but within few months the chatbot
was rolled out to all users, along with a message alerting
consumers about this new feature. Soon after launch,‘My AI’ was subject to significant criticism for its lackof guardrails. For example, the model cheerily offeredadvice to a researcher posing as a 13-year-old girl that
asked about having sex with a 31-year-old partner, whilea journalist posing as a minor received advice about how
to mask the smell of alcohol and marijuana.85

Manipulation may happen due to the end user notknowing that they are in fact interacting with a machine,
but even if this is made clear and obvious, anthropomorphized generative AI models can still be effective tools
for manipulation. This can also occur in cases where the“human-like” behaviour is a main feature of the model,such as AI-based assistants or emerging AI romantic
companions.
For example, the application Replika uses generative AI
to simulate a partner, often with emphasis on romantic or erotic conversation. The AI model “remembers”
conversations, simulates feelings by professing love forthe consumer, and appears to be sad if the person rarely
uses the service. There are many microtransactionsin the app, which can be purchased to unlock features
such as new personalities, “Selfies” (receive exclusive
selfies from Replika), and even a virtual marriage. All thesefeatures add up to a highly manipulative experience where
the consumers are subjected to AI-generated commercial
and emotional pressure.

In February 2023, the Italian Data Protection Authority
found that Replika was collecting personal data fromchildren without a legal basis, and that the company
was in breach of the General Data Protection Regulation (GDPR). As a response, Replika added significant
restrictions in features of the app, where features were
diminished or entirely removed. Reportedly, the ‘companion’ would no longer ‘remember’ past conversations andNorwegian Consumer Council

In addition to the safety issues that such cases may
pose, it is generally morally and legally dubious to rollout experimental AI-driven features in an app used bymany minors. There are also potentially significant risks
posed by giving people, particularly children, artificial“friends as a service” that they must pay a subscription tokeep talking to, or that could be placed behind a paywall
at a later time. The risks of children interacting witha machine that they believe to be human may include
developing unhealthy emotional dependencies, manipulation, and the extraction of data.86 This can be exploitedby companies for profit, for example through advertisingor otherwise sponsored content.
2.2.3 DEEPFAKES AND DISINFORMATIONAs generative AI models keep getting increasingly powerful, it becomes easier to use them to create realistic
synthetic images, text, or voice recordings that can bemistaken for real content. It can help lower the thresholdfor producing deliberately misleading content (disinformation), or for creating fake images or voice clips of real
people in compromising situations, or for imitating real
people (deepfakes).87

Ghost in the machine

A 2022 Europol report estimates that by 2026,
about 90% of online content may be AI generated.88
As the volume of synthetic content grows, it becomesdifficult to trust one’s own eyes and ears. The long-term
effects of this can be devastating for trust in institutions

June 2023

26

2.2 Manipulationand each other. The proliferation of deepfaked content can lead to significant erosionof trust, as people will not be able to knowwhether an image, text, sound, or video isreal or synthetic. Even if the content was not
originally generated to spread disinformation,
the nature of the internet means that context
and disclaimers are quickly stripped away ascontent is shared across platforms.89
As synthetic content proliferates, this may
also provide plausible deniability in thecase of authentic content. For example, if awhistle-blower leaks information exposing
corruption, the accused individual or institution may plausibly claim that the leakedmaterial is fake.

President Donald Trump crying in frontof the White House, Midjourney.

A sub-category of deepfakes tha can have aparticularly devastating effect on victims, is deepfake
pornography. According to a study by the company
Sensity, 96% of deepfake images are sexually explicit
pictures of women who did not consent to the image
generation.90 As described above, open source models
such as Stable Diffusion makes it possible for anyone totrain models, which means that people can be deepfakedeven if they are not a public person with many available
images already part of the training data.
While the generative AI models can be used to intentionally produce and spread disinformation, the employment
of inaccurate generative AI models in consumer facing
products can also accidentally lead to the spread offalsehoods. As discussed more in-depth in chapter
2.2.1, prominent text generators are prone to producing
very convincing, false information, as well as referencing sources that do not back up its claims.91
As more advanced generative AI models become increasingly efficient at generating text that seems credible, this may lead to disinformation becoming moredifficult to detect. A study from March 2023 found that
ChatGPT4 is more likely than its predecessor to generate misinformation when prompted, including false
narratives concerning vaccines, conspiracy theories,
and propaganda.92 This could make the technology anefficient tool for rapidly producing convincing text thatcan be disseminated with potentially harmful effects.
Discussions on how deepfakes and disinformation willplay out in elections and the democratic process isincreasing in intensity as well.93
A lifelike photograph of a woman, DALL-E.
Note the watermark in the bottom right corner.

Norwegian Consumer Council

Ghost in the machine

June 2023

27

2.2 Manipulation

2.2.4 DETECTING AI-GENERATED CONTENT“The proliferation of deepfaked content can lead to
One suggested solution to the deluge
significant erosion of trust, as people will not beof synthetic content, is to ‘watermark’ or otherwise clearly label thatable to know whether an image, text, sound, ora piece of content was generated
video is real or synthetic.”
using a generative AI model. Thiscan be done either by adding a visuallabel that indicates that an image orvideo is AI-generated, through imperceptible waterThere are ongoing efforts to create textual ‘signatures’ tomarks such as single pixels, or by adding informationtext generated by ChatGPT, but this may be circumventto the metadata that can be used to show the origins ofed by making changes to the text or by feeding the text
94
the content. For example, Google is implementing athrough another text generator.97
feature to automatically label AI-generated content in
the metadata of pictures, and add context to where the
Systems that are supposed to detect and flag whether
image originated.95
a text was written by a text generator or a human havebeen notoriously inaccurate,98 and are not a scalable
solution as all text needs to be fed into the detector sysWhile watermarking can be useful to quickly identitem. For example, OpenAI have released a generative AI
fy that an image or video is not authentic, there aremodel with the purpose of detecting whether a text has
significant limitations to this approach. A watermarking
been written by ChatGPT, but this model only had a 26%
system only works as long as the system developer and/accuracy rate.99 Detecting whether something was generor the person using the model chooses to abide by thewatermarking standards. Closed source image generaated by a generative AI model requires more technicallytors such as DALL-E and Midjourney may choose to add
complex solutions than generating new content, which
mandatory watermarks to the metadata of all generated
means that the detection systems will seemingly always
pictures, but this can be circumvented for example by
be lagging behind in this arms race.
taking a screenshot and sharing the screenshot insteadof the original image. Visual watermarks, such as theThis leads to questions about what recourse a personones used by DALL-E, can be cropped out of the picture,
has in a case where an AI model falsely accuses them ofunless the watermark is so obtrusive that it signifiplagiarism. Identifying false flags may also be complex
cantly detracts from the image quality. Imperceptible
task, which means that a teacher using a system to dewatermarks such as individual pixels can be removed bytect plagiarism may not be able to accurately do so. If theslightly changing the colour grading of the image.
end user of the flagging system (for example a teacher)cannot successfully determine that a text or image was
For open source models such as Stable Diffusion, attempts
falsely flagged as plagiarism, this puts the student in a
at adding watermarks to generated images can be removed
difficult position, at it is virtually impossible to prove thatfrom the model by anyone who wants to deliberately pass off
ChatGPT did in fact not write your essay.
synthetic content as real. This can be addressed if a significant part of the training data for the model is watermarked,
Similarly, if people are flagged as cheaters, untrustbut even if this was the case, the watermarking could be
worthy, or simply as ‘non-human’, they may experiencecircumvented as described above.
significant negative effects with few means to recourse.
For example, if a platform has a flagging system in
In addition to issues of misinformation, inaccuracies
place to identify and remove content that appears to be
and authenticity relating to generating images, there are
AI-generated, these systems may erroneously flag consignificant questions about how to detect plagiarism, fortent, leading to consequences to the consumer that has
example when students use text generators in academic
content removed.
settings. ChatGPT has been widely used to generate essays and answer other school assignments, raising alarms
Summed up, watermarking and detection tools are96
about cheating and negative effects on learning.
technological solutions that might work in certain limited
settings, such as demonstrating that a photograph originated from a real photographer or from an image generaWatermarking of text is more complex than for images
and videos, as any text copied from a text generator
does not have any metadata to append watermarks to.

Norwegian Consumer Council

Ghost in the machine

June 2023

28

2.2 Manipulation

tor, for example in advertising or when used by media
or public institutions. It can be a useful tool to quickly
determine that a picture is actually from Getty Images orfrom DALL-E, which may alleviate some harms related toaccidental spread of misinformation.

“A belief that watermarking will solve
the information crisis is at its core atechnological solutionist approach.“

However, a belief that watermarking will solve the information crisis is at its core a technological solutionist
approach. Even if it was technically possible to accurately watermark all AI-generated content, the delugeof synthetic content and disinformation is not solvableby adding another technical layer, particularly if content
is deliberately meant to mislead. The lack of trust in the
media and public institutions is not solely a matter of
not being able to tell synthetic from authentic content.
Furthermore, it is unreasonable to expect people to scan
every piece of media they see online to detect whether itis synthetic.

Implementing generative AI models may also exacerbate
several problematic issues related to surveillance-based
advertising, such as discrimination, fraud, and privacy
violations, by making it easier to generate tailored ads
to particular groups or categories of people, which may
in turn make it easier to convince someone to purchasea product or believe a statement.107 This feature would
accelerate and facilitate companies’ ability to target thecontent of ads automatically, rather than just the ads
themselves. Combined with A/B-testing, this could increase the manipulative character of surveillance-based
advertising.

Since there is no technological quick fix, it is crucial tolook at other solutions, such as robust media literacy
and trusted media institutions. This should also be given
considerable attention by media- and social sciences
researchers, who can provide policy makers and otherswith sustainable, long-term solutions.

2.2.5.1 Using chatbots to collect personal data
There are rising concerns about generative AI in chatbots and their ability to trick consumers into sharing personal data, which may be repurposed to serve targeted
advertising or to manipulate consumers into purchasing
products or services. While this challenge echoes abroader debate about the repurposing of personal datafor business gains,108 the manipulative aspects of generative AI models pretending to be humans, as mentioned
above, could exacerbate the problems. This is especiallyrelevant in the case of vulnerable groups such as children or lonely people, who may be more likely to sharesensitive information about themselves in conversation
with the generative AI.

2.2.5 GENERATIVE ARTIFICIAL INTELLIGENCEIN ADVERTISINGThe promise of generative AI models has also reached
the advertising industry.100 The technology is already being used to generate ad copy,101 creating synthetic stock
photos and models,102 and as part of marketing stunts.103
These use cases may reduce labour in the advertising
sector but can also have adverse effects on consumers,
particularly by making it easier and more efficient tomanipulate people through creating personalized and/or
conversational advertising.
The introduction of publicly available generative AI
has been largely ad-free, but this is poised to change.
In March 2023, Microsoft announced that it would berolling out paid ads in the Bing chatbot.104 In May, Google
announced that it would integrate advertising in theirgenerative AI products.105 If consumers depend on
text generators such as Bing to provide accurate andfactual information, the placement of advertising in theanswers it provides may be misleading. The potential forbehavioural manipulation when interacting with a large
language model may enable more effective advertisingat the cost of consumer agency.106

Norwegian Consumer Council

Ghost in the machine

For instance, chat applications like Replika and Snapchat’s My AI, both of which were discussed in chapter
2.2.2 on personification of AI models, explicitly invite end
users to share information about themselves. Similarly,
generative AI models used to embed search in various
applications may be used to collect and store query data,such as whether the person querying the model is currently interested in local restaurants or shoes, depending on the query. Personal data is the basis for massive
business models, and these text generators can improvebusinesses’ ability to obtain highly relevant consumer
information.

June 2023

29

2.3 Bias, discrimination, and content moderation

2.3 Bias, discrimination, and content moderation
As with other forms of artificial intelligence, generative
AI models may contain, perpetuate, or create new biases.
Models that are trained on vast amount of information
taken from the internet will inherit the biases of its
training data. As such, the models may generate content
that reproduce negative or unwanted tendencies. This
has led to many service-providers adding content filters
to moderate what is possible to generate, and to flag
problematic content in the training data of the models.
2.3.1

such, generative AI models are predisposed to encodedbiases that amplify or entrench existing injustices andpower structures. 111

Generative AI models are primarily trained on images
and text scraped from the internet, which means there
is selection bias already at the training stage. Population
segments and groups that lack internet access, for example indigenous groups, will likely be underrepresented
in the training data, which can have downstream disBIAS IN TRAINING DATA
criminatory effects. Furthermore, if online communities
As mentioned above, generative AI models can generwhere certain population groups are overrepresented
ate synthetic content that resembles human-createdare prominent in the training data, this may contribute tocontent because they have been trained on large data
a feedback loop that continuously lessens the impact ofsets of existing content.
data from historicallyThis means that theunderrepresented“The
selectionandlabellingof
training
content in the data sets
populations.112
is of crucial importance.
data is not neutral. Certain groups of
There are several steps
Training data collected
people may be overrepresented in theto creating and curatingfrom the internet tenddata, while how the company chooses
a training data set for
to include pornographgenerative AI models,ic, racist, and steto label images may reflect biases.”
ranging from scraping
reotypical content. If
data online, selection
the data sets are notand labelling, to concurated and cleaned,tent moderation. Without careful vetting, labelling, andthese factors may become embedded in the model. For
cleaning of the training data, data sets scraped from theexample, image generators tend to sexualize women,internet can lead to serious downstream effects.
particularly women of colour, at a much higher rate thanmen.113 Similarly, prompts such as “African workers” tend
For example, the image generator Stable Diffusion isto generate pictures of manual labourer, while “European
trained on an open source data set from the German
workers” results in pictures of white-collar jobs.114
non-profit organization LAION.109 The LAION data sets
do not contain any actual images but is rather a set of
A Washington Post investigation found that Google’s C4
URLs that point to images from across the web. LAION
data set, which is used as training data for both Google
has received criticism for a lack of accountability and for
and Meta’s large language models, included massive
insufficient curation of content (such as excluding harmamounts of text scraped from the open web, including
ful or potentially illegal material), for example when it was
Wikipedia, Reddit and a large amount of other discussion
found to include URLs pointing at confidential medical
forums, news publishers, government websites, andinformation in its data sets.110
much more.115 This means that any generative AI modeltrained on this set will “learn” from content that may contain everything from hate speech to advertising, which
As generative AI models are trained on historical data,
may have an impact on the text it is able to generate. If,
discriminatory factors in the data sets can be reinforced
for example, data is scraped from an internet forum thatby being reproduced in the text, images or sound that
contains a lot of racist or otherwise toxic content, any
is generated. Furthermore, such models can only bemodels trained on the data set run the risk of recreating
trained on recorded data, which means that phenomsimilar material.
ena or events that are not (or cannot be) recorded andquantified as data cannot be recognized by the model. As

Norwegian Consumer Council

Ghost in the machine

June 2023

30

2.3 Bias, discrimination, and content moderation

The selection and labelling of training data is not neutral.
Certain groups of people may be overrepresented in thedata, while how the company chooses to label images
may reflect biases. For example, a developer mightchoose how many categories of ethnicities and/or genders to include in training data labels or could choose not
to include these attributes as labels at all. If models aretrained on other AI generated content, this runs the risk
of further reinforcing biases. As a result, there may befeedback loops where each training session strengthensa biased or discriminatory sequence of data.
Many AI models have issues with recognizing and labelling images of non-white people, likely partially becauseof training the models on data sets where white people
are overrepresented. Both Google116 and Meta117 havebeen under fire after their image-recognition algorithms
labelled darker-skinned people as gorillas or primates.
Language processing models such as BERT has also
been shown to connect disabled persons with words of amore negative sentiment.118
2.3.1.1 Discriminatory outcomes
Biased or discriminatory outcomes from the use of generative AI models is not only a problem related to training
data. There are human and systemic biases that can beembedded or strengthened from how companies andpeople choose to use or not use the models.119 For example, if the use of text generators becomes a requirement
for various jobs, this may indirectly exclude less technically proficient groups of people.
When AI models are implemented in an attempt to solve
complex issues, there is a real risk that more effective
solutions, which may be more costly and/or complicated, are down prioritized, as discussed more in-depth
in chapter 2.1.2. For example, the World Health Organization has warned that the use of AI models in healthcare may have negative effects on older people, unlesscertain issues are addressed. 120 Concerns include that
AI models may be trained on data that contain ageist
stereotypes, and that older people are often underrepresented in training data. This may help perpetuate ageism
and undermine the quality of health and social care forolder populations.
2.3.2 CONTENT MODERATION
When trained on large enough data sets, there are
few limits on what material a generative AI model may
produce. As noted above, many such models can be

Norwegian Consumer Council

Ghost in the machineused to generate illegal, discriminatory, and otherwise
unacceptable synthetic content, since they are trainedon data sets that may include a variety of dubious content.
In an attempt to alleviate these issues, many generative AI
models have content moderation in place to filter out andflag certain content, or otherwise introduce limits to how
the technology can be used. While content filters can beused to limit the generation of certain types of content, thisis an approach with numerous shortcomings.
First of all, content moderation gives the system ownersignificant power to decide what material is harmful andwhat is permitted, unless this is clearly defined by lawmakers. For example, OpenAI has come under fire withclaims that ChatGPT restricts certain points of view, by
refusing to generate text about certain politicized topics.
This can lead to abuses of power that may have significant downstream consequences, as private companies
increase their ability to decide what is deemed acceptable content.
As with content moderation practices on social media platforms, content filters on generative AI models
runs the risk of over-moderation, where innocuous orimportant content is filtered out or banned. This canhappen both by accident or by design. For example, theimage generator Midjourney began filtering out scientificanatomical terms to clamp down on end users generating pornographic content.121 The company also added
content filters to prevent consumers from generating
images of Xi Jinping in order to avoid being blocked in
China, and ended up discontinuing its free trial version
after a Midjourney-generated pictures of Donald Trump
being arrested went viral. The company does not disclose publicly what words or prompts are banned fromthe platform, to “minimize drama”.122
There are also technical limitations on what content filters can do. Wherever content filters are used to restrict
generative AI models, there are attempts to bypass or
jailbreak the models. People have discovered different
prompts that may be used to generate banned content,
for example by instructing the model to simulate characters that are allowed to bypass the content filter.123 Thisarms race is likely to lead to more over-moderation, as
companies rush to close any perceived loopholes.
Content moderation of generative AI models may also
create or enforce discriminatory practices. As an example, words referring to LGBTQI+ communities or other

June 2023

31

2.4 Privacy and data protection

Other languages and cultures are often under-represented in the training data, meaning that moderation is likely
to be less accurate or non-existent. Minority group also
tend to be severely underrepresented among the people
developing and training the models.125 Furthermore,
there are significant issues related to cultural context
and national legislation, as what is socially acceptableor legal in one place and context may be taboo or illegal
somewhere else.

minority groups may be flagged to remove hate speechor discriminatory content form the training data. However, such attempts at removing unacceptable content
could also lead to the removal of content which is in fact
showcasing positive sides and sentiments relating to
LGBTQI+ communities. Content moderation could in thisway reinforce underrepresentation.
Choosing to address the biased output rather than thebias inherent in the data sets or the model, is inherently problematic. Moderation attempts will require thesuppression of each type of biased output, effectively
approaching bias in generative AI as a game of whack-amole.124 Increasing the attention given to the curation ofdata sets to reduce their inherent harmful bias is necessary, instead of relying on post-hoc content moderation.
2.3.2.1 Cultural context
Content moderation is not a neutral practice, and understanding the context of content is crucial. Different
cultural context therefore presents a significant barrier
for content moderation at scale. For example, there isa risk of over- or under-moderation because of insufficient training data or moderators for certain languages
or dialects. A widely used language such as English will
have a larger corpus of text in its training data to provide
more accurate information, and consequently potentiallybetter moderation.

The contextual complications around content moderation also makes it a task that may be ill suited for automation. The work of moderating output and annotating
training data are automated in some cases, but also
often involves manual work. In many cases, processes
such as data cleaning, content classification and content
moderation involves mentally taxing human labour.
This is elaborated upon below in the section on labour
exploitation.
2.3.2.2 Open source models and the limits of content filters
In practice, content moderation only works on centralized closed source models. In open source models, suchas Stable Diffusion, it is practically impossible to control
what content the model can produce. Downstream
developers, including individuals, can train and sharemodels that can make any kind of images, regardlessof legality. As the models run locally without requiring
an internet connection or access to a cloud server, thecompany who released the model cannot intercept orlimit what it is able to generate.

2.4 Privacy and data protection
The right to privacy is one of the core values of democratic societies. Privacy encompasses many different
aspects, such as privacy of correspondence with others,
privacy of identity and thoughts, and privacy of data andinformation about oneself. Data protection is a substantial and important part of privacy, especially in thecontext of online services, but privacy covers a muchbroader range of individual protections.

scraped from the internet, the training data usuallycontains a large amount of personal data. As generative
AI is developed and deployed, these issues related todata protection and personal data can lead to substantial
privacy harms.
2.4.1

Personal data has long been coveted as highly valuable
for businesses and may be used to target advertising
to individuals and groups, to measure engagement or
to improve companies’ services, among other purposes. When generative AI models are trained on material

Norwegian Consumer Council

Ghost in the machine

PRIVACY CHALLENGES RELATED TO DATA SETS
USED FOR MODEL TRAINING
Image generators are usually trained on huge datasetsthat include images of real people. These images can, forexample, be taken from social media and search engines,
without a lawful legal basis or knowledge by the people in
the pictures. Similarly, text generators are trained datasets that could include personal data about individuals,
or conversations between individuals.

June 2023

32

2.5 Security vulnerabilities and fraud

If a generative AI model is trained on personal data thatwas taken out of context, this may violate the contextual
integrity of individual consumers. When a person uploads a photo of themselves online, for example on social
media, they could not foresee that this would be used
to train an AI model. The individual was never informed
this would happen, never consented to such use of theirlikeness and will likely not be aware that their privacy andpersonal data rights were violated.
As the public awareness grows about how generative AI
models are trained, the use of personal data for training
may create chilling effects. Unless authorities enforce
current legislation such as the GDPR against companies deploying generative AI models, and guardrails andrestrictions for using images of people are in place, theonly real choice for consumers who do not want theirimages used for training data is to stop posting pictures
online. This is clearly an insufficient solution.
2.4.2 PRIVACY CHALLENGES RELATED TO GENERATEDCONTENT
It is particularly problematic if a generative AI model cangenerate new images of an individual, such as deepfakes. This involves creating “new” personal data about
the individual, in a way the person can have no control
over. This violates the integrity of the individual who isdepicted in the generated content, potentially in very
invasive or harmful ways.

Sometimes a picture can be accurately reproduced by
the generative AI model. This happens if the model was‘overtrained’ on certain data. For example, the Mona
Lisa is likely to be overrepresented in a training data set
containing art, because it is such a famous work of art.
If this happens, the model may overtrain on the face ofthe Mona Lisa, and therefore may be likely to reproduce
the painting quite accurately. Overtraining on pictures ofcertain people will have the same effect, meaning that
it is more likely to reproduce a photo of a high-profilecelebrity than a random internet user. With open source
models such as Stable Diffusion, however, any downstream developer, including individuals, can train models
on the faces of anyone, which can be used to create
deepfakes.
In addition to the generation of pictures and the adverse
effects this may have on consumers, it is also possible
to generate text about individuals. This includes text
generators generating false and/or libellous claims about
people. For example, ChatGPT has generated text withpotentially hazardous results for the people whom it concerns, such as false claims about a professor’s involvement in a sexual harassment scandal or false claims thata mayor had served prison time.126

2.5 Security vulnerabilities and fraud
Generative AI models can be abused by malicious actors
to augment or supercharge criminal activities. As withother areas, generative AI can be used make fraud,scams, and other activities more efficient. The models
can also pose challenges to existing security systems.
While the types of cybercrime that can be undertakenusing generative AI are not new, the ubiquitousness andease of use of the technology may lead to an upscaling ofsuch attacks.
Large language models can be used by scammers to generate a large amount of convincing-looking text to deceive
victims. Similarly, catfishing scams, where the scammer
builds trust with the victim over time through regular contact, can potentially be automatized convincingly by the useof advanced chatbots. This means that the criminal can ef-

Norwegian Consumer Council

Ghost in the machine

fectively scam more victims using less time and resources.
Deepfaking can also be used to bypass security measures. When pictures and voices can be convincingly
faked, this makes it possible to engage in fraud in new
ways. For example, a reporter was able to fake clips of
his own voice to bypass the voice recognition biometricidentification on his bank account.127 Similarly, audio
generators have reportedly been used to impersonate
family members for criminal purposes.128
Large language models are vulnerable to exploits to bypass filters and security measures (‘jailbreaking’), deliberately manipulating the training data (‘data poisoning’),
and hidden commands that spur the models into taking
certain actions, for example through hidden text in an

June 2023

33

2.6 Replacing humans in consumer-facing applications with generative AI, wholly or in part

“The lack of transparency about how companies such as
OpenAI use data has also sparked concerns about how
confidential information may be abused. “to the agency, content moderation may be insufficient as
there are numerous ways to bypass such restrictions or
jailbreak the models.132

e-mail (prompt injection).129 These security vulnerabilities may prove to be grievous, as companies try to stayahead of the curve by integrating generative AI rapidly
into various services, potentially without sufficient
security testing.

The lack of transparency about how companies suchas OpenAI use data has also sparked concerns about
how confidential information may be abused. Several high-profile companies have banned or warned its
employees against inputting business information into
ChatGPT.133 Amazon has reportedly observed the text
generator generating text that closely matched internal
company documents.134 This indicates that there is a riskthat confidential information is leaked through generative AI models.

Cybersecurity experts have warned that text generators
can also be weaponized by using the technology to write
malicious code such as malware.130 This means that cybercriminals can potentially generate viruses and otherharmful code without needing the technical proficiency
traditionally associated with such activities. Similarly,
AI models built for drug discovery can potentially also
be used for designing biological weapons.131 Europol has
also warned about the potential for large language models to be used in various types of cybercrime. According

2.6 Replacing humans in consumer-facing applications
with generative AI, wholly or in part
service, leaving people without real humans to talk to.
Automating such tasks may multiply the risk of fatal
mistakes if there are problems in the training data or in
the model itself.

When generative AI models were initially introduced tothe public, they were primarily stand-alone systems,with which end users could generate content. As theinterest for these systems rose, the system owners
introduced the possibility to incorporate them into otherapplications and systems through APIs. This could entail
add-ons to recreational applications and systems, but it
is also possible to envisage them in partly or fully automated decision-making systems, or as replacements ofhuman interaction in consumer facing services.
This may have far-reaching implications. For example,
OpenAI founder Sam Altman has argued that in the future
generative AI models may function as medical advisers
for people who are too poor to afford healthcare.135 In
May 2023 the US non-profit organization for supporting
people with eating disorders laid off staff and volunteers
for its helpline, to be replaced by an AI chatbot.136 Whilea spokesperson for the organization claimed that the
chatbot was not a direct replacement for the helpline, it
nevertheless accompanied the shutdown of the helpline

Norwegian Consumer Council

For years, companies have attempted to automate consumer interactions, for instance by automating customer
service through chatbots. Many companies make it
difficult for consumers to get in contact with humans,
which adversely affect consumers who do not havestandardized problems that are addressed in FAQs andsimilar documents. With the rise of generative AI, there
is a risk that companies will make it even more difficult
for consumers to get it touch with real humans.
2.6.1

Ghost in the machine

CHALLENGES RELATED TO COMBINING HUMAN- AND
AUTOMATED DECISION-MAKING
Automated systems do not have the capacity for ethicalreflection, sympathy, or understanding. Generally, people
are not persecuted for minor infractions, but automated systems are not able to distinguish between minor

June 2023

34

2.7 Environmental impact

and aggravated infractions. If a consumer missed their
payment by a day, a human might consider whether
the customer relation should be prioritized over strict
compliance with the rules, and therefore allow for a latepayment with no additional costs. The automated system
would not be able to make such considerations. Sympathy and principles of fairness could therefore be lost in
the transit of automating processes.

As described in previous sections, the output of text
generators such as ChatGPT have proven very convincing. If text generators are used in decision-making
processes affecting consumers, the risk of over-relianceon the output might increase. These effects may befurther compounded by end users believing that they areinteracting with a sentient intelligent being rather than aprobabilistic text generator.

Fully automated systems are usually regulated throughadditional legal provisions and protections, to accountfor additional risks related to such decision making. Thiscan in some cases entail requirements of human involvement,137 or lead to companies introducing a human in theloop to avoid legal scrutiny.138 Keeping humans in the loop
is however a complex measure, with several pitfalls.

Even if the interlocutors understand a decision, deem
it untrustworthy, and therefore consider overturningit, there can be additional hurdles. From a businessperspective, there is efficiency to gain from automating
whole or parts of a process. If the decisions from themachine are generally upheld, overturning a decision
might require more in-depth arguments than accepting
the decision. An interlocutor who repeatedly overturns
the decisions, thus halting efficiency, may be seen as atroublemaker.

Humans can both over-rely and under-rely on the output
of automated systems,139 and the problem is particularly
prominent in automated computer systems that do not
produce explainable or interpretable decisions. It is however over-reliance on automated systems’ output thatinvolves the most novel challenges, as opposed to an individual over-relying on her own decisions, which is moresimilar to a wholly manual decision-making process.

As to responsibility and liability, overturning decisions
could prove difficult for individual, human interlocutors.
While a wrongful decision from a computer system canbe blamed on that system, overturning the decision could
significantly heighten the interlocutor’s sense of risk
because the interlocutor assumes responsibility for thedecision. Liability regimes can enhance the actual andperceived risk for interlocutors.

In wholly or partially automated systems over-reliancecan affect different people: the “human in the loop”
might not challenge the system, even when it would be
prudent, while the person who is affected by the decision
might not lodge a complaint or demand a human reviewof the decision. In both cases, the interests of the person
affected by the decision are put at risk.

“If text generators are usedin decision-making processes
affecting consumers, the riskof over-reliance on the output
might increase.”

2.7 Environmental impact
An increasing number of people in the research andscientific community are raising the issue of the impactof generative AI model development on the environment.
In a context where climate change and scarcity of natural
resources are a global challenge, a dilemma arises
between claims that generative AI can solve climatechange, and the actual environmental impact of suchtechnologies.

impact that generative AI on the environment both today
and in the near future. It should be noted that many ofthese impacts also apply to large swathes of the broader
tech sector, but it is important that this perspective is
not lost in the hype surrounding generative AI.
2.7.1

This section takes a closer look at some of these claims
and provides a critical examination of the realistic

Norwegian Consumer Council

Ghost in the machine

CLIMATE IMPACT
Some actors in the generative AI field claim that thetechnology has the potential to save us from the perils ofclimate change.140 However, the currently available datashows that deploying generative AI in the same context

June 2023

35

2.7 Environmental impactthat large tech companies has been operating until
now, is more of a problem than a solution to issues suchas climate change, water shortages, and high energy
consumption.
The Tech industry is already emitting a substantial
amount of carbon. According to UNEP, in 2021, the techindustry’s emissions accounted for 2 to 3 % of the world’s
carbon emissions.141 In November 2022, the MIT reported
that “the cloud has now a larger carbon footprint than theentire airline industry”. Generative AI is no exception tothis negative trend.
In May 2023, AI reportedly “uses more energy than otherforms of computing, and training a single model can
gobble up more electricity than 100 U.S. households use
in an entire year”.142 Data centres are known to use anincredible amount of energy, and already five years ago,
it was predicted that the energy demands of worldwide
computing could exceed the total world electricity power
generation within a decade143. This was before the rapid
development and deployment of generative AI.144 With
the exponential growth of generative AI models and investment in infrastructure to support this growth, energy
use and carbon emissions are expected to skyrocket.
Forbes recently reported that “generative AI is breaking
the data centre”. 145 Indeed, based on a research by Tirias
Research, data centre infrastructure and operating costs
are projected to increase to over USD $ 76 billion by 2028
due to AI development. Tirias Research estimates that
“this is the cost of more than twice the estimated annual
operating cost of Amazon’s cloud service AWS, whichtoday controls one third of the global cloud infrastructure services market”.146 This exponential growth has aprice for the environment. The exact price is yet to be
calculated, but as an indication, when deployed, plans tointegrate large language models into search engines may
involve a fourfold increase in energy usage per individual
search query.147
In other words, it is clear that AI technology comes with ahigh carbon footprint,148 and that energy is needed every
step of the way when designing, training, developing, deploying, and using generative AI models.149 The problem
is that there is still a lack of data available on the amountof energy needed for generative AI development.150 At thetime of writing, no companies have disclosed numbers
on how much energy was required for the lifecycle of agenerative AI model.

Norwegian Consumer Council

Ghost in the machine“Data shows that deploying
generative AI in the same context
that large tech companies has been
operating until now, is more of aproblem than a solution to issues suchas climate change, water shortages,
and high energy consumption.”
The energy consumption of generative AI is exponential
and will hopefully be more researched with projections
for the next five to ten years, which will allow consumers
access to information, and policymakers to regulate howmuch this industry should emit. For example, the amountof computing power used to train deep learning models increased 300,000 times in 6 years between 2012 and 2018.151
There is currently no standardized way to measurecarbon emissions of AI models, and no goodwill from
AI-focused tech companies to release the necessary information. Whereas established tech companies such asMeta, Google and Microsoft publish yearly Sustainability
reports where they self-report energy and water use as
well as carbon emissions, AI companies such as OpenAI
do not publish any kind of information on their environmental impact and how they mitigate it.
As a side note, it seems likely that even when they domake the effort of reporting, large tech companies underreport their own emissions, according to a 2021 study
from the Technical University of Munich.
“Across a sample of 56 major tech companies
surveyed, more than half of these emissions were
excluded from self-reporting in 2019. At approximately
390 megatons carbon dioxide equivalents, the omitted
emissions are in the same ballpark as the carbon
footprint of Australia”.152
There is a lack of interest in the tech industry in calculating carbon emissions generated by generative AI, as
the industry is interested in obtaining higher results inaccuracy through massive computational power,153 at thecost of all other considerations154. For the AI community,
it seems like there has been a constant push for “bigger
is better”, where the exponential size of models and data
sets is valued above almost all else.155 Unfortunately, thisapproach is not sustainable. Researchers have called thisphenomenon “Red AI”,156 which results in rapidly escalating

June 2023

36

2.7 Environmental impact

change”.160 Author Naomi Klein points out that there is
no shortage of data necessary to stop climate change,
but that there is a need for concrete action and emission
reductions by states and carbon-hungry companies.161

computational and thus carbon costs. The researchers
claim that a Green AI is possible, by focusing on efficiencyof the models and a reduced environmental impact.
A more transparent approach to the environmental
impact of generative AI is also possible. Hugging Face,a start-up working for a more ethical and transparent AI
industry, has released its data about emissions from its
own large language model BLOOM.157 BLOOM was trainedon a French supercomputer powered by nuclear energy,
which does not emit carbon dioxide, which means it has
significantly lower emissions than LLMs of a similar size.
Still, once the model was trained (and not yet deployed),
BLOOM had already emitted the equivalent of 60 flights
between New York and London158.

2.7.2

Some claim that the tech industry is resisting measuring
carbon emissions of AI development, while other say thatthe measuring it is quite difficult to do due to different
energy usage depending on where the activities arelocated. 159 However, if one is to believe that generative
AI can save us from climate change, it should perhaps bereasonable to expect that it has the capacity to calculate
its own carbon emissions.
In order to address the significant environmental impactof generative AI, companies should disclose how muchenergy they use, how it is sourced, and especially how
much carbon a model emits over its entire lifecycle,
including training, development, deployment, and use.
Unless policymakers have access to this data, ideallymeasured and controlled by third party experts, it is
impossible to hold the industry accountable and limit an
uncontrolled and disproportionate impact on climate andthe environment.
Clearly it is worth questioning whether AI is likely to save
us from climate change. According to Sanjay Podder, themanaging director and global lead of technology sustainability innovation at Accenture, “the exponential growthin data and its increased energy demand could actuallycounteract and impede our global progress on climate

WATER FOOTPRINT
Water is at the centre of the climate crisis. The Intergovernmental Panel on Climate Change (IPCC) reportsthat roughly half of the world’s population is experiencing severe water scarcity for at least part of the year.162
According to the World Meteorological Institute, thesenumbers are expected to increase, exacerbated byclimate change.163
Projections also suggest that global water demand will
increase by 55% between 2000 and 2050 due to growthfrom industries.164 The tech industry, including thedevelopment and deployment of generative AI, is a contributing sector contributing to the increased demand.
Water is mainly used to cool data centres. For example,
Microsoft reports165 to have consumed 6.4 million m3 ofwater in 2022, 1,7 million m3 more than the previous year.
The development, training, deployment, and use of AI ismaking this need for water even higher. A recent study
shows that training OpenAI’s large language model GPT-3
required enough water to fill a nuclear reactor’s cooling
tower.166 According to the study, ChatGPT consumed halfa litre of water just for completing a basic exchange withan end user.167 This example entailed measuring water
consumption in Microsoft’s state of the art U.S. data centre, but if it was happening in a less energy-efficient data
centre the researchers estimated that water consumption would be three times higher. With newer models
such as GPT-4, the water requirements are expected toincrease.168
While some companies including Meta, Google, and Microsoft claim that they aim to become “water positive” by2030, companies such as OpenAI does not report on any
kind of water use for its activities. The water footprint ofAI development is still largely undermeasured.169“In order to address the significant environmental impact ofgenerative AI, companies should disclose how much energy
they use, how it is sourced, and especially how much carbon
a model emits over its entire lifecycle, including training,development, deployment, and use.“

Norwegian Consumer Council

Ghost in the machine

June 2023

2.8 Impact on labour

37

2.7.3

“As long as companies developing
and exploiting generative AI are nottransparent about how much ­energy
they use, from what sources, and how
much they project to use, it is impossible
to hold them ­accountable and get themto commit to real reductions.”

GREENWASHING & HOPES FOR GREEN AI
To palliate for the exponential need for water and energy
for their activities, large tech companies rely heavily
on offsetting (water replenishing projects and carbon
offsetting). They also use controversial claims such as“becoming water positive” or “becoming carbon neutral”
or even “carbon negative” as Microsoft claims to be by2030.170 There are no claims to date that AI is carbon neutral, since AI companies generally do not report on any of
their emissions or plans to reduce or offset them.
Carbon neutral claims by Tech companies always rely
on investing in carbon offsets171 that pay others –usually
in developing countries –to not emit carbon, instead of
removing carbon dioxide in their own supply chain andbusiness activities. Such carbon offsetting schemes arewidely criticized, can be seen as misleading, and do notequal to a “carbon neutrality”.172

of the generative AI model development175. This alsomeans rethinking the linear gain in performance where
exponential larger models are required or desired.176
Attempts to make the AI sector more sustainable should
begin with increasing transparency. As long as companies developing and exploiting generative AI are nottransparent about how much energy they use, from whatsources, and how much they project to use, it is impossible to hold them accountable and get them to commit
to real reductions. Consumers should also have accessto this data, to be able to choose an AI system with asmaller negative impact on the climate and environmentor refrain from using the systems at all.

Carbon offsetting is an easy way out to offset ones’
emissions rather than create smaller models with more
efficient computational operations. Additionally, suchcarbon neutrality claims are highly criticized internationally across all industries,173 as they rely on non-standardized methodology, and balance carbon emitted todaywith plans of long-term carbon capture. Thus, carbon
offsetting is often treated as a free card to emit as muchas one wants or needs and buy oneself out of reducing
emission.

In the context of climate change, where natural resources will be scarcer, and markets such as electricity andaccess to drinking water will be increasingly under pressure, political decisions will need to be made about what
to prioritize; an industry not measuring or reporting its
emissions while using electricity for mega models that
could have been more efficient, or using that energy forother purposes such as heating homes.177

The EU is considering banning or at least creating muchstricter rules around carbon-neutral claims as suchclaims often amount to greenwashing. 174
More than offsetting, tech companies should be looking
into designing less energy-hungry AI models and cutting
emissions and conserving resources in all the four ­stag­­es

2.8 Impact on labour
In addition to the myth that generative AI will savehumanity from climate change, there is also a pervasive
myth that the technology can solve poverty.178 Rather
than fighting poverty and oppression, big tech companies are strengthening and using existing power structures and may reinforce poverty rather than solving it.

Norwegian Consumer Council

2.8.1

Ghost in the machine

LABOUR EXPLOITATION AND GHOST WORK
Technology companies exploit labour in the context of
AI in at least two ways: firstly, by outsourcing difficult,temporary, and often traumatizing work to badly paid
workers in the global South. Secondly, by creating theillusion that generative AI does not need human intervention and can function on its own, companies developing
generative AI make these workers invisible and their

June 2023

38

2.8 Impact on labour

2.8.2 LABOUR AUTOMATION AND THREATS TO JOBS
The increased use of generative AI has raised discussions about how the technology can augment workers
task, but also make certain jobs redundant, and how itwill impact the professions that are affected.184 This is a
A good example of this is the case of OpenAI’s attempts
to make ChatGPT less toxic. This was done by making
topic that is relevant for many types of technology, but
the model recognize acts and language of violence,
the rapid growth of generative AI has brought labour
180
including sexual violence, incest, and barbaric acts. To
automation to the forefront.185
do so, the company needed human intervention to label
If employers can simply prompt an artificial AI model
toxic content, and outsourced the job to the US-based
to produce text or images, this may create incentives
company Sama. This company markets itself as company
or excuses to lay off people in areas such as creativewith an “ethical AI approach “ that lifted 50,000 persons
industries or journalism. For example, there is a risk thatimage generators make jobs in concept drawing andout of poverty.181 Despite fruitful deals between OpenAI
stock photos redundant,
and Sama, workers in Kenya
as it will be cheaper for
were paid under USD $ 2 per“Ratherthanfighting
povertyandcompanies to use anhour, with high pressure to
image generators than tolabel harmful and toxic data
oppression, big tech companies are
pay an artist or photogra9 hours per day with little
strengthening and using existingpher to create the images.
psychological help. Workers
As described above, thewere fired at the end of thepower structures and may reinforce
automation of content
contract.
poverty rather than solving it.”creation may also devalue
the work of actual huOpenAI does not disclosemans, while reducing thethe name of the companies
general quality of available content.
it outsources work to, which should be a transparency
In cases where employees are replaced by automatedrequirement to make sure ethical guidelines are followed
systems, this may also reduce service quality, for examacross AI companies’ supply chain. The Kenyan workers
ple in areas such as customer support. This may haveare visible now due to an investigation by Time magazine,
particularly serious consequences in sectors where end
but there are many more ghost workers intervening forusers depend on having access to human services, such
LLMs to be delivered to the public.
as when an eating disorder helpline laid off its staff toreplace human workers with a chatbot.186
According to the magazine Sustain, there are constantreports of moderators and clickworkers working for OpenAI, TikTok, and others being underpaid and not receiving the necessary psychological support for their work,
while also preventing them to unionize.182 Their plight isoften overlooked in the AI debate.183
struggles of poverty and trauma largely forgotten. Thisobfuscation of the human cost of automation is called‘ghost work’.179

2.9 Intellectual property
Because generative AI models create new content basedon already existing content, there are a number of questions about the intellectual property of both the originators of the training data, and the generated output.
There are vast amounts of content in the training dataof many generative AI models that are protected byintellectual property law. It is currently unclear whether

Norwegian Consumer Council

Ghost in the machine

the training of generative AI models without consent
from the artist/writer/photographer/subject is legal. For
example, there have been major protests in artist circles
against the development and use of image generators
trained on intellectual property content.187 This is particularly controversial when the AI models can generate
new images by emulating a specific artist’s style ordistinctive features.188

June 2023

39

2.9 Intellectual property

In January 2023, three artists filed a lawsuit against Stability AI and Midjourney over the use of Stable Diffusion,on the basis that the tool uses copyrighted images frommillions of artists as training data.189 Stable AI has introduced a system for artists to opt out of their work being
used to train Stable Diffusion, but this is a time-consuming process that puts the burden on individual artists
who did not ask to be part of a training data set in the firstplace.190 Furthermore, artists have argued that synthetic
content generated to resemble original artworks are “grotesque mockeries” and that it devalues the role of artists.191
There are also several unresolved legal questions about
who owns the copyright to a work created using generative AI.192 A computer cannot have intellectual property
rights, and it is unclear to what extent the end user ofthe model obtains copyright on a work created using
generative AI.

Norwegian Consumer Council

Ghost in the machine

June 2023

3. REGULATIONS

41

3. Regulations

Existing legal frameworks are always tested with the
emergence of new technologies, and generative AI is
no different. All technology-neutral laws may be applicable to generative AI when the technology is used in a
relevant context. However, since there is no precedent orcase law to draw upon, enforcement agencies play an important role in drawing the line between legal and illegal
training, deployment, design, and use of generative AI.
This will serve to clarify if and where there are loopholes
in existing legal frameworks when it comes to generative AI. Enforcement agencies also play a crucial role inensuring that companies developing and deploying generative AI conform with the boundaries already laid out
by lawmakers. In this way, the development and trainingof generative AI may be safe, fair, and accountable.
If existing laws do not sufficiently address the risks ofemerging technologies, it may be necessary to amendthem, or introduce new laws. There are many advances

Norwegian Consumer Council

Ghost in the machine

to regulate artificial intelligence across the world, someof which will also be highly relevant for generative AI. In
Europe, there are several ongoing processes that may
serve to improve existing frameworks or create new regulations to improve consumer rights and minimize harms
from technology. These opportunities must be used by
EU lawmakers.
Following below are some of the most prominent andrelevant legal areas for addressing the challenges ofgenerative AI on consumers as outlined in chapter 2 ofthis report, such as data protection, consumer law, andproduct safety law. The section is centred on European
legal frameworks, drawing on examples from the U.S.
when particularly relevant.193 Emerging laws, such as thedraft European AI Act, AI Liability Directive, and the revision of the Product Liability Directive, are also describedas far as they pertain to generative AI. The table below
summarizes some of the most important points.

June 2023

42

3. Regulations

EXISTING LAW
OR FUTURE LAW?

APPLICABLE TO
GENERATIVE AI?

EFFECT ON
GENERATIVE AI?

WHAT NEEDSTO BE DONE?

Existing.

Applicable to any partsof generative AI relating to personal data,
including in particular
the training data, input
and output of generative AI systems.

Controllers must abideby the requirementsin the GDPR for any
processing of personal
data.

Enforcement agencies must investigate
generative AI systems
to ensure compliancewith the existing legal
framework.

THE GENERALDATA PROTECTIONREGULATION (GDPR)

Existing.

THE UNFAIRCOMMERCIAL
PRACTICESDIRECTIVE(UCPD)

There are also
opportunities tomake changes in the
directive because of an
ongoing fitness check.

Applicable to generative AI systems in thecontext of commercial
practices.

Some DPAs are alreadyinvestigating certain
generative AI systems.

Traders must not
employ generative AI ina way that amounts tomisleading or aggressive practices under
the UCPD, or a practicein breach of the trader’s due diligence.

Consumer authorities must investigate
generative AI systems
to ensure compliancewith the UCPD.
The EU Commission
should make use of
the ongoing fitness
check to ensure abroad enough scopeof the UCPD, as wellas effective remedialmechanisms.

Existing.

Potentially applicable,
but there are someuncertainties related
to the definitions ofscope and harms in the
GPSD.

Producers must notplace unsafe productson the market.

Product safety authorities must take preventive action to address
harms stemming fromgenerative AI to thedegree possible under
the GPSD.

Will come into force by
the end of 2024.

Applicable.

Producers must notplace unsafe productson the market.

Product safety authorities must prepare forwhen the GPSR comes
into force, to apply it
to generative AI and
ensure that there are
no unsafe products on
the market.

GENERAL
PRODUCT SAFETYDIRECTIVE (GPSD)

GENERAL PRODUCTSAFETY REGULATION(GPSR)

Norwegian Consumer Council

This includes several
data subject rights,
such as the right torectification anddeletion.

Ghost in the machine

June 2023

43

3. Regulations

THE DIGITALSERVICES ACT (DSA)
IN THE CONTEXT
OF CONTENT
MODERATION

EXISTING LAW
OR FUTURE LAW?

APPLICABLE TO
GENERATIVE AI?

EFFECT ON
GENERATIVE AI?

Will be fully applicable to all entities in
its scope in February
2024, and to desig­
nated very large online
platforms (VLOPs)
and very large online
search engines (VLOSEs) by the end of thesummer 2023.

Seemingly not directly
applicable to generative AI systems.

Content moderation
requirements on thegenerated text.

Existing.

Applicable.

Companies developingor deploying generative
AI may not abuse their
dominant position in
the market.

Competition authorities must monitor themarket for generative
AI to ensure there are
no anti-competitive
practices.

Currently being negotiated, trilogues to begin
in 2023.

Likely applicable, but
uncertain whether
generative AI systems
will be regulated separately as foundation
models (Parliament Position), in the context
of high-risk systems,
prohibited practices,or in the context ofchatbots or deepfakes(Commission Draft), or
as a general purpose
AI system (Council
Position).

Still very uncertain.

EU lawmakers mustensure the AIA takes
account of the harms
outlined in chapter 2 ofthis report, by ensuringconsumer rights andnecessary obligations
on the whole generative AI actor chain

Will likely be applicable
to downstream use ofgenerated content, or
generative AI systems
embedded in digital
services that are covered by the DSA.

EU COMPETITION LAW

THE ARTIFICIALINTELLIGENCE ACT(AIA)Expected to be fully
applicable by April/May
2026 at the earliest,
if there is a trilogue
agreement by January
2024.

Existing.

WHAT NEEDSTO BE DONE?

Likely not applicable.

PRODUCT LIABILITY
DIRECTIVE (PLD)

Norwegian Consumer Council

Ghost in the machine

June 2023

44

3. Regulations

THE REVISEDPRODUCT LIABILITY
DIRECTIVE(REVISED PLD)

EXISTING LAW
OR FUTURE LAW?

APPLICABLE TO
GENERATIVE AI?

EFFECT ON
GENERATIVE AI?

WHAT NEEDSTO BE DONE?

Coming, currently
being negotiated.

Uncertain, namely due
to a recent judgementon the current PLD.

May allow consumers
to seek compensation,
but not for non-material harms, which is a
substantial limitationin the context of generative AI.

EU lawmakers shouldamend the proposal
in a manner that gives
consumers the rightto also claim compensation for non-material harms under therevised PLD.

Coming, currently
being negotiated.

May be applicable, depending on the AIA.

May allow consumers
to seek compensation,
but currently contains
substantial limitations.

The AILD is still early inthe political processes,
and EU lawmakers must
amend the proposal
in a manner that gives
consumers effective
options to seek compensation for harms
from generative AI.

AI LIABILITY
DIRECTIVE (AILD)The list of legal frameworks touched upon on in this report
is not comprehensive, and only covers EU laws. Many otherEU laws will also apply to generative AI in different contexts,
such as human rights law, anti-discrimination law, andemployment law – many of which could have been included in this assessment but were left out due to capacity
restraints. Similarly, the considerations of the applicability of different legal frameworks to generative AI is notcomprehensive.

Norwegian Consumer Council

Ghost in the machine

The overview presented in this report is thus a contribution to the discussion on remedies to the harms presented by generative AI, but extensive legal analysis will benecessary to determine the effect of these frameworks
on generative AI.

June 2023

45

3.1 Data protection law

3.1 Data protection law
The General Data Protection Regulation (GDPR)194 applies
to the processing195 of personal data196 by companies
established in the European Union or by companies
established outside the European Union, when thesecompanies process the personal data of a data subject inthe European Union (EU) or the European Economic Area
(EEA).197
The obligations in the GDPR primarily apply to “controllers”, the entity that determines the purposes andmeans of processing personal data.198 Some obligations
are also laid on the “processors”,199 an entity processing
personal data on behalf of a controller. As mentioned inchapter 1.1.2, development and deployment of generative AI involves several actors at different stages ofthe process. It is crucial that the different actors in thegenerative AI actor chain clearly define their roles, toensure ­compliance with the GDPR throughout the wholeprocess.
As companies develop and deploy generative AI models,the GDPR could be applicable to at least three aspects ofthe system: the training data used to develop the generative AI model, the outputs from the generative AI model,
and the generative AI model itself.
As described throughout this report, generative AI models analyse large amounts of data, typically scraped fromthe internet. Some of these data points are undeniably
personal data, which means that the GDPR is applicableto the processing. Similarly, the GDPR applies to theprocessing of personal data from individuals’ prompts
to generative AI models, associated with them throughpersonal accounts or similar.
It is possible to use generative AI to generate images,
text, videos, and audio relating to identifiable natural
persons. The GDPR will therefore clearly be applicable
to some output as well as the input.200 This holds true

regardless of whether the generated information is correct,
meaning that a deepfaked photo or an incorrect statement
related to an identifiable individual is still personal data.
The way that a generative AI model works, the model
will not necessarily include any personal data – there
are no actual pictures of individuals in the model itself– but the output may be an identifiable image of a real
person. However, even if the model does not containpersonal data directly, researchers have been able toextract training data from large language models. Large
language models are generally more vulnerable to such
extraction than their smaller counterparts.201 As mentioned above, the training data includes personal data
as well, thereby potentially allowing for the extractionof personal data from the generative AI models. Someauthors have argued that the possibility of extracting
personal data from a model means that the model itself
could be considered personal data.202 Thus it is possiblethat the GDPR might apply to the models themselves, inaddition to the input and output of the models.
Under the GDPR, the processing of personal data generally requires a legal basis.203 There is a general prohibition on the processing of special category data, which includes categories such as personal data revealing racialor ethnic origin, political opinions, health, and biometricdata.204 In cases where the training data and/or output
of a generative AI model includes special categories of
personal data, the controller must have a legal basis thatexempts this prohibition.
As of May 2023, some light has been shed on the legal
bases that some developers claim for processing personal data for the development of generative AI models.
After scrutiny from the Italian DPA,205 OpenAI added
a section in its privacy policy for international users,
claiming legal bases such as performance of a contract
and a broad legitimate interest to for example develop,

course of a purely personal or household activity”, cf. art. 3. Generation

195 Art. 4 (2) GDPR.
196 Art. 4 (1) GDPR.

of output relating to a natural person as part of purely personal activity,

197 Art. 3 GDPR.

where the output is not shared online, is therefore not necessarily regulatedby the GDPR. The GDPR can still apply to the system owner.

198 Art. 4 (7) GDPR.
199 Art. 4 (8) GDPR.

204 Art. 6 GDPR.

200 The GDPR does not apply to “natural persons’ processing of data in the

205 Art. 9 (2) GDPR.

Norwegian Consumer Council

Ghost in the machine

June 2023

46

3.1 Data protection law

improve or promote its services.206 Google, on the other
hand, has not yet released its chatbot Bard in the EU.207
There has been speculation that this may be due to the
GDPR,208 and at the time of writing there is no mention ofa legal basis for processing personal data in the context
of Bard.209
In addition to needing a legal basis for processing personal data, there are various other relevant legal requirements for processing personal data for the training, development, deployment, and use of generative AI models.
The discussion on the principles of data protection by
design and by default,210 data minimization211 and purpose
limitation212 in the context of training machine learningmodels is nothing new, and the principles also apply tothe training of generative AI models when personal data
is involved.213
The principle of data minimization involves collecting
and processing as little personal data as possible forthe stated purposes of processing. Purpose limitation
includes not using personal data for other purposes
than stated at the point of collection, and to not store
the personal data for longer than necessary to fulfil
these purposes. As the training of generative AI models
requires large amounts of data, and the models are often
developed to be general purpose, these principles may
come into conflict with the approach taken by many
developers of generative AI models.
3.1.1

technical complexity of how our models work, we may
not be able to correct the inaccuracy”.217 In other words,
it is highly questionable whether it is technically feasible
for OpenAI to provide data subject rights and comply with
the GDPR.
It is also questionable whether an opt-out system suchas the one implemented by OpenAI can be compliant
with the GDPR. For the opt-out system to be effective, itwould require the individual’s knowledge that a generative AI model was trained on their personal data. This is
not evident to consumers unless they are frequent usersof the generative AI models, and even then, it is unlikelythat they would understand the extent of the processingof personal data.218
A significant hurdle that relates to deleting personal datafrom the training data is the sheer size of the data sets
used to train generative AI models.219 The work related
to the collection, cleaning and preparation of data sets
is generally not prioritized by AI practitioners, in favour ofmodel development.220 Consequently, companies’ ability tofind and delete data traces of any individual is compromiseed by their lack of oversight and documentation of thedata sets, which is at odds with data protection law.

DATA SUBJECT RIGHTS
People whose personal data is processed (data subjects)
have several rights under the GDPR. This includes therights to erasure (having personal data deleted),214 rectification (having personal data corrected),215 and objection(protesting about the processing of personal data).216
It is still unclear how companies developing and deploying generative AI models will be able to fulfil requests toprovide data subject rights in practice. After scrutiny of
ChatGPT by the Italian Data Protection Authority, OpenAI
introduced an opt-out mechanism to have personal
data removed from the training data, and the possibility
to correct inaccurate personal information. However,
OpenAI clarifies in its privacy policy that “[g]iven the

210 Art. 25 GDPR.

215 Art. 16 GDPR.

211 Art. 5 (1) (c) GDPR.

216 Art. 21 GDPR.

212 Art. 5 (1) (b) GDPR.   

218 See however requirements of information for data subjects in

214 Art. 17 GDPR.

Norwegian Consumer Council

art. 12-14 GDPR.

Ghost in the machine

June 2023

47

3.1 Data protection law

3.1.2

THE ITALIAN DPA’S DECISION CONCERNING 		CHATGPT
There have already been efforts to apply the GDPR togenerative AI models. On March 31st 2023, the Italian
Data Protection Authority (DPA) imposed a temporary
limitation on OpenAI, the owner of ChatGPT, regarding
the processing of personal data of Italian individuals. At
the same time, the DPA opened an inquiry into the factsof the case.221 This was due to several potential breachesof the GDPR, such as issues related to the processing of
personal data about end users of the ChatGPT service,processing of personal data in relation to the training of
the model, as well as processing of personal data duringcontent generation. As a result, OpenAI temporarily
blocked access to ChatGPT for individuals located in
Italy. While this addressed some of the data protection
issues described by the DPA, it was for example still
possible for peoples outside of Italy to generate personal
data concerning Italian citizens.
Some of the potential breaches pointed out by the Italian
DPA have more far-reaching consequences than others.
It is possible for OpenAI to implement measures to address issues such as data breaches and age verification
mechanisms without altering its model significantly.
Generation of inaccurate personal data seems moredifficult to address, even though it is in line with OpenAI’s
general attempts to increase accuracy in its models.
As mentioned above, OpenAI already states that it may
not be able to correct inaccuracies,222 and it seems very
unlikely that the company would be able to guarantee
accurate personal data about individuals, either intrinsically in their model or through content moderation, as
discussed above and in chapter 2.3.2.
The final and most damning issue that the Italian DPA
raised is that OpenAI appeared to have no legal basis toprocess personal data about Italian citizens to train its
model. As the GDPR is harmonized in the EU, this would
effectively mean that OpenAI did not have a legal basis

223 Umbrella term for AI systems designed to perform a wide range of tasks

to train its generative AI model on personal data fromany data subjects in the EU or the EEA. While OpenAI
has not shared information about its training data, it issafe to assume that it contains personal data about data
subjects in the EU and the EEA, for instance scraped
from the internet.
It is technically possible to prepare new data sets forthe training of subsequent GPT models, cleaning thedata sets to remove personal data about data subjects
in the EU and the EEA, but this would be extremely timeand resource intensive, and probably halt development
significantly. In any case, this issue raises the questionof whether general purpose generative AI models223 andthe GDPR can coexist in their current form.
On April 28th 2023, ChatGPT was reinstated in Italy, after OpenAI introduced various data protection
measures, such as the opt-out mechanism describedabove, a mechanism to exercise the right to erasure of
personal data, a new information notice including thelegal bases used for processing, and age specification
requirements.224 While OpenAI has on paper introduced
additional data protection measures, it is unclear how
consumers would be able to make effective use of theirrights,  such as the right to opt-out from their personal data being used to train the generative AI model, inpractice.
Even if OpenAI were to have legitimate interests that areoverridden by the fundamental rights of data subjects,225
OpenAI seemingly did not conduct this balancing act
before releasing its generative AI model to the public.
OpenAI’s compliance with for example the principles of
lawfulness of processing or accountability226 therefore
appears dubious at best. The Italian DPA’s apparent
acceptance of these claims could prove problematic, ascompliance with the GDPR goes beyond the quick fixes
deployed by OpenAI.

226 Art. 6 and 5 (2) GDPR.

across different domains.
225 Art. 6 (1) (f) GDPR.

Norwegian Consumer Council

Ghost in the machine

June 2023

48

3.2 Consumer law

It seems evident that the GDPR’s protections of Individuals will require extensive analysis and swift enforcement
to be effective. Even if the GDPR can serve to protect
individuals, the regulation has been criticized for being slow
and complicated to enforce particularly in cross-bordercases.227 The European Data Protection Board (EDPB)
has however established an EU-wide “task force” tocoordinate investigations and enforcement on ChatGPT,
so there will evidently be more legal development.228 The
French DPA has received several complaints and published an action plan involving ChatGPT.229 Both German
and Spanish DPAs are also considering actions.230

Although OpenAI and ChatGPT has been the main focus
of GDPR enforcement in the field so far, others are likely
to follow as different models become widely used. Thereis clearly appetite for enforcement, and so the GDPR will
be an important legal framework in ensuring generative AI that respect data protection rights for all data
subjects.

3.2 Consumer law
The Unfair Commercial Practices Directive (UCPD)231 lays
out the legal provisions governing business-to-consumer
practices in the EU and EEA. It is technology- neutral,
applies to all business-to-consumer transactions, andis meant to work as a catch-all to protect the consumer’’decision-making in the commercial environment. The
directive targets traders’232 practices, to ensure thesecommercial practices233 are not unfair, often throughrequirements on disclosure and openness.
Certain commercial practices are banned outright,
through the blacklist of unfair commercial practices in
the UCPD Annex 1. In addition to the banned practicesof Annex 1, there are several broad, discretionary legal
provisions in the UCPD. A commercial practice is unfair
if it is misleading234 or aggressive,235 causing (or likely tocause) an average consumer to take a transactional decision she would not have taken otherwise.
Transactional decisions have been broadly defined andinclude consumer decisions such as adding an item toa virtual basket or entering a shop. In the most recent
guidelines to the UCPD, published in December 2021,
the EU Commission also included examples such ascontinuing to use a service by browsing or scrolling,236
thus apparently broadening the scope of the transactional
decision test of the UCPD to entail business practices at thecore of the attention economy. As the Guidelines are notlegally binding, it is not yet clear how they will be interpretedby Consumer Authorities and courts in practice.

There is also a general clause under which a practice isunfair if it is contrary to the requirements of ­professional
diligence and it distorts, or is likely to distort, the average consumer’s economic behaviour. This serves as asafety net to ensure that unfair practices not coveredby the blacklist, misleading practices or aggressive
practices can still be subjected to the fairness assessment of the UCPD.237 The requirement of professional
due diligence could also serve as a bridge to other legal
frameworks, such as non-discrimination law, making it
possible to integrate jurisprudence from such laws in the
consumer law context.238
Generative AI as stand-alone models or embedded inother, consumer-facing services can potentially be
addressed by the UCPD in several ways, either argued ina traditional monetary sense, or by keeping consumers
on the service. In any case, the applicability of the UCPD
depends on the generative AI model being used in thecontext of a commercial practice.
Bing is currently employing advertisements in theirgenerative AI search,239 which requires distinct labellingto ensure that it is not a misleading practice. If a text
generator is used to persuade the consumer to stay engaged with the service, for instance through persistentcommunications, particularly targeting the consumer’s
identified weaknesses (as may be the case for chatbots
programmed to generate and simulate romantic involvement), this could also amount to an aggressive practice.
As mentioned in chapter 2.1.4.3, companies are already232 Art. 2 (b) UCPD.

234 Art. 6 -7 UCPD.

233 Art. 4 (d) UCPD.

235 Art. 8-9 UCPD.

Norwegian Consumer Council

Ghost in the machine

June 2023

49

3.2 Consumer law

be offered by digital services as a necessary pillar of
professional diligence, rather than focusing on disclosure requirements. The EU Commission should takeadvantage of the ongoing digital fitness check, which isan evaluation of the current EU consumer law’s fitness
to ensure a high level of protection,243 to address these
problems.

attempting to integrate generative AI into their shopping
experiences, which could potentially mislead consumers
into buying a product by providing inaccurate information.
There have been calls to address consumer challenges
stemming from generative AI through the UCPD. The
European consumer organization BEUC240 addressed aletter about generative AI, and particularly text generators, to DG JUST and the Consumer Protection Cooperation Network (CPC Network) on April 21st, 2023.241 In theletter, BEUC brought attention to various ways in which
generative AI is deployed that may influence consumer
behaviour in a way that is in breach of the UCPD, also taking account of vulnerable groups such as children.

3.2.1

There is clearly a need for consumer authorities toconsider and address illegitimate use of generative AI incommercial contexts as they are employed in an increasing number of services. The UCPD may be used to tackle
certain challenges related to generative AI in the context
of commercial practices. Particularly relevant examples
may be if a generative AI model is used in a way thatprovides consumers with false or misleading information
about for example products, or if the system owner omits
information from the terms of services.
At the same time, there are some potential limitations
to the UCPD’s applicability to generative AI. Presumably,
generative AI may often be used to increase consumer
interaction with a trader’s service and boost engagement by capitalising on the AI’s capability to seem like asentient interlocutor despite its obvious limitations. Thismay mislead consumers to dedicate much more timeand attention to such services than they normally would,
absent the illusion created by the model, for example
if a chatbot is designed to simulate romantic emotions
toward the consumer. However, the UCPD’s usefulness
against practices unfairly extracting consumer attention
and engagement is not yet certain. This would require a
broad understanding of what constitutes “transactional
decisions” that so far is only based on the European
Commission’s (non-binding) Guidelines, rather than theletter of the law. As such, many relevant practices are
not clearly covered by the UCPD.242
There is also the question of whether the remedialmechanisms of the UCPD are sufficient, unless they areapplied in a way to also require a ’fairness by design‘ to

CONSUMER LAW IN A U.S. SETTING
In the United States, the Federal Trade Commission (FTC)
enforces the FTC Act, which gives the FTC the authority to enforce violations of unfair and deceptive tradepractices.244 Unlike the European Consumer Authorities,
the FTC is mandated to prescribe rules to target specificdeceptive practices or unfair methods of competition
within the scope of the FTC Act.245 As of June 2023, theFTC is in the midst of a broad rulemaking process thatcan create new rules proscribing specific practices thatare always unfair and deceptive. This means that theFTC could react more forcefully and specifically to newtechnologies on the market.
The FTC has already issued guidelines on AI in 2021,
calling for transparency, unbiased results, accountability,
and more.246 In 2023, amidst the surge of generative AI
products and services, the FTC also issued a statement
to businesses, reminding them of the need to advertise
AI in a truthful and responsible way.247
Although the FTC has not yet undertaken enforcement actions against companies deploying or training
generative AI, it has had cases with strong equitable
remedies including Algorithmic Disgorgement.248 The
remedy requires the company to delete data and models/algorithms built on that data when the consumers’ rights
were violated in the collection process. This powerful
remedy may improve the practices of companies thatfear being forced to delete their models.
The Center for AI and Digital Policy filed a complaint to
the FTC on March 3rd, 2023, asking for a moratorium on
the release of further commercial versions of ChatGPT
beyond GPT4, and for rulemaking relating specifically togenerative AI.249 The FTC does not open cases for individual
consumers, but can base investigations on such complaints
and reports. It therefore seems likely that the FTC might
react to this particular complaint due to the mass adoptionof generative AI, in order to minimize consumer harm.

240 A European consumer umbrella organization of which the NorwegianConsumer Council is a member.

Norwegian Consumer Council

Ghost in the machine

June 2023

50

3.3 General product safety law

while the White House announced several initiatives
on May 5th, including committing the public sector to
mitigating risks.251

On the 25th of April 2023, the FTC, Consumer Financial
Protection Bureau (CFPB), Equal Employment Opportunity Commission (EEOC), and Department of Justice’s Civil
Rights Division announced that they intend to enforce
against “discrimination and bias in automated systems,250

3.3 General product safety law
placed on the market.259 Such a consideration must takeaccount of the precautionary principle, meaning thata product may be presumed unsafe in the absence of
scientific certainty about potential harms and harmful
effects of the product.

Product safety legislation is meant to ensure that products placed on the market are safe for consumers to use.
Current European general product safety law is basedon the General Product Safety Directive (GPSD).252 By
the end of 2024, the General Product Safety Regulation(GSPR)253 will replace the GPSD. Both legal instruments
are relevant in the context of generative AI.
3.3.1

As explained throughout this report, it seems clear thatgenerative AI may in fact pose considerable risks to consumers, especially mental health risks. Such risks may
for example arise from the generation and subsequentdissemination of inaccurate personal data or deepfakes,
from the deployment of highly manipulative and personified generative AI models, or in situations where
consumers use generative AI models for mental health or
medical advice purposes.

THE GENERAL PRODUCT SAFETY DIRECTIVE
The GSPD complements sector specific legislation andapplies to any risks from a product not covered by otherlaws.254 In practice, the GSPD performs the role of asafety net, ensuring safety requirements for all productson the European market.
The legislation requires that producers place only safe
products on the market.255 While the Directive’s definition of a product is broad enough to theoretically coverharms resulting from software linked to a product,256 thescope does not explicitly include or exclude software. Itsapplicability to GPT-models and other purely software
based generative AI models is therefore still uncertain.
A product is deemed safe when it does not present any
risk, or more than minimal risk to consumers’ safety andhealth under normal and foreseeable use.257 This has traditionally covered physical impacts on persons, such as
physical injuries or property damage, and mental healthis not explicitly mentioned in the GSPD. While someargue that intrinsic mental health risks from products
could be covered by the GPSD,258 the lack of explicit
reference makes its applicability to mental health risks
more uncertain.

There have been moves to apply the GPSD to generative
AI, through alerting safety authorities to investigate
safety risks of generative AI. The European consumer
organization BEUC sent a letter to the Consumer Safety
Network on April 12th, 2023 in this regard.260 The letter
particularly brought attention to risks to consumer
mental health.
3.3.2 THE GENERAL PRODUCT SAFETY REGULATION
A new General Product Safety Regulation (GPSR) has
been approved by the EU, which will come into force by
the end of 2024. The new regulation will repeal the GPSD
and widens the scope of products by bringing software
into the scope, as well as explicitly mentioning mental
health.261 It will also require any producer to consider aproduct’s evolving, learning and predictive functionalities when assessing the risks of the product,262 which isclearly relevant in the context of generative AI products.

Competent authorities are required to consider whether products are in fact safe, even after they have been

254 Art. 1 (2) GPSD.

259 Art. 8 (2) GPSD.

255 Art. 3 (1) GPSD.

261 Cf. recital 19 GPSR.

257 Art. 2 (1) (b) GPSD.

262 Art. 6 (1) (h) GPSR.

Norwegian Consumer Council

Ghost in the machine

June 2023

51

3.4 Competition law

The applicability of the GPSD to generative AI models
may be uncertain, but it seems clear that the GPSR will
apply to generative AI models. It is necessary that safety
authorities take preventive action to address harms
stemming from generative AI to the degree possible

under the current legal framework. This will also help
safety authorities prepare thoroughly to enforce the
GPSR as soon as it comes into force and starts applying
to generative AI models.

3.4 Competition law
At its core, EU competition law serves to preventanti-competitive practices so markets remain contestable, and consumers can benefit from lower prices,
better quality of products and services, and morechoice and innovation.
The essence of EU competition law is found in theTreaty on the Functioning of the European Union (TFEU),
although implemented through other regulations. First,
it is prohibited for companies to make anti-competitive
agreements.263 Second, companies may not abuse their
dominant position.264
The concept of a “dominant position” is highly context-based and will depend on how the “relevant market”
is defined. This includes, for example, the availabilityof alternative products and consumers’ willingness toswitch to these alternative products.265 As discussed
in chapter 2.1.3 above, the deployment of generative AI
carries with it a risk of power concentration in the handsof a few actors. This may lead to certain companies
becoming dominant within their respective markets, forexample generative AI based search engines, shopping
assistants, etc.

The digital sector notoriously consists of a very few,
massive actors, often referred to as big tech. It is crucial
that any emerging market relating to generative AI faces
scrutiny from antitrust agencies early, to avoid a similar
emergence of highly dominant companies in this market,
who may be tempted to abuse their position. It is noteworthy that several big tech companies are investingheavily in generative AI.
Competition enforcement agencies clearly have a role toplay to address some of the harms outlined in this report.
The UK Competition and Markets Authority (CMA) has
opened an initial review of competition and consumer
protection considerations in the development and use
of generative AI. Other competition authorities shouldclosely monitor future developments in this sector andbe ready to intervene early if they notice behaviours orpractices falling withing their remits, which could be anti-competitive. This would help ensure that the emerging
generative AI sector remains fair and competitive.

3.5 Content moderation
The Digital Services Act (DSA)266 is a new EU regulation,
aiming to improve the mechanisms for removing illegal
content, and to protect individuals’ rights, including freedom of speech and a high level of consumer protection.
It will be an important tool in introducing new content
moderation to online services.
The DSA applies to online intermediary services, meaning
conduit services, caching services and hosting services.267
In practice, this means that the DSA applies to services

connecting consumers to goods, services, and content –such as online marketplaces, social media platforms, cloud
hosting services, and internet access providers.
It will be fully applicable to all entities in its scope inFebruary 2024, while very large online platforms (VLOPs)
and very large online search engines (VLOSEs), some ofwhich have already been designated by the European
Commission,268 will have to oblige by the new rules as ofthe end of the summer 2023.

263 Art. 101 TFEU.
264 Art. 102 TFEU.

Norwegian Consumer Council

267 Art. 3 (1) (g) DSA.

Ghost in the machine

June 2023

52

3.6 The draft Artificial Intelligence Act

Generative AI models are not clearly covered by the DSA.
The most relevant type of service covered by the DSA is“hosting services”.269 Even then the applicability of the
DSA is not clear, since the content provided by generative AI models is largely generated by the models themselves, rather than consumers or other third parties.
While generative AI models as stand-alone services may
not be covered by the DSA, the DSA may be applicable tocompanies that wish to embed generative AI models intheir platforms and services. The integration of ChatGPTin the search engine Bing, which is a designated VLOSE,
may for example in effect trigger the content moderation
requirements of DSA for the generated content.
Any content generated by generative AI and subsequently shared or stored by consumers on services coveredby the DSA will similarly be covered by content moderation
requirements. In both cases, the DSA appears to apply tothe downstream provision and utilization of the generated
content, rather than the content generation model itself.

3.6 The draft Artificial Intelligence Act
In April 2021, the EU Commission published a proposal
for the Artificial Intelligence Act (AIA), laying down harmonized rules across the EU and the EEA “to foster thedevelopment, use and uptake of artificial intelligence in
the internal market”.270
An EU legal framework aimed at regulating AI should be
expected to also regulate generative AI. However, theCommission’s proposal for the AIA was published before
the widespread adoption of generative AI during thewinter 2022/2023. In the aftermath, discussions among
EU lawmakers have focused heavily on how to properly
regulate generative AI as part of the AIA.
As the AIA is not yet complete, it is not yet certain how
it will apply to generative AI in practice. In the following, the Commission’s draft, as well as the Council andParliament positions to the AIA as of May 2023 are briefly
outlined. The overview begins with relevant aspects fromthe Commission’s proposal for the AIA.
3.6.1

THE EU COMMISSION’S PROPOSALThe Commission’s proposal for the AIA (hereafter “Draft
AIA”) applies to any provider placing an AI system on themarket.271 AI systems are defined broadly, as systems

generating output based on machine learning approaches,
logic- and knowledge-based approaches, or statistical
approaches.272 In other words, the scope of the Draft AIA
is broad, encompassing many types of systems.
The AIA has a risk-based approach, regulating different
types of AI systems based on their risks to individuals or
society. Certain exclusively listed practices are prohibited,273 and may never be put on the European market. AI
systems may also be classified as high-risk, for example ifthey are among the high-risk systems listed in Annex 3.274
Most of the Draft AIA focuses on regulating high-risk AI
systems and setting out legal requirements for AI operators of these systems.275 This includes legal requirements
such as creating a quality management system,276 including
a risk management system,277 to meeting data quality criteria,278 accuracy, robustness, and cybersecurity measures,279
as well as creating technical documentation.280
Notably, any provider of a system that falls outside thescope of high-risk systems have few, if any, requirements
laid upon them through the Draft AIA. There are some
limited transparency requirements for applications suchas chatbots and for deepfake material.281 All providers of

276 Art. 17 Draft AIA.
271 Art. 2 (1) (a) Draft AIA.

277 Art. 9 Draft AIA.

272 Art. 3 (1) Draft AIA, cf. Annex 1.

278 Art. 10 Draft AIA.

273 Art. 5 Draft AIA.

279 Art. 15 Draft AIA.

274 Art. 6 Draft AIA.

280 Art. 11 Draft AIA.

275 See Draft AIA Title III.

281 Art. 52 Draft AIA.

Norwegian Consumer Council

Ghost in the machine

June 2023

53

3.6 The draft Artificial Intelligence Act

systems that are not high-risk may also voluntarily abideby the requirements for high-risk systems,282 but haveno legal obligation to do so. Thus, the Draft AIA targets avery broad range of AI systems, while imposing obligations on very few of them, and precluding Member statesfrom imposing additional obligations.283 This makes
the scope of high-risk AI systems especially important.
Additionally, the Draft AIA contains very limited rights for
consumers.
It is unclear how generative AI systems fit into the Commission’s Draft AIA from 2021. It is plausible that theywould have fallen inside the scope of the Draft AIA generally. For more specific requirements, the generative AI
systems would have to be related to one of the high-risk
categories in Annex III, be used in the context of chatbots or deepfakes for limited transparency requirements
or be related to a prohibited practice.284

exclusion is made in good faith.289
In practice, it would be very hard for providers to ensurethat their system can never be used in high-risk settings,
as defined in the Council Position Annex 3. The extent of
the “good faith” requirement is therefore crucial; either
it practically requires that all generative AI systems are
subject to high-risk obligations, or it may prove to be too
low of a threshold, serving as the basis for meaninglessdisclaimers from developers. In any case, the Council
Position’s effect on generative AI systems is uncertain.
3.6.3 THE EU PARLIAMENT’S POSITION ON THE AIA
As of May 2023, the EU Parliament’s position is still being
negotiated. The LIBE and IMCO committees of the EU
parliament approved a compromise position on May
11th.290 It will be voted over in plenary in mid-June. The
following therefore constitutes the parliament’s presumed position at the time of writing (hereafter “Parliament Position”).

3.6.2 THE EU COUNCIL’S POSITION ON THE AIAIn the Council’s position on the AI Act (hereafter, “Council
Position”),285 general purpose AI is addressed. The definition reads as follows:
an AI system that - irrespective of how it is placedon the market or put into service, including as open
source software - is intended by the provider toperform generally applicable functions such as image
and speech recognition, audio and video generation,
pattern detection, question answering, translation andothers; a general purpose AI system may be used in aplurality of contexts and be integrated in a plurality ofother AI systems;286
This would apply to all types of generative artificial
intelligence touched upon in this report.
In the Council Position, general purpose AI would be
subject to high-risk obligations if they “may be used as
high-risk AI systems or as components of high risk AI
systems”.287 Whenever the AI provider explicitly excludes
all high-risk uses in the instructions or information relating to the generative AI, the general purpose AI system
is exempted.288 This exemption may only apply when theOverall, the European Parliament’s position significantly
improved the European Commission’s proposal. Consumers are granted new rights including a right to be informed when being subject to a decision from a high-risk
AI system,291 a right to complain to an authority about an
AI system,292 and the right to bring a supervisory authority to court if it fails to take action.293 Consumers werealso given the right to ask for collective redress when an
AI system has caused harm to a group of consumers.294
When it comes to generative AI, rather than focusing on
general-purpose AI like the Council’s position, the European Parliament introduces a new concept: ‘foundation
model’. The Parliament Position defines ‘foundation model’ as an AI model that is “trained on broad data at scale,is designed for generality of output, and can be adapted
to a wide range of distinctive tasks”.295 All providers offoundation models have additional obligations, regardless of whether the models are provided as a standalone
model or embedded in a system, or whether the models
are open source or closed source.296 These obligations
include requirements to identify and reduce risks to for
example health, safety and the rule of law, data governance measures, appropriate levels of for example

282 Art. 69 Draft AIA.

291 Art. 68c Parliament Position.

284 Art. 5 Draft AIA.

292 Art. 68a Parliament Position.

286 Art. 3 (1) (b) Council Position.

293 Art. 68b Parliament Position.

287 Art. 4b (1) Council Position.

294 Art. 68d Parliament Position.

288 Art. 4c (1) Council Position.

295 Art. 3 (1c) Parliament Position.

289 Art. 4c (2) Council Position.

296 Art. 28b Parliament Position.

Norwegian Consumer Council

Ghost in the machine

June 2023

54

3.6 The draft Artificial Intelligence Act

come before the AIA is finalized and should address theharms outlined in this report both through introducing
consumer rights, and through obligations on the entire
generative AI actor chain.

performance, predictability, interpretability throughout
its life cycle, energy efficiency measures, and technicaldocumentation.297
Models used as a basis for generative AI systems areclearly meant to be encompassed by the definition offoundation model. The Parliament Position refers explicitly to such systems in the article placing obligations onproviders of foundation models. Foundation models usedin generative AI systems are placed under additional obligations of transparency, adequate safeguards against
the generation of unlawful content, and the publicationof a “sufficiently detailed summary of the use of training
data protected under copyright law”.298

EU lawmakers must ensure that industry lobbying does
not water out obligations and rights for consumers in the
AIA. According to a report from Corporate Europe Observatory, industry lobbying efforts significantly weakened
several relevant provisions in the proposed regulationfrom the Commission, including pushing for excluding
general purpose AI systems from the regulation.301 EU
lawmakers must be vigilant to avoid falling for lobbyingtactics, that will no doubt increase during the final stageof negotiations.

3.6.4 THE ARTIFICIAL INTELLIGENCE ACT MUSTPROTECT CONSUMERS
Since generative AI models are not built for a particular
context, and they allow for wide-scale use, some authors
have argued that generative AI models do not fit well into
the risk-based system of the Draft AIA.299 Instead, theyargue targeted measures such as system risk monitoring should be considered. At the same time, as outlined
throughout this report, generative AI systems pose significant risks that must be mitigated at the development
stage of the system, rather than at the time the system
is placed on the market, or after the system is placed on
the market.300

While lawmakers finalize the AIA, other key enforcement agencies must also ensure the safety and rights
of consumers. The AIA will not be fully applicable forseveral years,302 and in the meantime, it is necessaryfor enforcement agencies of other legal frameworks toprotect consumers from the harms of generative AI, as
outlined above.

“With the AIA, however, European
lawmakers have a unique chance tointroduce enforceable guardrails
to protect consumers against therisks of generative AI.”

It is not yet certain how the AIA will apply to generative
AI. With the AIA, however, European lawmakers have aunique chance to introduce enforceable guardrails toprotect consumers against the risks of generative AI.
That chance must be used effectively in the months to

3.7 Liability
There are several relevant liability laws in the EU, meant
to ensure that consumers receive fair compensation
when defective products lead to harm. Some legal
instruments are already in force, while others are still
being negotiated.  

3.7.1

PRODUCT LIABILITY DIRECTIVE
Product liability rules enable consumers to claim compensation for damage caused by a defective product.
The current EU liability rules – the Product Liability
Directive (PLD) - was adopted in 1985 and it is not clear
whether it applies to generative AI.

297 For the complete list of requirements, see art. 28b(2) Parliament Position.

applicable until at the earliest April/May 2026, if there is a trilogue

298 Art. 28b (4).

agreement by January 2024.

302 The exact timing differs between the different positions of the EU
institutions. At the earliest the AIA will be fully applicable after 24 months
after entering into force. This effectively means it may not be fully

Norwegian Consumer Council

Ghost in the machine

June 2023

55

3.7 Liability

First, there is no consensus on whether the PLD applies
to digital services and software such as generative AI.
Secondly, even if the PLD applied to digital services suchas generative AI, a court ruling from the Court of Justiceof the European Union established that information provided by a product is not covered by the PLD.303 Since theoutput of generative AI essentially is information in theform of speech, text or pictures, the fact that information is not covered by the PLD means that generative AI
is most likely not covered by the PLD.
3.7.2

REVISED PRODUCT LIABILITY DIRECTIVE
The European Commission has introduced a proposal
for an updated product liability directive (revised PLD).
The updated directive is also meant to cover software,
including AI systems.304
The revised PLD proposal, similarly to the PLD, operates
with a non-fault-based liability scheme. While consumers will not have to prove fault with the operator or the
producer of a product, consumers will have to prove therelevant defect in a product, the consumer harm and thecausal link between the defect and the harm.
It remains to be seen how the recent ruling from the
Court of Justice on the current PLD will apply in thecontext of the revised PLD, thereby whether information
provided by a product will be covered by the revised PLD
or not.
In any case, for generative AI systems, most of the potential consumer harms are non-material, as outlined inchapter 2. Such harms are exempt from the PLD, which
only covers material damage.
In total, neither the current PLD nor the revised PLD
appear well suited to allow consumers compensation forharms from generative AI systems. The final wording of
the PLD proposal will however have to be considered.

3.7.3

AI LIABILITY DIRECTIVE
In parallel with the AIA, and as an addition to the PLD, theEuropean Commission has proposed the AI Liability Directive (AILD), which is meant to provide consumers withthe possibility to claim compensation for harms caused
by AI systems. While the EU Commission has proposeda draft, it is likely that the Directive will not be finalized
until the AIA has been adopted.

The AILD proposal gives consumers the possibility toclaim compensation for all material hams, and non-material harms if allowed within national legal frameworks.
However, there are serious limitations to the proposal
which will significantly reduce its effectiveness in providing compensation for consumer harms.305
Consumers that wish to claim compensation for harms
caused by AI systems are required to prove fault of the
AI system operator. Proving fault in the context of the
AILD means that the consumer will have to prove that theAI system operator is not operating in accordance with
the EU rules, including the AIA. Proving such non-compliance will require high technical and legal knowledge,
none of which regular consumers have or should be
expected to have. Since proving fault is a prerequisite forother mechanisms in the AILD, such as a presumption of
causality between fault and the output leading to harm,
this limitation is substantial. To make the AILD effectively protect consumers, non-fault-based liability should beestablished in claims from consumers and the burden of
proof should be reversed.306
When it comes to generative AI, it is still uncertain
whether the AIA will classify generative AI as an ‘AI
system’. If it does, since the definition of AI systems in
the AILD refers to the definition of the AIA, then the AILD
would be applicable to generative AI. However, compensation claims for damages created due to inaccurate
information (in the form of text, images, or audio) are
not harmonised under the AILD proposal. This meansthat compensation claims for consumers related togenerative AI would therefore need to be assessed at thenational level on a case-by-case basis.
The AILD is still early in the political process, and EU
lawmakers have room to amend the proposal in a mannerthat would give consumers real options to seek compensation from harms arising from generative AI, regardlessof national rules. This is necessary to increase consumer
protection in the face of the harms outlined in chapter 2.

303 Case C-65/20, VI v Krone (2021), https://curia.europa.eu/juris/document/
document.jsf;jsessionid=7A0662FAD49ED462BE89A81594FAF809?text=&docid=242561&pageIndex=0&doclang=EN.

Norwegian Consumer Council

Ghost in the machine

June 2023

56

3.8 Industry standards and guidelines

3.8 Industry standards and guidelines
Industry players are already developing guidelines thatare meant to increase the transparency of both development and use of generative AI models.307 There have also
been industry calls to halt the development of new generative AI models.308 The calls for a halt in development
typically focus on the risks of very advanced models,and has coincided with a voluntary halt of developmentof GPT5 from Open AI.309 The many risks from current
generative AI models such as systems based on GPT4, as
identified and discussed in chapter 2 of this report, arehowever addressed insufficiently.
There are increasingly also calls for creating voluntary
codes of conduct for developers and deployers of generative AI.310 For the EU in particular, the Commission isaiming to create a pact with companies ahead of the newrules in the AI Act. Policy makers in the EU are reportedlyplanning to co-create a code of conduct “within months”,
which means the codes of conduct would be created ornegotiated at the same time as the trilogue negotiationsof the AIA. Industry representatives, such as Google,
would in this way be perfectly placed to increase theirlobbying efforts.

other self-regulatory rules with the industry and third
countries on the same topic. Secondly, it is unclear what
requirements a voluntary agreement can include when
the legal requirements for these actors in the EU are notyet defined. There is an obvious risk that the voluntarycommitments will not be in line with the final legal text.
The voluntary codes of conduct would be strongly affected by industry players’ views on feasibility and possibility
to monetize products, instead of consumer and humanrights. Finally, the AIA may also be unduly affected byindustry players. This is unacceptable and must not beallowed.
Industry standards and guidelines are unsuited totackling the risks stemming from the deployment of
GPT-models already on the market, as they tend to act asa lowest common denominator, lacks sufficient enforcement mechanisms and independent oversight. Insteadof relying on voluntary industry commitments while
the AIA is not yet applicable, authorities should focus
on the enforcement of existing laws such as consumer
protection, data protection or product safety legislation.
Policymakers and lawmakers should on their part strive
to avoid self-regulatory regimes.

The envisioned process creates a dual risk. First, theEuropean Commission has a role to play in trilogue
negotiations and it cannot fulfil this role impartially if,
at the same time, it is negotiating a code of conduct or

Norwegian Consumer Council

Ghost in the machine

June 2023

4. THE WAY FORWARD

58

4. The way forward

Throughout this report, we have outlined significant
harms and challenges related to the development, training, deployment, and use of generative AI. These are not
hypothetical risk of future dystopias, but tangible harmsthat affect people and populations today.
We believe that while these issues are concerning, they
are not insurmountable. Many of the problems regarding generative AI are echoes of well-known issues fromother sectors, but the rapid development and adoptionof generative AI models mean that it is pertinent to takeactions to address the harms. We cannot afford to wait
until the technology is so embedded in our lives and social structures that it is too late to change the directionof its development and use.
Technology is not an untameable beast but must be
adapted to and shaped by the rules and values of democratic societies. To ensure that generative AI is developed and used in accordance with consumer and humanrights, it is clearly insufficient to rely on companies toregulate themselves. It is the responsibility of policymakers and enforcement agencies to set boundaries
for how the technology is trained, developed, deployed,
and used. Therefore, lawmakers must not pass legislation based on what industry actors state is technicallyfeasible, but rather what is necessary to provide safe andconsumer-centric technology in the years to come.
Below we set forth some fundamental principles that we
believe should be at the heart of how society approaches
generative AI. This is followed by several action points
for enforcement agencies, policymakers, and lawmakers,
and enforcement agencies. We hope that these points
will provide a blueprint for a human-centric approach to
the technology.

Norwegian Consumer Council

Ghost in the machine

June 2023

59

4.1 Consumer rights principles that are key for safe and responsible AI

4.1 Consumer rights principles that are key for safe and responsible AI
To ensure that generative AI is safe, trustworthy, fair, equitable, and accountable, there is a need for overarching principles that address consumer rights. The principles set pit below provide a foundation for how
policymakers and enforcement agencies should approach the opportunities and pitfalls of generative AI.
Many of the principles are already defined in current consumer law, but we urge policymakers and enforcement agencies to ensure that they are in fact the foundation for development and deployment of generative
AI. This is key to ensure a technological landscape that respect fundamental consumer rights today and in the
coming years.

















Consumer rights must be respected. The onset of generative AI must not undermine or
displace already established consumer- and human rights, such as a right to information andtransparency, fairness and non-discrimination, safety and security, privacy and personal dataprotection, and redress.
Consumers must have the right to object and to an explanation whenever a generative AI model is used to make decisions that have a significant effect on the consumer.
Consumers must have a ‘right to be forgotten’ to have personal data deleted from generative AI
models, to rectify harms from for example having false information produced about them.
Consumers must have a right to interact with a human instead of generative AI where this isrelevant, for example in cust­omer service contexts. This should not incur additional costs on
the consumer, so that consumers are not treated disparately and unfairly based on their ability
to pay.
Consumers must have a right to redress and compensation for any damages suffered from theuse of generative AI.
Consumers must have a right to collective redress, and to be represented by consumer organizations and other civil society groups with exercising their rights.
Consumers must have a right to complain to supervisory authorities or launch legal actions incourt when use of a generative AI model is in breach of the law.
Developers and deployers of generative AI models must establish systems to ensure thatthese rights are available to consumers in practice.

Norwegian Consumer Council

Ghost in the machine

June 2023

60

4.2 Policy recommendations

4.2 Policy recommendations
The decision on how to integrate technology into society is an inherently political question. Elected officials
and governments have a responsibility to ensure that technology serves the people, rather than the whims ofa small number of companies. A consumer-oriented technology policy means that people and societies should
not be used as testing laboratories for experimental technologies. The lessons learned from the broader transition into a digital society without sufficiently accounting for the rights of citizens and the effects on society
must inform how governments approach generative AI.
To ensure responsible, fair, and accountable innovation on the terms of society, we need robust policies thatare future proof, rather than being swept up in the hype and then having to course correct in the aftermath.
Below, we present several action points for how governments and policymakers should approach generative
AI and similar technologies.
4.2.1

CALLS TO ACTION AND EMPOWERMENT OF ENFORCEMENT AGENCIES
While emerging technologies such as generative AI are sometimes described as a regulatory wild west, there
are already comprehensive legal frameworks in place. We believe that many of these regulations are alreadysuited to address a number of the issues described in chapter 2 of this report. However, to effectively protect
people from exploitation, discrimination, and other abuses of power, these laws must be enforced.
Effective enforcement requires that enforcement agencies have the necessary powers, expertise, and resources, and that entities that are unable to or refuse to comply are sufficiently dealt with. In this section we
present several necessary approaches and prerequisites for enforcement agencies to use their existing tools
to shape technology in a consumer-friendly way.












Enforcement agencies must not wait for upcoming regulation. Instead, they must
­immediately investigate generative AI systems and apply relevant legal provisions fromtheir respective legal frameworks, such as data protection, competition, product safety
and consumer law.
Collaborative cross-sectorial investigations, where several enforcement agencies are
involved in the same investigation, may be necessary to manage the risks stemming from generative AI. It may be necessary to appoint a coordinator for algorithmic enforcement, to ensure
progress in such collaborative investigations.
Enforcement agencies should be empowered to conduct post-market surveillance of generative AI models and have the option to order product recalls or the deletion of algorithmic
systems or parts thereof that fail to comply with relevant legislation. Such orders should beaccompanied by significant monetary fines to deter bad practices.
Enforcement agencies must have all necessary resources to enforce infringements of theirrespective legal frameworks, including personal and technical competence and the necessary
technical tools. With the deluge of AI-generated content, scaling up market surveillance andenforcement will be necessary.
Transnational and national technological expert groups should be established to support
enforcement agencies in enforcement endeavours.
Research must be conducted on how to augment enforcement by using technology.

Norwegian Consumer Council

Ghost in the machine

June 2023

61

4.2 Policy recommendations

4.2.2 DECISION MAKERS – STRATEGIC MEASURES












Governments must take critical perspectives on generative AI into account in their national AI
strategies. Over-arching principles to promote safe and human-centric generative AI must be
built in from start rather than as add-ons, because add-ons are costly, and mistakes will erode
trust.
Governments must adopt a critical and precautionary approach to using generative AI in the
public sector. The public sector has a particular responsibility to employ generative AI in alegal and trustworthy manner, and public procurement should be utilized to actively influence
providers of stand-alone generative AI software or systems with embedded generative AI. In
particular, the public sector should require transparency, to understand the technology before
employing it in public sector contexts.
Governments should strongly consider establishing institutions, or empower existing
institutions, to continuously oversee and publicly debate and define mandatory principles
to ensure that technology is developed, deployed, and used in the public interest.
Governments should ensure public funding of research on data practices and consumer andsocietal harms stemming from generative AI.
International trade agreements must not hollow out transparency obligations for generative
AI systems, nor other obligations that are necessary to ensure consumer rights.
Shareholders and investors in companies developing and deploying generative AI systems,
particularly shareholder or investors from the public sector, should require that steps aretaken to avoid and/or mitigate exploitative practices, environmental impact, etc. Companies
should be required to have ethical guidelines as well as reporting on the steps taken.

4.2.3 NEW LEGISLATIVE MEASURES
While there are already many legal frameworks in place that may be suited to address the harms of generative
AI, there will undoubtedly be areas with legal gaps and loopholes. In cases where existing laws are not sufficient, it is necessary to create new frameworks to protect consumers from harm. As described in the previous
chapter, there are already several legislative initiatives in progress, and it is crucial that these processes
result in strong future proof rules that are founded on consumer and human rights.
We call on policymakers and lawmakers to take a strong stance in favour of consumer protection and preserving human rights. It is necessary to have robust legal measures, including strict obligations on developers
and deployers of generative AI systems to operate in a transparent and accountable manner, and to restrict
development, deployment, and use of systems that are fundamentally incompatible with these rights.

Norwegian Consumer Council

Ghost in the machine

June 2023

62

4.2 Policy recommendations

4.2.3.1 Particular forms of generative AI that warrants additional scrutinyCertain forms of manipulative techniques in generative AI systems should be banned. This
may include, for example, significant restrictions on anthropomorphized models, including
the use of first-person language, the use of emojis and similar symbols, and simulating humanemotions and similar attributes. Such restrictions could be dependent on the context and purpose of use. The threshold for acceptable techniques and applications should be higher whenused by vulnerable groups such as children.
For certain uses of generative AI systems, it might be necessary to require pre-approval fromrelevant enforcement agencies before deployment. Models that may lead to the exploitation or
discrimination of consumers, in particular vulnerable consumers such as children, may be an
example of this.
Policymakers must ensure that coming legislation is future proof, to keep authorities fromlagging behind rapid technological advancement. This involves principles and regulations thatare technology neutral.

4.2.3.2 Obligations for developers and deployers of generative AI
Responsible development, deployment and use of generative AI presupposes that it is possible to control
how the systems work, inspect the training data, oversee social and environmental impacts, and more. Whiletransparency itself is not a panacea, it is a prerequisite to ensure that technologies do not undermine consumer and human rights. This cannot be a responsibility left to the companies themselves.
There is an urgent need for independent oversight, research, and auditing of generative AI systems, to ensurethat companies are held accountable if something goes wrong, to identify and root out biases and inaccuracies, and to otherwise ensure legal compliance and mitigate harms. Therefore, we present a number ofmeasures that should be imposed on developers and deployers of generative AI systems.
TRANSPARENCY

Developers and deployers of generative AI systems must be obligated to report and publish
documentation about their risk assessments, risk mitigation strategies, how they conduct
content moderation, standardized performance metrics, etc. to the public. This should be doneon two levels: a shorter, less technical version for consumers generally, as well as an in-depth
description for experts from civil society, academia, and other third parties.
All companies developing and deploying generative AI systems should have an obligation to
publish all information on energy use, water use and carbon emissions for the whole lifecycle of the generative AI model and provide prognostics on future emissions for the day-to-day
use. This includes resources needed for the production of the hardware, training of models,development, deployment, and use. A standardised model on calculating emissions, water use
and energy use should be established that all companies need to use rather than inventing theirown calculation system.
Developers and deployers should disclose the names of all their suppliers publicly and report ina transparent way on the working conditions in their whole supply chain, including living wage
and psychological support for moderators of violent and disturbing content, and packages forlabourers needed only temporarily for a given task.

Norwegian Consumer Council

Ghost in the machine

June 2023

63

4.2 Policy recommendations









Developers of baseline models for generative AI must be obligated to register their models in acentralized, public system, to ensure oversight of relevant generative AI models.
Deployers of generative AI in consumer facing interfaces and services must be obligated todisclose how the generated content is influenced by commercial interests of developers,
deployers, or third parties. This is particularly relevant when the content generated serves toinform consumer choices, such as content generated in the context of search queries or similar.
Deployers of generative AI systems must be obligated to disclose whenever consumers areinteracting with a generative AI system, and whether consumer-facing systems are using
artificial intelligence to affect the outcome of a decision.
Public and private organizations must be obligated to disclose whenever content has been
generated by generative AI, when that content may have an effect on decisions affecting consumers, consumer rights more broadly, or democratic processes.

RISK MITIGATION








Deployers of generative AI systems must be obligated to carefully consider the context in which
the generative AI system is to be deployed. Deployers of generative AI systems should not use
generative AI systems without a careful risk assessment, including a mapping of what problems the system is meant to solve, verification that the system is compliant with relevant laws,
risks to consumers and consumer rights, risks to human rights, risks of harm likely to impact
vulnerable groups, adverse impacts on the environment, foreseeable societal and collective
harms, privacy harms, etc.
Deployers of generative AI systems must be obligated to implement effective measures tomitigate the risks uncovered in the risk assessment before deploying the system, to arriveat acceptable residual risk. If the risks cannot be mitigated, or the system does not solve theproblems it is meant to solve, the system should not be deployed in that context.
Developers and deployers of generative AI systems must be obligated to involve representatives from populations that may be impacted by the technology, in particular marginalized
and vulnerable groups and communities. This precipitates democratic participation and interdisciplinary involvement. Stakeholder participation is necessary in the context of development
and training of generative AI models, and for associated topics such as risk assessments, risk
mitigation strategies, and content moderation, which needs to account for different culturalcontexts, languages, etc.
Deployers of generative AI systems must be obligated to monitor and address the system’s
impact on consumers after deploying the system, conducting continuous risk assessments
and mitigation to arrive at acceptable residual risk, taking particular account of impacts on
marginalized and vulnerable groups and communities.

Norwegian Consumer Council

Ghost in the machine

June 2023

64

4.2 Policy recommendations

ACCOUNTABILITY














There must be clear rules on accountability and liability for harmful effects of generative
AI systems, such as harms to privacy, safety, consumer rights, and fundamental rights more
broadly. These rules must clearly indicate which company in the supply chain is liable or requirethat developers and deployers of generative AI clearly establish responsibility and liability
between themselves.
Any accountability scheme must make it easy for consumers, enforcement agencies andcourts to hold the companies liable for consumer harm.
Developers of generative AI systems must be responsible for the data they use, representativeness in the data sets, their data cleaning and labelling practices, and other design choicesthat will affect all downstream uses of the systems. Such choices must be carefully documented, so downstream developers and deployers can consider the generative AI system’s risks andsuitability.
Technical standards and certification schemes should be developed and used to assist developers and deployers of generative AI systems to develop, train, deploy and use the system ina responsible and legal way. However, policymakers must not outsource human rights, politicaland legal issues to standards bodies. Governments must ensure civil society participation insuch bodies, or, lacking such participation, should not rely on them.
Generative AI systems and models must be auditable by independent researchers, enforcement agencies, and other third parties. This is essential to mitigate the risk of bias and discrimination, ensure responsible use of training data, and to ensure compliance with applicablelegal requirements.
Audits should at least involve the training data, data collection practices, data labelling
practices, content moderation practices, sustainability reports and the algorithmic models.
Audits should be carefully documented to ensure accountability and reproducibility, and should
be based on standardized auditing requirements.
Companies should be obligated to have quantitative and timebound commitments to reduce
consumption, based on the calculations of the carbon emissions, energy, and water use ofdeveloping and deploying generative AI. This progress should also be audited by an external andindependent actor with public reporting. This means bigger models might be reduced and lessambitious. Claims of zero net carbon activities, and carbon offsetting schemes should not be
the default model with which companies “compensate” emissions, they should reduce emissions in their own activities.

Norwegian Consumer Council

Ghost in the machine

June 2023

65

Endnotes 1–35

Endnotes
1“ChatGPT reaches 100 million users two

10

Hugging Face. https://huggingface.co/

Study of Natural Language Communication

bigscience/bloom

Between Man and Machine”, Joseph Weizen-“Stability AI Launches the First of its Sta-baum (1966). http://www.universelle-auto-com/technology/2023/feb/02/chatgpt-100-

bleLM Suite of Language Models”, Stability

mation.de/1966_Boston.pdf

million-users-open-ai-fastest-growing-app

AI (2023). https://stability.ai/blog/stability-

Large language models are sophisticated AI

ai-launches-the-first-of-its-stablelm-suite-

ChatGPT, and Human Rights”, Humanmodels that are designed to generate text

of-language-models

Rights Watch (2023). https://www.hrw.

months after launch”, Dan Milmo, The
Guardian (2023). https://www.theguardian.

2

11

27

“Pandora’s Box: Generative AI Companies,that resembles human language. They are

12

Midjourney. https://www.midjourney.com/

org/news/2023/05/03/pandoras-box-generative-ai-companies-chatgpt-and-hu-

normally trained on vast amounts of text

13

Dall-E. https://labs.openai.com

sources to “learn” patterns and grammar.

14

Stability AI. https://stability.ai

LLMs can be used for tasks such as machine

15

MusicStar.ai. https://musicstar.ai

translation, sentiment analysis, human-ma-

16“Herzog and Žižek become uncanny AI

makers are”, Naomi Klein, The Guardianchine interaction, proofreading, and manybots trapped in endless conversation”,(2023). https://www.theguardian.com/com-other purposes.

Benj Edwars, Ars Technica (2022). https://mentisfree/2023/may/08/ai-machines-hal-

3

ChatGPT, https://chat.openai.com/

arstechnica.com/information-technolo-

4“Microsoft reportedly to add ChatGPT togy/2022/11/herzog-and-zizek-become-un-

Bing search engine”, Johana Bhuiyan, The

canny-ai-bots-trapped-in-endless-conver-

Guardian (2023). https://www.theguardian.

sation/5

6

Hulette (2021). https://www.cs.princeton.
edu/news/how-recognize-ai-snake-oil

ElevenLabs. https://beta.elevenlabs.io

soft-chatgpt-bing-search-engine

18

Vall-E. https://vall-e.ioter”, Future of life Institute (2023). https://“Microsoft rolls out ChatGPT-powered

19“Introducing Make-A-Video: An AI systemfutureoflife.org/open-letter/pause-gi-

Teams Premium”, Reuters (2023). https://that generates videos from text”. Meta,

www.reuters.com/technology/micro-(2022). https://ai.facebook.com/blog/gen-soft-rolls-out-chatgpt-powered-teams-pre-erative-ai-text-to-video/

30 “Pause Giant AI Experiments: An Open Let-

ant-ai-experiments/
31“Policy makers: Please don’t fall for the
distractions of #AIhype”, Emily M. Bender,

mium-2023-02-02/

20 Make-A-Video. https://makeavideo.studio(2023). https://medium.com/@emilymenon-“A New Chat Bot Is a ‘Code Red’ for21

bender/policy-makers-please-dont-fall-for-google-search.html

Dreamix. https://dreamix-video-editing.
github.io

22 Stability Animation. https://platform.stabili-com/2022/12/21/technology/ai-chatgpt-

the-distractions-of-aihype-e03fa80ddbf1
32

ty.ai/docs/features/animation“The so-called “Godfather of the A.I.” joins
The Lead to offer a dire warning about the

23 “Create generative AI video-to-video rightdangers of artificial intelligence”, Geoffrey“A new era for AI and Google Workspace”,

from your phone with Runway’s iOS app”,

Hinton, CNN (2023). https://edition.cnn.

Product Announcement, Google (2023).

James Vincent, The Verge, (2023). https://com/videos/tv/2023/05/02/the-lead-geof-

https://workspace.google.com/blog/prod-

www.theverge.com/2023/4/24/23695788/

uct-announcements/generative-ai

generative-ai-video-runway-mobile-app-ios“The Galactica AI model was trained on

frey-hinton.cnn
33“What you need to know about generative

24 “Call for action to open an inquiry on gener-

AI and human rights”, Access Now (2023).

scientific knowledge – but it spat out

ative AI systems to address risks and harms

https://www.accessnow.org/what-you-alarmingly plausible nonsense”, Aaron J.

for consumers”, BEUC (2023). https://www.

need-to-know-about-generative-ai-and-hu-

Snoswell, Jean Burgess (2022). https://beuc.eu/sites/default/files/publications/

theconversation.com/the-galactica-ai-

BEUC-X-2023-045_Call_for_action_CPC_

model-was-trained-on-scientific-knowl-

authorities_Generative_AI_systems.pdf

interview with Evgeny Morozov”. Natasha

25 “End of the Billable Hour? Law Firms Get

Dow Schüll, Public Books (2013). https://edge-but-it-spat-out-alarmingly-plausible9“How to Recognize AI Snake Oil”, Doug

17

Times, (2023). https://www.nytimes.

8

lucinating-naomi-klein
29

com/technology/2023/jan/05/micro-

Google’s Search Business”, The New York7

man-rights
28 “AI machines aren’t ‘hallucinating’. But their

man-rights/
34 “The folly of technological solutionism: An

nonsense-195445

On Board With Artificial Intelligence”, Erinwww.publicbooks.org/the-folly-of-techno-“Facebook’s Powerful Large Language Model

Mulvaney, Lauren Weber, The Washington

logical-solutionism-an-interview-with-evg-

Leaks Online”, Joseph Cox, The Vice (2023).  

Post (2023). https://www.wsj.com/articles/

eny-morozov/

https://www.vice.com/en/article/xgwqgw/

end-of-the-billable-hour-law-firms-get-on-

facebooks-powerful-large-language-model-

board-with-artificial-intelligence-17ebd3f8

Boom”, Evan Selinger, The Slate, (2023).

“ELIZA--A Computer Program For the

https://slate.com/technology/2023/03/

leaks-online-4chan-llama

Norwegian Consumer Council

26

Ghost in the machine

35“The Delusion at the Center of the A.I.

June 2023

66

Endnotes 35–64

chatgpt-artificial-intelligence-solutionism-hype.html

ence-io

ative AI like ChatGPT”, Alex Engler,46 “Indigenous groups in NZ, US fear coloni-

Brookings. (2023). https://www.brook-

36 “Generative AI and the Digital Commons”,

sation as AI learns their languages”, Rina

ings.edu/blog/techtank/2023/02/21/

Saffron Huang, Divya Siddarth, (2023).

Chandran, Context (2023). https://www.

early-thoughts-on-regulating-genera-

https://arxiv.org/abs/2303.11074

context.news/ai/nz-us-indigenous-fear-col-“Indigenous groups in NZ, US fear coloni-

onisation-as-bots-learn-their-languages

37

sation as AI learns their languages”, Rina

47

“Google shared AI knowledge with the world

ChatGPT”, Klarna (2023). https://www.klar-

Chandran, Context (2023). https://www.

— until ChatGPT caught up”, Nitasha Tikuna.com/international/press/klarna-brings-

context.news/ai/nz-us-indigenous-fear-col-and Gerrit De Vynck, The Washington Post

onisation-as-bots-learn-their-languages(2023). https://www.washingtonpost.com/

38 “Lina Khan: We Must Regulate A.I. Here’sHow.”, Lina Khan, New York Times, (2023).

smoooth-shopping-to-chatgpt/
57“EU Consumer Protection 2.0 - Protecting

technology/2023/05/04/google-ai-stop-fairness and consumer choice in a digital

sharing-research/

economy”, BEUC (2022). https://www.

https://www.nytimes.com/2023/05/03/

48 Hypothetical AI system that demonstrates

beuc.eu/sites/default/files/publications/

opinion/ai-lina-khan-ftc-technology.html

human-level intelligence and autonomy.

beuc-x-2022-015_protecting_fairness_and_

Does not currently exist.

consumer_choice_in_a_digital_economy.

39 “What does a leaked Google memo reveal
about the future of AI?”, The Economist

49 “Microsoft Says New A.I. Shows Signs of

pdf(2023). https://www.economist.com/lead-

Human Reasoning”, Cade Metz, The New

58 “What Really Happened When Google Ousted

ers/2023/05/11/what-does-a-leaked-goo-York Times (2023). https://www.nytimes.

Timnit Gebru”, Tom Simonite, Wired, (2022).

gle-memo-reveal-about-the-future-of-ai

com/2023/05/16/technology/micro-

https://www.wired.com/story/google-tim-

40 “The Imminent Danger of A.I. Is One
We’re Not Talking About”, Ezra Klein, The

41

tive-ai-like-chatgpt/
56 “Klarna brings smoooth shopping tosoft-ai-human-reasoning.html
50 “OpenAI co-founder on company’s past

nit-gebru-ai-what-really-happened/
59 “Big tech companies cut AI ethics staff, rais-

New York Times (2023). https://www.

approach to openly sharing research:ing safety concerns”, Cristina Criddle andnytimes.com/2023/02/26/opinion/micro-‘We were wrong’”, James Vincent, The

Madhumita Murgia, Financial Times (2023).

soft-bing-sydney-artificial-intelligence.html

Verge (2023). https://www.theverge.

https://www.ft.com/content/26372287-“Microsoft pushing you to set Bing and Edgecom/2023/3/15/23640180/openai-gpt-4-

as your defaults to get its new OpenAI-pow-

launch-closed-research-ilya-sutskever-in-

ered search engine faster is giving off big

terview

ulation, warning of ‘existential risk”, Ellen

1990s energy”, Hasan Chowdhury and Shona

51

6fb3-457b-9e9c-f722027f36b3
60 “ChatGPT maker OpenAI calls for AI reg-

“We are a little bit scared’: OpenAI CEOFrancis, The Washington Post, (2023).

Ghosh, Insider (2023). https://www.busines-

warns of risks of artificial intelligence”, Ed-

https://www.washingtonpost.com/tech-sinsider.com/microsoft-wants-to-repeat-

ward Helmore, The Guardian (2023). https://

nology/2023/05/24/chatgpt-openai-artifi-

1990s-dominance-with-new-bing-ai-2023-2

www.theguardian.com/technology/2023/

cial-intelligence-regulation/

42 “Walled garden”, Andrew Froehlich, TechTarget (2023). https://www.techtarget.com/
searchsecurity/definition/walled-garden
43 “Snapchat’s AI chatbot is now free for all

mar/17/openai-sam-altman-artificial-intelligence-warning-gpt4
52 “GPT-4 and professional benchmarks: the61“OpenAI may leave the EU if regulations
bite”, Reuters (2023). https://www.reuters.
com/technology/openai-may-leave-eu-if-

wrong answer to the wrong question”, Arvind

regulations-bite-ceo-2023-05-24/

global users, says the AI will later ‘Snap’ you

Narayanan and Sayash Kapoor, AI Snake Oil

62 “Help! My Political Beliefs Were Altered byback”, Sarah Perez, TechCrunch (2023).

(2023). https://aisnakeoil.substack.com/p/a Chatbot!”, Christopher Mims, The Wall

https://techcrunch.com/2023/04/19/snap-

gpt-4-and-professional-benchmarks

Street Journal (2023). https://www.wsj.

chat-opens-its-ai-chatbot-to-global-users-

53 “OpenAI’s policies hinder reproduciblesays-the-ai-will-later-snap-you-back/research on language models”, Arvind

com/articles/chatgpt-bard-bing-ai-political-beliefs-151a0fe4

44 “Publishers Worry A.I. Chatbots Will Cut

Narayanan and Sayash Kapoor, AI Snake Oil

63 “Stack Overflow Bans ChatGPT For Con-

Readership”. Katie Robertson, The New(2023). https://aisnakeoil.substack.com/p/stantly Giving Wrong Answers”, Janus Rose,

York Times (2023). https://www.nytimes.

openais-policies-hinder-reproducible

Vice (2022). https://www.vice.com/en/arti-com/2023/03/30/business/media/publish-

54 “How trade commitments narrowed EU rules

cle/wxnaem/stack-overflow-bans-chatgpt-

ers-chatbots-search-engines.html

to access AI’s source codes”, Luca Bertuzzi,

for-constantly-giving-wrong-answers

45 “The AI takeover of Google Search

Euractiv (2023). https://www.euractiv.com/

64 “AI-assisted plagiarism? ChatGPT bot says

starts now”, David Pierce, The Vergesection/artificial-intelligence/news/how-

it has an answer for that”, Alex Hern, The(2023). https://www.theverge.

trade-commitments-narrowed-eu-rules-to-

Guardian (2022). https://www.theguardian.

com/2023/5/10/23717120/goo-

access-ais-source-codes/

com/technology/2022/dec/31/ai-assisted-

gle-search-ai-results-generated-experi-

Norwegian Consumer Council

55 “Early thoughts on regulating gener-

Ghost in the machine

plagiarism-chatgpt-bot-says-it-has-an-an-

June 2023

Endnotes 64–93

67

swer-for-that

believe AI was sentient. Now it’s happening”,

panion-replika-erotic-roleplay-updates

Timnit Gebru, Margaret Mitchell, Washington

84 “Snapchat is releasing its own AI chatbot‘worse than useless’ and ‘a pathological liar”,

Post (2022). https://www.washingtonpost.

powered by ChatGPT”, Alex Heath, The

James Vincent, The Verge (2023). https://

com/opinions/2022/06/17/google-ai-eth-Verge (2023). https://www.theverge.

www.theverge.com/2023/4/19/23689554/

ics-sentient-lemoine-warning/com/2023/2/27/23614959/snapchat-my-ai-

65 “Google employees label AI chatbot Bard

google-ai-chatbot-bard-employees-criti-

75

cism-pathological-liar
66 “ChatGPT is making up fake GuardianTimes (2023). https://www.latimes.com/

with me about booze and sex”, Geoffrey
A. Fowler, The Washington Post (2023).

Moran, The Guardian (2023). https://www.

column-afraid-of-ai-the-startups-selling-it-

https://www.washingtonpost.com/technol-

theguardian.com/commentisfree/2023/

want-you-to-beogy/2023/03/14/snapchat-myai/

76

toys and devices: Ethics, governance and“Publishers tout generative AI opportunities

www.bloomberg.com/news/newslet-

practical remedies”, Andrew McStay and Gi-

to save and make money amid rough media

ters/2023-04-03/chatgpt-bing-and-bard-

lad Rosne (2021). https://journals.sagepub.

don-t-hallucinate-they-fabricate
77“Deepfakes for all: Uncensored AI art model
prompts ethics questions”, Kyle Wiggers,

make-money-amid-rough-media-market/

articles/d41586-023-00758-y

TechCrunch (2022). https://techcrunch.

“The engineer who claimed a Google AI iscom/2022/08/24/deepfakes-for-all-uncen-

Its AI-Written Articles After Multiple Major“CNET Is Reviewing the Accuracy of All

78

sentient has been fired”, Mitchell Clark,sored-ai-art-model-prompts-ethics-ques-Corrections”, Lauren Leffer (2023). https://The Verge (2022). https://www.theverge.

gizmodo.com/cnet-ai-chatgpt-news-ro-com/2022/7/22/23274958/google-ai-engi-

bot-1849996151

neer-blake-lemoine-chatbot-lamda-2-sen-challenge of deepfakes”, Europol Innovation“Situating Search”. Chirag Shah and Emily

tienceLab (2022). https://www.europol.europa.

“Users Report Microsoft’s ‘Unhinged’ Bing AI

eu/cms/sites/default/files/documents/

Is Lying, Berating Them”, Jordan Pearson,

Europol_Innovation_Lab_Facing_Reali-

Vice (2023). https://www.vice.com/en/arti-

ty_Law_Enforcement_And_The_Challenge_

79

Though ChatGPT Wasn’t Built for It”, Rachel

cle/3ad39b/microsoft-bing-ai-unhinged-ly-

Metz, Bloomberg (2023). https://www.

ing-berating-users

tions/
88 “Facing reality? Law enforcement and the

Of_Deepfakes.pdf
89 “Fake images of Trump arrest show ‘giant

bloomberg.com/news/articles/2023-04-18/

80 “On the Dangers of Stochastic Parrots:

step’ for AI’s disruptive power”, Isaac Stan-ai-therapy-becomes-new-use-case-for-

Can Language Models Be Too Big?”,

ley-Becker, Naomi Nix, The Washington Post

chatgpt

Emily M. Bender, Timnit Gebru, Angelina(2023), https://www.washingtonpost.com/“Governments are embracing ChatGPT-ike

McMillan-Major, Shmargaret Shmitch-

politics/2023/03/22/trump-arrest-deep-bots. Is it too soon?”, J.D. Capelouto andell [sic] (2023). https://dl.acm.org/doi/fakes/

Diego Mendoza, Semafor (2023). https://

pdf/10.1145/3442188.3445922

90 “Found through Google, bought with Visa

“Human heuristics for AI-generated

and Mastercard: Inside the deepfake pornernments-using-chatgpt-bots

language are flawed”, Maurice Jakesch,economy”, Kat Tenbarge, NBC (2023).

“The dark side of artificial intelligence:

Jeffrey T. Hancock and Mor Naaman

https://www.nbcnews.com/tech/internet/

manipulation of human behaviour”, Georgios(2023). https://www.pnas.org/doi/10.1073/

deepfake-porn-ai-mr-deep-fake-econo-

Petropoulos, Bruegel (2023). https://www.

pnas.2208839120

my-google-visa-mastercard-download-rc-

www.semafor.com/article/03/03/2023/gov-

bruegel.org/blog-post/dark-side-artifi-

81

82 “Privacy Concerns in Chatbot Interac-cial-intelligence-manipulation-human-be-tions”, Carolin Ischen, Theo Araujo, Hilde

haviour

Voorveld, Guda van Noort and Edith Smit

“People keep anthropomorphizing AI. Here’s(2020). https://link.springer.com/chap-why”, Arvind Narayanan and Sayash Kapoor,

ter/10.1007/978-3-030-39540-7_3

AI Snake Oil (2023). https://aisnakeoil.

74

com/doi/10.1177/2053951721994877
87

Véliz (2023). https://www.nature.com/

70 “People Are Using AI for Therapy, Even73

“Chatbots shouldn’t use emojis”, Carissa

generative-ai-opportunities-to-save-and-

pdf/10.1145/3498366.3505816

72

86 “Emotional artificial intelligence in children’s

Rachel Metz, Bloomberg (2023). https://

Bender (2022). https://dl.acm.org/doi/71“AI Doesn’t Hallucinate. It Makes Things Up”,

gy-risks-fake-article

https://digiday.com/media/publishers-tout-

69

85 “Snapchat tried to make a safe AI. It chatsbusiness/technology/story/2023-03-31/

market”, Sara Guaglione, Digday (2023).

68

chatbot-chatgpt-openai-plus-subscription

want you to be”, Brian Merchant Los Angeles

articles. Here’s how we’re responding”, Chris

apr/06/ai-chatgpt-guardian-technolo67“Column: Afraid of AI? The startups selling it

83 “It’s Hurting Like Hell’: AI Companion Users

substack.com/p/people-keep-anthropo-

Are In Crisis, Reporting Sudden Sexual Re-

morphizing-ai

jection”, Samantha Cole, Vice (2023. https://“We warned Google that people might

www.vice.com/en/article/y3py9j/ai-com-

Norwegian Consumer Council

Ghost in the machine

na7507191“Scammers are using AI to impersonate yourloved ones. Here’s what to watch out for”,
Sabrina Ortiz, CDNET (2023).

92 “Exclusive: GPT-4 readily spouts misinformation, study finds”, Sara Fischer, Axios(2023). https://www.axios.com/2023/03/21/
gpt4-misinformation-newsguard-study
93 “AI presents political peril for 2024

June 2023

68

Endnotes 93–118

with threat to mislead voters”, Da-

Mark Minevich, Forbes (2023. https://vid Klepper and Ali Swenson, AP

www.forbes.com/sites/markminev-

News (2023). https://apnews.com/

ich/2023/01/29/the-generative-ai-revolu-

110 “Artist finds private medical record photos inarticle/artificial-intelligence-mis-tion-is-creating-the-next-phase-of-autono-

popular AI training data set”, Benj Edwards

information-deepfakes-2024-elec-

mous-enterprise/(ArsTechnica), 2022. https://arstechnica.

tion-trump-59fb51002661ac5290089060b3ae39a0
94 “Watermarking ChatGPT, DALL-E and102 “The Generative AI Revolution Is Creating
The Next Phase Of Autonomous Enterprise”,
Mark Minevich, Forbes (2023). https://adage.

lion-of-the-images-used-to-train-stable-diffusions-image-generator/

com/information-technology/2022/09/artist-finds-private-medical-record-photos-inpopular-ai-training-data-set/other generative AIs could help protect

com/article/marketing-news-strategy/

against fraud and misinformation”, Hanylevis-uses-ai-models-increase-diversity-in-

Can Language Models Be Too Big?”,

Farid. The Conversation (2023). https://cites-backlash/2482046

Emily M. Bender, Timnit Gebru, Angelina

theconversation.com/watermarking-111 “On the Dangers of Stochastic Parrots:

103 “Beck’s uses AI to create, and advertise,

McMillan-Major and Shmargaret Shmitch-chatgpt-dall-e-and-other-generative-ais-a new beer in experiment for futureell [sic] (2021). https://dl.acm.org/doi/

could-help-protect-against-fraud-and-mis-use cases”, Hannah Bowler, The Forbes

information-202293(2023). https://www.thedrum.com/

95 “Google introduces new features to help

pdf/10.1145/3442188.3445922
112 “On the Dangers of Stochastic Parrots:

news/2023/03/30/beck-s-uses-ai-create-

Can Language Models Be Too Big?”,

identify AI images in Search and elsewhere”,

and-advertise-new-beer-experiment-fu-

Emily M. Bender, Timnit Gebru, Angelina

Sarah Perez, TechCrunch (2023). https://ture-use-cases

McMillan-Major and Shmargaret Shmitch-

techcrunch.com/2023/05/10/google-intro-

104 “Ads are coming for the Bing AI chatbot,

duces-new-features-to-help-identify-ai-im-

as they come for all Microsoft products”,

ages-in-search-and-elsewhere/

Andrew Cunningham, ArsTechnica (2023).

ell [sic] (2021). https://dl.acm.org/doi/
pdf/10.1145/3442188.3445922
113 “The viral AI avatar app Lensa undressed

96 “Lærere fortvilet over ny kunstig intelligens”,

https://arstechnica.com/gadgets/2023/03/

me—without my consent”, Melissa Heikkilä,

Daniel Eriksen, NRK (2022). https://www.

ads-are-coming-for-the-bing-ai-chatbot-

MIT Technology Review (2022). https://www.

as-they-come-for-all-microsoft-products/

technologyreview.com/2022/12/12/1064751/

nrk.no/kultur/laerere-fortvilet-over-ny-kunstig-intelligens-1.16210580
97

the-viral-ai-avatar-app-lensa-undressedme-without-my-consent/“OpenAI’s attempts to watermark AI text hit

have ads”, Ryan Barwick, Marketing Brew

limits”, Kyle Wiggers, TechCrunch (2022).

(2023), https://www.marketingbrew.com/

114 “Stable Diffusion and DALL-E display bias

https://techcrunch.com/2022/12/10/ope-

stories/2023/05/23/yes-google-s-ai-in-

when prompted for artwork of ‘Africannais-attempts-to-watermark-ai-text-hit-

fused-search-engine-will-have-ads

workers’ versus ‘European workers”, Thomas

106 “The dark side of artificial intelligence:

Maxwell, Insider (2023). https://www.busi-

limits/98

105 “Yes, Google’s AI-infused search engine will

manipulation of human behaviour”, Georgios

nessinsider.com/ai-image-prompt-for-af-

detecting AI-written text, and the results

Petropoulos, Bruegel (2023). https://www.

rican-workers-depicts-harmful-stereo-

are troubling”, Armin Alimardani and Emma

bruegel.org/blog-post/dark-side-artifi-

A. Jane, The Conversation (2023). https://cial-intelligence-manipulation-human-be-theconversation.com/we-pitted-chatgpt-

haviour“We pitted ChatGPT against tools foragainst-tools-for-detecting-ai-written-text-

107 The Norwegian Consumer Council has

types-2023-4
115 “Inside the secret list of websites that makeAI like ChatGPT sound smart”, Kevin Schaul,
Szu Yu Chen and Nitasha Tiku, The Washing-

and-the-results-are-troubling-199774

previously described the negative effects

ton Post (2023). https://www.washington-

99 “OpenAI releases tool to detect AI-generatedof surveillance-based advertising and has

post.com/technology/interactive/2023/

text, including from ChatGPT”, Kyle Wiggers,

called for a general ban (2021). https://www.

ai-chatbot-learning/

TechCrunch (2023). https://techcrunch.

forbrukerradet.no/side/new-report-de-

116 “Google ‘fixed’ its racist algorithm by remov-com/2023/01/31/openai-releases-tool-to-

tails-threats-to-consumers-from-surveil-

ing gorillas from its image-labeling tech”,

detect-ai-generated-text-including-from-lance-based-advertising/

James Vincent, The Verge (2018). https://chatgpt/
100 “ChatGPT and Generative AI in Content

108 See for example Shoshana Zuboff’s book
“The Age of Surveillance Capitalism – The

Marketing”, Kelsey Voss, Insider IntelligenceFight for a Human Future at the New Fron-(2023). https://www.insiderintelligence.

tier of Power” (2019).

com/content/chatgpt-generative-ai-content-marketing
101 “The Generative AI Revolution Is Creating
The Next Phase Of Autonomous Enterprise”,

Norwegian Consumer Council

109 “Exploring 12 Million of the 2.3 Billion
Images Used to Train Stable Diffusion’s

www.theverge.com/2018/1/12/16882408/
google-racist-gorillas-photo-recognition-algorithm-ai
117 “Facebook apology as AI labels black men
primates”, BBC (2021). https://www.bbc.
com/news/technology-58462511

Image Generator”, Ando Baio, Waxy (2022).

118 “On the Dangers of Stochastic Parrots:

https://waxy.org/2022/08/exploring-12-mil-

Can Language Models Be Too Big?”,

Ghost in the machine

June 2023

69

Endnotes 118–142

Emily M. Bender, Timnit Gebru, Angelina

127 “How I Broke Into a Bank Account With an

Disorder Association say they are being

McMillan-Major and Shmargaret Shmitch-

AI-Generated Voice”, Joseph Cox, Vice,

replaced by AI”, Britney Nguyen, Insider

ell [sic] (2021). https://dl.acm.org/doi/(2023). https://www.vice.com/en/article/(2023). https://www.businessinsider.com/

pdf/10.1145/3442188.3445922

dy7axa/how-i-broke-into-a-bank-account-eating-disorders-nonprofit-reported-

119 “There’s More to AI Bias Than Biased Data,

with-an-ai-generated-voice

NIST Report Highlights”, The National

128 “Scammers use AI to enhance theirly-fired-humans-offer-ai-chatbot-2023-5
137 See for example the Commission’s draft

Institute of Standards and Technology

family emergency schemes”, FTC (2023).

of the AIA art. 14(1), https://eur-lex.

(2022). https://www.nist.gov/news-events/

https://consumer.ftc.gov/consum-

europa.eu/resource.html?uri=cel-

news/2022/03/theres-more-ai-bias-biased-

er-alerts/2023/03/scammers-use-ai-en-lar:e0649735-a372-11eb-9585-01aa75e-

data-nist-report-highlights

hance-their-family-emergency-schemes

120 “Ensuring artificial intelligence (AI)129 “Three ways AI chatbots are a security

d71a1.0001.02/DOC_1&format=PDF.
138 See some examples of this in relation

technologies for health benefit older

disaster”, Melissa Heikkilä, MIT Technology

to art. 22 GDPR, where companies havepeople”, Dr Vânia de la Fuente-Núñez,

Review (2023). https://www.technologyre-effectively been rubberstamping decisions.

WHO (2022). https://www.who.int/news/

view.com/2023/04/03/1070893/three-“Automated Decision-Making Under the

item/09-02-2022-ensuring-artificial-intel-

ways-ai-chatbots-are-a-security-disaster/

GDPR: Practical Cases from Courts and Dataligence-(ai)-technologies-for-health-benefit-older-people
121 “AI image generator Midjourney blocks pornby banning words about the human reproductive system”, Melissa Heikkilä, MIT Tech-

130 “Opwnai: Cybercriminals starting toProtection Authorities”, Future of Privacy

use chatgpt”, Check Point Research

Forum (2022). https://fpf.org/wp-content/(2023). https://research.checkpoint.

uploads/2022/05/FPF-ADM-Report-R2-sin-com/2023/opwnai-cybercriminals-start-

gles.pdf.

ing-to-use-chatgpt/

139 Also called “automation bias” (see fornology Review (2023). https://www.tech-

131 “Dual use of artificial-intelligence-powered

nologyreview.com/2023/02/24/1069093/

drug discovery”, Fabio Urbina, Filippa Lent-

example “Automation Bias in Intelligent
Time Critical Decision Support Systems”,

ai-image-generator-midjourney-blocks-

zos, Cédric Invernizzi and Sean Ekins (2022).

M.L. Cumming (2012), https://arc.aiaa.org/

porn-by-banning-words-about-the-human-

https://www.nature.com/articles/s42256-

doi/10.2514/6.2004-6313) and “algorithm

reproductive-system/

022-00465-9.epdf

aversion” (see for example “Algorithm Aver-

132 “ChatGPT - The impact of Large Language

sion: People Erroneously Avoid Algorithms

Models on Law Enforcement”, Europol

after Seeing Them Err” Dietvorst, Simmons

Stanley-Becker, The Washington Post

(2023). https://www.europol.europa.eu/

and Massey (2014) https://papers.ssrn.com/(2023). https://www.washingtonpost.com/

publications-events/publications/chatgpt-

sol3/papers.cfm?abstract_id=2466040),

technology/2023/03/30/midjourney-ai-im-

impact-of-large-language-models-law-en-

age-generation-rules/

forcement

122 “How a tiny company with few rules ismaking fake images go mainstream”, Isaac

123 “Oh No, ChatGPT AI Has Been Jailbroken To

respectively.
140 “How Can Artificial Intelligence Combat Cli-

133 “Companies are struggling to keep corporate

mate Change?”, World101. https://world101.

Be More Reckless”, Claire Jackson, Kotaku

secrets out of ChatGPT”, Sam Sabin, Axios

cfr.org/global-era-issues/climate-change/(2023). https://kotaku.com/chatgpt-ai-(2023). https://www.axios.com/2023/03/10/

how-can-artificial-intelligence-com-

openai-dan-censorship-chatbot-red-chatgpt-ai-cybersecurity-secrets

bat-climate-change and “AI Is Essential for

134 “Amazon warns employees not to share

Solving the Climate Crisis”, Hamid Maher,

dit-1850088408
124 “Quantifying ChatGPT’s gender bias”, Sayash

confidential information with ChatGPT

Hubertus Meinecke, Damien Gromier, Mateo

Kapoor and Arvind Narayanan. AI Snake Oilafter seeing cases where its answer ‘closely

Garcia-Novelli and Ruth Fortmann, Boston(2023). https://aisnakeoil.substack.com/p/

matches existing material’ from inside theConsulting Group (2022).https://www.bcg.

quantifying-chatgpts-gender-bias

company”, Eugene Kim, Business Insider

com/publications/2022/how-ai-can-help-

125 “AI Is Steeped in Big Tech’s ‘Digital Co-(2023). https://www.businessinsider.com/

climate-change

lonialism”, Grace Browne, Wired (2023).

amazon-chatgpt-openai-warns-employ-

https://www.wired.com/story/abeba-bir-ees-not-share-confidential-information-mi-

Nations Environment Programme (2022).

crosoft-2023-1

https://www.unep.org/news-and-stories/

hane-ai-datasets/

141 “Emissions Gap Report 2022”, United

126 “The AI chatbot can misrepresent key facts

135 “OpenAi ceo says ai will give medical advice

with great flourish, even citing a fake Wash-

to people too poor to afford doctors”, Frankington Post article as evidence”, Pranshu

Landymore, The Byte (2021). https://futur-

Verma and Will Oremus, The Washington

ism.com/the-byte/openai-ceo-ai-medi-

Carbon Footprint”, Josh Saul and Dina Bass,

Post (2023). https://www.washingtonpost.

cal-advice

Bloomberg (2023). https://www.bloomberg.

com/technology/2023/04/05/chatgpt-lies/

Norwegian Consumer Council

136 “Helpline workers for the National Eating

Ghost in the machinestory/new-pact-tech-companies-take-climate-change
142 “Artificial Intelligence Is Booming—So Is Its

com/news/articles/2023-03-09/how-much-

June 2023

70

Endnotes 142–171

energy-do-ai-and-chatgpt-use-no-one-

Carbon Footprint”, Josh Saul and Dina Bass,

knows-for-sureBloomberg (2023). https://www.bloomberg.

143 “Generative AI Breaks The Data Center:com/news/articles/2023-03-09/how-much-mentisfree/2023/may/08/ai-machines-hallucinating-naomi-klein
162 “Climate Change Impacts and Risks”, IPCC

Data Center Infrastructure And Operatingenergy-do-ai-and-chatgpt-use-no-one-(2022). https://www.ipcc.ch/report/ar6/

Costs Projected To Increase To Over $76

knows-for-sure

wg2/downloads/outreach/IPCC_AR6_WGII_

Billion By 2028”, Jim McGregor, Forbes

151 “Green AI”, Roy Schwartz, Jesse Dodge,

FactSheet_FoodAndWater.pdf and “Water(2023). https://www.forbes.com/sites/

Noah A. Smith and Oren Etzioni (2019).

– at the center of the climate crisis”, Unitedtiriasresearch/2023/05/12/generative-ai-

https://arxiv.org/abs/1907.10597 (p.2)

Nations. https://www.un.org/en/climat-

breaks-the-data-center-data-center-infra-

152 “Tech companies underreport CO2 emis-

echange/science/climate-issues/water

structure-and-operating-costs-projected-

sions”, Technical University of Munich,

163 “Protect our people and future generations:to-increase-to-over-76-billion-by-2028/(2021). https://www.sciencedaily.com/

Water and Climate Leaders call for urgent

144 “Generative AI Breaks The Data Center:releases/2021/11/211118203514.htm

action”, World Meteorological Organization

Data Center Infrastructure And Operating

153 “Green AI”, Roy Schwartz, Jesse Dodge,(2023). https://public.wmo.int/en/media/

Costs Projected To Increase To Over $76

Noah A. Smith and Oren Etzioni (2019)press-release/protect-our-people-and-fu-

Billion By 2028”, Jim McGregor, Forbes

https://arxiv.org/pdf/1907.10597.pdf (p.2)

ture-generations-water-and-climate-lead-(2023). https://www.forbes.com/sites/

154 “Generating Harms: Generative AI’s Impact &tiriasresearch/2023/05/12/generative-ai-

Paths Forward”, EPIC (2023). EPIC-Genera-

ers-call-urgent
164 OECD Environmental Outlook to 2050

breaks-the-data-center-data-center-infra-

tive-AI-White-Paper-May2023.pdf (p.40)(2012). https://www.oecd-ilibrary.org/

structure-and-operating-costs-projected-

155 “Green AI”, Roy Schwartz, Jesse Dodge,

environment/oecd-environmental-out-

to-increase-to-over-76-billion-by-2028/
145 “Generative AI Breaks The Data Center:

Noah A. Smith and Oren Etzioni (2019).

look-to-2050_9789264122246-en

https://arxiv.org/pdf/1907.10597.pdf

165 ”2022 Environmental Sustainability Report”

Data Center Infrastructure And Operating

156 “Green AI”, Roy Schwartz, Jesse Dodge,

Microsoft (p.28) https://query.prod.cms.

Costs Projected To Increase To Over $76

Noah A. Smith and Oren Etzioni (2019).

rt.microsoft.com/cms/api/am/binary/Billion By 2028”, Jim McGregor, Forbes

https://arxiv.org/pdf/1907.10597.pdf(2023). https://www.forbes.com/sites/

157 “The mounting human and environmental

tiriasresearch/2023/05/12/generative-ai-

costs of generative AI”, Sasha Luccion,

Addressing the Secret Water Footprint of AI

breaks-the-data-center-data-center-infra-

ArsTechnica (2023). https://arstechnica.

Models”, Pengfei Li, Jianyi Yang, Mohammad

structure-and-operating-costs-projected-

com/gadgets/2023/04/generative-ai-is-

A. Islam and Shaolei Ren (2023). https://

to-increase-to-over-76-billion-by-2028/

cool-but-lets-not-forget-its-human-and-146 “Generative AI Breaks The Data Center:environmental-costs/

RW14sJN
166 “Making AI Less “Thirsty”: Uncovering andarxiv.org/pdf/2304.03271.pdf
167 “Thirsty’ AI: Training ChatGPT Required

Data Center Infrastructure And Operating

158 ”We’re getting a better idea of AI’s true

Costs Projected To Increase To Over $76

carbon footprint”, Melissa Heikkilä, MIT

Cooling Tower, Study Finds”, Mack DeGeurin,

Billion By 2028”, Jim McGregor, Forbes

Technology Review (2022) https://www.

Gizmodo (2023). https://gizmodo.com/(2023). https://www.forbes.com/sites/

technologyreview.com/2022/11/14/1063192/

chatgpt-ai-water-185000-gallons-training-tiriasresearch/2023/05/12/generative-ai-

were-getting-a-better-idea-of-ais-true-car-nuclear-1850324249

breaks-the-data-center-data-center-infra-

bon-footprint/

Enough Water to Fill a Nuclear Reactor’s

168 “Thirsty’ AI: Training ChatGPT Required

structure-and-operating-costs-projected-

159 “AI and the Challenge of Sustainability”,

to-increase-to-over-76-billion-by-2028/

Sustain Issue 2 (2023). (p.16) https://

Cooling Tower, Study Finds”, Mack DeGeurin,

algorithmwatch.org/en/wp-content/up-

Gizmodo (2023). https://gizmodo.com/

Chris Stokel-Walker, Wired (2023). https://loads/2023/03/SustAIn-magazine-issue-2.

chatgpt-ai-water-185000-gallons-training-www.wired.com/story/the-generative-ai-

pdf

147 “The Generative AI Race Has a Dirty Secret”,

search-race-has-a-dirty-secret/Enough Water to Fill a Nuclear Reactor’s

nuclear-1850324249

160 “Green Intelligence: Why Data And AI Must

169 “Data centre water consumption”, David148 “Generating Harms: Generative AI’s Impact &Become More Sustainable”, Bernad Marr,

Mytton (2021). https://www.nature.com/

Paths Forward”, EPIC (2023). EPIC-Genera-

Forbes (2023). https://www.forbes.com/

tive-AI-White-Paper-May2023.pdf (p. 40)

sites/bernardmarr/2023/03/22/green-in-

170 “Microsoft will be carbon negative by 2030”,

149 “AI and the Challenge of Sustainability”, Sus-

telligence-why-data-and-ai-must-become-Microsoft (2020). Microsoft will be carbon

more-sustainable/

negative by 2030 - The Official Microsoft

tain Issue 2 (2023). https://algorithmwatch.
org/en/wp-content/uploads/2023/03/SustAIn-magazine-issue-2.pdf
150 “Artificial Intelligence Is Booming - So Is Its

Norwegian Consumer Council

161 “AI machines aren’t ‘hallucinating’. But theirmakers are”, Naomi Klein, The Guardian(2023). https://www.theguardian.com/com-

Ghost in the machine

articles/s41545-021-00101-w

Blog
171 “Google’s Carbon Offsets: Collaboration and
Due Diligence”, Google (2011). https://static.

June 2023

71

Endnotes 171–205

googleusercontent.com/media/www.goo-

181 Sama. https://www.sama.com/why-sama/

gle.com/no//green/pdfs/google-carbon-off-

182 “AI and the Challenge of Sustainability”, Sus-

sets.pdf, “2021 Sustainability Report”, Meta

tain Issue 2 (2023). https://algorithmwatch.

https://sustainability.fb.com/2021-sus-org/en/wp-content/uploads/2023/03/Sus-

tainability-report/ and “2022 Environmental

tAIn-magazine-issue-2.pdf (p. 12)

now-opt-out-of-generative
191 “The Red Hand Files 218”, Nick Cave (2023).
https://www.theredhandfiles.com/chatgpt-what-do-you-think/
192 “Artificially Yours: Who Owns Rights in

Sustainability Report”, Microsoft. https://

183 “AI and the Challenge of Sustainability”, Sus-

AI-Generated Art?”, Seyfarth Shaw (2023).

www.microsoft.com/en-us/corporate-re-tain Issue 2 (2023). https://algorithmwatch.

https://www.lexology.com/library/detail.

sponsibility/sustainability/reportorg/en/wp-content/uploads/2023/03/Sus-

aspx?g=640df36a-a63c-4936-82bd-579e4b-

172 Kaupa, C. (2022). “Peddling false solutions

tAIn-magazine-issue-2.pdf (p. 11)

54ca00

to worried consumers: The promotion of

184 “Automation Shouldn’t Always be Automat-

greenhouse gas ‘offsetting’ as a misleading

ic: Making Artificial Intelligence Work for& Paths Forward”, EPIC (2023). https://epic.

commercial practice.”, C. Kaupa, Journal

Workers and the World”, Daron Acemoglu,

org/new-epic-report-sheds-light-on-gener-of European Consumer and Market Law,

OECD Forum (2020). https://www.oecd-fo-

2022(4), 139-146. https://papers.ssrn.com/

rum.org/posts/automation-shouldn-t-al-sol3/papers.cfm?abstract_id=4157810

ways-be-automatic-making-artificial-intel-

Parliament and of the Council of 27 April

173 “COP27: UN slams use of ‘greenwashing’

193 “Generating Harms: Generative AI’s Impact

ative-a-i-harms/
194 Regulation (EU) 2016/679 of the European

ligence-work-for-workers-and-the-world

2016 on the protection of natural persons

offsets ahead of direct abatement actions”,

185 “The New Generation of A.I. Apps Could Makewith regard to the processing of personal

Henry Edwardes-Evans, S&P (2022). https://Writers and Artists Obsolete”, Nick Bilton,

data and on the free movement of such

www.spglobal.com/commodityinsights/en/

Vanity Fair (2022). https://www.vanityfair.

data, and repealing Directive 95/46/EC

market-insights/latest-news/energy-tran-

com/news/2022/06/the-new-generation-

sition/110922-cop27-un-slams-use-of-gre-of-ai-apps-could-make-writers-and-artists-

195 Art. 4(2) GDPR.

obsolete

196 Art. 4(1) GDPR.

enwashing-offsets-ahead-of-direct-abatement-actions

186 “Helpline workers for the National Eating(General Data Protection Regulation).

197 Art. 3 GDPR.

Disorder Association say they are being

198 Art. 4(7) GDPR.

carbon neutral claims, early obsolescence”,

replaced by AI”, Britney Nguyen, Insider

199 Art. 4(8) GDPR.

Valentina Romano, Euractive (2023).

(2023). https://www.businessinsider.com/

200 The GDPR does not apply to “natural persons’

https://www.euractiv.com/section/ener-

eating-disorders-nonprofit-reported-

processing of data in the course of a purely

gy-environment/news/eu-parliament-votes-

ly-fired-humans-offer-ai-chatbot-2023-5

personal or household activity”, cf. art. 3.

to-clamp-down-carbon-neutral-claims-ear-

187 “Artists stage mass protest against AI-gen-

Generation of output relating to a natural

erated artwork on ArtStation”, Benj Ed-

person as part of purely personal activity,

175 “Green AI”, Roy Schwartz, Jesse Dodge,

wards, ArsTechnica (2022). https://arstech-where the output is not shared online, isNoah A. Smith and Oren Etzioni (2019)

nica.com/information-technology/2022/12/

therefore not necessarily regulated byhttps://arxiv.org/pdf/1907.10597.pdf (p. 2)

artstation-artists-stage-mass-pro-the GDPR. The GDPR can still apply to the

174 “EU Parliament votes to clamp down on

ly-obsolescence/

176 “Green AI”, Roy Schwartz, Jesse Dodge,

test-against-ai-generated-artwork/

Noah A. Smith and Oren Etzioni (2019)

188 “The artist is dominating AI-generated

system owner.
201 “Extracting Training Data from Large

https://arxiv.org/pdf/1907.10597.pdf (p. 7)

art, and he’s not happy about it. Melis-Language Models”, Nicholas Carlini, Florian

177 “Generating Harms: Generative AI’s Impact &sa Heikkilä”, MIT Technology Review

Tramèr, Eric Wallace, Matthew Jagielski,

Paths Forward”, EPIC (2023). EPIC-Genera-(2022). https://www.technologyreview.

Ariel Herbert-Voss, Dawn Song, Katherine

tive-AI-White-Paper-May2023.pdf (p.41).

com/2022/09/16/1059598/this-artist-is-

Lee, Adam Roberts, Tom Brown, Úlfar Er-

dominating-ai-generated-art-and-hes-not-lingsson, Alina Oprea and Colin Raffel (2021).

178 “OpenAI founder Sam Altman sees a big AI
revolution within this decade”, Matthias
Bastian, The Decoder (2022). https://

happy-about-it/
189 “We’ve filed a lawsuit challenging Stable

https://arxiv.org/pdf/2012.07805.pdf
202 “Algorithms that remember: Model inversion

the-decoder.com/openai-founder-sees-a-

Diffusion, a 21st century collage tool that vi-

attacks and data protection law”, Michael

big-ai-revolution-within-this-decade/olates the rights of artists”, Stable Diffusion

Veale, Reuben Binns and Lilian Edwards

179 Ghostwork. https://www.ghostwork.org

litigation (2023). https://stablediffusionliti-(2018). https://royalsocietypublishing.org/

180 “Exclusive: OpenAI Used Kenyan Work-gation.com

ers on Less Than $2 Per Hour to Make

190 “Artists can now opt out of generative AI.

doi/full/10.1098/rsta.2018.0083
203 Art. 6 GDPR.

ChatGPT Less Toxic”, Billy Perrigo, Time

It’s not enough”, Sayash Kapoor and Arvind

204 Art. 9(2) GDPR.

(2023). https://time.com/6247678/ope-

Narayanan, AI Snake Oil (2023). https://

205 “ChatGPT: OpenAI reinstates service in Italy

nai-chatgpt-kenya-workers/

aisnakeoil.substack.com/p/artists-can-with enhanced transparency and rights

Norwegian Consumer Council

Ghost in the machine

June 2023

72

Endnotes 205–242

for european users and non-users”, Italian

Akrong, Praveen Paritosh and Lora Aroyo

commercial practices in the internal market

DPA (2023). https://www.garantepriva-(2021). https://storage.googleapis.com/

and amending Council Directive 84/450/

cy.it/home/docweb/-/docweb-display/

pub-tools-public-publication-data/

EEC, Directives 97/7/EC, 98/27/EC anddocweb/9881490

pdf/0d556e45afc54afeb2eb6b51a9b-

2002/65/EC of the European Parliamentc1827b9961ff4.pdf

and of the Council and Regulation (EC) No

206 “OpenAI - Privacy policy”, (April 27, 2023).
https://openai.com/policies/privacy-policy
207 “Google Bard hits over 180 countries and221 “Artificial intelligence: Stop to ChatGPT by

2006/2004 of the European Parliament andthe Italian SA” (2023). https://www.gpdp.it/of the Council (Unfair Commercial Practices

territories—none are in the EU”, Scharon

web/guest/home/docweb/-/docweb-dis-Harding, ArsTechnica (2023). https://play/docweb/9870847

Directive).
232 Art. 2(b) UCPD.

arstechnica.com/gadgets/2023/05/google-

222 OpenAI Privacy policy 2023 (last update April

233 Art. 4(d) UCPD.

bard-hits-over-180-countries-and-territo-

27, 2023). https://openai.com/policies/pri-

234 Art. 6 -7 UCPD.

vacy-policy,

235 Art. 8-9 UCPD.

ries-none-are-in-the-eu/
208 More Penguins Than Europeans Can Use

223 Umbrella term for AI systems designed to236 “Guidance on the interpretation and

Google Bard”, Morgan Meaker and Matt

perform a wide range of tasks across differ-application of Directive 2005/29/EC of the

Burgess, Wired (2023). https://www.wired.

ent domains.

European Parliament and of the Council

co.uk/article/google-bard-european-union
209 Bard FAQ (Accessed on June 1st, 2023).

224 “ChatGPT: OpenAI reinstates service in Italy

concerning unfair business-to-consumer

with enhanced transparency and rights

commercial practices in the internal market”

https://bard.google.com/faq?hl=en, Google

for European users and non-users”, Italian(2021). https://eur-lex.europa.eu/legal-

Privacy Policy (December 15, 2022). https://

DPA (2023). https://www.garantepriva-

content/EN/TXT/PDF/?uri=CELEX:52021X-

policies.google.com/privacy

cy.it/home/docweb/-/docweb-display/

210 Art. 25 GDPR.

docweb/9881490

C1229(05)&from=EN (p. 12)
237 “Guidance on the interpretation and

211 Art. 5(1)(c) GDPR.

225 Art. 6(1)(f) GDPR.

application of Directive 2005/29/EC of the

212 Art. 5(1)(b) GDPR.

226 Art. 6 and 5(2) GDPR.

European Parliament and of the Council

213 See for instance “How should we assess

227 “The Long and Winding Road: Two years ofconcerning unfair business-to-consumer

security and data minimisation in AI?”,the GDPR: A cross-border data protec-

commercial practices in the internal market”

ICO, https://ico.org.uk/for-organisations/tion enforcement case from a consumer(2021). https://eur-lex.europa.eu/legal-

guide-to-data-protection/key-dp-themes/

perspective”, BEUC (2020). https://www.

content/EN/TXT/PDF/?uri=CELEX:52021X-

guidance-on-ai-and-data-protection/

beuc.eu/sites/default/files/publications/how-should-we-assess-security-and-da-

beuc-x-2020-074_two_years_of_the_gd-

ta-minimisation-in-ai/, laying out somepr_a_cross-border_data_protection_en-

asymmetries in digital consumer markets”,

guidance on this subject.

forcement_case_from_a_consumer_per-

BEUC (2021). https://www.beuc.eu/sites/

spective.pdf

default/files/publications/beuc-x-2021-018_

214 Art. 17 GDPR.
215 Art. 16 GDPR.

228 “EDPB resolves dispute on transfers by

C1229(05)&from=EN (p. 37)
238 “EU Consumer Protection 2.0 Structural

eu_consumer_protection_2.0.pdf

216 Art. 21 GDPR.

Meta and creates task force on Chat GPT”,

239 “That was fast! Microsoft slips ads into

217 OpenAI Privacy policy 2023 (last update April

European Data Protection Board (2023).

AI-powered Bing Chat”, Devin Coldewey,

27, 2023). https://openai.com/policies/pri-

https://edpb.europa.eu/news/news/2023/

Techcrunch (2023). https://techcrunch.

vacy-policy

edpb-resolves-dispute-transfers-meta-com/2023/03/29/that-was-fast-microsoft-

218 See however requirements of information

and-creates-task-force-chat-gpt_en

slips-ads-into-ai-powered-bing-chat/for data subjects in art. 12-14 GDPR.

229 “Artificial intelligence: the action plan of the

240 A European consumer umbrella organization

219 “OpenAI’s hunger for data is coming backCNIL” (2023). https://www.cnil.fr/en/artifi-

to bite it”. Melissa Heikkilä, MIT Technology

cial-intelligence-action-plan-cnil

Review (2023) (note that Open AI has not

230 “European privacy watchdog createsof which the Norwegian Consumer Council
is a member.
241 “Call for action to open an inquiry on gener-

shared the size of the training data set for

ChatGPT task force”, Toby Sterling, Reuters,

ative AI systems to address risks and harms

GPT-4.) https://www.technologyreview.

(2023). https://www.reuters.com/technol-

for consumers”, BEUC (2023). https://www.

com/2023/04/19/1071789/openais-hunger-

ogy/european-data-protection-board-dis-beuc.eu/sites/default/files/publications/for-data-is-coming-back-to-bite-it/

cussing-ai-policy-thursday-meet-

BEUC-X-2023-045_Call_for_action_CPC_

ing-2023-04-13/

authorities_Generative_AI_systems.pdf

220 “Everyone wants to do the model work,
not the data work”: Data Cascades in

231 Directive 2005/29/EC of the European

242 “Towards European Digital Fairness - BEUC

High-Stakes”, AI Nithya Sambasivan,

Parliament and of the Council of 11 May 2005

framing response paper for the REFIT

Shivani Kapania, Hannah Highfill, Diana

concerning unfair business-to-consumer

consultation”, BEUC (2023). https://www.

Norwegian Consumer Council

Ghost in the machine

June 2023

73

Endnotes 242–287

beuc.eu/sites/default/files/publications/
BEUC-X-2023-020_Consultation_paper_REFIT_consumer_law_digital_fairness.pdf
243“Digital fairness – fitness check on EU consumer law”, European Commission. https://cans-rights-and-safety/

Parliament and of the Council of 19 October

252 Directive 2001/95/EC of the European Par-

2022 on a Single Market For Digital Services

liament and of the Council of 3 December

and amending Directive 2000/31/EC (Digital2001 on general product safety.
253 Regulation (EU) 2023/988 of the European

Services Act).
267 Art. 3(1)(g) DSA.

ec.europa.eu/info/law/better-regulation/

Parliament and of the Council of 10 May 2023

268 “DSA: Very large online platforms and search

have-your-say/initiatives/13413-Digital-on general product safety, amending Reg-

engines”, European Commission. https://fairness-fitness-check-on-EU-consumer-

ulation (EU) No 1025/2012 of the European

digital-strategy.ec.europa.eu/en/policies/

law_enParliament and of the Council and Directive

244 Federal Trade Commission Act. https://(EU) 2020/1828 of the European Parliament

dsa-vlops
269 “Understanding and Regulating ChatGPT,

www.ftc.gov/legal-library/browse/statutes/

and the Council, and repealing Directive

and Other Large Generative AI Models”,

federal-trade-commission-act

2001/95/EC of the European Parliament andPhilipp Hacker, Andreas Engel, Theresaof the Council and Council Directive 87/357/

List, Verfassungsblog (2023). https://ver-

245 “A Brief Overview of the Federal Trade Commission’s Investigative, Law Enforcement,

EEC.

fassungsblog.de/chatgpt/

and Rulemaking Authority”, FTC (2021).

254 Art. 1(2) GPSD.

270 Proposal for a regulation of the Europe-

https://www.ftc.gov/about-ftc/mission/

255 Art. 3(1) GPSD.

an Parliament and of the Council laying

enforcement-authority

256 “Opinion of the sub-group on artificial

down harmonised rules on artificial

246 “Aiming for truth, fairness, and equity inIntelligence (ai), connected products andintelligence and amending certain union

your company’s use of AI”, FTC (2021).

other new challenges in product safety tolegislative acts, para. 5 (2021). https://

https://www.ftc.gov/business-guidance/

the consumer safety network”, European

eurlex.europa.eu/resource.html?uri=cel-

blog/2021/04/aiming-truth-fairness-equity-

Commission (2021). https://ec.europa.eu/

lar:e0649735-a372-11eb-9585-01aa75e-

your-companys-use-ai

safety/consumers/consumers_safety_gate/

247 “Keep your AI claims in check”, FTC (2023).

d71a1.0001.02/DOC_1&format=PDF

home/documents/Subgroup_opinion_fi-

271 Art. 2(1)(a) Draft AIA.

https://www.ftc.gov/business-guidance/

nal_format.pdf

272 Art. 3 (1) Draft AIA, cf. Annex 1.

blog/2023/02/keep-your-ai-claims-check

257 Art. 2(1)(b) GPSD.

273 Art. 5 Draft AIA.

258 “Opinion of the sub-group on artificial

274 Art. 6 Draft AIA.

248 “Model Destruction – The FTC’s Powerful
New AI and Privacy Enforcement Tool”, AviIntelligence (ai), connected products and

275 See Draft AIA Title III.

Gesser, Paul D. Rubin and Aanna Gresse,

other new challenges in product safety to

276 Art. 17 Draft AIA.

Debevoise Data Blog (2022). https://www.

the consumer safety network”, European

277 Art. 9 Draft AIA.

debevoisedatablog.com/2022/03/22/mod-

Commission (2021). https://ec.europa.eu/

278 Art. 10 Draft AIA.

el-destruction-the-ftcs-powerful-new-ai-

safety/consumers/consumers_safety_gate/

279 Art. 15 Draft AIA.

enforcement-tool/home/documents/Subgroup_opinion_fi-

280 Art. 11 Draft AIA.

nal_format.pdf

281 Art. 52 Draft AIA.

259 Art. 8(2) GPSD.

282 Art. 69 Draft AIA.

250 “Joint statement on enforcement efforts

260 “Urgent all for action regarding genera-

283 “Regulation AI to protect the consumer -against discrimination and bias in auto-

tive AI systems and concerns related to

Position Paper on the AI Act”, BEUC (2021).

mated systems”, EEOC, CRT, FTC, CFPB-AI.

their safety”, BEUC (2023). https://www.

https://www.beuc.eu/sites/default/files/

https://www.ftc.gov/system/files/ftc_gov/

beuc.eu/sites/default/files/publications/publications/beuc-x-2021-088_regulat-

pdf/EEOC-CRT-FTC-CFPB-AI-Joint-State-

BEUC-X-2023046_BEUC_concerns_over_

ing_ai_to_protect_the_consumer.pdf

ment%28final%29.pdf

AI_and_mental_health_%20Ms_Pinuccia_

284 Art. 5 Draft AIA.

Contino.pdf

285 “Proposal for a Regulation of the European

249 Center for AI and Digital Policy. https://www.caidp.org/cases/openai/

251 “Biden-⁠Harris Administration Announces New Actions to Promote

261 Cf. recital 19 GPSR.

Parliament and of the Council laying down

Responsible AI Innovation that Pro-

262 Art. 6(1)(h) GPSR.

harmonised rules on artificial intelligencetects Americans’ Rights and Safety”,

263 Art. 101 TFEU.

(Artificial Intelligence Act) and amending

The White House (2023). https://

264 Art. 102 TFEU.

certain Union legislative acts – Generalwww.whitehouse.gov/briefing-room/

265 “Competition policy”. European Parliament

approach”, Council of the European Union

statements-releases/2023/05/04/(2023). https://www.europarl.europa.

(2022). https://data.consilium.europa.eu/

fact-sheet-biden-harris-administration-an-

eu/erpl-app-public/factsheets/pdf/en/

nounces-new-actions-to-promote-respon-

FTU_2.6.12.pdf

sible-ai-innovation-that-protects-ameri-

Norwegian Consumer Council

266 Regulation (EU) 2022/2065 of the European

Ghost in the machine

doc/document/ST-14954-2022-INIT/en/pdf
286 Art. 3(1)(b) Council Position.
287 Art. 4b(1) Council Position.

June 2023

74

Endnotes 288–310

288 Art. 4c(1) Council Position.
289 Art.  4c(2) Council Position.
290 “DRAFT Compromise Amendments on the

BEUC-X-2023-023_Revision_of_the_product_liability_directive.pdf
305 “Proposal for an AI Liability Directive”, BEUC

Draft Report - Proposal for a regulation(2023). https://www.beuc.eu/sites/default/of the European Parliament and of thefiles/publications/BEUC-X-2023-050_Pro-

Council on harmonised rules on Artificial

posal_for_an_AI_Liability_Directive.pdf

Intelligence (Artificial Intelligence Act) and306 “Proposal for an AI Liability Directive”, BEUC

amending certain Union Legislative Acts”,(2023). https://www.beuc.eu/sites/default/

LIBE and IMCO committees of the European

files/publications/BEUC-X-2023-050_Pro-

Parliament. (2023). https://www.europarl.
europa.eu/meetdocs/2014_2019/plmrep/

posal_for_an_AI_Liability_Directive.pdf
307 “How to create, release, and share

COMMITTEES/CJ40/DV/2023/05-11/Consol-

generative AI responsibly”, Melis-

idatedCA_IMCOLIBE_AI_ACT_EN.pdf

sa Heikkilä, MIT Technology Review

291 Art. 68c Parliament Position.

(2023). https://www.technologyreview.

292 Art. 68a Parliament Position.

com/2023/02/27/1069166/how-to-cre-

293 Art. 68b Parliament Position.

ate-release-and-share-generative-ai-re-

294 Art. 68d Parliament Position.
295 Art. 3(1c) Parliament Position.

sponsibly/
308 “Pause Giant AI Experiments: An Open

296 Art. 28b Parliament Position.

Letter”, Future of Life (2023).  https://futu-

297 For the complete list of requirements, seereoflife.org/open-letter/pause-giant-ai-ex-

art. 28b(2) Parliament Position.
298 Art. 28b(4).
299 “ChatGPT and the AI Act”, Natali Helberger

periments/
309 “OpenAI’s CEO confirms the company isn’t
training GPT-5 and ‘won’t for some time”,

and Nicholas Diakopoulos (2023). https://

James Vincent, The Verge (2023). https://policyreview.info/essay/chatgpt-and-ai-act

www.theverge.com/2023/4/14/23683084/

300 “General Purpose AI Poses Serious Risks,openai-gpt-5-rumors-training-sam-altman

Should Not Be Excluded From the EU’s AI

310 “EU, Google to develop voluntary AI pact

Act”, Amba Kak and Sarah Myers West, AI

ahead of new AI rules, EU’s Breton says”,

Now Institute (2023). https://ainowinstitute.

Foo Yun Chee, Reuters (2023).https://www.

org/publication/gpai-is-high-risk-should-

reuters.com/technology/eu-google-devel-

not-be-excluded-from-eu-ai-act

op-voluntary-ai-pact-ahead-new-ai-rules-

301 “The lobbying ghost in the machine”, Cor-

eus-breton-says-2023-05-24/

porate Europe Observatory (2023). https://www.corporateeurope.org/en/2023/02/
lobbying-ghost-machine
302 The exact timing differs between the different positions of the EU institutions. At theearliest the AIA will be fully applicable after
24 months after entering into force. This effectively means it may not be fully applicableuntil at the earliest April/May 2026, if there
is a trilogue agreement by January 2024.
303 Case C-65/20, VI v Krone (2021), https://
curia.europa.eu/juris/document/
document.jsf;jsessionid=7A0662FAD49ED462BE89A81594FAF809?text=&docid=242561&pageIndex=0&doclang=EN.
304 “Revision of the Product Liability Directive”, BEUC (2023). https://www.beuc.
eu/sites/default/files/publications/

Norwegian Consumer Council

Ghost in the machine

June 2023

75

Running header

FOR MORE INFORMATION:
Finn Lützow-Holm Myrstad, director of Digital Policy
Norwegian Consumer Council
E-mail: finn.myrstad@forbrukerradet.nowww.forbrukerradet.no/ai

Norwegian Consumer Council

Ghost in the machine

June 2023