# LLM-based Interaction for Content Generation: A Case Study on the Perception of Employees in an IT department

Alexandre Agossah, Frédérique Krupa, Matthieu Perreira da Silva, Patrick Le Callet

► To cite this version:

Alexandre Agossah, Frédérique Krupa, Matthieu Perreira da Silva, Patrick Le Callet. LLM-based Interaction for Content Generation: A Case Study on the Perception of Employees in an IT department. ACM IMX 2023 - Interactive Media Experience, Association for Computing Machinery, Jun 2023, Nantes (France), France. 10.48550/arXiv.2304.09064. hal-04616965

HAL Id: hal-04616965

https://hal.science/hal-04616965

Submitted on 19 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# LLM-based Interaction for Content Generation:
A Case Study on the Perception of Employees in an IT
department

Alexandre Agossah<sup>a,b,c</sup>, Frédérique Krupa<sup>b,d</sup>, Matthieu Perreira Da Silva<sup>a</sup>,
Patrick Le Callet<sup>a</sup>

<sup>a</sup>Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes,
France, Nantes, France

<sup>b</sup>Digital Design Lab, L'École de design Nantes Atlantique, Nantes, France

<sup>c</sup>Groupe SIGMA, La Chapelle-sur-Erdre, France

<sup>d</sup>Université Paris 1 Pantheon-Sorbonne, Paris, France

## Abstract

In the past years, AI has seen many advances in the field of NLP. This has led to the emergence of LLMs, such as the now famous GPT-3.5, which revolutionise the way humans can access or generate content. Current studies on LLM-based generative tools are mainly interested in the performance of such tools in generating relevant content (code, text or image). However, ethical concerns related to the design and use of generative tools seem to be growing, impacting the public acceptability for specific tasks. This paper presents a questionnaire survey to identify the intention to use generative tools by employees of an IT company in the context of their work. This survey is based on empirical models measuring intention to use (TAM by Davis, 1989, and UTAUT2 by Venkatesh and al., 2008). Our results indicate a rather average acceptability of generative tools, although the more useful the tool is perceived to be, the higher the intention to use seems to be. Furthermore, our analyses suggest that the frequency of use of generative tools is likely to be a key factor in understanding how employees perceive these tools in the context of their work. Following on from this work, we plan to investigate the nature of the requests that may be made to these tools by specific audiences.

**Keywords:** Acceptability, Computer-Human Interaction, Large Language Models, Professional context
# LLM-based Interaction for Content Generation:
A Case Study on the Perception of Employees in an IT
department

Alexandre Agossah<sup>a,b,c</sup>, Frédérique Krupa<sup>b,d</sup>, Matthieu Perreira Da Silva<sup>a</sup>,
Patrick Le Callet<sup>a</sup>

<sup>a</sup>Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes,
France, Nantes, France

<sup>b</sup>Digital Design Lab, L'École de design Nantes Atlantique, Nantes, France

<sup>c</sup>Groupe SIGMA, La Chapelle-sur-Erdre, France

<sup>d</sup>Université Paris 1 Pantheon-Sorbonne, Paris, France

## Abstract

In the past years, AI has seen many advances in the field of NLP. This has led to the emergence of LLMs, such as the now famous GPT-3.5, which revolutionise the way humans can access or generate content. Current studies on LLM-based generative tools are mainly interested in the performance of such tools in generating relevant content (code, text or image). However, ethical concerns related to the design and use of generative tools seem to be growing, impacting the public acceptability for specific tasks. This paper presents a questionnaire survey to identify the intention to use generative tools by employees of an IT company in the context of their work. This survey is based on empirical models measuring intention to use (TAM by Davis, 1989, and UTAUT2 by Venkatesh and al., 2008). Our results indicate a rather average acceptability of generative tools, although the more useful the tool is perceived to be, the higher the intention to use seems to be. Furthermore, our analyses suggest that the frequency of use of generative tools is likely to be a key factor in understanding how employees perceive these tools in the context of their work. Following on from this work, we plan to investigate the nature of the requests that may be made to these tools by specific audiences.

**Keywords:** Acceptability, Computer-Human Interaction, Large Language Models, Professional context
# 1. Introduction

The rise of artificial intelligence (AI) and automation in various sectors, such as IT, health or even retail, invites us to question our work habits and consider greater collaboration with increasingly powerful systems to assist us with certain tasks. For example, to help with medical diagnosis or even sales prediction (Agossah et al., 2022a). However, there are already many concerns about the use of these tools, particularly in the face of systems perceived as black boxes whose functioning is difficult to understand. The need for transparency is now a major issue to contribute to making AI more acceptable, given the tools for which: 1) it is difficult to understand how they work (Arrieta et al., 2020) (Rudin, 2019), 2) the certainty rate of the tool in its own behaviours and the error rate are still underutilized data (Agossah et al., 2022b) (Yin et al., 2019), and 3) the decision-making biases of these tools still question the moral and ethical considerations surrounding their design and use. There are also new concerns emerging around the use of text, code and image generation tools such as ChatGPT from OpenAI. These tools appear to be increasingly capable of understanding and generating human-like content, opening up many possibilities, but also concerns about the use of AI in the workplace.

AI can greatly benefit workers by automating simple and/or time-consuming tasks, freeing up time for more complex or higher value-added tasks. However, the strength of some AI-based tools, such as ChatGPT, which generates content, is to allow its users to access information. Like a search engine, these tools have the ability to position themselves between the requester and the information transmitter. But in this case, the response transmitted is moderated by the tool's learning from the transmitter's information. In addition, it seems important to take account individuals' perceptions and concerns when using these tools (Gamkrelidze et al., 2021). One of the main challenges concerning practical cases of AI use is the need to ensure the accuracy and reliability of these tools, in order to ensure the quality of the results. Furthermore, questions arise about the trust that can be placed in these tools such as ethical issues, data security and privacy protection, or concerns about job losses due to automation (Agossah et al., 2022a) (Ferguson, 2019). It is also essential to have more transparency about how these tools work to clarify the responsibility attributed to the human who cooperates with these systems.

In this article, we propose to study the perception of this new type of
# 1. Introduction

The rise of artificial intelligence (AI) and automation in various sectors, such as IT, health or even retail, invites us to question our work habits and consider greater collaboration with increasingly powerful systems to assist us with certain tasks. For example, to help with medical diagnosis or even sales prediction (Agossah et al., 2022a). However, there are already many concerns about the use of these tools, particularly in the face of systems perceived as black boxes whose functioning is difficult to understand. The need for transparency is now a major issue to contribute to making AI more acceptable, given the tools for which: 1) it is difficult to understand how they work (Arrieta et al., 2020) (Rudin, 2019), 2) the certainty rate of the tool in its own behaviours and the error rate are still underutilized data (Agossah et al., 2022b) (Yin et al., 2019), and 3) the decision-making biases of these tools still question the moral and ethical considerations surrounding their design and use. There are also new concerns emerging around the use of text, code and image generation tools such as ChatGPT from OpenAI. These tools appear to be increasingly capable of understanding and generating human-like content, opening up many possibilities, but also concerns about the use of AI in the workplace.

AI can greatly benefit workers by automating simple and/or time-consuming tasks, freeing up time for more complex or higher value-added tasks. However, the strength of some AI-based tools, such as ChatGPT, which generates content, is to allow its users to access information. Like a search engine, these tools have the ability to position themselves between the requester and the information transmitter. But in this case, the response transmitted is moderated by the tool's learning from the transmitter's information. In addition, it seems important to take account individuals' perceptions and concerns when using these tools (Gamkrelidze et al., 2021). One of the main challenges concerning practical cases of AI use is the need to ensure the accuracy and reliability of these tools, in order to ensure the quality of the results. Furthermore, questions arise about the trust that can be placed in these tools such as ethical issues, data security and privacy protection, or concerns about job losses due to automation (Agossah et al., 2022a) (Ferguson, 2019). It is also essential to have more transparency about how these tools work to clarify the responsibility attributed to the human who cooperates with these systems.

In this article, we propose to study the perception of this new type of
interaction for accessing and generating information. We therefore conduct
a case study within an IT company to study employees' perception of gen-
erative tools in terms of acceptability. After exploring what we consider to
be generative tools, we will describe the methodology used and then present
our results.

## 2. The emergence of LLM-based tools for code, text and image generation

Artificial intelligence (AI) is a field that aims to design programs capa-
ble of performing tasks normally reserved for humans, and without explicit
instructions. Tools for generating text, code and images are among the ap-
plications of AI that are increasingly present. These tools are based on AI
models called *Large Language Models* (LLMs), which are trained on huge
amounts of text to learn linguistic structures and patterns. With this knowl-
edge, LLMs are able to use advanced algorithms to automatically generate
content from instructions such as keywords or phrases, creating text, code
and images that appear to have been produced by humans.

For code generation, generation tools most often create computer code
from natural language, comments in the code or specification. They can
thus facilitate the work of developers by allowing them to generate code more
quickly than in peer-programming, which avoids them writing repetitive or
complex code, but the quality still seems lower than when it is exclusively
produced by humans in the same given timeframe (Imai, 2022). In order
to study the accuracy and comprehensibility of the code provided by Github
Copilot, Nguyen and Nadi (2023) conducted a comparative study of Copilot's
performance on four programming languages (Nguyen and Nadi, 2022). The
researchers gave multiple code generation requests per language to Copilot
and analyses of the generated code. They conclude their study by presenting
Copilot as a good starting point, capable of delivering between 60% and
91% of correct code depending on the language. But they did not find any
significant difference between languages in terms of complexity.

To scale the performance of Copilot, Dakhel and his collaborators con-
ducted a comparative study with human performance (Dakhel et al., 2022).
The objective was also to identify whether Copilot is a good programming
companion. The researchers evaluated the tool's ability to suggest correct,
efficient and reproducible solutions for fundamental programming problems.
In a second step, the researchers wanted to identify whether the solutions
interaction for accessing and generating information. We therefore conduct
a case study within an IT company to study employees' perception of gen-
erative tools in terms of acceptability. After exploring what we consider to
be generative tools, we will describe the methodology used and then present
our results.

## 2. The emergence of LLM-based tools for code, text and image generation

Artificial intelligence (AI) is a field that aims to design programs capa-
ble of performing tasks normally reserved for humans, and without explicit
instructions. Tools for generating text, code and images are among the ap-
plications of AI that are increasingly present. These tools are based on AI
models called *Large Language Models* (LLMs), which are trained on huge
amounts of text to learn linguistic structures and patterns. With this knowl-
edge, LLMs are able to use advanced algorithms to automatically generate
content from instructions such as keywords or phrases, creating text, code
and images that appear to have been produced by humans.

For code generation, generation tools most often create computer code
from natural language, comments in the code or specification. They can
thus facilitate the work of developers by allowing them to generate code more
quickly than in peer-programming, which avoids them writing repetitive or
complex code, but the quality still seems lower than when it is exclusively
produced by humans in the same given timeframe (Imai, 2022). In order
to study the accuracy and comprehensibility of the code provided by Github
Copilot, Nguyen and Nadi (2023) conducted a comparative study of Copilot's
performance on four programming languages (Nguyen and Nadi, 2022). The
researchers gave multiple code generation requests per language to Copilot
and analyses of the generated code. They conclude their study by presenting
Copilot as a good starting point, capable of delivering between 60% and
91% of correct code depending on the language. But they did not find any
significant difference between languages in terms of complexity.

To scale the performance of Copilot, Dakhel and his collaborators con-
ducted a comparative study with human performance (Dakhel et al., 2022).
The objective was also to identify whether Copilot is a good programming
companion. The researchers evaluated the tool's ability to suggest correct,
efficient and reproducible solutions for fundamental programming problems.
In a second step, the researchers wanted to identify whether the solutions
proposed by Copilot are competitive with those given by a human for the
same given problems. To do this, they mobilised computer science students.
The researchers concluded that the students kept a higher ratio of correct
and diverse answers than the generation tool. Nevertheless, the need to re-
pair buggy solutions and the complexity of the generated codes remain lower
for Copilot. The number of studies investigating Copilot's capabilities is still
relatively small at present.

Until 2022, most generative tools belonged to a particular generation
category. But OpenAI's ChatGPT, which has been released to the general
public since November 2022, is capable of generating text and code. It is a
chatbot that now uses the GPT-4 language model to understand and inter-
pret user queries and generate relevant responses. This tool is specialised in
dialogue and can adapt to different topics and styles, but it cannot generate
images by itself. ChatGPT can thus be integrated into many other fields
than programming.

We believe that generation tools offer a differentiating interaction experience, becoming a media through their ability to transmit and disseminate information. In particular, by generating content that responds to specific queries, these tools can help to communicate more effectively with target audiences and provide useful, engaging and personalised information in real time. These tools represent a major advance in the field of AI and offer new opportunities for professionals and amateurs. However, they also raise ethical and social issues related to the quality, reliability, accountability and intellectual property of the content generated. For example, the question of who is responsible for errors in the generated code, or how to deal with plagiarism issues in texts produced by LLMs.

In this study, we will focus on the perceptions of these generation tools
within a department of an IT company. By examining current research and
developments in the field of AI-based authoring tools, we seek to better un-
derstand how these tools are perceived by a specific professional audience. To
this end, we will present our survey methodology to explore these perceptions
through the lens of acceptability.

3. Methodology

3.1. *Measurement of acceptability*

The use of generative tools is becoming increasingly common in the IT
field because of their potential to help generate and optimise code, create
proposed by Copilot are competitive with those given by a human for the
same given problems. To do this, they mobilised computer science students.
The researchers concluded that the students kept a higher ratio of correct
and diverse answers than the generation tool. Nevertheless, the need to re-
pair buggy solutions and the complexity of the generated codes remain lower
for Copilot. The number of studies investigating Copilot's capabilities is still
relatively small at present.

Until 2022, most generative tools belonged to a particular generation
category. But OpenAI's ChatGPT, which has been released to the general
public since November 2022, is capable of generating text and code. It is a
chatbot that now uses the GPT-4 language model to understand and inter-
pret user queries and generate relevant responses. This tool is specialised in
dialogue and can adapt to different topics and styles, but it cannot generate
images by itself. ChatGPT can thus be integrated into many other fields
than programming.

We believe that generation tools offer a differentiating interaction experience, becoming a media through their ability to transmit and disseminate information. In particular, by generating content that responds to specific queries, these tools can help to communicate more effectively with target audiences and provide useful, engaging and personalised information in real time. These tools represent a major advance in the field of AI and offer new opportunities for professionals and amateurs. However, they also raise ethical and social issues related to the quality, reliability, accountability and intellectual property of the content generated. For example, the question of who is responsible for errors in the generated code, or how to deal with plagiarism issues in texts produced by LLMs.

In this study, we will focus on the perceptions of these generation tools
within a department of an IT company. By examining current research and
developments in the field of AI-based authoring tools, we seek to better un-
derstand how these tools are perceived by a specific professional audience. To
this end, we will present our survey methodology to explore these perceptions
through the lens of acceptability.

3. Methodology

3.1. *Measurement of acceptability*

The use of generative tools is becoming increasingly common in the IT
field because of their potential to help generate and optimise code, create
documentation, search for information, etc. We developed a customised ques-
tionnaire to explore the perception of employees of an IT company towards
these tools in the context of their work. The objective of this questionnaire
was to study the perception of these tools from the point of view of accept-
ability (intention to use). Acceptability is often presented as the sum of the
representations that an individual has with regard to a technology, leading
to an intention to use the latter (Tricot et al., 2003) (Bobillier-Chaumon and
Dubois, 2009) (Pasquier, 2012). This concept is also approached from a de-
sign perspective, as the characteristics of a tool that lead to its target users
intending to use it (Nielsen, 1994).

To study the acceptability of generative tools, we explored some of factors
influencing the acceptance or rejection of these tools (perceived ease of use,
perceived usefulness, social influence, hedonic motivation) according to the
literature, using the Technology Acceptance Model (TAM) (Davis et al.,
1989) (Atarodi et al., 2019) and Unified Theory of Acceptance and Use of
Technology model 2 (UTAUT 2) (Venkatesh et al., 2012). TAM measures
intention to use based on perceived usefulness and ease of use of the device,
while the UTAUT 2 model focuses on more diverses determinants such as
hedonic motivation and social influence. We included social influence and
hedonic motivation in our study, assuming that, due to the recent emergence
and widespread use of LLM-based generation tools, the social environment
and attractiveness may be dominant in the intention to use. In summary, our
study focuses on the acceptability of generative tools, exploring the factors
that influence their adoption or rejection, using the TAM and UTAUT 2
models. We chose to explore these concepts by applying a questionnaire to
the observed service.

3.2. *Hypothesis*

In this study, we have formulated several hypotheses concerning the fac-
tors that may influence the perception of generation tools within an NSE.
We hypothesised that job experience, type of job, having tried a generation
tool before (in a generic way), the degree of importance given to the tasks
for which generation tools are used, and the frequency of use of generation
tools to perform one's tasks may all have an impact on the perception of
these tools. These hypotheses will be tested through the survey conducted
with the employees concerned. The hypotheses are set out as follows:

H1: Experience in the job can influence the perception of the tool. The
more experience a person has in their job, the more informed they can be
documentation, search for information, etc. We developed a customised ques-
tionnaire to explore the perception of employees of an IT company towards
these tools in the context of their work. The objective of this questionnaire
was to study the perception of these tools from the point of view of accept-
ability (intention to use). Acceptability is often presented as the sum of the
representations that an individual has with regard to a technology, leading
to an intention to use the latter (Tricot et al., 2003) (Bobillier-Chaumon and
Dubois, 2009) (Pasquier, 2012). This concept is also approached from a de-
sign perspective, as the characteristics of a tool that lead to its target users
intending to use it (Nielsen, 1994).

To study the acceptability of generative tools, we explored some of factors
influencing the acceptance or rejection of these tools (perceived ease of use,
perceived usefulness, social influence, hedonic motivation) according to the
literature, using the Technology Acceptance Model (TAM) (Davis et al.,
1989) (Atarodi et al., 2019) and Unified Theory of Acceptance and Use of
Technology model 2 (UTAUT 2) (Venkatesh et al., 2012). TAM measures
intention to use based on perceived usefulness and ease of use of the device,
while the UTAUT 2 model focuses on more diverses determinants such as
hedonic motivation and social influence. We included social influence and
hedonic motivation in our study, assuming that, due to the recent emergence
and widespread use of LLM-based generation tools, the social environment
and attractiveness may be dominant in the intention to use. In summary, our
study focuses on the acceptability of generative tools, exploring the factors
that influence their adoption or rejection, using the TAM and UTAUT 2
models. We chose to explore these concepts by applying a questionnaire to
the observed service.

3.2. *Hypothesis*

In this study, we have formulated several hypotheses concerning the fac-
tors that may influence the perception of generation tools within an NSE.
We hypothesised that job experience, type of job, having tried a generation
tool before (in a generic way), the degree of importance given to the tasks
for which generation tools are used, and the frequency of use of generation
tools to perform one's tasks may all have an impact on the perception of
these tools. These hypotheses will be tested through the survey conducted
with the employees concerned. The hypotheses are set out as follows:

H1: Experience in the job can influence the perception of the tool. The
more experience a person has in their job, the more informed they can be
about the usefulness and effectiveness of the tool.

**H2:** The type of job can also have an impact on the perception of the tool. Depending on the tasks and responsibilities associated with a job, the use of a generation tool may be perceived differently.

**H3:** The frequency of use of the generation tools to carry out one's tasks can also have an impact on their perception. A person who uses these tools frequently may have a more positive view of their usefulness.

**H4:** The degree of importance attached to the tasks for which generation tools are used can influence the perception of these tools. If a person places a lot of importance on these tasks, he or she may be more critical of the use of a tool to perform them.

# 4. Results

In the context of our study, we felt it was important to ask respondents about their working methods, and in particular about the strategies they use to facilitate their task when they encounter difficulties. Searching for information online and asking a colleague for help were the most common actions taken (see Figure 1).

Of the 41 people in the department, 19 responded to the survey. Of these 19 respondents, 9 use digital solutions that are not provided by the organisation to carry out their note-taking, documentation and time management tasks. Regarding code, text or image generation tools, 16 respondents already know ChatGPT and have already tried it, while 7 respondents know GitHub Copilot, of which 2 have already tried it. Finally, 6 respondents mention other generative tools such as DALL-E and Google BARD.

When asked about the frequency of use of such tools in their work, the majority of respondents stated that they never use them, while the other participants use them on an occasional basis (see Figure 2). Interestingly, 13 respondents found generative tools useful for their work, with various uses such as development support (5 respondents), optimising information retrieval (4 respondents), writing emails (3 respondents) and pre-sales, management and POC (proof of concept) advice (3 respondents). These results suggest that generative tools have potential to improve efficiency and productivity at work and can be considered acceptable by users.
about the usefulness and effectiveness of the tool.

**H2:** The type of job can also have an impact on the perception of the tool. Depending on the tasks and responsibilities associated with a job, the use of a generation tool may be perceived differently.

**H3:** The frequency of use of the generation tools to carry out one's tasks can also have an impact on their perception. A person who uses these tools frequently may have a more positive view of their usefulness.

**H4:** The degree of importance attached to the tasks for which generation tools are used can influence the perception of these tools. If a person places a lot of importance on these tasks, he or she may be more critical of the use of a tool to perform them.

# 4. Results

In the context of our study, we felt it was important to ask respondents about their working methods, and in particular about the strategies they use to facilitate their task when they encounter difficulties. Searching for information online and asking a colleague for help were the most common actions taken (see Figure 1).

Of the 41 people in the department, 19 responded to the survey. Of these 19 respondents, 9 use digital solutions that are not provided by the organisation to carry out their note-taking, documentation and time management tasks. Regarding code, text or image generation tools, 16 respondents already know ChatGPT and have already tried it, while 7 respondents know GitHub Copilot, of which 2 have already tried it. Finally, 6 respondents mention other generative tools such as DALL-E and Google BARD.

When asked about the frequency of use of such tools in their work, the majority of respondents stated that they never use them, while the other participants use them on an occasional basis (see Figure 2). Interestingly, 13 respondents found generative tools useful for their work, with various uses such as development support (5 respondents), optimising information retrieval (4 respondents), writing emails (3 respondents) and pre-sales, management and POC (proof of concept) advice (3 respondents). These results suggest that generative tools have potential to improve efficiency and productivity at work and can be considered acceptable by users.
### 4.1. *Correlation between the studied dimensions*

The correlation between the dimensions studied was assessed using Pearson's correlation. The results show that all the dimensions studied are moderately correlated with each other, with correlations ranging mainly from 0.5 to 0.846, with the exception of social influence which has the lowest correlation scores (r between .263 and .539) with all other dimensions. A correlation table is presented below:

<figure><img src="image_1.png" /><figcaption>Figure 1: Correlogramm of Pearson correlation between the studied dimensions of acceptability (PU: Perceived Usefulness; PEU: Perceived Ease of Use; SI: Social Influence; HM: Hedonic Motivation; UI: Usage Intention)</figcaption></figure>

These results suggest that the different dimensions studied are interconnected and therefore likely to have an impact on each other. However, social influence does not seem to be related to the other dimensions, which may indicate that social factors play a different or distinct role in the perception of LLM tools at work than the other dimensions studied.

### 4.2. *Perception of the generation tools according to the respondents' classification*

In order to further explore the perception of LLM-based generation tools (see figure 2), we considered our sample from different perspectives. The aim was to identify whether there are differences in perceptions according to
### 4.1. *Correlation between the studied dimensions*

The correlation between the dimensions studied was assessed using Pearson's correlation. The results show that all the dimensions studied are moderately correlated with each other, with correlations ranging mainly from 0.5 to 0.846, with the exception of social influence which has the lowest correlation scores (r between .263 and .539) with all other dimensions. A correlation table is presented below:

<figure><img src="image_2.png" /><figcaption>Figure 1: Correlogramm of Pearson correlation between the studied dimensions of acceptability (PU: Perceived Usefulness; PEU: Perceived Ease of Use; SI: Social Influence; HM: Hedonic Motivation; UI: Usage Intention)</figcaption></figure>

These results suggest that the different dimensions studied are interconnected and therefore likely to have an impact on each other. However, social influence does not seem to be related to the other dimensions, which may indicate that social factors play a different or distinct role in the perception of LLM tools at work than the other dimensions studied.

### 4.2. *Perception of the generation tools according to the respondents' classification*

In order to further explore the perception of LLM-based generation tools (see figure 2), we considered our sample from different perspectives. The aim was to identify whether there are differences in perceptions according to
objective factors such as job experience (see figure 3), the type of job (see figure 4), frequency of use of generation tools (see figure 5) and the degree of importance attached to the task for what employees already use that kind of tools (see figure 6).

<figure><img src="image_2.png" /><figcaption>Figure 2: Perception of generation tools</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 3: Perception of the generation tools according to experience in current job</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 4: Perception of the generation tools according to type of job</figcaption></figure>
objective factors such as job experience (see figure 3), the type of job (see figure 4), frequency of use of generation tools (see figure 5) and the degree of importance attached to the task for what employees already use that kind of tools (see figure 6).

<figure><img src="image_3.png" /><figcaption>Figure 2: Perception of generation tools</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 3: Perception of the generation tools according to experience in current job</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4: Perception of the generation tools according to type of job</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>Figure 5: Perception of the generation tools according to estimate frequency of usage of generation tools in their job</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 6: Perception of the generation tools according to degree of importance attached to the task</figcaption></figure>

It is interesting to note certain trends in the scores presented. For example, the more experience respondents have in their jobs, the less useful they perceive generative tools to be. Or, respondents who use generative tools for high-value tasks perceive these tools as useful and easy to use. However, the social influence seems to vary very little between the different conditions. We will further analyse this in section 4.3.

### 4.3. Hypothesis testing

As previously stated, we chose to explore the perception of generative tools in the context of work is impacted by the experience in the job (H1), the type of job (H2), the frequency of use of generative tools (H3), the degree of importance given to the tasks for which the respondents use generative tools (H4). To test our hypotheses, we used the Kruskall Wallis test. This is a non-parametric statistical test, serving as an alternative to the one-factor ANOVA (see table 1).
<figure><img src="image_6.png" /><figcaption>Figure 5: Perception of the generation tools according to estimate frequency of usage of generation tools in their job</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 6: Perception of the generation tools according to degree of importance attached to the task</figcaption></figure>

It is interesting to note certain trends in the scores presented. For example, the more experience respondents have in their jobs, the less useful they perceive generative tools to be. Or, respondents who use generative tools for high-value tasks perceive these tools as useful and easy to use. However, the social influence seems to vary very little between the different conditions. We will further analyse this in section 4.3.

### 4.3. Hypothesis testing

As previously stated, we chose to explore the perception of generative tools in the context of work is impacted by the experience in the job (H1), the type of job (H2), the frequency of use of generative tools (H3), the degree of importance given to the tasks for which the respondents use generative tools (H4). To test our hypotheses, we used the Kruskall Wallis test. This is a non-parametric statistical test, serving as an alternative to the one-factor ANOVA (see table 1).
<table><caption>Table 1: Results of Kruskal-Wallis tests assessing the effect of different respondent classifications on the measured dimensions of acceptability (p value) (PU: Perceived Usefulness, PEU: Perceived Ease of Use, SI: Social Influence, HM: Hedonic Motivation, UI: Usage Intention)</caption><thead><tr><th></th><th>Work experience</th><th>Job Type</th><th>Estimate frequency of usage</th><th>Degree of importance attached to the tasks</th></tr></thead><tbody><tr><td>PU</td><td>5.365 (.147)</td><td>3.032 (.220)</td><td>7.85 (.049)</td><td>11.79 (.067)</td></tr><tr><td>PEU</td><td>1.755 (.625)</td><td>4.710 (.095)</td><td>5.51 (.138)</td><td>6.84 (.336)</td></tr><tr><td>SI</td><td>.310 (.958)</td><td>.597 (.742)</td><td>1.00 (.801)</td><td>4.25 (.643)</td></tr><tr><td>HM</td><td>3.990 (.263)</td><td>.835 (.659)</td><td>5.28 (.152)</td><td>9.15 (.166)</td></tr><tr><td>UI</td><td>3.925 (.270)</td><td>2.391 (.303)</td><td>10.15 (.017)</td><td>10.44 (.107)</td></tr></tbody></table>

The results of the hypothesis tests presented in the table 1 indicate that some of our hypotheses are supported by the data, while others are not. First, regarding job experience (H1), none of the tests resulted in a p-value of less than .05, suggesting that there is no significant difference in the perception of generative tools based on job experience.

In terms of job type (H2), no p-value less than .05 was observed either, indicating that perceptions of generative tools do not differ significantly by job type.

However, the frequency of use of generative tools (H3) shows significant differences in perceived usefulness (H(3) = 7.85, p < .05) and intention to use (H(3)=10.15, p < .05). This suggests that respondents who use generative tools more frequently perceive them as more useful and have a higher intention to use.

Finally, for the degree of importance placed on the tasks for which respondents use generative tools (H4), although only one p-value is slightly greater than the threshold of .05 (perceived usefulness: H(6) = 11.79, p = .067). This may indicate a trend in which respondents who use these tools for more important tasks perceive greater usefulness. However, it is important
<table><caption>Table 1: Results of Kruskal-Wallis tests assessing the effect of different respondent classifications on the measured dimensions of acceptability (p value) (PU: Perceived Usefulness, PEU: Perceived Ease of Use, SI: Social Influence, HM: Hedonic Motivation, UI: Usage Intention)</caption><thead><tr><th></th><th>Work experience</th><th>Job Type</th><th>Estimate frequency of usage</th><th>Degree of importance attached to the tasks</th></tr></thead><tbody><tr><td>PU</td><td>5.365 (.147)</td><td>3.032 (.220)</td><td>7.85 (.049)</td><td>11.79 (.067)</td></tr><tr><td>PEU</td><td>1.755 (.625)</td><td>4.710 (.095)</td><td>5.51 (.138)</td><td>6.84 (.336)</td></tr><tr><td>SI</td><td>.310 (.958)</td><td>.597 (.742)</td><td>1.00 (.801)</td><td>4.25 (.643)</td></tr><tr><td>HM</td><td>3.990 (.263)</td><td>.835 (.659)</td><td>5.28 (.152)</td><td>9.15 (.166)</td></tr><tr><td>UI</td><td>3.925 (.270)</td><td>2.391 (.303)</td><td>10.15 (.017)</td><td>10.44 (.107)</td></tr></tbody></table>

The results of the hypothesis tests presented in the table 1 indicate that some of our hypotheses are supported by the data, while others are not. First, regarding job experience (H1), none of the tests resulted in a p-value of less than .05, suggesting that there is no significant difference in the perception of generative tools based on job experience.

In terms of job type (H2), no p-value less than .05 was observed either, indicating that perceptions of generative tools do not differ significantly by job type.

However, the frequency of use of generative tools (H3) shows significant differences in perceived usefulness (H(3) = 7.85, p < .05) and intention to use (H(3)=10.15, p < .05). This suggests that respondents who use generative tools more frequently perceive them as more useful and have a higher intention to use.

Finally, for the degree of importance placed on the tasks for which respondents use generative tools (H4), although only one p-value is slightly greater than the threshold of .05 (perceived usefulness: H(6) = 11.79, p = .067). This may indicate a trend in which respondents who use these tools for more important tasks perceive greater usefulness. However, it is important
to note that these results are not statistically significant and should therefore
be interpreted with caution.

Overall, these analyses suggest that frequency of use of generative tools seems to be a key factor in understanding how respondents perceive these tools in the context of their work. Further research could focus on why frequency of use influences perceived usefulness and intention to use, as well as how to encourage wider adoption of these tools among employees.

**5. Discussion**

In conclusion, this study aimed to explore the perception of LLM-based generative tools in the professional context. We examined the dimensions of perceived usefulness, perceived ease of use, social influence, hedonic motivation and intention to use, and if they were impacted by objective factors such as job experience, type of job, frequency of use of generative tools and the degree of importance attached to the tasks for which these tools are used.

Our results show that generative tools are generally perceived as moderately useful and easy to use, although their adoption within the organisation studied remains limited. The dimensions studied are also moderately correlated with each other, except for social influence, which has the lowest correlation scores. Moreover, the frequency of use of the generation tools turns out to be a key factor in understanding the perception of these tools by employees. People who use them more frequently perceive these tools as more useful and have a higher intention to use them.

These findings suggest that LLM-based generation tools have potential to improve efficiency and productivity at work, but their adoption could be hindered by factors such as resistance to change, lack of awareness or lack of adequate training. Organisations seeking to encourage the use of these tools could consider initiatives to promote their adoption, such as training, workshops, or internal communication campaigns. It still conditionnate by the purpose of the implementation (improving workers agentivity, helping them for specific tasks, remplacing them, etc.).

It is important to note that this study has some limitations, including the sample size and the fact that it focuses on a single organisation. Future research could extend this study to other work contexts and include larger samples to examine the generalisability of our findings. In addition, qualitative studies could be conducted to further our understanding of why
to note that these results are not statistically significant and should therefore
be interpreted with caution.

Overall, these analyses suggest that frequency of use of generative tools seems to be a key factor in understanding how respondents perceive these tools in the context of their work. Further research could focus on why frequency of use influences perceived usefulness and intention to use, as well as how to encourage wider adoption of these tools among employees.

**5. Discussion**

In conclusion, this study aimed to explore the perception of LLM-based generative tools in the professional context. We examined the dimensions of perceived usefulness, perceived ease of use, social influence, hedonic motivation and intention to use, and if they were impacted by objective factors such as job experience, type of job, frequency of use of generative tools and the degree of importance attached to the tasks for which these tools are used.

Our results show that generative tools are generally perceived as moderately useful and easy to use, although their adoption within the organisation studied remains limited. The dimensions studied are also moderately correlated with each other, except for social influence, which has the lowest correlation scores. Moreover, the frequency of use of the generation tools turns out to be a key factor in understanding the perception of these tools by employees. People who use them more frequently perceive these tools as more useful and have a higher intention to use them.

These findings suggest that LLM-based generation tools have potential to improve efficiency and productivity at work, but their adoption could be hindered by factors such as resistance to change, lack of awareness or lack of adequate training. Organisations seeking to encourage the use of these tools could consider initiatives to promote their adoption, such as training, workshops, or internal communication campaigns. It still conditionnate by the purpose of the implementation (improving workers agentivity, helping them for specific tasks, remplacing them, etc.).

It is important to note that this study has some limitations, including the sample size and the fact that it focuses on a single organisation. Future research could extend this study to other work contexts and include larger samples to examine the generalisability of our findings. In addition, qualitative studies could be conducted to further our understanding of why
some people adopt these tools more quickly than others, and to identify best practices for promoting their adoption.

In our perspectives, this study is the first step of a larger investigation on the acceptability of that kind of tool as a professional media. For future work, we aim to focus on the type of requests that developers would prompt to ChatGPT specifically. Investigate this point could permit us to have a better overview on the nature of the interaction with that tool for this specific public.

Ultimately, this study provides valuable insight into how employees perceive LLM-based generation tools. But it also offers resources of reflexion for organisations looking to leverage these technologies to improve the productivity and experience of their employees in getting their jobs done.

## Acknowledgement

We acknowledge the support of Groupe SIGMA.

## References

Agossah, A., Krupa, F., Deconde, G., Perreira da Silva, M., Le Callet, P., 2022a. Déploiement de l'IA en situation de travail : une trop faible considération des employés ?, in: 33ème Conférence Internationale Francophone sur l'Interaction Humain-Machine (IHM 2022), Namur, Belgium. URL: https://hal.science/hal-03789529.

Agossah, A., Lévêque, L., Perreira da Silva, M., Le Callet, P., Krupa, F., Deconde, G., 2022b. Liens entre confiance et acceptabilité dans un dispositif IA, in: 33ème Conférence Internationale Francophone sur l'Interaction Humain-Machine (IHM 22), Namur, Belgium. URL: https://hal.science/hal-03789503.

Arrieta, A.B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al., 2020. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion 58, 82–115.

Atarodi, S., Berardi, A.M., Toniolo, A.M., 2019. Le modèle d'acceptation des technologies depuis 1986: 30 ans de développement. Psychologie du Travail et des Organisations 25, 191–207.
some people adopt these tools more quickly than others, and to identify best practices for promoting their adoption.

In our perspectives, this study is the first step of a larger investigation on the acceptability of that kind of tool as a professional media. For future work, we aim to focus on the type of requests that developers would prompt to ChatGPT specifically. Investigate this point could permit us to have a better overview on the nature of the interaction with that tool for this specific public.

Ultimately, this study provides valuable insight into how employees perceive LLM-based generation tools. But it also offers resources of reflexion for organisations looking to leverage these technologies to improve the productivity and experience of their employees in getting their jobs done.

## Acknowledgement

We acknowledge the support of Groupe SIGMA.

## References

Agossah, A., Krupa, F., Deconde, G., Perreira da Silva, M., Le Callet, P., 2022a. Déploiement de l'IA en situation de travail : une trop faible considération des employés ?, in: 33ème Conférence Internationale Francophone sur l'Interaction Humain-Machine (IHM 2022), Namur, Belgium. URL: https://hal.science/hal-03789529.

Agossah, A., Lévêque, L., Perreira da Silva, M., Le Callet, P., Krupa, F., Deconde, G., 2022b. Liens entre confiance et acceptabilité dans un dispositif IA, in: 33ème Conférence Internationale Francophone sur l'Interaction Humain-Machine (IHM 22), Namur, Belgium. URL: https://hal.science/hal-03789503.

Arrieta, A.B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al., 2020. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion 58, 82–115.

Atarodi, S., Berardi, A.M., Toniolo, A.M., 2019. Le modèle d'acceptation des technologies depuis 1986: 30 ans de développement. Psychologie du Travail et des Organisations 25, 191–207.
Bobillier-Chaumon, M., Dubois, M., 2009. L'adoption des technologies en situation professionnelle: quelles articulations possibles entre acceptabilité et acceptation? Le travail humain 72, 355-382.

Dakhel, A.M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M.C., Ming, Z., et al., 2022. Github copilot ai pair programmer: Asset or liability? arXiv preprint arXiv:2206.15331.

Davis, F.D., et al., 1989. Technology acceptance model: Tam. Al-Suqri, MN, Al-Aufi, AS: Information Seeking Behavior and Technology Adoption, 205-219.

Ferguson, Y., 2019. Ce que l'intelligence artificielle fait de l'homme au travail. visite sociologique d'une entreprise. Les Mutations du travail, Paris, Éd. La Découverte, 23-42.

Gamkrelidze, T., Barcellini, F., Zouinar, M., 2021. Intelligence artificielle dans les activités professionnelles: quelles visions des acteurs concernés?, in: 55ème congrès de la SELF.

Imai, S., 2022. Is github copilot a substitute for human pair-programming? an empirical study, in: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, pp. 319-321.

Nguyen, N., Nadi, S., 2022. An empirical evaluation of github copilot's code suggestions, in: Proceedings of the 19th International Conference on Mining Software Repositories, pp. 1-5.

Nielsen, J., 1994. Usability engineering. Morgan Kaufmann.

Pasquier, H.M.L., 2012. Définir l'acceptabilité sociale dans les modèles d'usage: vers l'introduction de la valeur sociale dans la prédiction du comportement d'utilisation. Ph.D. thesis. Université Rennes 2.

Rudin, C., 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence 1, 206-215.

Tricot, A., Plégat-Soutjis, F., Camps, J.F., Amiel, A., Lutz, G., Morcillo, A., 2003. Utilité, utilisabilité, acceptabilité: interpréter les relations entre trois
Bobillier-Chaumon, M., Dubois, M., 2009. L'adoption des technologies en situation professionnelle: quelles articulations possibles entre acceptabilité et acceptation? Le travail humain 72, 355-382.

Dakhel, A.M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M.C., Ming, Z., et al., 2022. Github copilot ai pair programmer: Asset or liability? arXiv preprint arXiv:2206.15331.

Davis, F.D., et al., 1989. Technology acceptance model: Tam. Al-Suqri, MN, Al-Aufi, AS: Information Seeking Behavior and Technology Adoption, 205-219.

Ferguson, Y., 2019. Ce que l'intelligence artificielle fait de l'homme au travail. visite sociologique d'une entreprise. Les Mutations du travail, Paris, Éd. La Découverte, 23-42.

Gamkrelidze, T., Barcellini, F., Zouinar, M., 2021. Intelligence artificielle dans les activités professionnelles: quelles visions des acteurs concernés?, in: 55ème congrès de la SELF.

Imai, S., 2022. Is github copilot a substitute for human pair-programming? an empirical study, in: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, pp. 319-321.

Nguyen, N., Nadi, S., 2022. An empirical evaluation of github copilot's code suggestions, in: Proceedings of the 19th International Conference on Mining Software Repositories, pp. 1-5.

Nielsen, J., 1994. Usability engineering. Morgan Kaufmann.

Pasquier, H.M.L., 2012. Définir l'acceptabilité sociale dans les modèles d'usage: vers l'introduction de la valeur sociale dans la prédiction du comportement d'utilisation. Ph.D. thesis. Université Rennes 2.

Rudin, C., 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence 1, 206-215.

Tricot, A., Plégat-Soutjis, F., Camps, J.F., Amiel, A., Lutz, G., Morcillo, A., 2003. Utilité, utilisabilité, acceptabilité: interpréter les relations entre trois
dimensions de l'évaluation des eiah, in: Environnements Informatiques pour l'Apprentissage Humain 2003, ATIEF; INRP. pp. 391-402.

Venkatesh, V., Thong, J.Y., Xu, X., 2012. Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. MIS quarterly, 157-178.

Yin, M., Wortman Vaughan, J., Wallach, H., 2019. Understanding the effect of accuracy on trust in machine learning models, in: Proceedings of the 2019 chi conference on human factors in computing systems, pp. 1-12.
dimensions de l'évaluation des eiah, in: Environnements Informatiques pour l'Apprentissage Humain 2003, ATIEF; INRP. pp. 391-402.

Venkatesh, V., Thong, J.Y., Xu, X., 2012. Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. MIS quarterly, 157-178.

Yin, M., Wortman Vaughan, J., Wallach, H., 2019. Understanding the effect of accuracy on trust in machine learning models, in: Proceedings of the 2019 chi conference on human factors in computing systems, pp. 1-12.