# Technology-Enhanced Organizational Learning: A Systematic Literature Review

Michail N. Giannakos, Patrick Mikalef, Ilias O. Pappas

► To cite this version:

Michail N. Giannakos, Patrick Mikalef, Ilias O. Pappas. Technology-Enhanced Organizational Learning: A Systematic Literature Review. 18th Conference on e-Business, e-Services and e-Society (I3E), Sep 2019, Trondheim, Norway. pp.573-584, 10.1007/978-3-030-29374-1_46. http-02510139

HAL Id: hal-02510139

https://inria.hal.science/hal-02510139v1

Submitted on 17 Mar 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
Technology-Enhanced Organizational Learning: A
Systematic Literature Review

Michail N. Giannakos<sup>1[0000-0002-8016-6208]</sup>, Patrick Mikalef<sup>1[0000-0002-6788-2277]</sup> and Ilias
O. Pappas<sup>1 2 [0000-0001-7528-3488]</sup>

¹ Norwegian University of Science and Technology, Trondheim, NO-7491, Norway

² University of Agder, Kristiansand, NO-4639, Norway
michailg@ntnu.no

**Abstract.** E-Learning systems are receiving ever increasing attention in academia, businesses as well as in public administrations. Managers and employee who need efficient forms of training as well as learning flow within the organization, do not have to gather in a place at the same time, or to travel far away for attending courses. Contemporary affordances of e-learning systems allow them to perform different jobs or tasks for training courses according to their own scheduling, as well as collaborate and share knowledge and experiences that results rich learning flow within the organization. The purpose of this article is to provide a systematic review of empirical studies in the intersection of e-learning and organizational learning in order to summarize the current findings and guide future research. Forty peer-reviewed articles were collected from a systematic literature search and analyzed based on a categorization of their main elements. This survey identifies five major directions Technology-Enhanced Organizational learning has been focused during the last decade. Future research should leverage on big data produced from the platforms and investigate how the incorporation of advanced learning technologies (e.g., learning analytics, personalized learning) can help increasing organizational value.

**Keywords:** Organizational Learning, E-learning, Literature Review, Learning Environments.

# 1 Introduction

E-Learning covers the integration of Information and Communication Technology (ICT) in environments with the main goal to foster learning [1]. The term E-Learning is often used interchangeably with the term Technology-Enhanced Learning (TEL), to portray several modes of such environments (i.e., online, virtual learning environments etc.). The digitalization of resources and processes enables flexible ways to foster learning across the different sections and personnel inside an organization. Learning has long been associated in the past with formal or informal education and training. However organizational learning is much more than that. It can be defined as “the process of improving actions through better knowledge and understanding” [2].
Organizational learning is extremely important in an organization, since it is associated
with the process of creating value from an organizations' intangible assets; it combines
notions from several different domains, such as organizational behavior, human
resource management, artificial intelligence and information technology [3].

In this work, we define Technology-Enhanced Organizational Learning (TEOL) as “the utilization of digital technologies to enhance the process of improving actions through better knowledge and understanding in an organization”. During the last years, there is a significant body of research focusing in the intersection of TEL and organizational learning (i.e., TEOL) [4, 9, 10]. However, there is systematic work that summarizes and conceptualizes the results in order to reinforce the swift of enterprises that want to move from information-based to knowledge-based enterprises [3]. Thus, the study addresses the following research questions:

* What is the current status of Technology-Enhanced Organizational Learning research, seen through the lens of areas of implementation (e.g., industries, public sector), technologies used, and methodologies (e.g., types of data and data analysis techniques employed)?

Our motivation for this work is based on the emerging developments in the area of
learning technologies, creating momentum for their adoption in organizations. The
purpose of this paper is to provide a review of research on the Technology-Enhanced
Organizational Learning research in order to summarize the findings and guide future
studies. This study can provide a springboard for other scholars and practitioners,
especially in the area of knowledge-based enterprises, to examine TEL approaches by
taking into consideration the prior and ongoing research efforts.

The rest of the paper is organized as follows. In the next section we present the
related background work; the third section describes the methodology used for the
literature review describing how the studies were selected and analyzed. The fourth
section presents the research findings derived from the data analysis based on the
specific areas of focus. Finally, in the last section, the authors discuss the results and
make suggestions for future work.

## 2 Background

Switching from the information-based enterprise to the knowledge-based enterprise is a major challenge for today's companies [3]. Uni-directional learning flows, such as formal and informal training, is surely important but not sufficient to cover the needs enterprises face [4]. To uphold enterprises' competitiveness, enterprise staff have to operate in highly intense information and knowledge-oriented environments. Traditional learning approaches fail to substantiate learning flow on the basis of daily evidence and experience. Thus, novel, ubiquitous and flexible learning mechanisms are needed, placing the human (e.g., employees, managers, civil servants etc.) at the center of the information and learning flow and bridging traditional learning with experiential, social and smart learning.
Organizations consider the lack of skills and competences as being the major knowledge-related factors hampering innovation today [3]. Thus, the implementation of solutions supporting informal, everyday and work training (e.g., social learning, VR/AR solutions etc.) in order to develop individual staff competences as well to upgrade the competence affordances at the organization level.

TEOL has been delivered primarily in the form of web-based learning [3]. More recently, the TEL tools portfolio is rapidly expanding to make more efficient joint use of novel learning concepts, methodologies and technological enablers to achieve more direct, effective and lasting learning impact. Employing virtual learning environments, mobile-learning solutions and AR/VR technologies and head-mounted displays so trainees are empowered to follow their own training pace, learning topics and assessment tests that fit their needs [6, 34, 35]. The spread of use of social networking tools has also brought attention to the contribution of social and collaborative learning [17, 38].

Contemporary learning systems supporting adaptive, personalized and collaborative learning, expand the toolset available in TEOL and contribute to the adoption, efficiency and general prospects of the introduction of TEL in the organization [19]. During the last years, TEOL has put particular emphasis in the form of sharing internal and external to the enterprise knowledge, with systems that leverage collaborative learning and social learning functionalities [25, 32]. This is the essence of Computer Supported Collaborative Learning (CSCL), CSCL literature has developed a framework that combines individual learning, organizational learning and collaborative learning, facilitated by establishing adequate learning flows and emerges effective learning in an enterprise learning [5], in Fig. 1.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Representation of the combination of enterprise learning and knowledge flows (adapted from [5]).</figcaption></figure>

Establishing efficient knowledge and learning flows is a prime target for future data-driven enterprises [3]. Considering the involved knowledge, the human resources and their required skills in an enterprise, a clear need for continuous, flexible and efficient learning exists. This can be achieved by contemporary learning systems and practices, that provide high adoption, smooth usage, high satisfaction and alignments with the
current practices of the enterprise. Since, the required competences in an enterprise are evolving over time, the development of competence models needs to be agile and leverage on state-of-the art technologies that align with organization's processes and models. Therefore, with this paper we attempt to provide a review of the TEOL research in order to summarize the findings and guide the development of organizational learning in future enterprises as well as future studies.

# 3 Methodology

To answer the research questions, the authors decided to conduct a systematic review of the literature by following transparent procedure adopted in the field of information systems and software engineering in order to minimize potential researcher biases and support reproducibility [7].

## 3.1 Articles collection

Several procedures were followed to ensure high quality review of the literature of TEOL. A comprehensive search of peer-reviewed articles was conducted through February 2019, (short papers, posters, dissertations and reports were excluded), based on a relatively inclusive range of key terms: "organizational learning" "elearning", "organizational learning" "e-learning", "organisational learning" "elearning" and "organisational learning" "e-learning". The term "elearning" (also written as e-learning) was selected, since it's an umbrella term that always captures articles that use different terminology (e.g., learning technology, educational technology, technology enhanced learning). Publications were selected from 2010 onwards, since after 2010 we saw tremendous advancements (e.g., MOOCs, learning analytics, personalized learning) in the area of learning technologies. A wide variety of databases were searched, including the SpringerLink, Wiley, ACM Digital Library, IEEE Xplore, Science Direct, SAGE, ERIC, AIS eLibrary and Taylor & Francis. The search process uncovered 2.347 peer-reviewed articles.

## 3.2 Inclusion and exclusion criteria

The selection phase determines the overall validity of the literature review, and thus it's important to define specific inclusion and exclusion criteria. As Dybå and Dingsøyr [8] specified, the quality criteria needs to cover three main issues (i.e. rigour, credibility, and relevance) that needs to be considered when evaluating the quality of the selected studies. We applied eight quality criteria informed by the proposed Critical Appraisal Skills Programme (CASP) and related works [8]. You can see those criteria in Table 1.

<table><caption>**Table 1.** Quality Criteria</caption><tr><td>1.</td><td>Does the study clearly address the research problem?</td></tr><tr><td>2.</td><td>Is there a clear statement of the aims of the research?</td></tr><tr><td>3.</td><td>Is there an adequate description of the context in which the research was carried out?</td></tr></table>
4. Was the research design appropriate to address the aims of the research?

5. Does the study clearly determine the research methods (subjects, instruments, data collection, data analysis)?

6. Was the data analysis sufficiently rigorous?

7. Is there a clear statement of findings?

8. 8. Is the study of value for research or practice?

Therefore, studies were eligible for inclusion if they were focused on TEOL. The aforementioned criteria were applied in stage 2 and stage 3 of the selection process (figure 2), when the researcher had to assess the papers based on their titles and abstracts, and the full papers.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Stages of the selection process</figcaption></figure>

3.3 **Analysis**

Each collected study was analyzed based on the following elements: study design
(e.g., experiment, case study), area (e.g., IT, healthcare), technology (e.g., wiki, social
media), population (e.g., managers, employees), sample size, unit of analysis
(individual, firm), data collections (e.g., surveys, interviews), research method, data
analysis and the main research objective of the study. It is important to highlight, that
the articles were coded based on the reported information, different authors reported
information in different level of granularity (e.g., an online system vs the name of the
system), while in some case the information was missing from the paper. Overall the
authors did their best to code the article as accurately and complete as possible.

4 Findings

In this section, Table 2 presents the detailed results. Analysis of the studies was
performed using non-statistical methods considering the variables reported in Table 2.
This section follows by an analysis and discussion of the categories.
<table><caption>**Table 2.** Table captions should be placed above the tables.</caption><thead><tr><th>Study</th><th>Study Design</th><th>Area / Topic</th><th>Technology</th><th>Population</th><th>Sample</th><th>Unit of analysis</th><th>Data Collect.</th><th>Method</th><th>Analysis</th><th>Objectives</th></tr></thead><tbody><tr><td>[9]</td><td>Survey</td><td>ND</td><td>Web</td><td>Mixed</td><td>222</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SEM</td><td>ItU</td></tr><tr><td>[10]</td><td>Survey</td><td>MGM</td><td>generic</td><td>mg</td><td>380</td><td>Individ</td><td>Surv</td><td>Quant</td><td>Reg</td><td>Sat</td></tr><tr><td>[11]</td><td>Survey</td><td>Telec</td><td>generic</td><td>empl</td><td>128</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SEM</td><td>Sat</td></tr><tr><td>[12]</td><td>Exp</td><td>Bsn</td><td>Online</td><td>stud</td><td>143</td><td>Individ</td><td>Surv</td><td>Quant</td><td>A-VA</td><td>Flearn</td></tr><tr><td>[13]</td><td>Survey</td><td>IT</td><td>Online</td><td>empl</td><td>500</td><td>Individ</td><td>Surv</td><td>Quant</td><td>Reg</td><td>OV</td></tr><tr><td>[14]</td><td>Survey</td><td>Univ.</td><td>Web</td><td>stud</td><td>832</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SEM</td><td>OV</td></tr><tr><td>[15]</td><td>Exp</td><td>IT</td><td>Web</td><td>empl</td><td>24</td><td>Individ</td><td>Surv</td><td>Quant</td><td>Descr</td><td>Per</td></tr><tr><td>[16]</td><td>Exp</td><td>IT</td><td>Web</td><td>empl</td><td>24</td><td>Individ</td><td>Surv/Int</td><td>Mixed</td><td>Descr</td><td>Per</td></tr><tr><td>[17]</td><td>Exp</td><td>IT</td><td>Podcast</td><td>empl</td><td>26</td><td>Individ</td><td>Surv/Int</td><td>Mixed</td><td>CA</td><td>ItU</td></tr><tr><td>[18]</td><td>CaseST</td><td>Cons.</td><td>ABAP</td><td>empl</td><td>12</td><td>Session</td><td>Int</td><td>Quall</td><td>Descr</td><td>Per</td></tr><tr><td>[19]</td><td>Survey</td><td>Cons.</td><td>Web</td><td>clients</td><td>222</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SEM</td><td>ItU</td></tr><tr><td>[20]</td><td>Survey</td><td>ND</td><td>m-learn</td><td>mg.em</td><td>342</td><td>Individ</td><td>Surv</td><td>Quant</td><td>Descr</td><td>Usage</td></tr><tr><td>[21]</td><td>Survey</td><td>ND</td><td>collabor</td><td>ND</td><td>130</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SEM</td><td>Per</td></tr><tr><td>[22]</td><td>Survey</td><td>Health</td><td>KRS</td><td>empl</td><td>800</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SEM</td><td>ItU</td></tr><tr><td>[23]</td><td>CaseST</td><td>Public</td><td>OERs</td><td>civil</td><td>68</td><td>Individ</td><td>Int/FG</td><td>Quall</td><td>CA</td><td>Barr.</td></tr><tr><td>[24]</td><td>Survey</td><td>Ent</td><td>generic</td><td>empl</td><td>317</td><td>Firm</td><td>Surv</td><td>Quant</td><td>SEM</td><td>Per</td></tr><tr><td>[25]</td><td>Exp</td><td>ND</td><td>social</td><td>empl</td><td>28</td><td>Individ</td><td>Surv</td><td>Quant</td><td>Descr</td><td>Per</td></tr><tr><td>[26]</td><td>Survey</td><td>Ent</td><td>smart</td><td>mg</td><td>120</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SEM</td><td>ItU</td></tr><tr><td>[27]</td><td>Exp</td><td>Bsn</td><td>e-portf.</td><td>stud</td><td>7</td><td>Individ</td><td>Surv</td><td>Quant</td><td>Descr</td><td>Flearn</td></tr><tr><td>[28]</td><td>CaseST</td><td>Autom</td><td>collabor</td><td>empl</td><td>3</td><td>Firm</td><td>Int/FG</td><td>Quall</td><td>Descr</td><td>Usage</td></tr><tr><td>[29]</td><td>CaseST</td><td>IT</td><td>collabor</td><td>empl</td><td>202</td><td>Individ</td><td>Surv</td><td>Quant</td><td>Reg</td><td>Per</td></tr><tr><td>[30]</td><td>CaseST</td><td>ND</td><td>generic</td><td>empl</td><td>126</td><td>Individ</td><td>Int</td><td>Mixed</td><td>CA</td><td>Flearn</td></tr><tr><td>[31]</td><td>CaseST</td><td>ND</td><td>Wiki</td><td>res</td><td>16</td><td>Individ</td><td>Surv</td><td>Quant</td><td>Descr</td><td>Flearn</td></tr><tr><td>[32]</td><td>Survey</td><td>Ent</td><td>social</td><td>empl</td><td>97</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SEM</td><td>Flearn</td></tr><tr><td>[33]</td><td>Survey</td><td>IT</td><td>Podcast</td><td>empl</td><td>12</td><td>Individ</td><td>Int</td><td>Quall</td><td>CA</td><td>ItU</td></tr><tr><td>[34]</td><td>Survey</td><td>ND</td><td>generic</td><td>learn</td><td>12</td><td>Firm</td><td>Int</td><td>Quall</td><td>CA</td><td>align</td></tr><tr><td>[35]</td><td>CaseST</td><td>IT</td><td>SL</td><td>ND</td><td>16</td><td>Individ</td><td>Int</td><td>Quall</td><td>CA</td><td>feas</td></tr><tr><td>[36]</td><td>CaseST</td><td>MGM</td><td>Web</td><td>ND</td><td>22</td><td>Individ</td><td>Surv/log</td><td>Quant</td><td>Descr</td><td>Usage</td></tr><tr><td>[37]</td><td>Survey</td><td>several</td><td>smart</td><td>empl</td><td>342</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SEM</td><td>ItU</td></tr><tr><td>[38]</td><td>Survey</td><td>ND</td><td>generic</td><td>learn</td><td>83</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SEM</td><td>ItU</td></tr><tr><td>[39]</td><td>Survey</td><td>Telec</td><td>Online</td><td>empl</td><td>294</td><td>Individ</td><td>Surv</td><td>Quant</td><td>A-VA</td><td>Usage</td></tr><tr><td>[40]</td><td>Survey</td><td>IT</td><td>generic</td><td>empl</td><td>550</td><td>Individ</td><td>Surv</td><td>Quant</td><td>Descr</td><td>feas</td></tr><tr><td>[41]</td><td>CaseST</td><td>Health</td><td>Web</td><td>empl</td><td>40</td><td>Individ</td><td>log</td><td>Quant</td><td>Descr</td><td>Usage</td></tr><tr><td>[42]</td><td>CaseST</td><td>ND</td><td>generic</td><td>empl</td><td>15</td><td>Individ</td><td>Int</td><td>Quall</td><td>CA</td><td>Usage</td></tr><tr><td>[43]</td><td>Survey</td><td>Electr.</td><td>generic</td><td>empl</td><td>379</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SEM</td><td>Flearn</td></tr><tr><td>[44]</td><td>Survey</td><td>ND</td><td>generic</td><td>empl</td><td>120</td><td>firm</td><td>Surv</td><td>Quant</td><td>SEM</td><td>Benef</td></tr><tr><td>[45]</td><td>Exp</td><td>IT</td><td>Web</td><td>empl</td><td>24</td><td>Individ</td><td>Surv/Int</td><td>Mixed</td><td>Descr</td><td>Per</td></tr><tr><td>[46]</td><td>Survey</td><td>Telec</td><td>generic</td><td>ND</td><td>297</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SEM</td><td>Sat</td></tr><tr><td>[47]</td><td>Survey</td><td>ND</td><td>Web</td><td>civil</td><td>439</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SEM</td><td>ItU</td></tr><tr><td>[48]</td><td>CaseST</td><td>IT</td><td>generic</td><td>empl</td><td>93</td><td>Individ</td><td>Surv</td><td>Quant</td><td>SNA</td><td>Flearn</td></tr></tbody></table>

Survey, Survey study; Exp, - Experiment; CaseSt, Case Study; ND, Non-Defined; MGM, Management; Telec, Telecommunication; Bsn, Business; Univ, University; Cons., Consulting; Public, Public Sector; Ent., Enterprise; Web, Web-based; KRS, knowledge repository system; OERs, Open Educational Resources; SL, Second Life, mg, Managers; empl, employees; stud, students; res, Researchers; learn, learning specialists; Indiv., Individual; Surv, Surveys; Int, Interviews; FG, Focus Groups; Log, Log files; Reg, Regression Analysis; Descr, Descriptive Statistics; A-VA, Analysis of variances/covariance; CA, Content Analysis; ItU, Intention to Use; Sat, Satosfaction; OV, Organizational Value; Per, Performance; Flearn, foster learning; Benef, Benefits; align, alignment; feas, feasibility; Barr, Barriers
## 4.1 Sample size and population involved

The categories related to the sample of the articles, include the number of participants in each study (size), their background (e.g., managers, employees) and the area/topic the study was conducted. The majority of the studies involve employees (24), with few studies involve managers (3), civil servants (2), learning specialists (2), clients and researchers. Regarding the sample size, approximately half of the studies (19) have been conducted with less than 100 participants, few (9) can be considered large scale studies (more than 300 participants) and few (8) can be considered small scale (less than 20 participants). When it comes to the area/topic the study was conducted, most of the studies (9) were conducted in the context of the IT industry, but also having a good coverage of other important areas (i.e., healthcare, telecommunications, business, public sector). Interestingly, there are several studies that either didn't define the area or they were implemented in a generic context (e.g., participants from different sections or companies).

## 4.2 Research methods

When assessing the status of research of an area, one of the most important aspects is the methodology used. By "method" in table 1, we refer to the distinction among quantitative, qualitative, and mixed research. In addition to the method, in our categorization protocol we also "study design", referring to the distinction among survey study (i.e., gathers data by asking a group of participants), experiment (i.e., creation of situations to record beneficial data) and case study (i.e., closely studies a group of individuals).

Based on this categorization, we can see from Table 2 that the majority of the papers are quantitative (29) and qualitative (7) with few studies (4) utilizing mixed methods. Regarding the study design, most of the studies were survey studies (22), 11 were the case studies and fewer were experiments (7). The unit of analysis in most of the studies was the individual participant (35), with fewer studies having the firm as the unit of analysis and one study the training session. Regarding the measures used on the studies, most of them utilized surveys (32), followed from interviews (11), with few studies using field notes from focus groups (2) and log files from the systems (2). We only had 6 studies were the researchers used different measures to triangulate or extend their findings. The majority of the articles used Structural Equation Modeling (SEM) (15) to analyze their data, with 11 studies employing descriptive statistics, 7 content analysis, 5 regression analysis or analyses of variances/covariance and 1 study using social network analysis (SNA).

## 4.3 Technologies

Concerning the technology used, most of the studies (12) didn't study a specific system, but they referred in their investigation to a generic e-learning or technological solution. Several studies (8) named web-based learning environment, without describing the functionalities of identifying the system. The rest of the studies focused on online
learning environment (3), collaborative learning systems (3), social learning systems (2), smart learning systems (2), podcasting (2) and the rest of the studies used a specific system (e.g., a wiki, mobile learning system, ABAP, e-portfolios, second life).

## 4.4 Research objectives

Concerning the research objectives of the studies, we can separate them in six main categories. The first category focuses intention of the employees to use the technology (9), the second focuses in the performance of the employees (8), the third focuses in the value/outcome of the organization (2), the fourth focuses in the actual usage of the system (6), the fifth in employees satisfaction (3) and the sixth in the ability of the proposed system to foster learning (7). In addition to the six aforementioned categories, we also saw studies focusing in potential barriers for TEOL in the organization [23], the various benefits associated with the successful implementation of TEOL [44], the feasibility [35, 40] and the alignment of the proposed innovation with the other processes and systems in the organization [34].

# 5 Discussion and Conclusion

After reviewing the 40 identified articles in the area of TEOL, we can agree on the importance of the affordances offered from different e-learning technologies, as well as, the importance of the relationship between TEOL and employees satisfaction and performance, and benefits associated with organizational value and outcome. TEOL provides employees, managers and even clients opportunities to learn in a more differentiated manner, compared to the formal learning form. However, how the organization adopts these capabilities and leverages on them to achieve its goals is a complex and challenging procedure.

Several studies [7, 22, 26] focused on the positive effect of perceived managerial support, perceived usefulness, perceived ease of use and other Technology Acceptance (TAM) related constructs of the e-learning system in supporting all the three levels of learning (i.e., individual, collaborative and organizational). Another interesting dimension highlighted from many studies [12, 21, 31] is the role of socialization in the adoption and usage of the e-learning system. Building connections and creating a shared learning space in the e-learning system is critical [12]. This is consistent with the expectancy-theoretical explanation of the social context impacts on employees motivation to participate in learning [7, 26].

Organizational learning literature suggests that e-learning may be more appropriate for the acquisition of certain types of knowledge than others (e.g. procedural vs. declarative, or hard-skills vs. soft-skills), however there is no empirical evidence for that [12]. However, the literature highlights the importance of integrating organizational, individual, and social perspectives [25] in TEOL. In addition, for the development TEOL approach, the organization needs to consider the alignment of individual learning needs, organizational objectives, and social networking [25]. To achieve this, it is advisable for the organization to define the expected objectives and
technologies that have the capacity to support them, and enrich them with self-directed
and socially constructed learning practice in the organization [25].

**5.1 Status of TEOL research**

The current review suggests that, while the efficient implementation of TEOL entails
certain challenges, there is also a great potential for employees' performance and
organizational outcome and value overall. There are also opportunities for improving
organizations learning flow that might not be feasible for formal learning and training.
Overall the research conducted during the last decade in TEOL has focused on the
following directions:

1. **Investigating the affordances of different technologies in different organizations.**

In particular focusing in aspects of how easy to use or useful the technology is, or
how well aligned / integrated to other systems and processes within the
organization. In addition, studies focused on how different learning technologies
(e.g., smart, social, personalized etc.) contribute to organizational learning in
different contexts and needs.

2. **Enriching the learning flow and learning potential in different levels.**

How different factors contribute to different levels of organizational learning, and
practices to address individual, collaborative and organizational learning within
the structure of the organization.

3. **Identifying critical aspects for successful TEOL.**

There is a considerable amount of mainly qualitative studies, focusing on potential barriers of TEOL implementation as well as risks and requirements associated with the feasibility. In the same vain, there was an emphasis in the importance of alignment of TEOL (in both processes and technologies) within the organization.

4. **Implementing employee-centric TEOL.**

In most of the studies, the main objective was to increase employee's adoption,
satisfaction and usage of the learning system. In addition, several studies focused
on TEOL's ability to increase employee's performance as well as to increase
knowledge flow and foster learning.

5. **Achieving goals associated with the value creation of the organization.**

A considerable number of studies, that utilized the firm as a unit of analysis (and
not the individual employee) focused on TEOL's capacity to increase
organizational value and customer value.

**5.2 Implications and future work**

Several implications for TEOL were revealed through this literature review. First,
most of the studies agree that employees/trainees experience is extremely important for
the successful implementation of TEOL. Thus keeping them in the design and
implementation cycle of TEOL will increase the adoption and satisfaction as well as
reduces the risks and barriers. Another important implication relates with the qualities
of the technologies, easy to use, useful and social technologies result more efficient
TEOL. Thus it's important for the organization to incorporate these functionalities in the platform and reinforce them within appropriate content and support. This should not only benefit learning outcomes, but it can also provide the networking opportunities for employees to broaden their personal networks, that are often lost when companies move to e-learning.

A number of suggestions for further research have emerged from reviewing prior and ongoing work on TEOL. One recommendation for future researchers is to clearly describe the TEOL approach by providing detailed information for the technologies and materials used, as well as the organizations. This will allow us to conduct meta analyses and identify potential effects of firm's size or area on the performance and other aspects related with organizational value. Future work should also focus on collecting and triangulating different types of data from different sources. The reviewed studies have been conducted using mainly survey data and limited usage of data coming from the platforms, thus the interpretations and triangulation between the different types of the collected data were limited.

## Acknowledgments

This work has received funding from the Norwegian Research Council under the project FUTURE LEARNING (number: 255129/H20) and Xdesign (290994/F20).

## References

1. Rosenberg, M. J., & Foshay, R. (2002). E-learning: Strategies for delivering knowledge in the digital age. Performance Improvement, 41(5), 50-51.

2. Fiol, C. M., & Lyles, M. A. (1985). Organizational learning. Academy of management review, 10(4), 803-813.

3. El Kadiri, S., Grabot, B., Thoben, K. D., Hribernik, K., Emmanouilidis, C., Von Cieminski, G., & Kiritsis, D. (2016). Current trends on ICT technologies for enterprise information systems. Computers in Industry, 79, 14-33.

4. Manuti, A., Pastore, S., Scardigno, A. F., Giancaspro, M. L., & Morciano, D. (2015). Formal and informal learning in the workplace: a research review. International journal of training and development, 19(1), 1-17.

5. Goggins, S. P., Jahnke, I., & Wulf, V. (2013). Computer-supported collaborative learning at the workplace. New York, NY: Springer.

6. Muller Queiroz, A. C., Moreira Nascimento, A., Tori, R., Brashear Alejandro, T., Veloso de Melo, V., de Souza Meirelles, F., & da Silva Leme, M. I. (2018). Immersive Virtual Environments in Corporate Education and Training. In AMCIS.

7. Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering.

8. Dybå, T., & Dingsøyr, T. (2008). Empirical studies of agile software development: A systematic review. Information and software technology, 50(9-10), 833-859.

9. Cheng, B., Wang, M., Moormann, J., Olaniran, B. A., & Chen, N. S. (2012). The effects of organizational learning environment factors on e-learning acceptance. Computers & Education, 58(3), 885-899.
10. Mitić, S., Nikolić, M., Jankov, J., Vukonjanski, J., & Terek, E. (2017). The impact of information technologies on communication satisfaction and organizational learning in companies in Serbia. *Computers in Human Behavior*, 76, 87-101.

11. Navimipour, N. J., & Zareie, B. (2015). A model for assessing the impact of e-learning systems on employees' satisfaction. *Computers in Human Behavior*, 53, 475-485.

12. Yanson, R., & Johnson, R. D. (2016). An empirical examination of e-learning design: The role of trainee socialization and complexity in short term training. *Computers & Education*, 101, 43-54.

13. Iris, R., & Vikas, A. (2011). E-Learning technologies: A key to Dynamic Capabilities. *Computers in Human Behavior*, 27(5), 1868-1874.

14. Alsabawy, A. Y., Cater-Steel, A., & Soar, J. (2013). IT infrastructure services as a requirement for e-learning system success. *Computers & Education*, 69, 431-451.

15. Jia, H., Wang, M., Ran, W., Yang, S. J., Liao, J., & Chiu, D. K. (2011). Design of a performance-oriented workplace e-learning system using ontology. *Expert Systems with Applications*, 38(4), 3372-3382.

16. Wang, M., Vogel, D., & Ran, W. (2011). Creating a performance-oriented e-learning environment: A design science approach. *Information & Management*, 48(7), 260-269.

17. Wei, K., & Ram, J. (2016). Perceived usefulness of podcasting in organizational learning: The role of information characteristics. *Computers in Human Behavior*, 64, 859-870.

18. Bologa, R., & Lupu, A. R. (2014). Organizational learning networks that can increase the productivity of IT consulting companies. A case study for ERP consultants. *Expert Systems with Applications*, 41(1), 126-136.

19. Cheng, B., Wang, M., Yang, S. J., & Peng, J. (2011). Acceptance of competency-based workplace e-learning systems: Effects of individual and peer learning support. *Computers & Education*, 57(1), 1317-1333.

20. Lee, J., Kim, D. W., & Zo, H. (2015). Conjoint analysis on preferences of HRD managers and employees for effective implementation of m-learning: The case of South Korea. *Telematics and Informatics*, 32(4), 940-948.

21. Choi, S., & Ko, I. (2012). Leveraging electronic collaboration to promote interorganizational learning. *International Journal of Information Management*, 32(6), 550-559.

22. Tsai, C. H., Zhu, D. S., Ho, B. C. T., & Wu, D. D. (2010). The effect of reducing risk and improving personal motivation on the adoption of knowledge repository system. *Technological forecasting and Social change*, 77(6), 840-856.

23. Stoffregen, J. D., Pawlowski, J. M., Ras, E., Tobias, E., Šćepanović, S., Fitzpatrick, D., ... & Friedrich, H. (2016). Barriers to open e-learning in public administrations: A comparative case study of the European countries Luxembourg, Germany, Montenegro and Ireland. *Technological Forecasting and Social Change*, 111, 198-208.

24. López-Nicolás, C., & Meroño-Cerdán, Á. L. (2011). Strategic knowledge management, innovation and performance. *International journal of information management*, 31(6), 502-509.

25. Wang, M. (2011). Integrating organizational, social, and individual perspectives in Web 2.0-based workplace e-learning. *Information Systems Frontiers*, 13(2), 191-205.

26. Lee, J., Choi, M., & Lee, H. (2015). Factors affecting smart learning adoption in workplaces: comparing large enterprises and SMEs. *Information Technology and Management*, 16(4), 291-302.

27. Wang, S., & Wang, H. (2012). Organizational schemata of e-portfolios for fostering higher-order thinking. *Information Systems Frontiers*, 14(2), 395-407.
28. Siadaty, M., Jovanović, J., Gašević, D., Jeremić, Z., & Holocher-Ertl, T. (2010). Leveraging semantic technologies for harmonization of individual and organizational learning. In European Conference on Technology Enhanced Learning (pp. 340-356). Springer.

29. Subramaniam, R., & Nakkeeran, S. (2019). Impact of Corporate E-Learning Systems in Enhancing the Team Performance in Virtual Software Teams. In Smart Technologies and Innovation for a Sustainable Future (pp. 195-204). Springer.

30. Kaschig, A., Maier, R., Sandow, A., Lazoi, M., Barnes, S. A., Bimrose, J., ... & Schmidt, A. (2010). Knowledge maturing activities and practices fostering organisational learning: results of an empirical study. In European Conference on Technology Enhanced Learning (pp. 151-166). Springer.

31. Khalili, A., Auer, S., Tarasowa, D., & Ermilov, I. (2012). SlideWiki: elicitation and sharing of corporate knowledge using presentations. In International Conference on Knowledge Engineering and Knowledge Management (pp. 302-316). Springer.

32. Qi, C., & Chau, PY. (2016). An Empirical Study of the Effect of Enterprise Social Media Usage on Organizational Learning. In PACIS (p. 330).

33. Wei, K., Sun, H., & Li, H. (2013). On the Driving Forces of Diffusion of Podcasting in Organizational Settings: A Case Study and Propositions. In PACIS (p. 217).

34. Costello, J. T., & McNaughton, R. B. (2018). Integrating a dynamic capabilities framework into workplace e-learning process evaluations. Knowledge and Process Management, 25(2), 108-125.

35. Mueller, J., Hutter, K., Fueller, J., & Matzler, K. (2011). Virtual worlds as knowledge management platform—a practice-perspective. Information Systems Journal, 21(6), 479-501.

36. Hung, Y. H., Lin, C. F., & Chang, R. I. (2015). Developing a dynamic inference expert system to support individual learning at work. British Journal of Educational Technology, 46(6), 1378-1391.

37. Lee, J., Zo, H., & Lee, H. (2014). Smart learning adoption in employees and HRD managers. British Journal of Educational Technology, 45(6), 1082-1096.

38. Hester, A. J., Hutchins, H. M., & Burke-Smalley, L. A. (2016). Web 2.0 and Transfer: Trainers' Use of Technology to Support Employees' Learning Transfer on the Job. Performance Improvement Quarterly, 29(3), 231-255.

39. Gal, E., & Nachmias, R. (2011). Online learning and performance support in organizational environments using performance support platforms. Performance Improvement, 50(8), 25-32.

40. Kim, M. K., Kim, S. M., & Bilir, M. K. (2014). Investigation of the dimensions of workplace learning environments (WLEs): Development of the WLE measure. Performance Improvement Quarterly, 27(2), 35-57.

41. Rober, M. B., & Cooper, L. P. (2011, January). Capturing knowledge via an" Intrapedia": A case study. In 2011 44th Hawaii International Conference on System Sciences (pp. 1-10). IEEE.

42. Michalski, M. P. (2014). Symbolic meanings and e-learning in the workplace: The case of an intranet-based training tool. Management Learning, 45(2), 145-166.

43. Joo, Y. J., Lim, K. Y., & Park, S. Y. (2011). Investigating the structural relationships among organisational support, learning flow, learners' satisfaction and learning transfer in corporate e-learning. British Journal of Educational Technology, 42(6), 973-984.

44. Liu, Y. C., Huang, Y. A., & Lin, C. (2012). Organizational factors' effects on the success of e-learning systems and organizational benefits: An empirical study in Taiwan. The International Review of Research in Open and Distributed Learning, 13(4), 130-151.

45. Wang, M., Ran, W., Liao, J., & Yang, S. J. (2010). A performance-oriented approach to e-learning in the workplace. Journal of Educational Technology & Society, 13(4), 167-179.
46. Lin, C. Y., Huang, C. K., & Zhang, H. (2019). Enhancing Employee Job satisfaction via E-learning: The Mediating Role of an Organizational Learning Culture. International Journal of Human-Computer Interaction, 35(7), 584-595.

47. Lai, H. J. (2017). Examining civil servants’ decisions to use Web 2.0 tools for learning, based on the decomposed theory of planned behavior. Interactive Learning Environments, 25(3), 295-305.

48. Škerlavaj, M., Dimovski, V., Mrvar, A., & Pahor, M. (2010). Intra-organizational learning networks within knowledge-intensive learning environments. Interactive Learning Environments, 18(1), 39-63.