# 3D Interactions between Virtual Worlds and Real Life in an E-Learning Community

Authors: Ulrike Lucke; Raphael Zender
Journal: Advances in Human-Computer Interaction
 (2011)

---

## Abstract

Virtual worlds became an appealing and fascinating component of today's internet. In particular, the number of educational providers that see a potential for E-Learning in such new platforms increases. Unfortunately, most of the environments and processes implemented up to now do not exceed a virtual modelling of real-world scenarios. In particular, this paper shows that Second Life can be more than just another learning platform. A flexible and bidirectional link between the reality and the virtual world enables synchronous and seamless interaction between users and devices across both worlds. The primary advantages of this interconnection are a spatial extension of face-to-face and online learning scenarios and a closer relationship between virtual learners and the real world.


---

## Body

## 1. Introduction

Interactivity is closely related to aspects of networking and interdisciplinary development, bringing together researchers from engineering, computer science, media art and design, and social sciences. Here, the importance of computer science needs to be emphasized along with the growing immersion of digital systems in our daily life: dealing with computers can be seen as a new cultural technique besides reading, writing, and calculating [1, 2]. There is a strong mutual penetration of the digital and physical world, leading to phenomena like virtual reality (computers mirroring the real world) or augmented reality (real-world objects enriched with digital information). For several years, virtual 3D worlds gained significant public attention. The most recent and most famous of these worlds is Second Life, but several text- or graphic-based environments in the web existed before. As first euphoria and commercial initiatives calm down [3], the scientific interest in media and information theory is raising (e.g., for educational applications [4, 5] and for the problem of multiple identities [6]).From a cultural or psychological perspective, virtual 3D worlds allow to study human behaviour in a decoupled, reversible way—like mirroring the reality, including other people’s intimate thoughts, by interacting through a the 3D interface [7]. In this way, there is a significant change in that virtual items with an artificial, digital environment become more and more reality [8]. This opens a new perspective to the pervasiveness of human-computer interfaces: It does not matter where, how, and what type of interface the user is interacting with; his intuitive movement (input like turning around, entering an area, or touching the screen) is immediately followed by a reaction (output like showing a text or an image, playing an audio or video stream). The media architecture connects people, space, and data by interleaving physical and virtual reality creating an extended sphere of (inter)action. This relates to the theory of cognition where receiving, processing, and transmission of information are understood as a sensuous, somatic experience [9]. From this point of view, the tangibility of an object is not restricted to physical environments but extends into virtual worlds—provided that these offer users the possibilities to (inter)act like in real life. This contrasts strongly to the common vision of tangible user interfaces [10] as graspable instantiation of graphical user interfaces. But, at a closer look it is the implementation of the same concept: to transparently couple bits and atoms in order to let the users interact with a computer in the same way they do with the real world.Considering this, a systematic combination of real life and virtual interaction is promising a huge benefit for electronic learning, in terms of (not only virtually) tangible E-learning interfaces that enrich the experiences of learners—and probably also those of teachers. By a felt-as-somatic interaction with the learning environment the cognitive capabilities of students can be exhausted to a much larger extent than in traditional classroom settings, where learners are typically acting in a much more passive and less individual way. In the following, this paper demonstrates how the tangibility of real-life objects can be closely interweaved with elements in a virtual 3D world. The goal of our work was to systematically interconnect classroom and virtual learning in order to provide a higher level of individuality and flexibility to the user—not only in terms of a “3D remote control,” but as a generalized architecture for flexible, bidirectional interchange of media between different environments (like classroom, media lab, learning platform, and virtual world). The many technical facets of our architectural framework are outside the focus of this paper, like service mash-ups [11], context awareness [12], and streaming media [13]. We will rather concentrate on integrating real-life settings and the virtual world Second Life. Thus, the main contribution of this paper is to explain the benefits of this approach for education. We are going to describe how the virtual and real-life environment fit into the general concept, which possibilities for interaction they offer to the users, how they can be interconnected, how we designed the 3D user interface, and finally which use cases become possible. Thus, the paper combines several perspectives from computer science, social sciences, and media design in an interdisciplinary approach.


## 2. Related Work

A widely accepted model for interconnecting different teaching/learning settings is blended learning [14], where several phases of face-to-face and online learning are arranged in a predefined order. However, teachers and students are bound to certain phases and platforms/tools at every moment of a course.We believe that the direct interconnection of different educational settings allows for a seamless combination of synchronous scenarios during the lecture (with interaction between teachers and learners) and asynchronous scenarios before and after (individual or collaborative preparation and wrap-up)—regardless of the used platform. This enhances the learning comfort, increases the scope and quality of a lecture, and advances mobility and equality of opportunities for learners and lecturers. Current developments in this area can be divided into two groups: point-to-point connections and systematic redesigns.There exist a number ofdedicated point-to-point connections between different platforms and tools.(i)
Based on the practical need to simplify and accelerate the processes to deploy teaching and learning material, there have been some developments to automatically integrate lecture recordings into learning management systems [15, 16]. These solutions strongly rely on the used tools and techniques (i.e., recording software and learning platform).(ii)
To provide another example, there are some mash-ups between virtual worlds and other platforms, like for 3D visualization of large data sets [17] or for establishing links from the 3D world to traditional content on a learning platform [18]. Also, there is no general approach behind these point-to-point connections.All these solutions suffer from limited extensibility and complex maintenance due to their dependence from tools and technologies.Systematic integrations of different platforms are rare, no matter if designed for education or for other purposes.(i)
An abstract specification of educational presentation systems helps to identify related components and methods, for example, using an object-oriented model [19].(ii)
Basic features of a learning platform can be identified and invoked from various external sources [20] instead of implementing isolated solutions in every learning platform or web portal, again.(iii)
A generic middleware [21] can help to interconnect online games that are based on multiple platforms.Usually, these solutions follow an approach of fundamental platform decomposition for a later flexible recombination of modules. Here, object-, service-, or peer-to-peer-based architectures come into play. However, this is hard to realize with existing tools and infrastructures.Considering state-of-the-art design principles and sustainability of developments, a systematic integration is desirable. Services have proven to be a valid mechanism for enhancement of existing platforms [22], if applied in a coarse-grained manner.


## 3. Bidirectional Interconnection of Face-to-Face and Virtual Learning

### 3.1. A Service-Oriented Architecture for Cross-Platform Media Distribution

Distributed application scenarios consist of a high number of tools, platforms, and infrastructures. Especially, network-based environments are characterized by a high degree of heterogeneity and dynamics, which requires a systematic approach for conception and implementation of a well-suited architectural model. Otherwise, performance, scalability, and long-term sustainability cannot be guaranteed. Recent developments often show an unstructured aggregation of dedicated point-to-point connection between specific systems—though the theory of distributed systems offers a pool of general models for different requirements and conditions of the application scenario [23]. To mention just a few popular architectures: client/server, publisher/subscriber, peer-to-peer, or broker architecture. The pros and cons of these models affect issues like required knowledge on communication partners, existence of bottle necks, or completeness and timeliness of responses.Selection of an appropriate model comes along with a systematic analysis of the application scenario, usually with the help of formal models for actors, use cases, components, and processes. Depending on the nature of the application, this can be achieved, for instance, with graphical modelling techniques [24] as well as using algebraic structures [25]. Besides computer scientists, this involves several representatives of the application scenario in system development, and thus reduces the risk of technically driven aberrations. Especially, the broker model—as the concept behind the service oriented architecture (SOA)—requires substantial process modelling by domain experts, complementary change management from a social and organizational point of view, and a so-called SOA governance by the upper management level [26].We chose the broker model for systematic interconnection of different interaction spaces mainly because of its high degree of heterogeneity, agility, scalability, and transparency. Therewith, we dynamically redirect interactions between different locations (represented as media and control services) without predefined knowledge on any site. Figure1 shows this scenario using the example of telelecturing. Communication between involved parties basically takes place in three steps. (1) A provider registers its (lecturing) services along with their characteristic at the broker. (2) A consumer (prospective participant) asks the broker for a desired service (lecture). The broker returns information on location, syntax, and terms of use of the currently best-suited service. (3) The consumer connects to the provider and invokes its service.Figure 1
A Service-Oriented Architecture transparently interconnects different service providers and consumers (here: for telelecturing) without requiring the clients to have specific knowledge on given infrastructure and protocols.This architecture unifies the interaction between different types of providers, consumers, and services, as all steps described above are based on abstract service descriptions and platform-independent communication. Consumer and provider are just required to implement a minimal service interface. Thus, existing infrastructures and tools do not need to be redesigned.In an educational application scenario, there are some additional points that allow or even require the use of SOA. First of all, there are a number of established network addresses that are known to all clients and servers (like learning platforms); they can be used as brokers. Moreover, the content of a lecture typically is not security relevant, which simplifies the implementation and practical use of a prototype. (Nevertheless, a cross-institutional scenario requires basic services for authentication and accounting.) Another aspect is the large number of potential users and services that demands a scalability and agility impossibly provided by conventional models (like point-to-point connection of clients and servers). Finally, a strong requirement in educational scenarios is the acceptance and effective learning outcome by the users, not only those with a less technical background, which requires an intuitive and satisfying client-side interface. Here, the bow is drawn back to the desired tangibility and somatic, sensual perception of rich interaction spheres, which we tried to transfer from physical to virtual environments.Beyond simple brokerage, our service-based middleware offers some unique features which we would like to point out here.(i)
Different network technologies are bridged transparently [27] (like Ethernet, WiFi, Bluetooth, and ZigBee), since there are many specific devices and networks in an educational context that have to be involved.(ii)
Moreover,interoperability on service level is realized by an abstract layer for unifying service publication, brokerage, and use [27]. This was necessary because of the increasing number of service technologies.(iii)
Finally,context awareness both on network and on service layer is integrated [13] in order to adapt the system behaviour to the current situation of the users, their environment, and the system itself.That is why we called this central instance not only a broker, but a University Service Bus—indicating that there are complex functions performed by the middleware on behalf of the other system components (and finally of the users themselves).


### 3.2. Interactions in Virtual Worlds

From the users’ point of view, there are three levels to deal with a digital system [28], which can be also applied to a virtual 3D world.(i)
They may staypassive without influencing their environment. This includes to move around (e.g., walk, drive, or fly), to examine persons or objects (like exhibits in a museum), or to consume presented content (e.g., textual or audiovisual elements).(ii)
They mayactively shape their environment according to their own visions. Typical possibilities are presentation of any content (like presenting a poster or giving a talk) and creation or modification of certain elements (e.g., 3D models or simulations).(iii)
They mayinteractively communicate with other users (e.g., by text or voice chat) and objects (not necessarily just in the virtual world itself but maybe also in external environments).In general, the degree of intensity is rising from level to level. As a specialty of virtual 3D worlds, navigation, and interaction are similar to our actions in reality and thus are perceived to be more simple, natural, and intensive [5] than traditional computer-based interaction patterns. As a consequence, a particular suitability of virtual 3D worlds for teaching and learning can be seen. Typical application scenarios from an educational point of view are the following:(i)
distribution of previous lectures (in terms of a slide show or video) for passive, asynchronous reception, similar to a PodCast [29],(ii)
synchronous transmission of video data and moves of an avatar, similar to a virtual video conference [30],(iii)
additional discussions and/or reflections among students or with the tutor [27].As far as we know, existing scenarios are restricted to a single virtual world up to now. In principle, a connection to other environments (virtual or physical) is possible, too. We built an infrastructure that fulfills all of the above-mentioned tasks. Primarily, ongoing face-to-face lectures are provided as a service and can thus be invoked by any platform in real-time. For Second Life, this is realized by presenting the slides and annotations of the lecturer on the virtual canvas, and by mapping the lecturer’s voice to the avatar in the virtual world. Also, additional sounds can be mapped to the virtual lecture hall. Secondarily, all lectures are recorded and stored in an archive. In case that there is no ongoing live event, these recordings can be accessed from Second Life, too. For the users, there is no difference to be seen between this asynchronous playback and a live transmission except missing features for interaction. These interactions are the third field of our developments. We support a transparent, personal communication between users in the virtual world and onsite. Again, all these features are not limited to a special lab on the campus or to Second Life as virtual counterpart. The broker architecture allows for a highly flexible and dynamic deployment/invocation of services no matter from their origin and the targeted platform. The only prerequisition is a registration of the event at the broker, carried out by the lecturer prior to the presentation.The added value a virtual world provides in contrast to conventional face-to-face teaching is not only to copy a classroom or lab setting and to broadcast a lecture in the Web. Of course, this scenario is important especially for inexperienced users—teachers as well as students—in order to orientate themselves. The advantage is the almost unlimitedchangeability of the environment. From an educational point of view, this prevents the teacher from circumstantially explaining unknown situations (e.g., for exploring foreign cultures) or tedious theory (e.g., traffic rules for a driving license)—an appropriate scenario can be created for the students. From a psychological point of view, this requires the users to deal with a potentially instable environment. Moreover, the efforts for the teacher shall not be underestimated. Providing well-known, unchangeable conditions and locations (like a virtual lecture hall with a connection to real-life settings) as a framework to embed and to experience alternating scenarios may help them to cope with this demand.Another important point iscommunication. The platform offers possibilities to get in contact with a huge number of individuals in an informal way. In contrast to traditional learning platforms (with email, forums, chats, whiteboards, and so on), the 3D modelling of objects, persons, and their behaviour creates a kind of social presence. The apparent visibility and tangibility of objects and persons makes it easier for users to (inter)act in an unconstrained and natural way. Nevertheless, interaction always requires a counterpart, which in turn requires a given organizational structure (like lecture or consultation hours) to tackle the problem of lonely avatars in an almost unlimited space. We focus the combination of presentation and communication processes known from a classroom or lab with those in virtual worlds (here, Second Life). This implies a close correlation between appearance and behaviour of persons as well as objects in both environments. Tutor and students can individually choose to join the combined scenario either from the face-to-face or the virtual setting. Moreover, mutual control on the ongoing lecture and the media equipment, respectively, is possible from both environments.


### 3.3. System Interfaces

We had to consider the interfaces of existing systems in order to develop our framework, namely, the media equipment and control centre in our lab and the virtual world. The challenge was to interconnect these worlds using the SOA without touching regular operation of these systems.The media control centreonsite acts as a provider of the media services. It contains an audio and video crossbar switch, which assigns all available inputs/outputs. This is controlled using a wireless panel via the internet control system protocol (ICSP). Instead of this panel, our service wrapper sends such ICSP commands to the switch, and thus is able to access and control the media streams even from remote. Other services have been developed for audio/video streaming [13]. This includes live streams of ongoing lectures as well as ondemand streams of archived e-lectures from a central media server. Though the internal realization is completely different, the use is fully transparent for the consumer. The user may only recognize additional interaction possibilities for live streams. We use a Darwin streaming server to broadcast streams, since it offers various data formats and simultaneous client connections, in contrast to other solutions.Second Life is a complexvirtual world model that is completely hosted, simulated, and rendered on servers of the vendor Linden Lab. Though there are some possibilities for own extensions in the dedicated Second Life client software, this does not apply to server-side communication mechanisms that have to be facilitated for realization of an SOA. That is why we introduced a so-called surrogate as a transmitter between available services and Web Service protocols on the one hand and SL’s native communication mechanisms on the other hand. With the help of the Linden scripting language (LSL) 3D objects can been extended by interactive functions [31] for HTTP connection to the surrogate and thus to the SOA. Events can be triggered by mouse clicks, timers, or messages on the SL-internal communication channels.Moreover, we intended to integrate mobile devices of onsite students (cell phones) in order to enable personal communication with virtual participants. Here, we had to bridge heterogeneity on network and service level (Bluetooth and its services on the phones, and IP and Web Services in the Internet). We developed a so called general purpose access Point (GPAP) to tackle this problem; this core element of our infrastructure lies outside the focus of this paper.


### 3.4. Realization of the 3D User Interface

The design of a Second Life interface to an SOA covers three major aspects. First of all, the overall appearance must be appealing and—for better acceptance—without an explicit reference to E-learning. Associations with an object or region from real-life create curiosity and may lead to a better identification. They are feasible to express local affiliations in virtual worlds that are assembled from a diversity of different countries and regions. For this reason, we decided to model two famous landmarks at the beach of Rostock Warnemünde: the historic lighthouse and the so called “Teepott” with its remarkable roof (Figure2).The lighthouse and “Teepott” have been transferred from real-life (as a nautical and touristical landmark) to our virtual site in Second Life (as a learning and communication space).
(a)(b)The second aspect concerns the interior of the buildings. It should reflect the purpose of buildings and elements and should encourage an (inter)active participation. We modelled a virtual media lab with table, several chairs and a canvas to build an open learning and communication environment (Figure3).The main equipment of the media lab as well as additional elements for room decoration and interaction is modelled in Second Life, after all providing a much more attractive atmosphere than the original lab.
(a)(b)Additionally, all internal functions must be distinguishable and intuitionally controllable. The service-based communication with other environments should occur seamlessly. We use an access model that is based on the Second Life group model: Registered members of our group are considered as trustable avatars and are allowed to control the virtual and real equipment. They can also authorize other avatars for specific events or lectures. Guest avatars can only consume content and communicate with other students or the lecturer by text and voice chat.


### 3.5. Exemplary Use Cases

From the many functions the virtual environment can fulfill we would like to explain four in more detail:(i)
controlling media equipment in the onsite lab from the virtual world,(ii)
playing archived lectures in the virtual world,(iii)
participating in ongoing lectures from the virtual world,(iv)
communicating with other students across both worlds.Themedia equipment in the onsite lab is controlled by a so called head-up display (HUD) which appears when an avatar touches a dedicated 3D object. We chose a keyboard as 3D object to combine familiar functionality with the aesthetics and beauty of personal media [32]. It acts as an interface to the media control service provided by the onsite lab. The HUD dialog shows the different sources and drains for multimedia signals (audio and video), and the user can assign signal routings. This is depicted in Figure 4. Technically, this is realized by invoking the media control service which transmits the commands to the lab.As an additional 3D element that attracts the user’s attention a keyboard encapsulates interaction with invisible objects and non-3D, dialogue-based interaction (e.g., for controlling the technical equipment of the real-life media-lab).
(a)(b)The main use case the system was designed for is streaming of lecture recordings, which can take place live and ondemand.Lecture streaming from archive is a very simple scenario regarding both implementation of the system and interaction in the virtual world. An intuitive touch on the canvas of the virtual world [33] lets an avatar open a dedicated HUD for directly selecting from available video sources to be displayed on the canvas. The broker provides a list of currently available streams in the system, and the user selects from this list. Afterwards, the URL of this streaming service is mapped to the 3D model of the canvas, and playback begins. Depending on the media configuration, this contains only the slides (including digital annotations) or a combination with the lecturer’s video. Simultaneously, the voice of the lecturer is mapped to the virtual speakers. Here, the user stays passive throughout the lecture and is just consuming the multimedia content. However, the learning effect is the same as with traditional lecture recordings—but playful students may find it more attractive to follow them in a lab-like setting than in a classical web browser. However, the immanent benefits of the virtual world do not have an effect here.An enhanced use case islive streaming of an ongoing lecture. Typically, this is associated with the lecturer being present both in the classroom and in the virtual world (by means of an avatar). He starts recording and live streaming onsite, which is recognized by the broker. Thus, the stream is automatically available for playback on the virtual canvas. We experimented with different onsite configurations and considered a scenario with two screens (one for the slides of the lecture and one for depicting the virtual world—please refer to the photograph in Figure 3) to be the best solution. Now, all onsite and virtual participants can directly follow the lecture as if they were really together, which offers an added value compared to the more isolated ondemand settings. Social presence and intuitive interaction with the system create a comfortable atmosphere, which motivates the students to play a more active role and thus to intensify the process of knowledge acquisition.There are also some possibilities for interactivity, to a limited extent. First, the Second Life (text or audio) chat will directly reach onsite participants if the virtual world is projected onsite. Audio messages can be handled in the same way as requests to speak by the audience onsite. But, we experienced that text messages can produce a high cognitive load for lecturers who have to be aware that there may appear comments or questions behind their back on the screen. Second, interactions with the students onsite need to find a way to the virtual world. We experimented with two mechanisms: The camera and microphone can be switched from the lecturer to the audience, which results in some delay of the lecture, and it is better handled by an additional technician than by the lecturer. Alternatively, the lecturer may simply repeat any question of the audience in order to transmit it to the virtual world. This does not require much effort. Nevertheless, we found these interaction possibilities insufficient.That is why we additionally developed an innovativecross-technology communication concept [27]. We implemented a chat service running in the learning environment onsite. A messaging client on the users’ mobile phones enables them to find other users by service discovery. Second Life also provides a chat interface to its users. We developed a virtual cell phone as messaging object which is able to find other messaging objects. If an avatar is “wearing” this object, it connects to the messaging service in the lab onsite and requests for the list of available messaging partners. Furthermore, it registers itself as messenger. Both in Second Life and on the Bluetooth phones the group of available chat partners is displayed as list, and the users (or their avatars, resp.) are able to send messages to all chat partners no matter where they are (Figure 5). This is the basis for collaborative settings beyond traditional instruction. In the general pedagogical model as well as in our specific scenario, the teacher steps back acting rather like a trainer or moderator. Such arrangements have proven to be suited for in-depth learning experiences resulting in highly transferable and applicable knowledge.The cross-platform message exchange bridges between a mobile phone (Bluetooth) and the Second Life chat functions (Ethernet and Web Services). Students make use of their personal phone or corresponding virtual devices.
(a)(b)


## 3.1. A Service-Oriented Architecture for Cross-Platform Media Distribution

Distributed application scenarios consist of a high number of tools, platforms, and infrastructures. Especially, network-based environments are characterized by a high degree of heterogeneity and dynamics, which requires a systematic approach for conception and implementation of a well-suited architectural model. Otherwise, performance, scalability, and long-term sustainability cannot be guaranteed. Recent developments often show an unstructured aggregation of dedicated point-to-point connection between specific systems—though the theory of distributed systems offers a pool of general models for different requirements and conditions of the application scenario [23]. To mention just a few popular architectures: client/server, publisher/subscriber, peer-to-peer, or broker architecture. The pros and cons of these models affect issues like required knowledge on communication partners, existence of bottle necks, or completeness and timeliness of responses.Selection of an appropriate model comes along with a systematic analysis of the application scenario, usually with the help of formal models for actors, use cases, components, and processes. Depending on the nature of the application, this can be achieved, for instance, with graphical modelling techniques [24] as well as using algebraic structures [25]. Besides computer scientists, this involves several representatives of the application scenario in system development, and thus reduces the risk of technically driven aberrations. Especially, the broker model—as the concept behind the service oriented architecture (SOA)—requires substantial process modelling by domain experts, complementary change management from a social and organizational point of view, and a so-called SOA governance by the upper management level [26].We chose the broker model for systematic interconnection of different interaction spaces mainly because of its high degree of heterogeneity, agility, scalability, and transparency. Therewith, we dynamically redirect interactions between different locations (represented as media and control services) without predefined knowledge on any site. Figure1 shows this scenario using the example of telelecturing. Communication between involved parties basically takes place in three steps. (1) A provider registers its (lecturing) services along with their characteristic at the broker. (2) A consumer (prospective participant) asks the broker for a desired service (lecture). The broker returns information on location, syntax, and terms of use of the currently best-suited service. (3) The consumer connects to the provider and invokes its service.Figure 1
A Service-Oriented Architecture transparently interconnects different service providers and consumers (here: for telelecturing) without requiring the clients to have specific knowledge on given infrastructure and protocols.This architecture unifies the interaction between different types of providers, consumers, and services, as all steps described above are based on abstract service descriptions and platform-independent communication. Consumer and provider are just required to implement a minimal service interface. Thus, existing infrastructures and tools do not need to be redesigned.In an educational application scenario, there are some additional points that allow or even require the use of SOA. First of all, there are a number of established network addresses that are known to all clients and servers (like learning platforms); they can be used as brokers. Moreover, the content of a lecture typically is not security relevant, which simplifies the implementation and practical use of a prototype. (Nevertheless, a cross-institutional scenario requires basic services for authentication and accounting.) Another aspect is the large number of potential users and services that demands a scalability and agility impossibly provided by conventional models (like point-to-point connection of clients and servers). Finally, a strong requirement in educational scenarios is the acceptance and effective learning outcome by the users, not only those with a less technical background, which requires an intuitive and satisfying client-side interface. Here, the bow is drawn back to the desired tangibility and somatic, sensual perception of rich interaction spheres, which we tried to transfer from physical to virtual environments.Beyond simple brokerage, our service-based middleware offers some unique features which we would like to point out here.(i)
Different network technologies are bridged transparently [27] (like Ethernet, WiFi, Bluetooth, and ZigBee), since there are many specific devices and networks in an educational context that have to be involved.(ii)
Moreover,interoperability on service level is realized by an abstract layer for unifying service publication, brokerage, and use [27]. This was necessary because of the increasing number of service technologies.(iii)
Finally,context awareness both on network and on service layer is integrated [13] in order to adapt the system behaviour to the current situation of the users, their environment, and the system itself.That is why we called this central instance not only a broker, but a University Service Bus—indicating that there are complex functions performed by the middleware on behalf of the other system components (and finally of the users themselves).


## 3.2. Interactions in Virtual Worlds

From the users’ point of view, there are three levels to deal with a digital system [28], which can be also applied to a virtual 3D world.(i)
They may staypassive without influencing their environment. This includes to move around (e.g., walk, drive, or fly), to examine persons or objects (like exhibits in a museum), or to consume presented content (e.g., textual or audiovisual elements).(ii)
They mayactively shape their environment according to their own visions. Typical possibilities are presentation of any content (like presenting a poster or giving a talk) and creation or modification of certain elements (e.g., 3D models or simulations).(iii)
They mayinteractively communicate with other users (e.g., by text or voice chat) and objects (not necessarily just in the virtual world itself but maybe also in external environments).In general, the degree of intensity is rising from level to level. As a specialty of virtual 3D worlds, navigation, and interaction are similar to our actions in reality and thus are perceived to be more simple, natural, and intensive [5] than traditional computer-based interaction patterns. As a consequence, a particular suitability of virtual 3D worlds for teaching and learning can be seen. Typical application scenarios from an educational point of view are the following:(i)
distribution of previous lectures (in terms of a slide show or video) for passive, asynchronous reception, similar to a PodCast [29],(ii)
synchronous transmission of video data and moves of an avatar, similar to a virtual video conference [30],(iii)
additional discussions and/or reflections among students or with the tutor [27].As far as we know, existing scenarios are restricted to a single virtual world up to now. In principle, a connection to other environments (virtual or physical) is possible, too. We built an infrastructure that fulfills all of the above-mentioned tasks. Primarily, ongoing face-to-face lectures are provided as a service and can thus be invoked by any platform in real-time. For Second Life, this is realized by presenting the slides and annotations of the lecturer on the virtual canvas, and by mapping the lecturer’s voice to the avatar in the virtual world. Also, additional sounds can be mapped to the virtual lecture hall. Secondarily, all lectures are recorded and stored in an archive. In case that there is no ongoing live event, these recordings can be accessed from Second Life, too. For the users, there is no difference to be seen between this asynchronous playback and a live transmission except missing features for interaction. These interactions are the third field of our developments. We support a transparent, personal communication between users in the virtual world and onsite. Again, all these features are not limited to a special lab on the campus or to Second Life as virtual counterpart. The broker architecture allows for a highly flexible and dynamic deployment/invocation of services no matter from their origin and the targeted platform. The only prerequisition is a registration of the event at the broker, carried out by the lecturer prior to the presentation.The added value a virtual world provides in contrast to conventional face-to-face teaching is not only to copy a classroom or lab setting and to broadcast a lecture in the Web. Of course, this scenario is important especially for inexperienced users—teachers as well as students—in order to orientate themselves. The advantage is the almost unlimitedchangeability of the environment. From an educational point of view, this prevents the teacher from circumstantially explaining unknown situations (e.g., for exploring foreign cultures) or tedious theory (e.g., traffic rules for a driving license)—an appropriate scenario can be created for the students. From a psychological point of view, this requires the users to deal with a potentially instable environment. Moreover, the efforts for the teacher shall not be underestimated. Providing well-known, unchangeable conditions and locations (like a virtual lecture hall with a connection to real-life settings) as a framework to embed and to experience alternating scenarios may help them to cope with this demand.Another important point iscommunication. The platform offers possibilities to get in contact with a huge number of individuals in an informal way. In contrast to traditional learning platforms (with email, forums, chats, whiteboards, and so on), the 3D modelling of objects, persons, and their behaviour creates a kind of social presence. The apparent visibility and tangibility of objects and persons makes it easier for users to (inter)act in an unconstrained and natural way. Nevertheless, interaction always requires a counterpart, which in turn requires a given organizational structure (like lecture or consultation hours) to tackle the problem of lonely avatars in an almost unlimited space. We focus the combination of presentation and communication processes known from a classroom or lab with those in virtual worlds (here, Second Life). This implies a close correlation between appearance and behaviour of persons as well as objects in both environments. Tutor and students can individually choose to join the combined scenario either from the face-to-face or the virtual setting. Moreover, mutual control on the ongoing lecture and the media equipment, respectively, is possible from both environments.


## 3.3. System Interfaces

We had to consider the interfaces of existing systems in order to develop our framework, namely, the media equipment and control centre in our lab and the virtual world. The challenge was to interconnect these worlds using the SOA without touching regular operation of these systems.The media control centreonsite acts as a provider of the media services. It contains an audio and video crossbar switch, which assigns all available inputs/outputs. This is controlled using a wireless panel via the internet control system protocol (ICSP). Instead of this panel, our service wrapper sends such ICSP commands to the switch, and thus is able to access and control the media streams even from remote. Other services have been developed for audio/video streaming [13]. This includes live streams of ongoing lectures as well as ondemand streams of archived e-lectures from a central media server. Though the internal realization is completely different, the use is fully transparent for the consumer. The user may only recognize additional interaction possibilities for live streams. We use a Darwin streaming server to broadcast streams, since it offers various data formats and simultaneous client connections, in contrast to other solutions.Second Life is a complexvirtual world model that is completely hosted, simulated, and rendered on servers of the vendor Linden Lab. Though there are some possibilities for own extensions in the dedicated Second Life client software, this does not apply to server-side communication mechanisms that have to be facilitated for realization of an SOA. That is why we introduced a so-called surrogate as a transmitter between available services and Web Service protocols on the one hand and SL’s native communication mechanisms on the other hand. With the help of the Linden scripting language (LSL) 3D objects can been extended by interactive functions [31] for HTTP connection to the surrogate and thus to the SOA. Events can be triggered by mouse clicks, timers, or messages on the SL-internal communication channels.Moreover, we intended to integrate mobile devices of onsite students (cell phones) in order to enable personal communication with virtual participants. Here, we had to bridge heterogeneity on network and service level (Bluetooth and its services on the phones, and IP and Web Services in the Internet). We developed a so called general purpose access Point (GPAP) to tackle this problem; this core element of our infrastructure lies outside the focus of this paper.


## 3.4. Realization of the 3D User Interface

The design of a Second Life interface to an SOA covers three major aspects. First of all, the overall appearance must be appealing and—for better acceptance—without an explicit reference to E-learning. Associations with an object or region from real-life create curiosity and may lead to a better identification. They are feasible to express local affiliations in virtual worlds that are assembled from a diversity of different countries and regions. For this reason, we decided to model two famous landmarks at the beach of Rostock Warnemünde: the historic lighthouse and the so called “Teepott” with its remarkable roof (Figure2).The lighthouse and “Teepott” have been transferred from real-life (as a nautical and touristical landmark) to our virtual site in Second Life (as a learning and communication space).
(a)(b)The second aspect concerns the interior of the buildings. It should reflect the purpose of buildings and elements and should encourage an (inter)active participation. We modelled a virtual media lab with table, several chairs and a canvas to build an open learning and communication environment (Figure3).The main equipment of the media lab as well as additional elements for room decoration and interaction is modelled in Second Life, after all providing a much more attractive atmosphere than the original lab.
(a)(b)Additionally, all internal functions must be distinguishable and intuitionally controllable. The service-based communication with other environments should occur seamlessly. We use an access model that is based on the Second Life group model: Registered members of our group are considered as trustable avatars and are allowed to control the virtual and real equipment. They can also authorize other avatars for specific events or lectures. Guest avatars can only consume content and communicate with other students or the lecturer by text and voice chat.


## 3.5. Exemplary Use Cases

From the many functions the virtual environment can fulfill we would like to explain four in more detail:(i)
controlling media equipment in the onsite lab from the virtual world,(ii)
playing archived lectures in the virtual world,(iii)
participating in ongoing lectures from the virtual world,(iv)
communicating with other students across both worlds.Themedia equipment in the onsite lab is controlled by a so called head-up display (HUD) which appears when an avatar touches a dedicated 3D object. We chose a keyboard as 3D object to combine familiar functionality with the aesthetics and beauty of personal media [32]. It acts as an interface to the media control service provided by the onsite lab. The HUD dialog shows the different sources and drains for multimedia signals (audio and video), and the user can assign signal routings. This is depicted in Figure 4. Technically, this is realized by invoking the media control service which transmits the commands to the lab.As an additional 3D element that attracts the user’s attention a keyboard encapsulates interaction with invisible objects and non-3D, dialogue-based interaction (e.g., for controlling the technical equipment of the real-life media-lab).
(a)(b)The main use case the system was designed for is streaming of lecture recordings, which can take place live and ondemand.Lecture streaming from archive is a very simple scenario regarding both implementation of the system and interaction in the virtual world. An intuitive touch on the canvas of the virtual world [33] lets an avatar open a dedicated HUD for directly selecting from available video sources to be displayed on the canvas. The broker provides a list of currently available streams in the system, and the user selects from this list. Afterwards, the URL of this streaming service is mapped to the 3D model of the canvas, and playback begins. Depending on the media configuration, this contains only the slides (including digital annotations) or a combination with the lecturer’s video. Simultaneously, the voice of the lecturer is mapped to the virtual speakers. Here, the user stays passive throughout the lecture and is just consuming the multimedia content. However, the learning effect is the same as with traditional lecture recordings—but playful students may find it more attractive to follow them in a lab-like setting than in a classical web browser. However, the immanent benefits of the virtual world do not have an effect here.An enhanced use case islive streaming of an ongoing lecture. Typically, this is associated with the lecturer being present both in the classroom and in the virtual world (by means of an avatar). He starts recording and live streaming onsite, which is recognized by the broker. Thus, the stream is automatically available for playback on the virtual canvas. We experimented with different onsite configurations and considered a scenario with two screens (one for the slides of the lecture and one for depicting the virtual world—please refer to the photograph in Figure 3) to be the best solution. Now, all onsite and virtual participants can directly follow the lecture as if they were really together, which offers an added value compared to the more isolated ondemand settings. Social presence and intuitive interaction with the system create a comfortable atmosphere, which motivates the students to play a more active role and thus to intensify the process of knowledge acquisition.There are also some possibilities for interactivity, to a limited extent. First, the Second Life (text or audio) chat will directly reach onsite participants if the virtual world is projected onsite. Audio messages can be handled in the same way as requests to speak by the audience onsite. But, we experienced that text messages can produce a high cognitive load for lecturers who have to be aware that there may appear comments or questions behind their back on the screen. Second, interactions with the students onsite need to find a way to the virtual world. We experimented with two mechanisms: The camera and microphone can be switched from the lecturer to the audience, which results in some delay of the lecture, and it is better handled by an additional technician than by the lecturer. Alternatively, the lecturer may simply repeat any question of the audience in order to transmit it to the virtual world. This does not require much effort. Nevertheless, we found these interaction possibilities insufficient.That is why we additionally developed an innovativecross-technology communication concept [27]. We implemented a chat service running in the learning environment onsite. A messaging client on the users’ mobile phones enables them to find other users by service discovery. Second Life also provides a chat interface to its users. We developed a virtual cell phone as messaging object which is able to find other messaging objects. If an avatar is “wearing” this object, it connects to the messaging service in the lab onsite and requests for the list of available messaging partners. Furthermore, it registers itself as messenger. Both in Second Life and on the Bluetooth phones the group of available chat partners is displayed as list, and the users (or their avatars, resp.) are able to send messages to all chat partners no matter where they are (Figure 5). This is the basis for collaborative settings beyond traditional instruction. In the general pedagogical model as well as in our specific scenario, the teacher steps back acting rather like a trainer or moderator. Such arrangements have proven to be suited for in-depth learning experiences resulting in highly transferable and applicable knowledge.The cross-platform message exchange bridges between a mobile phone (Bluetooth) and the Second Life chat functions (Ethernet and Web Services). Students make use of their personal phone or corresponding virtual devices.
(a)(b)


## 4. Evaluation of the System

The prototypical solution of the presented system was developed in a course on Web 2.0 and Second Life. For the students, this was associated with a wide range of actions that took place in the physical as well as the virtual environment: listening to introductory talks, preparing and giving their own talks on advanced topics, discussing these topics, extending the conception, implementing the virtual environment itself, testing and presenting the results of their work. We evaluated the infrastructure in three subsequent master courses with all together 25 students and different lecturers; two with traditional instruction and one with a game-based setting [34]. All talks have been recorded and deployed in Second Life as well as in our local learning platform, both live and ondemand. We are now going to describe our experiences from these courses with a focus on the virtual domain.


### 4.1. Technical Aspects

The frequency of using the service-based dissemination of lectures did continuously rise during the tests (compared to traditional linking of material in an E-learning platform) though students initially signalled no willingness to make use of these mechanisms. Comparing different transmission techniques, streaming was recognized as appropriate especially for participating and frequent reworking of a lecture, while download was considered as helpful mostly for targeted revision of lectures prior to exams.As Figure6 shows, the technical solution was rated mostly average to good. In general, the lecture integration into Second Life (right) polls worse than the compared version for the learning platform (left). As weak points, mainly the image and sound quality as well as the interaction facilities were mentioned. This helped us to fine-tune the resolution and sampling rate in the streaming server and to define some guidelines for a lecturer how to deal with annotation features of the recording software and with interactive elements in the pedagogical setting. Moreover, the students provided a number of suggestions for improvement especially of the Second Life solution and the overall organization of the lecture.Figure 6
Student ratings for the quality of integrating onsite lectures into a learning platform (left) and a virtual world (right) showed relevant points for further improvement of our systems.After all, we asked the students if they would make use of such offers during their studies, again. Regarding lecture streaming in the traditional learning platform 75% said yes and 25% perhaps. Regarding lecture streaming in Second Life, 50% said yes, 25% said perhaps, and 25% said no. Our conclusion is that using the virtual world rather makes sense in highly interactive settings like project-based or game-based learning.


### 4.2. Social and Pedagogical Aspects

In an initial inquiry, we asked the students for theirpersonal  attitudes towards innovative E-learning materials and settings. The answers ranged from open minded (15%), to deliberate (35%), to sceptical (50%)—in contrast to conventional slides and scripts which were generally rated as important. The final attitude after finishing our tests changed, where lecture recordings gained the same relevance as slides and scripts (almost 100%), and also live transmissions and virtual lectures were rated as mostly important (75%). This demonstrates that the results of our evaluation were not biased by a general affinity of the students towards E-learning mechanisms or material.Our subjective observations from asocial point of view can be divided into three main phases of the course:(i)
In the first phase lasting a few weeks, the natural instinct of participants to play around in the virtual world dominated their behaviour. To a certain extent this also affected the instructors. For instance, many students experimented with the optical appearance of their avatar, widely exceeding limits given in real-life by biology, culture, or personal concerns. Some even acted in a more aggressive way, for example, tried to change, remove, or destroy virtual objects. This was encouraged by the prototypical nature of the technical realization as well as their feeling to be unobserved resulting from a lack of organizational structure in the new environment compared to traditional classroom or lab settings.(ii)
In a second phase that lasted till the end of the course, we saw a significantly increased productivity of the students. They started to build up a team rather than a number of individuals, surely as a result of successful interaction and cooperation. Their identification with the project (visible, for instance, in dedicated logo shirts they designed and exchanged) seemed to be much higher in the virtual than in the physical environment, which we explain with the given potential to personally act out. Here, the innovative interaction sphere is a major benefit to exploit the students’ individuality and creativity.(iii)
Finally, even a third phase taking place after finishing the course was identified. Students continued to work and play with the system even beyond their official schedule. The team kept meeting physically as well as virtually, further refining the system, and there existed a strong interest to continue work in consecutive courses or projects which we are familiar only from other “tangible” projects, for example, mobile robots or field trips.From our subjective perspective, the averagelearning outcome (in terms of given marks in the exams as well as in terms of obtained personal competencies of the students) was higher than in previous courses on similar topics. Unfortunately, because of the small number of participants in our master courses, we could not divide the students in test group and control group in order to quantify an increased learning outcome. However, some qualitative statements from students provided after the courses may validate our observations. In general, students reported that the interconnection of physical and virtual settings was “a valuable add-on to traditional lectures”, and that “theory was more aligned to practice”. One student involved in the game-based project confirmed: “Virtual projects in Second Life are a good idea and very exciting. It’s annoying to spend more time with peculiarities of the SL engine than with the course topic itself. However, this is exactly what we as computer scientists have to cope with, so it’s interesting to try that.” Another one commented: “It was amazing to actually experience the results of my programming in a virtual environment. This was definitely worth the efforts.” These observations are consistent with other experiments and studies in this field [35].Our conclusion is that tangibility—no matter if physical or virtual—helps to fosteridentification with the project and intrinsic motivation of the students. This results in a better performance in single tasks as well as in increased self-confidence. This is in turn beneficial for the learning outcome, which is also compliant with other studies [36, 37]. In addition, well-known spatial as well as temporal conditions facilitate orientation and satisfaction of users in the virtual world.We feel that these findings from our experiments as well as from educational research in general confirm the efficacy of our system. However, the contribution of our work is rather technical, a systematical approach to interconnect not only different tools, but also different teaching/learning settings which were isolated before.


## 4.1. Technical Aspects

The frequency of using the service-based dissemination of lectures did continuously rise during the tests (compared to traditional linking of material in an E-learning platform) though students initially signalled no willingness to make use of these mechanisms. Comparing different transmission techniques, streaming was recognized as appropriate especially for participating and frequent reworking of a lecture, while download was considered as helpful mostly for targeted revision of lectures prior to exams.As Figure6 shows, the technical solution was rated mostly average to good. In general, the lecture integration into Second Life (right) polls worse than the compared version for the learning platform (left). As weak points, mainly the image and sound quality as well as the interaction facilities were mentioned. This helped us to fine-tune the resolution and sampling rate in the streaming server and to define some guidelines for a lecturer how to deal with annotation features of the recording software and with interactive elements in the pedagogical setting. Moreover, the students provided a number of suggestions for improvement especially of the Second Life solution and the overall organization of the lecture.Figure 6
Student ratings for the quality of integrating onsite lectures into a learning platform (left) and a virtual world (right) showed relevant points for further improvement of our systems.After all, we asked the students if they would make use of such offers during their studies, again. Regarding lecture streaming in the traditional learning platform 75% said yes and 25% perhaps. Regarding lecture streaming in Second Life, 50% said yes, 25% said perhaps, and 25% said no. Our conclusion is that using the virtual world rather makes sense in highly interactive settings like project-based or game-based learning.


## 4.2. Social and Pedagogical Aspects

In an initial inquiry, we asked the students for theirpersonal  attitudes towards innovative E-learning materials and settings. The answers ranged from open minded (15%), to deliberate (35%), to sceptical (50%)—in contrast to conventional slides and scripts which were generally rated as important. The final attitude after finishing our tests changed, where lecture recordings gained the same relevance as slides and scripts (almost 100%), and also live transmissions and virtual lectures were rated as mostly important (75%). This demonstrates that the results of our evaluation were not biased by a general affinity of the students towards E-learning mechanisms or material.Our subjective observations from asocial point of view can be divided into three main phases of the course:(i)
In the first phase lasting a few weeks, the natural instinct of participants to play around in the virtual world dominated their behaviour. To a certain extent this also affected the instructors. For instance, many students experimented with the optical appearance of their avatar, widely exceeding limits given in real-life by biology, culture, or personal concerns. Some even acted in a more aggressive way, for example, tried to change, remove, or destroy virtual objects. This was encouraged by the prototypical nature of the technical realization as well as their feeling to be unobserved resulting from a lack of organizational structure in the new environment compared to traditional classroom or lab settings.(ii)
In a second phase that lasted till the end of the course, we saw a significantly increased productivity of the students. They started to build up a team rather than a number of individuals, surely as a result of successful interaction and cooperation. Their identification with the project (visible, for instance, in dedicated logo shirts they designed and exchanged) seemed to be much higher in the virtual than in the physical environment, which we explain with the given potential to personally act out. Here, the innovative interaction sphere is a major benefit to exploit the students’ individuality and creativity.(iii)
Finally, even a third phase taking place after finishing the course was identified. Students continued to work and play with the system even beyond their official schedule. The team kept meeting physically as well as virtually, further refining the system, and there existed a strong interest to continue work in consecutive courses or projects which we are familiar only from other “tangible” projects, for example, mobile robots or field trips.From our subjective perspective, the averagelearning outcome (in terms of given marks in the exams as well as in terms of obtained personal competencies of the students) was higher than in previous courses on similar topics. Unfortunately, because of the small number of participants in our master courses, we could not divide the students in test group and control group in order to quantify an increased learning outcome. However, some qualitative statements from students provided after the courses may validate our observations. In general, students reported that the interconnection of physical and virtual settings was “a valuable add-on to traditional lectures”, and that “theory was more aligned to practice”. One student involved in the game-based project confirmed: “Virtual projects in Second Life are a good idea and very exciting. It’s annoying to spend more time with peculiarities of the SL engine than with the course topic itself. However, this is exactly what we as computer scientists have to cope with, so it’s interesting to try that.” Another one commented: “It was amazing to actually experience the results of my programming in a virtual environment. This was definitely worth the efforts.” These observations are consistent with other experiments and studies in this field [35].Our conclusion is that tangibility—no matter if physical or virtual—helps to fosteridentification with the project and intrinsic motivation of the students. This results in a better performance in single tasks as well as in increased self-confidence. This is in turn beneficial for the learning outcome, which is also compliant with other studies [36, 37]. In addition, well-known spatial as well as temporal conditions facilitate orientation and satisfaction of users in the virtual world.We feel that these findings from our experiments as well as from educational research in general confirm the efficacy of our system. However, the contribution of our work is rather technical, a systematical approach to interconnect not only different tools, but also different teaching/learning settings which were isolated before.


## 5. Summary and Further Work

The service-based linking of face-to-face learning scenarios and different virtual learning environments described in this paper goes far beyond previous approaches. Connections between Second Life and traditional E-learning platforms already exist, but they are restricted to cross references or a common database. There are no systematic approaches to combine synchronous and asynchronous learning processes of both paradigms.The presented system achieves a flexible and systematic coupling of platforms and tools of computer-aided teaching and learning in classrooms and virtual worlds. It consequently makes use of a service-oriented architecture (SOA). An intermediate service layer between the different environments contains all services that are provided by the specific platforms. Each environment is furthermore able to consume services available in this layer. For the first time, learner and lecturer can shape the specific learning and teaching processes in an ad hoc manner beyond predefined phases (Blended Learning) or environments (decoupled face-to-face and virtual learning processes). The individual arrangement can be modified during the lecture. In addition, the emerged independency allows a unification of synchronous and asynchronous learning and teaching scenarios. This can be easily realized across different educational providers. An interference of administrative areas of responsibility is not longer required thanks to the transparent encapsulation in an SOA.Besides several courses at the University of Rostock, we also used the developed system for events like virtual online conferences or for remote (i.e., distributed) defence of students’ theses, to name just two examples.Nevertheless, the prototype can be extended at several points. A service-based feedback channel from the virtual learning environment into the presence learning environment does not yet exist. Although currently not required (clients and browser satisfactorily perform this task) it is desirable with regards to higher flexibility. A direct integration of SOA mechanisms into virtual worlds like Second Life would also increase the flexibility of the developed system, just like an extension of virtual environments to further data formats (e.g., PDF or Flash).Furthermore, the interaction between teachers and learners can be designed more intuitionally. This is feasible by a fusion of the presented approach with principles and technologies of self organization and pervasive computing [38].Finally, an advanced state of the system will allow us to transfer the scenario from Master to Bachelor courses (in order to attract more students and thus to gain a broader statistical base). We are confident that the service-based interconnection of virtual 3D worlds and real-life locations is an excellent interaction concept for different E-learning communities.After all, the most obvious benefit of virtual worlds (to freely create and modify a 3D environment, for example, forsimulations and role playing games [34]) is yet rarely utilized for learning activities. Though technical preconditions are given, this will require additional efforts from social and educational sciences as well as from media theory and design.



---

*Source: 684202-2011-05-09.xml*