# Developing Mobile Remote Collaboration Systems for Industrial Use: Some Design Challenges

Leila Alem, Weidong Huang

► To cite this version:

Leila Alem, Weidong Huang. Developing Mobile Remote Collaboration Systems for Industrial Use: Some Design Challenges. 13th International Conference on Human-Computer Interaction (INTER-ACT), Sep 2011, Lisbon, Portugal. pp.442-445, 10.1007/978-3-642-23768-3_53. hal-01596991

HAL Id: hal-01596991

https://inria.hal.science/hal-01596991

Submitted on 28 Sep 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Developing Mobile Remote Collaboration Systems for
Industrial Use: Some Design Challenges

Leila Alem and Weidong Huang

CSIRO ICT Centre
PO Box 76
Epping NSW 1710 Australia
{Leila.Alem; Tony.Huang}@csiro.au

**Abstract.** Many real-world collaboration scenarios involve a helper remotely guiding a worker performing a task requiring the manipulation of physical objects-devices. Systems currently available for remote guiding have limitations for industrial use, particularly in terms of supporting the mobile aspect of work. To meet these needs and as part of our work for the mining industry, we have developed two systems. These systems use a wearable computer and a wearable display to support the mobile aspect of the remote collaboration. In this paper, we review existing work in remote guiding and state their limitations in relation to industrial needs. We then present two mobile remote collaboration systems that we have designed and developed to support the maintenance of mining equipments. Designing for real world use is challenging, systems need to be easy to use and be able to operate in various environmental conditions. Systems also need to support the mobility aspect of work and support different roles of the collaborators.

**Keywords:** Remote collaboration; mobile collaboration; remote gestures; augmented reality; usability

## 1 Introduction

There are a range of real world situations in which remote expert guidance is required for a local novice to complete physical tasks. In maintenance, for example, an expert may be guiding remotely a technician into repairing a piece of equipment. There is a growing need for systems or platforms that support collaboration between a worker and a helper, ensuring the effective delivery, just in time, of remote guidance.

Prior research on remote collaboration has demonstrated that providing access to a shared visual space and supporting the richness of hand gestures are critical to the success of remote guiding [2, 6, 7, 9]. However, existing systems either confine collaborators in fixed desktop settings (e.g., [2, 9]), or support only the pointing gesture (e.g., [8]). This limits their use in real world applications. For example, in the mining industry, when a sophisticated machine is not operational, onsite maintenance technicians require urgent input from a remote specialist/expert, as the time lost in the machine not being used translates into a loss in productivity. In addition, on-site technicians are highly mobile workers; in order to repair a machine, they are often
required to go to the location of the machine, move around the machine, fetch tools and inspect equipment. This in turn requires that the systems be wearable and wireless, and that the systems function in various working conditions.

To meet these mobility requirements, we developed two systems: HandsOnVideo for a fixed remote helper guiding a mobile on-site worker [1, 4] and HandsInAir for a mobile remote helper guiding a mobile on-site worker [3]. These two systems were developed as part of our Human System Integration project within the CSIRO's Minerals Down Under (MDU), a National Research Flagship Program.

## 2 HandsOnVideo

<figure><img src="image_2.png" /><figcaption>**Fig. 1** Left: helmet with camera and near-eye display. Right: system work flow.</figcaption></figure>

Workers in mine sites wear helmets for safety. HandsOnVideo uses the helmet as a support for mounting a camera and a near eye display (see the left image in Figure 1). This setup supports the mobility of the worker. It is also hands free, hence allowing the worker to use his/her hands for maintenance operations. The near eye display is a small device with two small screens at the top of the worker field of view. By looking up, the worker can see what is displayed on a virtual screen, that is, the video of the workspace overlaid with the helper gestures (pointing to objects and showing orientation). With this setup, the worker is also able to be aware of his/her workspace and the surrounding environment.

The helper station is a large touch-display table located in an office or a workshop. It consists of three components: 1) a shared visual space that displays the video of what the worker sees (captured by the camera on the helmet of the worker); 2) a panoramic view of the worker's workspace, which the helper can use to maintain situation awareness; and 3) four storage areas on both sides of the shared visual space, which allow the helper to store specific scenes of the workspace for future use.

The right image in Figure 1 illustrates how the system works. First, the video of the worker's camera is sent to the helper side and displayed on the shared visual space (arrow 1) of the touch table. The helper uses his/her hands over the shared visual space to show/explain how a procedure should take place. The helper hand gestures are captured by a camera mounted on top of the display (arrow 2). The captured hand gestures are then sent to the worker side and displayed on the near eye display (arrow 3).
## 3 HandsInAir

Expert helpers may be out of the office and on the move. HandsInAir was developed to explore the feasibility of supporting the mobility of the helper while using an interface/system similar to the one worn by the worker. The collaborators have the same hardware configuration as shown in the left image in Figure 1. This configuration not only allows the helper to move around, but also allows him/her to point to remote objects and use more complex hand gestures on a virtual display of the workspace, removing the need for a large physical display.

<figure><img src="image_3.png" /><figcaption>Fig. 2 Camera captures and hand gestures showed on the near eye displays.</figcaption></figure>

Figure 2 illustrates how the system works. Once a connection between the helper and the worker is established, the system initializes two video streams between the two sides. First the video of the worker camera is sent to the helper side and displayed on the near eye display. The helper sees the workspace of the worker and uses his/her hands to support instructions. The helper's hand gestures are captured by the helper camera and overlaid on the local videos of both the worker and helper (see Figure 2). Initial testing of HandsInAir suggests that it is easy for the users to point to remote objects using the virtual display, and that it is easy and natural to perform gestures on the virtual display in order to show, for example, how two pieces of Lego toys should be assembled in a specific way.

## 4 Designing for real world use: some issues

Designing for real world use is challenging, systems not only need to be easy to use, but also need to support the mobile and the flexible aspect of work.

Supporting mobility of the helper comes with a price. The helper unit in HandsOnVideo provides a panoramic view of the workspace of the worker; this allows the helper to point to the area of attention outside the field of view of the worker. However, this awareness support is no longer available in the current setting
of HandsInAir. Research is needed to explore how awareness of the worker's environment should be provided to the helper.

Supporting the flexibility of work means that the systems for real world use should be able to operate in various environmental conditions, specifically lighting conditions. Previous work has demonstrated the value of unmediated gestures. For example, the projection of hand gestures into the workspace may lead to a gain in task performance [5, 6]. The display of the workspace with an overlay of the hand gestures may lead to an increase in satisfaction with the collaborative effort [9]. We investigated the option of projecting the hands using Augmented Reality as well as the option of using a see through system. The dependence on the right lighting conditions for the projection option and the limitation of the field of view for the see through system resulted in disregarding these options.

Both HandsOnVideo and HandsInAir make use of a near eye display. The use of the near eye display allows these systems to operate in various lighting conditions. However, this comes at a price. In both systems, the hand gestures are displayed on a small virtual screen, in which the workspace plus the hands of both the worker and the helper are displayed. Our users playing the role of the worker reported that sometimes they had to remove their hands from the workspace in order to leave some space for the hands of the helper.

In summary, we have presented two systems that we have developed to support the mobility aspect of work. These two systems draw on the latest research in remote guiding systems. While they are designed to be suitable for industrial use, some research effort still remains for further improvements and refinements.

## References

1. Alem, L., Tecchia, F. and Huang, W. (2011) HandsOnVideo: Towards a Gesture based Mobile AR System for Remote Collaboration. In Alem, L. and Huang, W. (eds): Mobile Collaborative Augmented Reality: Recent Trends, 127-138.

2. Fussell, S. R., Setlock, L. D., Yang, J., Ou, J., Mauer, E. and Kramer, A. D. I. (2004) Gestures over video streams to support remote collaboration on physical tasks. In Human-Computer Interaction, 19, 273-309.

3. Huang, W. and Alem, L. (2011) HandsInAir: A Wearable System for Remote Collaboration. In APWCS11, 171-174.

4. Huang, W. and Alem, L. (2011) Supporting Hand Gestures in Mobile Remote Collaboration: A Usability Evaluation. In BCS HCI11.

5. Kirk, D. S., Rodden, T. and Stanton Fraser, D. (2007) Turn It This Way: Grounding Collaborative Action with Remote Gestures. In CHI07, 1039-1048.

6. Kirk, D. S. and Stanton Fraser, D. (2006) Comparing Remote Gesture Technologies for Supporting Collaborative Physical Tasks. In CHI06, 1191-1200.

7. Kraut, R. E., Fussell, S. R. and Siegel, J. (2003) Visual information as a conversational resource in collaborative physical tasks. In Human-Computer Interaction, 18, 13-49.

8. Sakata, N., Kurata, T., Kato, T., Kourogi, M. and Kuzuoka, H. (2003) WACL: supporting telecommunications using wearable active camera with laser pointer. In ISWC03, 53-56.

9. Alem, L. and Li, J. (2011) A Study of gestures in a video mediated collaborative assembly task. In International journal on Advances in Human Computer Interaction. Volume 2011 (2011), Article ID 987830, 7 pages, doi:10.1155/2011/987830.