# Designing Snakey: A Tangible User Interface Supporting Well Path Planning

John Harris, James Young, Nicole Sultanum, Paul Lapides, Ehud Sharlin,
Mario Costa Sousa

► To cite this version:

John Harris, James Young, Nicole Sultanum, Paul Lapides, Ehud Sharlin, et al.. Designing Snakey: A Tangible User Interface Supporting Well Path Planning. 13th International Conference on Human-Computer Interaction (INTERACT), Sep 2011, Lisbon, Portugal. pp.45-53, 10.1007/978-3-642-23765-2_4. hal-01591831

HAL Id: hal-01591831

https://inria.hal.science/hal-01591831

Submitted on 22 Sep 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Designing *Snakey*: A Tangible User Interface
## Supporting Well Path Planning

John Harris¹, James Young², Nicole Sultanum¹, Paul Lapides¹,
Ehud Sharlin¹, Mario Costa Sousa¹

¹ University of Calgary, Canada
{harrisjj, nbsultan, ehud, smcosta}@ucalgary.ca
paul.lapides@gmail.com
² University of Manitoba, Canada
young@cs.umanitoba.ca

**Abstract.** We present our ongoing design of *Snakey*, a tangible user interface (TUI) dedicated to the field of reservoir geosciences and engineering. Our design of *Snakey* emphasizes intuitive manipulation and interaction with 3D curves, common to underground well path exploration and planning. *Snakey* is not designed to replace the traditional CAD reservoir well path planning tools, but rather to augment them with tangible and collaborative interaction, facilitating spatial exploration during the multi-stage planning processes involved in well path design. Our short paper presents design goals and prototyping solutions relating to *Snakey*'s physicality, sensing technology, input/output mapping, and multi-modal feedback, as well as our findings from a preliminary evaluation of the latest *Snakey* prototype in a design critique session by experienced reservoir engineers.

**Keywords:** Well path planning, Tangible user interfaces (TUIs), physical interaction, real-time 3D curve manipulation, interactive visualization techniques, reservoir and geosciences engineering

## 1 Introduction

The petroleum industry has been exploring and proposing different solutions for well planning tasks [1]. Well planning is a fundamental task present in different stages of oil/gas field development from early exploration to production. Traditionally, well planning is an iterative process, involving teams of multi-disciplinary experts with specific constraints and goals. The 3D well trajectory is often laid out only in 2D maps and cross sections, making it difficult to understand the exact spatial relationships between the 3D reservoir model (synthesized from geological survey data) and the planned well-bore. Desktop-based solutions for well planning have been adopted by the industry, typically with WIMP interfaces, with wells trajectories being represented on the screen as 3D curves located inside a 3D digital model of the reservoir geology and its flow properties. The task of positioning well trajectories using the current WIMP interfaces involves a large number of control-point manipulations applied along each curve representing the well, leading to a time consuming and non-intuitive operation.
Well planning tasks have benefited from the use of various virtual reality (VR) interaction systems which allowed designers increased immersion in the path exploration task [1,2]. However, these VR solutions are not physical and do not provide tangible affordances. *Snakey* (Fig. 1) is being designed in order to match some of the unique properties of tangible user interfaces (TUIs) to the inherent physical and collaborative nature of 3D well-path planning in reservoir geosciences and engineering (G&E). The *Snakey* interface consists of 1) a graspable, semi-rigid physical device (similar to a long plastic “snake”) which mirrors the shape of an underground well path, 2) a visual display directly behind the hanging physical device which reflects the users’ physical manipulations onto a virtual model of a well path and its surrounding geological survey data, and 3) output elements (LED lights and vibration motors) embedded in the physical device that provide direct haptic and visual feedback to the user.

The *Snakey* TUI attempts to leverage users’ natural spatial perception and manipulation skills, mapping the task of virtual 3D well path onto a meaningful physical representation [3]. *Snakey* directly couples the 3D well path perception and action spaces into a single physical object, lowering the cognitive load required for interaction and improving the spatial mapping and ease of use [4]. *Snakey* is not designed to replace traditional CAD well path planning tools, but is an exploratory application of the tangible, collaborative, and immediate nature of TUIs to the complex, multi-stage planning processes involved in well path design. It is aimed at the reservoir engineering stage, when fluid flow simulations are performed and wells are manipulated to allow for different simulation runs and better understanding on the flow behavior across the reservoir.

<figure><img src="image_2.png" /><figcaption>**Figure 1** – The *Snakey* interface, visual display, and Vicon mocap camera array</figcaption></figure>
## 2 Related Work

Physical manipulation of 3D curves was one of the early focuses of TUI research. In 1999, Balakrishnan et al. [5] presented their work on using Measurand's ShapeTape™, an input device that can sense its 3D bend and twist, for interactive manipulation of 3D curves and surfaces. ShapeTape is a rubber ribbon which integrates internal tracking via embedded fiber optic bend and twist sensors. Balakrishnan et al. mentioned attaching spring steel to ShapeTape to provide more physical constraints to the rubber core [5], however to our knowledge there is no published work on integrating ShapeTape into an interface that can physically hold its shape consistently in a manner required by our design goals.

AR-Jig [6] provides interactive physical mapping to 2D curves using a handheld pin-array, which can be used to control 3D surfaces. However, the pins provide a relatively small, and physically discrete spatial control area that we believe would be difficult to meaningfully map to direct interaction with large 3D curves.

Closer to our applied domain of reservoir G&E, GeoTUI [7] provides a tabletop TUI for interaction with geological data, however the interface does not support 3D physical interaction and spatial manipulation of the 3D curves of well paths. Tateosian et al. [8] allow users to manipulate and explore detailed geological tasks using physical 3D surface sculpting, however their interface does not allow physical interaction with a consistent, shape holding 3D curve that can be mapped to spatial explorations of wells in reservoir G&E.

Like others [7], we believe that the reservoir G&E domain can benefit from the exploration of new physical interfaces that will afford a more intuitive and natural interaction with the well path planning task.

## 3 *Snakey's* Design Goals and Implementation

The primary design goal of *Snakey* was selecting the TUI's properties to create a valid interaction metaphor for the design and planning of 3D well paths in reservoir G&E visualization tasks. Derived from this goal, the device should afford simple and intuitive physical manipulation of 3D curves in real time *while maintaining* its 3D shape throughout the interaction session, unless directly manipulated by the user. We envisioned reservoir engineers, a single user or a collaborative group, being able to relate to the 3D physical *Snakey* as a constant and consistent representation of the virtual well path. Devising a means for *Snakey* to persistently represent the well, while at the same time allowing users to manipulate and interact with its curve, was our main design challenge.

Another design goal was to allow flexible and easy view and physical access to *Snakey*, enabling reservoir engineers to visualize the overall 3D trajectory of the well from different viewpoints in relation to the reservoir model. We also wanted to support modularity, reconfigurability and branching, as wells are often planned with junctions and forks, establishing multilateral 3D configurations (Fig. 3).

Furthermore we investigated the integration of other physical modalities and properties, beyond the spatial, into *Snakey*. Embedded lighting indicators could highlight sections of the well path that needed further attention. Embedding haptic feedback could allow users to have direct insight relating to some of the well
properties, such as fluid flow through the well path or its vicinity, or providing indications of having reached one of the well's or the reservoir's physical constraints.

Early in our implementation we chose the Vicon™ 3D motion capture camera system [9] for tracking the 3D spatial layout of *Snake*. With this system a number of retro-reflective markers are placed within a given tracking volume and an encompassing array of infrared emitters and cameras detect their 3D positions. The system uses triangulation algorithms to collectively reconstruct the positions of the markers in 3D space and in real-time. For the *Snake* TUI, the discrete 3D positions of the tracking markers were then interpolated to yield the continuous 3D curve path.

The Vicon 3D camera system provided a large interaction volume for the *Snake* device, adaptability (e.g. multi-prong curves), and practically unlimited curve lengths within our task by adding more markers. The visual tracking system introduced the problem of temporary occlusions such as when users cover a marker with their hands during manual manipulation, a problem we partly solved through the use of smoothing and interpolation algorithms. However, occlusions remain one of the *Snake* prototype's reoccurring problems.

The TUI is based on Joby's *GorillaPod™* [10]: flexible camera tripods which consist of a set of modular interlocking plastic orbs (a series of ball and socket joints) which are exceptionally light weight and are both easily manipulated and able to hold rigid shapes even when under significant loads. The *Snake* prototype, using the mid-range *GorillaPod SLR-Zoom* model, can form curves with a minimum bend radius of 3.5cm and when used to form a single, continuous chain it can support the weight of approximately 25 additional orbs arranged in a horizontal line. (Fig. 2)

In terms of modularity, sections of the orb links can be manually pulled apart with sufficient force and then easily popped/pushed back together. Combined with special branching nodes, this provided a quick and easy “building blocks” style method of creating multi-lateral/branching well path formations; an important component of most reservoir drilling operations. (Fig. 3) While the *GorillaPod* links were physically capable of forming “loops” and other more complex formations, our applied task of well path exploration precludes our need to consider them (as they do not make practical sense for the current *Snake* task) and as such were not explored in detail.

<figure><img src="image_3.png" /><figcaption>**Figure 2** – The “GorillaPod” rigidity, even when heavily cantilevered (top), and its ability to be easily manipulated into complex shapes (bottom).</figcaption></figure>
Additional visual and haptic interactive modalities (in parallel with the on-screen
display) were also directly integrated into the physical *Snakekey* device. Visual
feedback was provided by a set of programmable tri-colour, wide-angle LED lights.
Outputting approximately 8000 millicandelas of light intensity (sufficiently bright that
they can be painful to look at directly at full power), the LEDs could each be
individually programmed to take on a range of colours and intensities using a daisy-
chained, two-wire communication bus. Haptic feedback was provided by a set of
10mm diameter, button-style vibration motors, each controlled in parallel (along with
the LED lights) by a central microcontroller. Both the LED lights and vibration
motors were small enough to be mounted directly onto individual GorillaPod orbs
without interfering with *Snakekey*'s flexibility of motion. (Fig. 1)

*Snakekey* was mounted to a rigid, reconfigurable aluminum frame allowing the “fixed end” of the *Snakekey* chain to be position above, below, or to the side of the working volume, as necessary. (Fig. 1) This approach provides a fixed 3D position in space by which to correlate the physical object with its virtual, on-screen counterpart. It also helps create an open workspace around the physical object wherein multiple collaborating users can interact with, and manipulate the *Snakekey* device simultaneously.

<figure><img src="image_4.png" /><figcaption>**Figure 3** – (Left) A branched configuration of the *Snakekey* object. (Right) Common multilateral well path configurations (a) Multibranched, (b) Forked, (c) Laterals into horizontal hole; (d) Laterals into vertical hole; (e) Stacked laterals; (f) Dual-opposing laterals.</figcaption></figure>

*Snakekey*'s mounting frame was positioned directly in front of a large, high
resolution display which, with the help of the 3D position data provided by the Vicon
tracking system, superimposes the reconstructed virtual 3D curve onto the 3D
reservoir model (Fig. 1). The displayed virtual model describes both the geology and
flow behavior as simulated by high-end systems used in industry, e.g. [11]. As the
user manipulates the *Snakekey* device in the physical space, its virtual representation
updates within the reservoir model in real time. This facilitates a direct input/output
mapping between the *Snakekey* object's physical shape and its virtual representation.
Users can explore the virtual reservoir model immediately and easily by grasping and
manipulating the physical *Snakekey* device. The interface's physical constraints are
expected to translate directly into similar constraints for the virtual model: for example, limited bend radius and avoidance of self-intersection are inherent restrictions both in real-world well paths scenarios, as well as in the *Snakekey* TUI.

During interaction the *Snakekey* TUI can augment the visual display with further real-time feedback which is physically embedded into the interface. For example by indicating through the rumble intensity of the attached vibration motors the different fluid flow rates in different sections of the well, or by indicating that the well path intersects with a dangerous terrain volume using glow of a section of the integrated LED lights.

# 4 Preliminary Evaluation of *Snakekey*

The design of the *Snakekey* prototype was participatory, with close involvement of reservoir engineers and domain experts in all the design iterations. In this section we include the findings, criticism and limitations of *Snakekey*, as reflected via a design critique session which was conducted with the help of two experienced reservoir engineers. Our preliminary evaluation revealed that the experts were inspired by the concept of having a graspable physical manifestation of their three-dimensional well path plan. They highlighted that *Snakekey*, once we tackle some of its current limitations, can signify a major step forward in terms of usability and interaction with 3D well paths. Below, we provide a collection of important observations gathered from our users. Additionally, we feel that the abundance of ideas and enthusiasm which our users provided reflects the impact that a more refined interface could have on the well path planning process.

While the users were able to easily view the physical *Snakekey* device from any angle simply by walking around it, they were unhappy with the current need to rotate and move the geological data on the visual display using a desktop mouse and keyboard, taking attention away from the TUI and introducing a critical disconnect between the mental models of the physical device and the on-screen representation of the virtual data. In order to mitigate the physical inconvenience of having to shift between the *Snakekey* device and a desktop mouse, it was proposed that a viewpoint controller would be mounted to the frame holding the *Snakekey* device.

Our expert users also asked for added functionality that would allow them to control interface attributes beyond the 3D well curvature and path. For example, indicating where a specific operation takes place along the 3D well by physically squeezing a region of the *Snakekey* chain, or sliding one's hand along a region of the TUI to mark an area in the geological data.

Another major design feature that was requested is the ability to store the shape of the physical device and to facilitate comparison of stored alternative well paths. Once the shape had been stored, users would have liked to physically “load” the well shape back onto *Snakekey*. Supporting such a task may be accomplished if a future *Snakekey* becomes mechanically actuated, in essence turning it into a robotic interface capable of taking on its own formation (e.g. [12]). Simpler physical synchronization schemes were proposed, requiring the user to be actively involved in the physical well “loading” operation, for example by providing indications of when the physical device has approximately matched its digital parallel by showing green (match) or varying shades of yellow-orange-red (no match) using *Snakekey's* sectional LED lights.
Another finding relates to *Snakekey's* physical scale limitation. In practice, wells can be many hundreds of meters long, with each and every part of their huge extent often being carefully designed. Currently, the physical *Snakekey* affords only a one-to-one mapping to the entire length of the well. However, because the distance between adjacent GorillaPod orb joints is fixed (i.e. each link is a solid piece of plastic), users cannot specify curves any smaller than a single link. A possible solution is to support the design of wells in various stages of scale and granularity. The first stage would map the interface to the well as a whole singular entity; establishing the well's basic position and shape. The next stage would map *Snakekey* to only a portion of the complete well path, allowing a subsection of the well to be designed with higher precision. This "zooming in and out" approach revisits the aforementioned need for the physical interface to match previous, stored curves.

The experts were very positive regarding the inclusion of the LED lights and vibration motors as additional multimodal feedback integrated into *Snakekey*. They considered scenarios such as having the visual display show pipeline curvature tolerances while the vibration motors indicate fluid flow and the LED lights display rock densities. Following, *Snakekey* could arguably allow the user to perceive a fusion of properties that are interrelated, and would have been difficult to visualize simultaneously using a single, traditional display.

The reservoir engineers were also particularly enthused about the inherently collaborative nature of *Snakekey*. Any user within a team, regardless of professional training or project role, could instantly step in and manipulate the physical curve, provide meaningful input, and have the result be immediately visible and apparent to everyone involved.

The experts also proposed additions to the capabilities of the virtual screen model that took advantage of the unique *Snakekey* interface characteristics but would otherwise be difficult to control/express using a traditional WIMP interface. For example, while the resulting 3D curve can show one-dimensional geological information along its length (both virtually on-screen and physically using the attached LED lights), it was proposed that a 2D cross-section view of the virtual data would also reveal more information. This could be accomplished by treating the planar projection of the *Snakekey* chain's 3D form as a curtain that would "cut" the reservoir voxel data at the location of the curve, providing a richer visualization that shows the internal, "under the surface" data to the user. Alternative visualization schemes might include a translucent "tunnel" of surrounding voxel information that is shown within a set radius of the 3D curve.

Finally, the experts saw the immediacy and intuitiveness of the *Snakekey* interface as an opportunity to employ more scalable reservoir simulation solutions and provide much faster design feedback than is currently found industry. For example, instead of using traditional CAD software to simulate a complete well path using extremely detail physical parameters and finite-element methods (a process which can take days to compute), a proposed *Snakekey* well path could be simulated at a more general level with fewer parameters, having results quickly visualized and reviewed by the design group and allowing *Snakekey* TUI to be immediately adjusted for another simulation cycle. With this, engineers can interactively "zero in" on better well path designs in a matter of seconds before investing the time to run a traditional full-scale simulation.
# 5 Conclusion

We presented *Snakekey*, a tangible user interface supporting physical manipulation and planning of 3D well paths. Our current *Snakekey* prototype allows users to interact with and manipulate a physical curve-like artifact which is tracked in real-time and spatially mapped to a virtual 3D curve. In turn, this allows users to intuitively explore complex 3D geological data and collaborate in well path planning tasks with a multi-disciplinary team. *Snakekey* affords flexible physical manipulation and its main advantage is its ability to maintain its shape even when not being held by a user. Other features explored in the *Snakekey* TUI are integrated haptic feedback and dynamic visual cues within the physical interface itself, as well as dynamically changing *Snakekey's* topology as afforded by its modular nature. Preliminary evaluation emphasized many of the prototype's limitations, outlined some short-term goals for improvement and also highlighted the promise *Snakekey*-like TUIs hold for the domain of reservoir geosciences and engineering, and the task of 3D well path planning.

# 6 Acknowledgements

We would like to thank our colleagues and anonymous reviewers for their valuable input. This research was supported by the Alberta Innovates Academy (AITF) / Foundation CMG Industrial Research Chair in Scalable Reservoir Visualization.

# References

1. Lidal, E.M., Langeland, T., Giertsen, C., Grimsgaard, J., Helland, R., "A Decade of Increased Oil Recovery in Virtual Reality", *IEEE Computer Graphics and Applications*, (2007)

2. Loftin, R.B., Bavinger, B.A., LeRoy, S.D., Nelson Jr., H.R. "Advanced Visualization Techniques for Exploration and Production", *Offshore Technology Conference* (1997)

3. Ullmer, B., Ishii, H., "Emerging Frameworks for Tangible User Interfaces", *Carroll JM (ed) Human-Computer Interaction in the New Millenium*. pp 579-601 (2001)

4. Sharlin, E., Watson, B., Kitamura, Y., Kishino, F., Itoh, Y., "On Tangible User Interfaces, Humans, and Spatiality", *Personal Ubiquitous Computing*. pp 338-346. (2004)

5. Balakrishnan, R., Fitzmaurice, G., Kurtenbach, G., Singh, K., "Exploring Interactive Curve and Surface Manipulation Using a Bend and Twist Sensitive Input Strip". *Proc. of the 1999 Symposium on Interactive 3D Graphics*. Atlanta, Georgia, United States. April 26-29 (1999)

6. Anabuki, M., Ishii., H., "AR-Jig: A Handheld Tangible User Interface for Modification of 3D Digital Form via 2D Physical Curve." *Proc. of the 2007 6th IEEE and ACM international Symposium on Mixed and Augmented Reality* (2007)

7. Couture, N., Rivière, G., Reuter, P., "GeoTUI: a tangible user interface for geosciences". In *Proc. of the 2nd international Conference on Tangible and Embedded interaction* (Bonn, Germany, February 18 - 20, 2008). TEI '08. ACM, New York, NY, 89-96 (2008)

8. Tateosian, L.G., Mitasova, H., Harmon, B., Fogleman, B., Weaver, K. and Harmon, R.S. "TanGeoMS: Tangible geospatial modeling system" IEEE Transactions on Visualization and Computer Graphics (TVCG). (2010).

9. Vicon Motion Capture System: http://vicon.com/

10. Joby Gorilla Pods: http://joby.com/

11. Computer Modeling Group Ltd. http://www.cmgroup.com/

12. Ota, T., Degani, A., Schwartzman, D., Zubiate, B., McGarvey, J., Choset, H., Zenati, M., "A Highly Articulated Robotic Surgical System for Minimally Invasive Surgery". In *Annals of Thoracic Surgery*. April 2009; 87 (4); 1253-1256