Towards Augmented Reality User Interfaces in 3D Media Production
Max Krichenbauer∗

Goshiro Yamamoto†Takafumi Taketomi‡

Christian Sandor§

Hirokazu Kato¶

Interactive Media Design Laboratory
Nara Institute of Science and Technology

Figure 1: Our goal is to establish Augmented Reality as a user interface for professional media production. Our approach consists of three steps:
We conducted an online survey with 3D media professionals from all around the world (a). Based on their responses, we derived requirements
and designed a user interface (b). We then implemented a prototype (c) and evaluated it with media professionals in a pilot study.

A BSTRACTThe idea of using Augmented Reality (AR) user interfaces (UIs) tocreate 3D media content, such as 3D models for movies and games
has been repeatedly suggested over the last decade. Even though
the concept is intuitively compelling and recent technological advances have made such an application increasingly feasible, very
little progress has been made towards an actual real-world application of AR in professional media production. To this day, no
immersive 3D UI has been commonly used by professionals for 3D
computer graphics (CG) content creation.
In this paper, we are first to publish a requirements analysis forour target application in the professional domain. Based on a surveythat we conducted with media professionals, the analysis of professional 3D CG software, and professional training tutorials, we identify these requirements and put them into the context of AR UIs.
From these findings, we derive several interaction design principles
that aim to address the challenges of real-world application of AR
to the production pipeline. We implemented these in our own prototype system while receiving feedback from media professionals.
The insights gained in the survey, requirements analysis, and userinterface design are relevant for research and development aimed atcreating production methods for 3D media production.
Keywords: Augmented Reality, Immersive Authoring
Index Terms:
H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems—Artificial, Augmented,
and Virtual Realities; I.3.6 [Computer Graphics]: Methodology andTechniques—Interaction techniques∗ e-mail: max-k@is.naist.jp† e-mail:goshiro@is.naist.jp‡ e-mail:takafumi-t@is.naist.jp§ e-mail:sandor@is.naist.jp¶ e-mail:kato@is.naist.jp

1 I NTRODUCTION
Even before the advent of AR, the idea of creating an immersive
3D modeling application for creating 3D content was proposed by
Clark [7]. However, even though AR has been shown to be excellent for 3D tasks thanks to correct spatial alignment [24, 26], there
is, to our knowledge, no fully functioning immersive 3D UI available for professional 3D content creation, including film, TV, orgames. A number of AR CAD applications were created and tested
by researchers, but none of them was based on a requirements analysis of professional 3D design work or was further developed for
professional production. In order to bridge the gap between researchers and studios, we analyzed the requirements of digital content creation and derived a UI design from them.
We limit our work to the 3D content creation part of the
production: modeling, rigging, animation, simulation, and lighting/rendering. We do not consider 2D elements of the production
pipeline such as compositing or color-grading, since this work isnaturally two-dimensional and thus differs greatly from the scrutinized tasks. We also do not consider authoring of AR experiences:
this paper is about using AR as a means of producing 3D content,
not about using 3D content to create AR applications.
In order to provide a scientifically sound analysis of professional
3D design work, we have included several sources of information.
Primarily, we performed a survey with 3D professionals worldwide(section 3). We also analyzed the UIs of existing professional 3D
content creation software. Furthermore, to gain more insight into
the common use of the software, we reviewed tutorial videos aimed
at professional 3D design training, which cover the use of the software as well as production principles. Based on these sources, we
deduced requirements for UIs for 3D design, and discuss a set of
UI concepts that aim to address these issues.
Contribution: Our main contribution is a thorough requirements analysis of UIs for professional 3D media design. Our findings will allow more application oriented research and developmentin this area. Furthermore, we suggest a number of interface designs
addressing the challenges, which we identified. Taking feedback
from media professonals, we combined these designs into a integrated work environment. Our findings can help advance the searchfor universal UI principles for AR 3D modeling.

2

R ELATED W ORKPrevious publications related to AR UIs for professional 3D media creations can be categorized as follows: reviews that give an
overview and analysis of previous attempts at create AR UIs; authoring tools that allow a limited form of content creation by assembling predefined objects; AR UIs for engineering CAD purposes;
immersive 3D modelers; AR related UIs targeted at professional
CG software and procedures; and novel UIs for 3D design that are
not AR or Virtual Reality (VR) related. In the following, we discuss
the most prominent publications from each category.
Reviews of real-world applications of AR. It has been a common problem of research efforts to bridge the gap to commercial
application. Fite-Georgel [11] surveyed a large number of systems
on their applicability and successful adoption for industrial applications such as manufacturing and construction. Our focus is on
virtual 3D media content creation, which is more of an artistic discipline than precise engineering. LaViola and Keefe [19] give an
overview of past and recent 3D UIs for art and design purposes,
without having a specific viewpoint of media production.
AR authoring tools. Another area of vibrant research are immersive authoring systems. In these systems, the user assembles aworld out of provided predefined content by interactive placement
and changing specific attributes such as texture. FingARtips [4] offers glove-based within-arms-reach object manipulation in an AR
tabletop environment, aimed at urban planning. Thus the interaction is limited to placing houses and streets. Two recent systems
were presented at the IEEE 3D UIs Contest: the DIY World Builder[27] and the Wonderland Builder [3], targeting level creation forgames and creating fantasy worlds. While there is some requirement in professional 3D design to incorporate previously preparedor acquired objects, UIs that are limited to content assembly are notsufficient for professional design.
AR UIs for engineering. Engineering also has a high demandfor creation of virtual 3D models, and a number of AR based UIs
were developed. These systems are commonly called ARCAD andfocus on precision and physical correctness instead of artistic expression. Girbacia [13] developed an ARCAD system to cooperatewith available legacy 3D CAD software, offering only 3 features(creating a primitive object, extruding 2D closed profiles and revolving a 2D profile around an axis). Gao [12] proposed a semiimmersive virtual environment for solid modeling using voice commands to extend the UI. Shen et al. [23] developed a system thatcombines an AR UI with remote collaboration for several engineers, exploring principles how to protect technical correctness in acollaborative set-up.
Immersive 3D modelers. Using AR as a UI for artistic 3D modeling, animation, etc. as we intend it has been proposed by several
authors, however without concerning the professional application.
3Dm [5] was one of the first systems that explored the design spaceof immersive modeling in a VR setting. The THRED system [22]offers bi-manual modeling using 3D tracked “bats”. Similarly, JDCAD+ [14] is a non-immersive modeling and animation system thatuses a 6 degrees of freedom (DOF) tracked “bat” to create content for virtual environments. 3darmodeler [9] is a stand-alone AR
based modeler directed toward amateurs. Kim et al. [16] developed
immersive VR modeling system that is based on 5 hand gestures
for basic Subdivision-Surface modeling. The outdoor AR system
Tinmith [20] was also used for creating 3D models in a very limited way. Construct3D [15] is a modeling system which aims formathematical education. ICOME [21] explored remote collaboration in immersive modeling environments. None of these systems
concerns itself with professional application in media production.
AR UIs for CG software. Only a small number of publications
concerned itself with the possibilities and demands of professional
application. ARpm[10] is an AR front-end to Autodesk R 3D Studio Max R , which functions by taking screenshots of the softwareand sending Windows system calls to control it which greatly limits performance and usability. Takala et al. [25] implemented asemi-immersive 3D UI for the Blender modeling software using a
Playstation3 and a number of PlayStation Move controllers. Theiraim was to create an inexpensive set-up. Most interestingly, theytested their prototype with 7 professional 3D artists. However, theirevaluation only concerned the use of the prototype, whether it was“fun” or whether they experienced “fatigue” during their 25 minute
use time. Barakonyi published a system that targets character animation exclusively by using a physical puppet [2]. The concept of
tracking a physical rig — originally developed by Knep et al. [18]— was improved by applying AR technology. Such an approach
is logically limited to a very narrow application: only animation ofvirtual content which can be represented by a physical puppet (nofluids, hair or muscle rigs) and no possibility to play back recorded
animation, forcing the animator to work “straight-ahead” which is
uncommon in professional animation. Most importantly, none ofthese publications includes an analysis of requirements for professional application. The only commercially available AR application
directed toward professional 3D CG content creation known to us
are the ARplugins. However, these plug-ins to 3D content creation
software merely offer visualization of the content. There is no way
to create or edit models in the AR environment.
Non-AR novel 3D user input devices. Another area of researchand development are novel user input devices for 3D design. These
commonly try to offer more DOF while still using a normal monitor
set-up. Autodesk Inc. R offers a Plugin for their Maya R modelingsoftware that uses a Leap Motion controller to track the users hands,
allowing some interaction with the virtual content [1]. The Didjiglove [8] is a glove-based user input system to control animationsin professional 3D software by using it as an advanced programming interface. Similar to the previously mentioned concept of employing a physical puppet, QUMARION [6] is a currently commercialized version without the AR overlay proposed by Barakonyi.
3

S URVEY ON 3D DESIGN WORK

We performed a survey with 3D media professionals in order togather a solid empirical base of information. In the following, we
give information on how we performed the survey and present themost interesting results, including questions related to the work
load and health issues, common forms of collaboration in the studio environment, amount of functionality habitually used by artists,
complexity of the content worked with, as well as expectations on
future UIs. The complete survey results are provided as supplementary material.
We used an online survey platform in order to reach professionals from around the globe. The questions included a selfassessment of professionalism and skills in order to get an impression of the participant. Where the self-assessment did not suit ourrequirements for industry professionals we did not include the data.
We distributed the survey via various channels including our own
personal contact network, 3D art forums and freelancer recruiting
websites. In order to ensure authenticity we asked the name andcompany or portfolio website of all participants. We did so at theend of the survey after all answers had been completed and ensured
the participant that their names would not be published in order toavoid social desirability bias. Answers of participants who could
not be identified were then removed from the sample. We did notcontact each individual artist, but the timing of responses after informing different groups of people about the survey gave us confidence that the person did in fact take the survey. The final sample
consisted of 54 participants from 17 countries (Figure 1(a)).
We asked a number of questions regarding the artists work environment and daily work context. When asked for their average
daily work hours, the average time stated was 6.87 hours (standard
deviation (SD) 2.27 hours ), where the common duration of one

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2: (a) Frequency of various types of collaboration. (b) Perceived importance of various types of collaboration. (c) “Please estimate thecomplexity/size of content you are working with.” (d) “Estimate how many different tools and functions of 3D CG software you regularly use.” (e)“Estimate how many keyboard hot-keys for your 3D software you use regularly.” (f) Estimation and classification of additional scripts or plug-ins
used with 3D software; (normal) means publicly available, (custom) means developed in-house or ordered exclusively.

work session until taking a break was on average 3.02 hours (SD2.25 hours). When asked for their current main input device, allartists expressed working with a 2D device, 75.47% with a mouseand 24.53% with a pen tablet. Not a single participant stated using
an alternative or 3D user input device for work. More than 16.67%
of artists reported that they previously tried 3D user input devices,
either haptic 3D input device (3.70%), 3D mouse (11.11%), or a 3D
spheric mouse (5.56%). 1.85% stated to have used a glove based input device. When asked how their arms, hands and wrists feel after
several days of working with 3D software, 46.67% stated “normal”,
46.67% expressed that their arms felt “tired” and 6.67% even statedthat their hands “hurt”. However, when asked whether they perform
some kind of countermeasures to deal with work related problems
in their arms, wrists or hands, a total of 31.48% stated that they
employ some kind of counter measure (special hand-gear: 1.85%;
special exercises: 29.63%; take medication: 1.85%; seeing a physician or health expert: 3.70%). We further found that 38.89% of
the participants are missing a third dimension in their UI (monitor: 18.52%, mouse: 35.19%). However, 27.78% also expressed
explicitly positive opinions about 2D UIs.
To get an impression of collaboration in the studio environment,
we divided the various types of collaboration into four categories.
Non-interactive review was defined to be the process of presenting
the content without being able to edit it at the same time. Interactive
review similarly was described as receiving feedback while simultaneously being able to edit the content. Divided collaboration was
defined as the process of multiple artists working on the same content by means of dividing it either by separation or taking turns.
Full collaboration finally defined working together on the samecontent at the same time. We asked the participants both how regularly they performed each type of collaboration and how importantthey consider it—whether based on experience or expectation—for
their professional work. The non-interactive review and interactive
review were considered the most common and important form ofcollaboration, but even full collaboration, while relatively rare, wasconsidered important by the majority of artists. (Figure 2(a), 2(b)).
One set of questions related to the way that artists used their professional tools daily. We asked them how many of the functions that

their 3D software offers they actually use regularly. We discoveredthat it is common for 3D artist to use up to a hundred functions(Figure 2(d)). Furthermore, we asked how many keyboard hot-keys
they habitually used to increase their efficiency. We found that using 16 or more hot-keys was not a rare exception, whereas using
less than or 3 hot-keys is uncommon (Figure 2(e)). When asked
whether they use scripts or plug-ins for their software that were
not part of the original set of functionality, only 3.70% stated not
to require any additional functionality, and many artists use morethan just a few additional scripts or plug-ins (Figure 2(f)). Similarly, 52.83% of the participants in our survey stated to use custom
scripts or plug-ins that where developed at their own company, or
were ordered to be created exclusively for their own requirements.
77.78% stated that they believe custom scripts and plug-ins will always be a requirement in order to adjust the software to production
requirements.
We asked about the use of translational navigation and zooming/scaling during work on a 5-bin Likert scale, ranging from “constantly” to “never” (Figure 2(c)).
Finally, we presented artists with a series of novel user input devices, as they were previously proposed by scientists: AR overlay
applied to a haptic 3D input device, 3D mouse, gloves, and a mannequin, as well as voice input. We asked the participants to rate
their agreement to common arguments regarding the advantages
and disadvantages of these systems in form of a Likert Scale. It
became apparent that media professionals agree considerably morewith the shortcomings of novel user input devices than with their
expected merits. We also asked them to rate the usefulness of thepresented devices on a Likert scale from 1 (“very useless”) to 7(“very useful”), comparing them to each other as well as to the
traditional mouse as a user input device for 3D content creation.
Strikingly, all 3D input devices scored significantly lower (between
4.50 and 4.25) than the traditional 2D mouse (6.59). Voice input
scored even lower than the 3D UI devices (3.30).
4

D ERIVED R EQUIREMENTS

In this section we present information on specific requirements forthe professional 3D content creation workflow. These are: an er-

gonomic design that can be used comfortably for a long periodof time, support for collaboration among artists and supervisors, ahigh amount of features, support for navigating the content, support
for 2D and alphanumeric operations, and an increase in productivity over the traditional 2D UI. The subsections are in no particularorder, as all points are independent requirements.
Ergonomic design. As our survey has shown, 3D artists tend
to work for several hours before taking a break and spend a lot oftime working with their software every day. Even with the quite
comfortable and sophisticated traditional UIs, this becomes an issue for the artists health and well-being. It is therefore important toprovide UIs that are comfortable to use over an extended time without strain or tiredness. Previous publications only provided quickuser tests and thus never discussed problems encountered duringextended use, which are vital for professional application. For example, some researchers suggested systems where the users interact while standing [15, 20]. While this may be comfortable or even
healthy for a short time period, it is unlikely to be adopted for extended use. Furthermore, the device used for viewing the AR must
be considered. Using a tablet or other mobile device requires theartists to move it around whenever the viewing position must bechanged. As our survey has shown, this is basically constantly thecase. Hence, a head-mounted display (HMD) is more appropriate.
Collaboration support. Since professional production is not an
individual effort of a single artist, some kind of collaboration is obviously necessary. According to our survey and own experience,
many types of collaboration are commonly performed and necessary for producing high-quality results. In the most basic way, thisconsists of showing the content to another person while the user isable to perform changes simultaneously. The traditional use casebeing that the supervisor or a colleague visits the artist at his work
desk, and comments on the current state of the work. Traditional
2D UIs support this naturally because the monitor screen is visible
to everyone, and via pointing and talking, feedback can be communicated. Novel UIs however must be designed with this requirementin mind. This requires some way of bidirectional information transfer: other people than the artists must be able to perceive the virtual
content, and must be able to communicate with the artists about
the content. AR has an inherent advantage over VR in this respect
because all users in a shared augmented environment can see eachother without need for virtual representation. Hence, pointing at
specific parts of the content while talking with each other directlyis possible. For VR and remote collaboration, similar communication methods must be provided. However, reviews are not the only
type of collaboration used or wished for by artists. While traditional PC UIs are inherently single-user (even when several mouses
and keyboards are connected, the single focus makes real collaboration impossible), some types of simultaneous work through quick
alteration, file versioning or separation of tasks are also common orat least estimated to be useful by most artists. Novel systems canbreak with the traditional single-user paradigm and offer real time
multi-user collaboration.
High amount of features. While in industrial applications correctness is usually the greatest challenge [23], media production
challenges us with an intense amount of functionality. Because
there is a huge pressure to produce more and more stunning expressive imagery, the accumulated feature set by far exceeds any existing AR or VR application. These functions include tools like translation or rotation, actions like extruding or collapsing a polygon,
operational modes like interacting with single vertices or complex
group hierarchies of objects, mathematical operations like boolean
difference of two bodies, computer graphics features like different
types of light sources, materials or cameras, or support for industry standards like automation APIs, scripting or file types. As oursurvey illustrates, these features are actively used by the artists,
and that fast input methods like hot-keys are a necessity. Even

more problematic: since most artists extend their feature set withadditional scripts and plug-ins (sometimes even individually customized) it is impossible to anticipate the required features. While
the technical challenge seems daunting, it is actually easily solved:
instead of trying to implement all required functionality, one caneasily build on existing software either by integrating open-sourcecode or by offering the UI in the form of a plug-in for existing
software. The real challenge lies in the UI design: no previous system known to us attempted to provide access to a similar amountof functionality while still being intuitive. Novel approaches are
required to organize complexity in 3D UIs.
Fast and intuitive navigation. Another important factor thatstems from the complexity of the content processed in professional
production is the need to navigate through the content. In this context we use the term navigation in its traditional meaning of changing the relative position of the users viewpoint and the virtual environment or content without altering the content (editing). In immersive VR environments, the term and reasoning is intuitive to understand because the user has a sense of “moving” through the environment. AR applications typically don’t incorporate any navigation other than physical (by moving the camera or moving a fiducial
marker) because the user can still see the real environment, so there
would be no sense of personal motion. However, in digital media
production, the content is too big and complex for physical navigation to be feasible. Some scenes may incorporate whole cities,
where the artist is working on figures in individual streets while
trying to keep an overview of the complete scene. Our survey confirms this: no one stated that they can perform their work without
any form of zooming or navigation – on the contrary, most artists
navigate their content often or constantly during work. Therefore,
applications have to implement intuitive and quick user navigation.
Support for 2D and alphanumeric operations. 3D design consists of a greater number of sub-tasks, not all of whichare 3-dimensional in themselves. Some tasks are inherently 2dimensional in nature, even though they relate to 3D CG. An example for this is texture mapping, where the “UV” coordinates ofvertices of polygonal models must be laid out on a 2D map. Here,the 3D artist tries to achieve a balance between distortions in the
texture, seams in texture mapping, limited space on the 2D map
and ease-of-use for texturing artists. Another problem that is difficult to realize in 3D UIs is the need for alphanumerical input. This
is often required in 3D design in the form of file names or inputting
exact parameter numbers. Moreover, in rigging, it is often required
to perform some minor scripting on the 3D content, such as defining
a relationship between several objects to implement some mechanics. One way to offer alphanumeric input is via voice input overa microphone, as proposed by Girbacia et al. [13]. However, atalking-out-loud approach is not very suited for a shared workplaceenvironment. Throat microphones might alleviate the problem atthe expense of increased cost, decreased comfort and a loss in reliability. Our survey shows that 21.2% of the participating artists
disagree to any usefulness of voice input, and only 36.4% see it as
a possible replacement for traditional mouse-and-keyboard UIs.
Increase in productivity. Surprisingly, one of the greatest challenges is to design a UI that actually matches or even outperforms
traditional 2D UIs in terms of efficiency. It is easy to believe thata 3D UI will in itself increase ease of use and user performance
thanks to spatial understanding. In our user tests however, we had
to discover that this is not the case: spatial understanding is limited,even when stereo vision HMDs are used, and the limited advantage
of some depth perception is hardly enough to encourage switchingsystems. As our survey shows, artists are open to alternative 3D
input devices, but critical in their adoption. They have developed
skills to work with 2D UIs with high speed and precision. Our
survey has shown the limited hope of artists that alternative UI concepts can live up to working with a traditional 2D mouse.

Figure 3: UI design concepts. (a) Mapping of functions to the hand. (b) 2D window interaction. Inactive windows face the user beyond arms
reach. Ray casting from the dominant eye over the cursor is used for interaction. When a window becomes active, it is moved forward, changing
its size accordingly so that the optical impression will not change. (c) Navigation concept: the world origin is transformed to give the illusion of
positioning the objects without editing. Top: Single handed navigation, “zooming” with the Shift key (right). Bottom: Bi-manual navigation.

5

U SER I NTERFACE D ESIGN

Based on this set of requirements, we developed a concept for a
3D UI to meet the requirements for professional 3D design work.
Figure 1b shows an illustration of the suggested UI. In our system,
the artist is comfortably seated at his work desk, wearing a stereo
video-see-through HMD. The artists hands are tracked to allow direct manipulation within arms reach. An AR system is used over
VR to allow interaction with everyday objects (eg. a coffee cup) andother people (important for collaboration). The 3D AR UI shouldbe implemented as a plug-in or extension of the respective software package that the artist uses. This allows seamless transition
between traditional and AR UI, and automatically provides all thefeatures and integration requirements of the production.
Ergonomic direct hand interaction. We advocate bi-manual
direct hand interaction within arms reach as main input vector, as
it is the most natural and versatile way of interaction. The user canrest the elbows or forearms on the table while working to reduce
the strain during extended work. The thumbs of both hands are theframe of reference for pinch-based interaction, which allows a natural “grabbing” interaction metaphor. The main interaction functionality (editing 3D content, using 2D elements and navigation of
the scene) is implemented on both hands in the same way, allowing simultaneous bi-manual interaction. Glove based systems havebeen shown to support a great number of buttons comfortably [8],which supports our requirement to offer a high amount of functionality. In our UI design, we implement 8 pinch-buttons per hand,with two complementary contact areas: the tip of the thumb and thepalm of the hand. The tip of the little finger assumes the function ofa Shift key, temporarily altering the assigned functions of the otherbuttons. Thus, we offer an effective number of 14 keys per hand.
Figure 3(a) shows the mapping of functions on the hands.
Navigation. When the “navigation” button on top of the handis pressed, the whole 3D content (ie. world origin) will change its
position in the work area as if being grabbed by the user’s hand,in 6DOF (translation and rotation), realizing the “grabbing-theair” concept. This technique is intuitive not only in immersive virtual environments—where it gives the impression of pulling oneself
through the environment—but also when working with a single virtual object in an AR environment, where it feels similar to grabbing
and positioning the object in the work space. Thus, it automatically realizes a “frame-of-reference” concept for hand interaction:
the user holds the object (in fact: the whole virtual scene) with thenon-dominant hand while working on it with the dominant hand.
Zooming is performed with the Shift button, or by using two hands
simultaneously (Figure 3c).
Collaboration. Novel 3D UIs can support collaboration on twolevels: interactive reviews and full real-time collaboration. In themost simple case, it should be possible to perform interactive reviews by displaying the virtual content on the computer monitor bycloning the video stream of the artists HMD, or by employing additional cameras and displays. In order to allow full collaboration, simultaneous interaction is required which is harder to achieve since
it conflicts with the requirement for simple navigation: if several
artists share the same global coordinate system, they would interfere with each others work when they change it. Kiyokawa’s proposed separation between personal and public spaces [17] can beapplied here, offering separated coordinate systems to every user aswell as a global coordinate system for exchange.
Menus offering a high amount of operations. As shown earlier, the range of tools and operations provided by professional 3D
software is huge. We propose a palette way of organizing functionality that uses the natural range of freedom of the hand interaction.
The visual feedback was found to be most convenient when 2D,
facing the user, and using the third DOF of hand motion (distance)
to navigate several layers of tools and options.
2D editor windows. Apart from the obvious advantage of AR
that the user is still able to use traditional UIs without having toswitch, we also developed a 3D approach to 2D interactions. All 2D
window elements in the AR view face the user at all times. Wheninactive, they reside at a background plane beyond arms reach sothey don’t interfere with the interaction. The user can point at thesewindows by hovering the thumb cursor over them, as seen by thedominant eye (i.e. by casting a ray from the dominant eye overthe cursor onto the window). When a window gains focus, it isbrought forward towards the cursor to eliminate the binocular disparity. When the window comes forward or returns to the background, it is re-scaled accordingly to keep its apparent 2D size to
avoid irritating the user (Figure 3(b)).
Efficient 9DOF manipulation. Traditional 2D UIs can only offer two DOF at a time and even 3D user input devices usually onlyoffer 6DOF (more DOF may be theoretically available, but the user
has to switch interaction and is thus is unable to control them simultaneously). Our bimanual 3DUI can offer object manipulation
with all 9DOF simultaneously in a intuitive and efficient way: one
hand controls translation and rotation, the relative position of thesecond hand controls scaling in three dimensions. It is intuitive tounderstand: one hand grabs and moves an object, and the second
hand stretches or squashes it.
Improved animation time control. Time control for animation is usually implemented in the form of a 1D slider operated
with the mouse. We implement an animation key, which allows“flipping” through the animation with a simple wave of the nondominant hand, similar to traditional hand-drawn animation where
the artist would flip though a number of frames. Since the dominant hand can still be used to manipulate objects simultaneously,
the artist can create coarse animations very fast (Figure 4).

Figure 4: The user’s view through the HMD while animating. Top row: The left hand controls the time. Bottom row: The right hand can be used
to simultaneously manipulate objects, creating rough animation keyframes very fast. Right: Resulting animation.

6

P ROTOTYPE

We implemented our recommendations on 3D AR UIs in a prototype (Figure 1(c)) in the form of a plug-in for Autodesk R Maya R .
We used thin cotton gloves as user input devices by sewing conductive materials on them. Tracking of cameras and hands wasachieved with ARToolKit. Throughout the development of the prototype, we stayed in contact with media professionals who agreed totest and comment on the system. This gave us valuable insights that
helped advance both theoretical understanding as well as practicalimplementation.
7

C ONCLUSION

In this paper we presented information on applying AR UIs to digital media content creation. We showed the challenges and problems
for such applications and proposed several solutions which we havetested in form of a prototype.
As future work, we plan to perform a mature user study withmedia professionals.
R EFERENCES[1] Autodesk Inc. Leap Motion Plug-in for Autodesk R Maya R 2014.
http://area.autodesk.com/MayaLeapPlugin/, 2014.
[2] I. Barakonyi and D. Schmalstieg. Augmented reality in the character animation pipeline. In ACM SIGGRAPH 2006 Sketches, page 75.
ACM, 2006.
[3] C. Barot, K. Carpentier, M. Collet, A. Cuella-Martin, V. Lanquepin,
M. Muller, E. Pasquier, L. Picavet, A. Van Ceulen, and K. Wagrez.
The wonderland builder: Using storytelling to guide dream-like interaction. In 3D User Interfaces (3DUI), 2013 IEEE Symposium on,
pages 201–202. IEEE, 2013.
[4] V. Buchmann, S. Violich, M. Billinghurst, and A. Cockburn. Fingartips: gesture based direct manipulation in augmented reality. In Proceedings of the 2nd international conference on Computer graphics
and interactive techniques in Australasia and South East Asia, pages
212–221. ACM, 2004.
[5] J. Butterworth, A. Davidson, S. Hench, and M. T. Olano. 3dm: A three
dimensional modeler using a head-mounted display. In Proceedingsof the 1992 symposium on Interactive 3D graphics.
[6] CELSYS Inc.
QUMARION — Humanoid input device.
http://www.clip-studio.com/quma/en/, 2014.
[7] J. H. Clark. Designing surfaces in 3-d. Communications of the ACM,
19(8):454–460, 1976.
[8] L. Dipietro, A. M. Sabatini, and P. Dario. A survey of glove-based
systems and their applications. Systems, Man, and Cybernetics, Part
C: Applications and Reviews, IEEE Transactions on, 38(4).
[9] T. V. Do and J.-W. Lee. 3darmodeler: a 3d modeling system in augmented reality environment. International Journal of Computer Systems Science and Engineering, 4(2):145–154, 2008.
[10] P. Fiala and N. Adamo-Villani. ARpm: an augmented reality interface
for polygonal modeling. In Mixed and Augmented Reality, 2005. Pro-

ceedings. Fourth IEEE and ACM International Symposium on, pages
196–197. IEEE, 2005.
[11] P. Fite-Georgel. Is there a reality in industrial augmented reality?
In Mixed and Augmented Reality (ISMAR), 2011 10th IEEE International Symposium on, pages 201–210. IEEE, 2011.
[12] S. Gao, H. Wan, and Q. Peng. An approach to solid modelingin a semi-immersive virtual environment. Computers & Graphics,
24(2):191–202, 2000.
[13] F. Girbacia. An approach to an augmented reality interface for computer aided design. Annals of DAAAM & Proceedings, 2010.
[14] M. Green and S. Halliday. A geometric modeling and animation system for virtual reality. Communications of the ACM, 39(5).
[15] H. Kaufmann. Construct3d: an augmented reality application formathematics and geometry education. In Proceedings of the tenth
ACM international conference on Multimedia.
[16] H. Kim, G. Albuquerque, S. Havemann, and D. W. Fellner. Tangible 3d: Hand gesture interaction for immersive 3d modeling. In
IPT/EGVE, pages 191–199, 2005.
[17] K. Kiyokawa. 3d collaboration using mixed reality technology. In
Proc. of the First International Symposium on Universal Communication, pages 100–109, 2007.
[18] B. Knep, C. Hayes, R. Sayre, and T. Williams. Dinosaur input device. In Proceedings of the SIGCHI conference on Human factors
in computing systems, pages 304–309. ACM Press/Addison-Wesley
Publishing Co., 1995.
[19] J. J. LaViola and D. F. Keefe. 3d spatial interaction: applications forart, design, and science. In ACM SIGGRAPH 2011 Courses, page 1.
ACM, 2011.
[20] W. Piekarski. 3d modeling with the tinmith mobile outdoor augmented reality system. Computer Graphics and Applications, IEEE,
26(1):14–17, 2006.
[21] C. Raymaekers, T. De Weyer, K. Coninx, F. Van Reeth, and E. Flerackers. Icome: an immersive collaborative 3d object modelling environment. Virtual Reality, 4(4):265–274, 1999.
[22] C. Shaw and M. Green. Two-handed polygonal surface design. In
Proceedings of the 7th annual ACM symposium on User interface software and technology, pages 205–212. ACM, 1994.
[23] Y. Shen, S. Ong, and A. Nee. Augmented reality for collaborative
product design and development. Design Studies, 31(2):118–145,
2010.
[24] R. Shepard and J. Metzler. Mental rotation of three-dimensional objects. Science, 1971.
[25] T. M. Takala, M. Makarainen, and P. Hamalainen. Immersive 3d modeling with blender and off-the-shelf hardware. In 3D User Interfaces(3DUI), 2013 IEEE Symposium on, pages 191–192. IEEE, 2013.
[26] M. Tonnis, C. Sandor, C. Lange, and H. Bubb. Experimental evaluation of an augmented reality visualization for directing a car driver’s
attention. In Proceedings of the 4th IEEE/ACM International Symposium on Mixed and Augmented Reality, pages 56–59. IEEE Computer
Society, 2005.
[27] J. Wang, O. Leach, and R. W. Lindeman. DIY World Builder: an
immersive level-editing system. In 3D User Interfaces (3DUI), 2013
IEEE Symposium on, pages 195–196. IEEE, 2013.