Venue: Graphics_Interface 2022 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The submission presents an experimental comparison of 3 user interfaces for selecting points on a 3D model, motivated by potential applications in jaw osteotomy surgery planning (section 2.3).  The submission refers to 'manipulation' and 'marking' as tasks, where 'manipulation' seems to mean rotating and translating the 3D model (a skull), and 'marking' seems to refer to selecting small targets or points on the surface of the 3D model.  (These terms are not standard in the HCI literature: in other contexts or other papers, 'manipulation' could refer to changing an object's shape, and 'marking' could refer to drawing or writing or annotating.)  The experiment was done in VR and involved 12 participants.  The 3 user interfaces compared were:

1. 'mouse': a single 2D mouse, where the right button was for rotation of the 3D model, the scroll button was for translation of the 3D model, and selection was done with some kind of raycast and the left button (the raycast is shown in figure 1, right, but it's unclear if/how the ray could be tilted up/down)

2. 'hands': 2 bare hands, where the 'left' hand could pinch to grab the 3D model and then translate and rotate it, whereas the 'right' hand held a virtual stylus device (figure 2 right) used to point at the target and click a virtual button to select a point on the surface

3. 'controller+stylus': 2 hardware devices (figure 3), where the controller in the 'left hand' could rotate and translate the 3D model, and the hardware stylus in the other hand could point at and select a target

The submission seems to be unaware of the literature on bimanual interaction (according to Google Scholar, Guiard 1987 is cited more than 1000 times, and Balakrishnan + Kurtenbach (1999) is cited more than 200 times) which would predict that the 'mouse' condition would not perform as well as the bimanual techniques ('hands' and 'controller+stylus').  The submission also never mentions Fitts law (searching for the keywords: fitts raycast 3d, in Google Scholar, https://scholar.google.com/scholar?q=fitts+raycast+3d , finds more than 200 references).  Fitts' law predicts the time to select a target as a function of the distance to the target and the size of the target.  In the submission's experiment, users were asked to select points as close as possible to the targets, however the Fitts' law literature tells us that there is a tradeoff between the time spent pointing toward a target and the precision of the final location (=size of target), and that furthermore, different users tend to choose different tradeoffs in time/precision.  The Fitts law literature also proposes a way to calculate 'effective' target size, to normalize data from different users and compare them more directly (section 3.4 in MacKenzie 1992).  In summary, there is a lack of connection between the submission's topic and existing literature.

A separate problem with the submission is the lack of details reported about the user interfaces, making it impossible to reproduce the experiment.  Figure 1, right, makes it appear that the ray was somehow emitting from the front of the mouse, but it's unclear if there was any way to tilt the ray up/down or if the user had to translate the 3D model of the skull to be in front of the mouse.
In the 'hands' user interface, we read that "the [virtual] pen moved according to the moves [movements] of the palm", but this is vague, and it is unclear how difficult it was for the user's finger to press the virtual button of the pen, and if pressing the virtual button could disturb the orientation of the virtual ray being cast by the stylus.  There is no video to demonstrate the techniques.

Based simply on the literature on bimanual interaction, and the fact that bare hand interfaces tend to suffer from tremor/noise, I would predict that the 'controller+stylus' interface would yield the best performance.  Thus, the submission's results in favor of 'controller+stylus' are not surprising.

There are many variations on the user interfaces that are possible, many of which have been investigated in previous literature, such as the Rockin' Mouse (Balakrishnan et al. 1997, cited more than 200 times) which is designed to select in 3D while avoiding the fatigue that comes from lifting ones arm into the air.  Another possible variation on the proposed techniques is that, during rotation of the 3D model with the non-dominant hand (the 'left' hand), either a bare hand or a hand holding a hardware controller device, we can imagine that if the user wants to rotate the 3D model 180 degrees, it can be difficult to rotate their hand through such an angle, so simply applying a multiplicative gain factor to the rotation could help.  Also, when moving a raycast to select a target in 3D, the gain factor can be made proportional to the speed of the mouse or speed of the device, to make it easier to make small adjustments when the ray is close to a small target.  Another possible variant would be to have the user hold a hardware controller in their non-dominant hand (so they can move the 3D model with 6 degrees of freedom) while performing rayrast selection with a mouse in their right hand (avoiding fatigue from lifting their arm, and avoiding hand tremor which woud be filtered out by the mouse's mass).  Another variant could allow the user to make the 3D model of the skull bigger, to make it easier to point at a precise location on the surface.  All of these variants have been studied, in some fashion, in the extensive literature on Fitts law and pointing techniques.  The user interfaces chosen for evaluation in the submission lack a convincing justification given all the alternatives available.

Guiard, Yves. "Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model." Journal of motor behavior 19, no. 4 (1987): 486-517.

Balakrishnan, Ravin, and Gordon Kurtenbach. "Exploring bimanual camera control and object manipulation in 3D graphics interfaces." In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pp. 56-62. 1999.

MacKenzie, I. Scott. "Fitts' law as a research and design tool in human-computer interaction." Human-computer interaction 7, no. 1 (1992): 91-139.

Balakrishnan, Ravin, Thomas Baudel, Gordon Kurtenbach, and George Fitzmaurice. "The Rockin'Mouse: integral 3D manipulation on a plane." In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems, pp. 311-318. 1997.

Casiez, Géry, Daniel Vogel, Ravin Balakrishnan, and Andy Cockburn. "The impact of control-display gain on user performance in pointing tasks." Human–computer interaction 23, no. 3 (2008): 215-250.

König, Werner A., Jens Gerken, Stefan Dierdorf, and Harald Reiterer. "Adaptive pointing: implicit gain adaptation for absolute pointing devices." In CHI'09 Extended Abstracts on Human Factors in Computing Systems, pp. 4171-4176. 2009.
review_title: comparison of selection techniques for VR

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The paper is presents a study in which 3 different 3D object manipulation techniques are presented: mouse, tracker+stylus, and hands. 

I am not aware of any studies in which a similar technique (3D controller and stylus) were used, and thus the study has an important practical value, as it can generalize to other  combinations of devices of the corresponding type. I cannot identify any obvious flaws, and the paper is well written; however, please consider revising how the stats results are presented: Chi2 = XX, df = 2, p < .0XXX (df and p in italics; Chi is a Greek letter and not in italics).
review_title: Seems like a clear accept after trivial issues are fixed.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper reports on a study of three input devices (mouse, hands, controller+stylus) for pointing at a 3D object in VR. The results of the study are that the mouse was the most accurate, but users preferred the controller+stylus. The paper's contribution is in adding empirical data to the existing area of pointing performance for 3D objects, although the performance results are primarily a reinforcement of what is already known (i.e., the mouse is accurate). 

There are several general problems with the paper that reduce the novelty and authority of the contribution:

1. Related work. The paper has a very brief related-work section that focuses on the input devices themselves rather than on the problem of pointing to 3D objects. The problem of pointing in 3D has been considered for a long time in HCI, with many important insights reported in prior work (e.g., performance of different devices, performance of different interaction techniques, methods for improving pointing, etc.). Here are some example references that have not been included in the paper, but should be considered in terms of the framing of the research problem (in addition, there are numerous performance results in prior literature that should have been considered when designing the study)

Hinckley, Ken, et al. "A survey of design issues in spatial input." Proceedings of the 7th annual ACM symposium on User interface software and technology. 1994.

Goble, John C., et al. "Two-handed spatial interface tools for neurosurgical planning." Computer 28.7 (1995): 20-26.

Baloup, Marc, Thomas Pietrzak, and Géry Casiez. "Raycursor: A 3d pointing facilitation technique based on raycasting." Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 2019.

Dang, Nguyen-Thong. "A survey and classification of 3D pointing techniques." 2007 IEEE international conference on research, innovation and vision for the future. IEEE, 2007.

2. Experimental setup. The authors do not provide adequate detail to judge the setup and realism of the experiment. For example, there is no information about whether head tracking was used (i.e., whether participants could use head parallax to look around the object) or whether the VR headset was only used as a display (which to some degree defeats the purpose of using a headset). Second, using a visible marker on the 3D object means that much of the task performance would be governed by the participant's ability to look for the target (and the techniques' ability to support this manipulation) rather than the targeting time. The authors do state that these two aspects are combined in the task, but because they are so different in terms of the device characteristics, these need to be better separated in the study report. Third, the tasks are intended to "simulate a task that medical professionals would do during virtual surgery planning", but it is clear that medical professionals do not search for existing targets in order to point to them. Fourth, more figures are needed to clearly indicate how each of the devices / techniques worked in the study.

3. Study results. The performance results of the study are primarily that the mouse is more accurate than the Hands technique. No results are given for the other comparisons, and in general there is a lack of detail in the reporting of the results (e.g., reporting NS results). Finding that the existing standard input device is better than novel devices does underline the strong performance of the mouse, but adds little to our knowledge of how devices affect 3D pointing. There are some odd statements presented regarding the results, e.g., "Haptic feedback was provided with Mouse", but nothing was introduced to indicate that this was a haptic feedback device (it may be that the authors simply mean that the mouse is a physical object, but this is not the typical use of "haptic" for this area of research). There is little discussion of how the results will generalize to actual surgery-preparation tasks or to other 3D pointing contexts.

Finally, there are numerous writing problems (grammatical problems, odd constructions, difficult-to-follow sentences) that should be corrected. Overall, the results of the study do not provide a clear and valuable contribution for HCI researchers or practitioners.



review_title: Several problems add up to a weak contribution overall

Decision: Reject