dataesr/scientific-paragraphs-categorization huggingface dataset:
We hope to bridge this gap, paying special attention to spatial user interfaces. We believe there is significant potential to make 3D interactive visualization systems more effective by leveraging new readily available sensing technologies [Bes17, LKM * 17] and adapting 3D interaction techniques developed in other contexts  [JH13]  to work for the spe-cial needs of interactive data visualization tasks. Such an approach would make use of the skills to interact with the physical 3D world that people naturally possess, and, thereby, have potential for great positive impact since so many important datasets have an inherent 3D structure: data acquired from simulations as well as spatial data, medical data, or biological data. To contribute to this future, this state of the art report surveys the spatial 3D interaction techniques that have been presented in the literature, presents a task-based framework for guiding new research on vis-specific spatial 3D interaction techniques, and repeats the call for additional research on spatial 3D interfaces specifically to support 3D visualization tasks.
Spatial 3D datasets are particularly challenging to visualize. Unlike general 3D interaction, visualization of 3D datasets is less focused on creation than it is on sense-making. Making sense of 3D datasets requires an ability to manipulate the data or the view, to select in 3D specific regions of interest, and to place and manipulate visualization widgets to better understand the inherent structure of the dataset or some of its internal properties. While 3D interaction techniques address some of these challenges on pre-defined objects, 3D visualization techniques should enable users to achieve all operations on non-predefined structures. This additional requirement is not satisfied by most of the classical 3D interaction techniques when used in a spatial visualization context. Moreover, 3D interaction, such as manipulation, selection and annotation, becomes more challenging when applied to complex features or structures of 3D VIS datasets, especially when more precise interaction is needed.
For instance, the survey from Oeltze-Jafra et al. [OJMN * 19] focuses on medical data generation and its analysis without highlighting the large body of work done on interactive visualization tasks for 3D datasets. Our paper is focused on spatial interfaces, which include interaction paradigms such as: tangible interaction (e. g. , [HPGK94, DGHCM * 03, SGF * 11, JLS * 13, IGA14b, HRD * 19]), mid-air gestures (e. g. , [GPC11, TTSI * 12, LGK * 13, NHH * 16]), haptic interfaces (e. g. , [LCP * 07, PCT * 07]), or hybrid interaction paradigms (i. e. , combining several input technologies; e. g. ,  [PTW98, OBF03, SSSS11, BIAI17a] ). These spatial interaction paradigms provide several theoretical advantages: tangible interaction mimics everyday interaction with the real-world  [Fit96, BIAI17b] , mid-air gestures enable hands-free interaction with medical data during surgeries [NHH * 16], and hybrid interaction leverages the benefits of multiple interaction paradigms  [Bes17].
Tangible props may take the form of the data, serving as both a physical representation of data and a means of interacting with the data, or their physical form may be more abstract, providing passive haptic context or support for the interface, but without much visual feedback on the prop itself e. g. , [HPGK94,GHP * 95,FBZ * 99, IGA14a, JLS * 13, IGA14b]. Extending beyond passive haptic aids, we also include active haptics in this paradigm. Haptic devices enable 3D manipulation and tactile feedback within a restricted interaction space (due to the limited range of robotic arms, cables, etc. ). Manipulations with these devices can be programmed to feel realistic, as they would in the real world, or "extra" effects can be added using programmatically controlled vibrations and forces. This property has been used for visualizing 3D datasets.
The first two categories are out of the scope of this paper as they tend to be used for browsing through 2D images [HPMD13, YBM13, NHH * 16] or controlling medical instruments [NHK * 03] rather than manipulating 3D medical datasets. However, hand and body gestures are quite relevant. They have been used for visualization tasks (e. g. , [KRW * 11, RRA * 12, Gal13, YBM13]) but also for a variety of other tasks in the operating room, as recently surveyed by Mewes et al. [MHWH17].
Task However, Volumetric view and object manipulation goes beyond "simple" 3D manipulations and raises specific challenges that are not typically present in general spatial interfaces for navigation (e. g. , redirected walking techniques, WIMs, wand-based flying). While simple rotations and/or translations make it possible to view 3D data externally, many 3D spatial datasets are dense, and relevant internal aspects of the data are, therefore, naturally occluded. Interactions for Volumetric view and object manipulation should directly address this need. Cutting planes or transfer function editors are often used for this purpose, and since these are widget-based, one might consider these as falling under a visualization widget manipulation task. However, from the standpoint of the user's cognitive approach they are tied so tightly to view manipulations (e.
g. , moving the camera inside volumetric data necessarily involves clipping) that it can be useful to think of these as integral Volumetric view and object manipulation tasks. In fact, we argue that this is the type of insight that is useful when determining the best ways to translate 3D user interfaces created for more generic 3D environments (e. g. , architectural walkthroughs, simulations) to 3D visualization applications. Many 3D visualization view manipulations consider only cutting planes to slice through the data, but it is interesting to notice that some experts might need non-planar or free-form surface slicing of their data to provide an easier and more natural analysis of some datasets [PTH98, GPB99, PSOP01, MFOF02, MFF03, SGH03, KVLP04, RRRP08, REM11, KGP * 12, LSG * 16, PCE * 17].
These approaches can be linked to techniques such as peeling, which can be useful for surgical planning [SGH03, KVLP04, BHWB07, MRH08, REM11, HMP * 12, PCE * 17], but can also be used in other domains, such as reservoir visualizations  [SSSS11]. Non-planar slices are often defined relative to the data but can also be specified with significant user input (e. g. ,  [BHWB07]  Beyer combines 2D mouse input with medical scan data). Specifying, modifying, and positioning non-planar slicing objects or without relying on underlying data poses an interesting challenge for spatial interfaces, and this is a topic that we return to in later sections (see Sect. 3. 2 and Sect. 3. 1). Another challenge is the manipulation of data and cutting planes with axis-based constraints  [BKBN12]. 3D visualization users must often manipulate/zoom heterogeneous datasets, including many manipulations along a single axis  [FGN10] , and more generic 3D manipulation techniques, as typically studied in the user interface literature, do not often address this latter point.
Automated algorithms have been developed to minimize occlusion (see e. g. , [TEC * 16]), but the issue can be avoided altogether with the help of interactive placement. Interactive visualization widgets have also been used in other contexts. The Glyph Lens technique uses a magic lens effect to overcome issues of occlusion for viewing volumetric tensor fields  [TLS17]. A full overview of lenses and their use in visualization is available in the survey from Tominski et al. [TGK * 17]. In addition to these primitives, domain expert sometimes need to assess the values of specific points in their datasets, a feature that is often implemented with a probe widget. Interactively positioned 3D probe widgets have been used to facilitate the computationally-heavy inspection of 4D MRI Blood-Flow [vPOBB * 11] and other complex data [MEV * 06, KGP * 12].
The widget contains multiple interaction zones and is oriented to match the orientation of the scene being viewed. One-finger manipulations along the edge of the cube translate along the parallel axis. One-finger manipulations on the sides of the cube control single-axis rotations. Scaling is controlled by pinching the cube, on the cube sides for uniform scaling while a pinch gesture and on opposite edges will initiate a non-uniform scaling. In an other study  [LODI16] , the tBox technique was found to increase the feeling of precision for 3D interaction. As noted by Yu et al. [YSI * 10], dataset-specific interaction techniques are sometimes needed. Fu et al. [FGN10]  present an example, combining trackball rotations with a custom "powers-of-ten-ladder" (see Fig. 2b ). The technique facilitates exploration of astronomical datasets, which require rotations and scaling operations that span large magnitudes (translations are less useful in this scenario).
Arcball rotations are controlled using a single finger, panning operations are controlled with 5-finger gestures, and zooming operations are controlled using a bimanual two-finger pinch. Two-finger inputs activate the ladder widget, where each region corresponds to a power of ten zoom level. Fu et al. created this technique to alleviate the strain on users' hands when performing large zooming operations in astronomical datasets. While Fu et al. extend more traditional tactile interaction to support the special needs of astronomical spatial data, Kim et al. designed a tactile interface to support the special needs of navigating through and comparing spatial datasets that change over time [KJK * 15]. The approach, applied to historical architectural reconstructions across different time periods, combines a timeline widget, multi-layer map-based navigation, and immersive visual-ization with staged, animated transitions between datasets.
Other dataset-specific, or dataset-inspired, interfaces include the work by Sultanum et al. [SVBCS13] , which addressed the challenge of navigating within geological outcrops via a two-step technique; users first indicate a navigation surface onto which the camera will be constrained, and touch gestures are then used to tilt, zoom, or pan the camera with respect to the x-/y axes. Finally, some tactile interactions for visualization take the approach of augmenting tactile 2D input with additional inputs. This has most frequently been done on tabletops and in hybrid virtual environments (e. g. , [BI05, SAL06, HIW * 09, MJGJ11]) with hand tracking to augment touch input. Jackson et al. [JSK12]  applied this concept to 3D data visualization, using the posture of the hand above a 3D stereoscopic table to allow users to tilt, bend or twist datasets within the 3D space (see Fig.
Butkiewicz and Ware then use data exploration widgets called "dye poles" placed at the surface, with controls to create and manipulate seed point placement at varying ocean depths. Other widgets can be used to specify points or paths in 3D space, e. g. , using Butkiewicz et al. 's  [BSW19]  recent Pantograph technique. A different approach to particle seeding was taken by Klein et al. [KGP * 12]. They use a monoscopic display, but treat the cutting plane that they place in the projected view of a generic flow dataset as a proxy to specify 3D locations for seed point placement. This is combined with an unprojected view of the same cutting plane that acts as a widget. Using this widget, users can place particles around a small region (single-finger input), along a line embedded into the cutting plane (two-finger input), or around a larger circular volume (input from three or more fingers).
3D data selection & annotation tasks with tactile input In addition to view changes and data object manipulation, one of the most essential tasks in visualization is data selection and picking. While they can be achieved with established techniques such as ray casting (we review some of the main selection metaphors in Sect. 2. 2. 3) for datasets that consist of explicit objects, additional techniques are needed for continuous data, such as volumetric scalar fields, particle clouds, or flow fields. Tactile input selection techniques often mirror selection techniques developed for more traditional input modalities [Wil96b, AA09, AA13, Ban14], but researchers showed that the more direct style of control often possible with tactile input as compared to mouse input leads to benefit for visualization  [BIRW19].
Bonanni et al. 's Handsaw [BAC * 08] prototype made it possible to obtain slices of the data by interacting with hand-held objects (such as a laser). Despite the physical ability to move the handheld object in any direction, the virtual slices were restricted to move only along a normal direction. Spindler and Dachselt [SSD09] make the slicing plane itself physical by supporting interaction with a tracked, physical, paper-like prop (called PaperLens). Their hardware includes a 2D tabletop augmented with a projector and sensors. Multiple interactions are possible and are visualized by projecting imagery directly onto the paper. For example, users can select which layer of multi-layered data to view simply by changing the height of the paper with respect to the table. This constitutes another interesting example of constraining tangible interaction (or at least the interpretation of the users' interaction) rather than treating the 6-DOF manipulation of tangible objects quite so literally.
Another interesting use of tangible interaction for visualization is to use multiple tangible objects to represent different portions of the data. For example, the tangible system developed by Reuter et al. [RCR08]  used props to help archaeologists virtually reassemble fractured artifacts, like a 3D puzzle. Following a similar motivation, Khadka et al. [KMB18]  use hollow tangible props worn around the wrist to represent individual slices or fields of data. Users can add or remove these from the visualization by manipulating the props. Interaction using generic tracked VR controllers, AR markers, and the like can also be viewed as a form of tangible interaction as the shape of the controllers or surface the markers are printed on convey some tangible information, even if not dataset or taskspecific.
For interaction in AR, Tawara and Ono  [TO10]  relied on a simple visual marker to enable users to manipulate medical data with 6 DOF (see Fig. 4b ). In a Desktop AR context, markers have also been used as metaphors for cutting planes to provide arbitrary slicing position and orientation of volumetric datasets, such as tomographies. Moving beyond a flat marker, while still acting as a generic prop, Chakraborty et al. [CGM * 14] used a physical wireframe cube prop in AR for 3D manipulation of chemistry data. The cube is used as a container for the visualized dataset. Issartel et al. [IGA14b]  used a cuboctahedron to manipulate fluid dynamic data with 6DOF in AR, also proposing different slicing techniques for us with handheld AR visualization. The manipulated cuboctahedron is covered with markers and tracked with a tablet's camera.
Their approach enables slicing through the data by treating the tablet as a cutting plane or by using an optically-tracked stylus. Interaction with generic VR controllers is also common, and the research includes techniques for simultaneously manipulating views of multiple volumetric datasets or 3D scenes in order to support comparative visualization. Bento Box [JOR * 19] accomplishes this via a bimanual interface for quickly selecting and arranging sub-volumes of interest in a grid. Another approach, Worlds-in-Wedges [NMT * 19], accomplishes a similar task by combining a custom world-in-miniature interface with a pie-slice view of several worlds at once. In both cases, generic VR controllers provide 6 DOF pointing and grabbing inputs that are interpreted relative to the data. As mentioned in Sect.
4c ). The stylus and the dataset prop are visually tracked, thanks to visual markers, and a see-through tablet is used to provide Augmented Reality. Because the data are represented by a physical volume, the seed point origin must be offset from the stylus tip so that it can points can be placed inside the volume, but users still benefit from the tangible aspects and can push a button on the stylus to start emitting particles from the point of origin. A similar approach is used by Tawara and Ono  [TO10] , who make use of a wiimote augmented with visual markers to provide a seeding point origin visualized in AR with a head-mounted display. Similar to Issartel's approach, the location of the seeding point is not directly located on the wiimote, though this is not because of physical limitations in this case as the data is simply manipulated through a flat 2D marker.
A similar setup is used by Nyström et al. [NMVB09]. Gomez et al. [GJL10]  propose to facilitate selections with two tracked props, a In addition to guiding 3D drawing, haptic devices can use datadriven feedback to further assist with making accurate 3D selections, helping to overcome the challenges of occlusion and cluttering. Zhou et al. [ZCL08]  use a Phantom force feedback device with stereoscopic glasses to draw 2D lassos that are then connected to select DTI fiber tracts. Jackson et al. [JCK12b]  introduced Force Brushes, which uses progressive data-driven haptics provided by a Phantom to select subsets of 3D lines in dense datasets (see Fig. 5c ). Chakraborty et al. [CGM * 14] combined a Phantom and a visuallytracked cube prop in an Augmented Reality environment for 3D manipulation and selection of chemistry data.
For example, Kukimoto et al. [KNEK05]  use a tracked PDA in a collaborative VR environment to capture annotations. A single button press activates the note taking that is done by moving the PDA in the 3D space. Similar to this, Benko and Wilson  [BW10]  use an infra-red laser-pointer with dome-based visualizations of 3D graphs or astronomical data. In their system, a presenter can use the laser to write annotations directly on the dome surface, which are captured by a camera and integrated into the computer graphics renderings shown during public presentations. Using tracked 3D paintbrushes or generic VR controllers, in Scientific Sketching [KAM * 08] 3D sketches are used not just to annotate 3D datasets but also as a visualization design tool. The system aims at involving artists and other visual experts in the task of designing the most effective uses of color, texture, form, and metaphor for multivariate VR visualizations.
6b ). The visualized data is visible on a traditional 2D display. Since they implemented many different interaction techniques, they defined translations as twohanded (palm facing forward) concurrent manipulations. Zooming is achieved through the same posture but by moving both hands closer or further away. Rotations are also achieved with both hands, but with clenched fists. Similarly, Lubos et al. [LBLS14]  proposed a set of bimanual gestures to support 3D manipulation of point clouds for users wearing HMDs (see Fig. 6c ). Laha et al. [LB13]  proposed Volume Cracker, which consists of bi-manual gestures to crack open a dataset to explore its internal structure. The authors aimed to replace traditional slicing plane manipulations with this approach. Users have to close both hands to crack the visualized volume in two and then can manipulate each individual cracked part or iteratively crack them into smaller subsets.
Visualization widget tasks with mid-air input Our survey uncovered far less on mid-air gestural interaction interaction with visualization widgets. Nonetheless, as explained in Sect. 2. 1. 3, to date, most of the applications of mid-air gestural interaction involve 2D image browsing. An exception is the work of Gallo et al. [GPC11]  who proposed a specific gesture to facilitate measurements between points in 3D (see Fig. 6b ). Users have to keep a hand outside of the interaction space and use the other hand to point at specific locations. The locations are then recorded, and the distance is measured and shown to the users. Malkawi et al. [MS05]  also proposed to use mid-air gestural interaction to provide data read-outs (see Fig. 7a ). Among the many commands they propose to support with their system, they explored a specific gesture to allow users to obtain detailed information of a specific 3D point in space, visualized through an AR headset.
Designing custom devices is less common than utilizing the mobile devices that have, for more than a decade now, been able to provide both tactile and tangible input. The Natural Material Browser  [FFH13]  combines spatial/tangible manipulation of a tablet and multitouch gestures for volumetric material science datasets. The rotations tracked with the inner sensors of the tablet allow domain scientists to visualize different slices of the data on the tablet, while touch-interaction is simply used to either provide rotation information or change datasets and parameters (a system control task). Pushing this approach further, López et al. [LODI16]  investigated touch input for 3D manipulations on a tablet, in the context of an additional vertical stereoscopic screen. They provide 3D manipulations through tactile input using the tBox method  [CDH11]  on the tablet, but also add the possibility to rotate the data by directly rotating the tablet.
Moving away from custom-designed hardware, Veit et al. [VC14]  proposed 3D selection and annotation of the selected subsets with a spatially tracked multi-touch device in a 3D stereoscopic setting. Their 3D selection technique focuses on point cloud data and proposes to manipulate a selection sphere that is attached to a ray controlled by the tablet's movement. Tactile manipulation changes the size of the selection sphere on the fly. This approach is versatile and can adapt to different datasets and regions of interest. Pushing the versatility even further, Besançon et al. [SAIB16, BSY * 19, GSBI20] proposed to use a spatially tracked tablet to provide free form 3D selection of volumetric data. Their approach is not data-dependent and allows users to first trace a lasso on the tablet to obtain a 2D shape that is then extruded in 3D using 6DOF tablet manipulations.
This approach relies on a combined tablet and large screen and enables entirely user-controlled free-form selection. Although their research focuses more on the specific workflow of annotations and their review and modifications, Pick et al. [PWHK16]  combine custom CAVE hardware with a smartphone to support annotation tasks. Many different visualization systems rely on immersive displays, such as VR headsets. Taking notes can be particularly challenging in these environments, and hybrid interaction paradigms often provide a solution. The early work of Harmon et al. [HPRB96]  and Poupyrev et al. [PTW98]  was pioneering in this regard. They used a spatially tracked device to annotate (medical) data for VR environments (with headsets). The users could navigate the environment by walking inside of it, position themselves and the tracked-tablet at a specific location, and start their note-taking process with a pen on the tablet.
A need to focus on visualization widgets To facilitate discussions of the state of the art of 3D spatial visualization, we have gathered the approaches mentioned in our survey in Table  1 , following the same classification as in Sect. 3 (an aggregated version was presented in Fig. 1 ). To obtain this   1 , we clearly see that HCI and VIS researchers have rightfully investigated several interaction paradigms, each of which is valuable, especially considering that experts in different domains have different needs. For instance, surgeons are more likely to be interested in mid-air gestural interaction, while researchers in geology, fluid dynamics, or archaeology can make use of props or screens. Our final report on hybrid interaction paradigms (see Sect. 3. 4) and Table  1  also clearly highlight that researchers in VIS and HCI have been leveraging the potential of hybrid interaction paradigm for improving visual exploration of 3D datasets.