dataesr/scientific-paragraphs-categorization huggingface dataset:
Tasks Both experiments rely on the same tasks. For each mapping, participants had three training trials with simple shapes to ensure that they were well-trained. The first training session was the longest (up to 30 minutes) as participants also had to learn the tablet's interface and the tasks. Then, after a break, they performed the selections on the three datasets (Fig. 3 ), and we measured the quantitative metrics. For each dataset, participants performed two consecutive trials, resulting in six trials per mapping per participant. We asked participants to select the red and avoid the blue dots, and to be as accurate as possible without noticeably slowing down. To avoid overly long trials, we explained that it was nearly impossible to be perfectly accurate. We rendered selected dots with a lower saturation (i.
Experiment 1: Best AR Mapping We recruited 18 participants (ages 20-45, AVG=26, SD=5. 8) to study the three AR mappings in our first experiment, pre-registered [BPSS * 21, Har22] at osf. io/pwauq. We documented differences with the pre-registration on OSF (see Errata. docx). Participants performed all the trials for a mapping, before moving to the next one. For a given participant, we presented the datasets in the same order for each mapping. We counter-balanced the dataset order using a cyclic Latin square on (P ID /3) mod 3 (integer division), and the technique order on P ID mod 3, where ID is unique and ranges from 0 to 17. After each mapping, participants answered the corresponding part of the questionnaire (see additional materials). At the end of the experiment, participants answered the remaining general questions.
Results We gathered participants × conditions × datasets × trials = 18 × 3 × 3 × 2 = 324 data points. We report our results using estimation techniques based on 95% confidence intervals (CIs) instead of p-values [Dra16, AGM19, BD19]. We refrain from dichotomously interpreting our results to follow current best practices [BD19, CDBG20, HHC * 21]. Except where stated otherwise, we computed CIs from bootstrap resampling and computed all within-subjects pairwise comparisons by bootstrapping geometric means on ratios to emphasize effect sizes. We removed one trial (P ID ; Mapping; Data; Trial) = (P 1 ; RA;Cylinder; 1) as the participant selected only a very small portion of the data, surely due to miss-clicks with the buttons "reset selection" or "end-task. " Completion Time. We started the timer at the beginning of each trial, which we paused during the computations of volumetric selections.
To correct for positive skewness, we analyzed log-transformed speed metrics  [SL10]  and present the CIs of their anti-logged results (Fig. 6 ), which is standard for such data  [JDF13, BIAI17]. We did not find evidence of one mapping being faster than another, not supporting H1. We experienced a learning effect (Fig. 7 ) as participants tried to understand and find a strategy for each dataset to solve the problems during their first respective mapping, before sticking to it for the others. Especially for the Galaxies and the Spring, the possible solutions were not obvious to find or to apply, when compared to the Cylinder where participants understood directly that they (1) can select the outer part by extruding a cylinder and (2) remove its inner part with a smaller cylinder. To a lesser extent, users got used to the devices as all techniques use the same input M and output O modalities, strengthening again this learning effect.
We found strong evidence that the Spring dataset was the longest to complete, which participants solved using two strategies. Some defined it mathematically using constrained Boolean operators. The number of operations is here higher than in the other datasets. Others followed the Spring using 6 DoF unions, which require slow movements as unconstrained operations need more control for accuracy. Workload. We used the NASA-TLX  [Har06]  questionnaire to measure the participants' workload. We performed pairwise comparisons as a within-subject design to calibrate participants (Fig. 8(b) ). Fig. 8  shows strong evidence that NA felt the least performant, some evidence that RA felt the most performant, and some evidence that RA required less effort than RF to participants. Finally, we found no evidence of a difference between NA, RA, and RF w.
This is especially true for NA (Spring and Cylinder), which we explain by its level of directness with respect to the AR view. As the Galaxies dataset contains much empty spaces compared to the others, participants could   see the screen of the tablet as they interacted within the dataset. This may explain why they focused more on the tablet in the Galaxies dataset compared to the others using NA. Participants marked both answers in the questionnaire if they focused similarly on both views. Constrained. The level of directness may explain why NA and RA might invite more unconstrained operations than RF (Fig. 11 ). We were surprised to see that participants solved the Spring primarily using constrained operations, while we designed it to encourage unconstrained ones. P 0 , P 9 , and P 17 stated that it was difficult to perform unconstrained operations for this dataset, for which they spent a lot of time.
P 0 found RA and RF to be similar in most cases where she relied on constrained operations. In our video recordings we saw participants P 0-1 , P 3-5 , P 7-10 , and P 12-16 , in some trials with RA and RF, operating the 2D virtual plane in a sideways fashion. This gave them a better perspective (and thus accuracy; H3) compared to the most direct mapping (NA). It is also useful, e. g. , for the Cylinder whose inner part relies on two parallel planes easily visible from the side. In particular in RF, P 3-6 , P 10 , and P 12 aligned the virtual tablet with the cylinder's circle side in the Cylinder dataset. They then physically moved around it to face the virtual plane at an angle of 90 •. They then moved the tablet during the extrusion toward the cylinder's rectangular side, i. e. , the tablet's normal.
In the Galaxies data, users can select the red galaxy by extruding along the normal of the almost-planar blue galaxy, where users stop to maximize their accuracy. P 3 , P 9-10 , and P 17 rotated the dataset before starting the selection. P 17 , in addition, lowered the dataset to put the red galaxy at a comfortable position. He then selected it from top to bottom. P 0 , P 3 , and P 4 placed the virtual plane near the blue galaxy, before selecting the red one which was on top. They kneeled or lowered themselves to place the virtual plane and check its position. Finally, P 11 selected the red galaxy using the unconstrained mode. In RA, he focused on the AR view sideways while moving the tablet around.
Discussion Our participants mainly performed constrained operations, which correspond to target selection tasks. Reflecting on studies about Fitt's law [Bal04, MAK * 88] and P 9 's comments about her screen focus, we hypothesize that users focus on the AR view during their initial coarse but fast movements (ballistic phase  [GH88] ), and later on the tablet for precise movements. Indeed, the HMD facilitates a good 3D understanding of the dataset and its spatial relationship with the virtual plane. These facilitate fast movements towards the target, while the tablet, with its near-clipping plane, renders a precise position of where the virtual plane is within its neighborhood. We considered this hypothesis in the design of our second experiment. The NA condition was not the preferred one (H5) mostly because of a lack of scene overview and because the AR hologram is rendered on top of the tablet, making both hardly readable.
Experiment 2: AR vs. Original We recruited 18 participants, as pre-registered at osf. io/rvpuc. We had to replace one participant with another as this person did not respect our experiment protocol, but we include their results on OSF. The second experiment used the same protocol, tasks, and datasets as the first. We counter-balanced techniques (our AR technique and the original 2D one) using cyclic Latin squares on P ID mod 2 and datasets on (P ID /2) mod 3. In the 2D condition, participants could use the complete set of translations provided by the FI3D widgets. Participants (ages 21-36, AVG=25, SD=4. 1) were unfamiliar with AR (1-2 on a 5-point Likert Scale, AVG=1. 4, SD=0. 50) and VR headsets (1-3 on a 5-point Likert Scale, AVG=1. 7, SD=0. 69), but rated the VICON tracking as accurate (3-7 on a 7-point Likert Scale, AVG=5.
4, SD=1. 1). We hypothesize that: H6 AR will outperform 2D both in cognitive load and participant preferences as participants now have better depth perception through the 3D stereoscopic view, and that AR is more direct than 2D when focusing on the AR view. H7 AR will be more accurate than 2D for the same reasons, and because the input modalities are similar. H8 AR will be faster than 2D as AR allows participants to place the virtual tablet in a 1:1 mapping prior to extrusions, compared to 2D which needs to either place the tablet using relative translations, or to move the dataset using the FI3D widgets. H9 During extrusions, participants will use the AR view for coarse movements and use the tablet's view for refinements.
Results We use the same reporting and analysis as for our first experiment. Completion Time. We did not find evidence that one technique would be faster (Fig. 13 ), not supporting H8. We again observed a learning effect across all datasets between the first and the second techniques as ordered per participant (Fig. 14 ). Finally, we again see strong evidence that the Spring dataset was the slowest to solve. Workload. We found strong evidence that users felt more performant, were less frustrated, needed to work less hard, and needed less mental effort at solving tasks using AR compared to 2D (Fig. 15 ). We also note that the difference is large in most cases (Fig. 15(b) ). This may be due to the AR view offering strong spatial cues about the spatial relationship between virtual tablet and 3D visualization and helps users to understand the 3D visualization, and due to the difference of the directness of the techniques.
These diverging results may come from the participants' strategies concerning the use of constrained operations. Constrained. When examining the ratio of useful constrained operations over total number of useful operations (Fig. 17 ), it seems that AR calls for more unconstrained operations compared to 2D (Fig. 17(b) ). This may be due to users having a better 3D understanding of the dataset in AR, especially for the Spring, and thus making them willing to work with 6 DoF. Moreover, the level of directness of RA (AR) over 2D may invite more 6 DoF operations. However, unconstrained operations for the Spring are less accurate than constrained ones. This may explain the difference in accuracy. Participants performed unconstrained operations in a Naïve Approach in AR: they placed the tablet near the Spring, drew a circle, and again replaced the tablet near the Spring before extruding.
P 0 tried twice to solve the Spring in unconstrained mode in AR. He then tried and succeed at solving the Spring using the constrained mode. P 9 used constrained operations for the Spring in 2D (his first condition) and used unconstrained operations in AR, where the 3D geometry was better understood. P 6 and P 8 used unconstrained operations in AR for their first trial of the Spring, but reverted to constrained operations for all the three other trials involving the Spring. screen) when placing the virtual tablet before starting an extrusion. Fig. 19  also shows the AR view supported our participants better than the 2D view. In AR, the 3D view gives strong spatial cues about where the virtual tablet resides w. r. t. the dataset, compared to the 2D screen. In 2D, however, the camera associated with the FI3D widgets on the tablet matches its virtual position, which helps users to orient the dataset with respect to the virtual tablet.
Preferences. Fig. 20  shows strong evidence that users prefer AR over 2D. Participants could rate each technique as being equal,  which we encoded with a value of "1 st position" for both techniques. The effect size, however, is weaker for the Cylinder dataset. As 2D allowed participants to manipulate the 3D visualization with respect to the static virtual tablet, some first oriented the Cylinder to face the virtual tablet along its extrusion axis and then moved it along the depth axis. Using the constrained mode, they only needed to move the tablet forward to first add the red dots, and then to remove the inner part of the Cylinder, which is a simple strategy. Observations. Extrusion in AR: P 0-6 , P 8-14 , and P 16 primarily focused on the AR view when extruding. P 0 and P 10 said that they looked at the tablet only to remove the inner part of the Cylinder, which helped them with its near-clipping plane.
Interestingly, P 8 used the AR view to check when to stop and if she was doing as expected, and used the tablet to know where to start using its nearclipping plane. P 1 used the AR view to understand the relationship between the virtual tablet and the dataset. She said that she rarely looked at the tablet compared to 2D, except when she needed accuracy. P 5 used the tablet only to check if he had reached the end of the extrusion. P 13 stated that constrained selections are like sliding movements, and he did not look much at both screens, except to check the start and the end of the movement on the AR view when required, e. g. , with the inner part of the Cylinder. P 2 , P 6 , and P 9 focused on the tablet only during unconstrained operations (for the Spring), to check that their lasso was aligned with the circle the orthographic view of the tablet renders.
P 2 also commented on the conflict between the AR and tablet views when both are merged. In addition, some participants used the AR view during the whole process or to refine their movements. P 2 looked at it to check when the extrusion should end. P 3 , P 4 , P 11-12 , P 14 , and P 16-17 used the AR view during the whole process. Finally, P 7 and P 15 focused on the tablet most of the time and used the AR view to check their selections. P 15 said that he especially looked at the tablet at the beginning and the end of his selections. Overall, we found conflicting evidence regarding H9, depending on participants' preferences and strategies. We therefore cannot conclude anything about it. Extrusion in 2D: Before starting the extrusion, some participants sometimes placed the dataset (using the FI3D widgets) with respect to the fixed virtual tablet, instead of placing the virtual tablet with respect to the fixed dataset.
They first oriented the dataset correctly, before translating it on the depth axis of the virtual plane, resulting in few but accurate operations for the placement before starting an extrusion. P 13 commented he used the 2D to adapt the depth position (with respect to the virtual tablet) of the dataset, before starting an extrusion. Once the dataset and the virtual tablet were aligned, P 0 used "Tangible Rotation" of the dataset in the 2D condition to arrange the plane of the virtual tablet to be parallel to the view shown on the external screen, for Cylinder and Spring (where he used only constrained movements). This allowed him to reduce the problem along one dimension to remove, e. g. , the inner part of the Cylinder. P 7 used the same strategy all over her 2D condition trials. This "Tangible Rotation" can thus reduce the users' workload and can improve the original technique.
Discussion While we did not find evidence that AR would improve the users' accuracy and speed, our results answer our main research question about the users' workload (H6). Users largely preferred and felt more comfortable using a 3D AR view over a 2D screen. By providing them with an AR-HMD, we shifted the users' attention from the tablet to the AR view. This is a fundamental difference compared to the original Tangible Brush where most users primarily relied on the view of the tablet for almost everything. Our evidence shows that the AR-HMD and the tablet can be used jointly with different available strategies. Some users used the near-clipping plane of the tablet to be accurate, and others relied on the AR-HMD for most interactions. We saw three strategies to place the virtual tablet with respect to the 3D dataset.
The first one relies on the FI3D widgets. Designers may consider what parameters to give to the FI3D camera, which should depend on the users' tasks (e. g. , placing the dataset in the 3D space or with respect to the virtual tablet). While Besançon et al. 's [BSA * 19] participants seemed to have troubles with rotating their datasets using tangible rotations, we did not experience such an issue with the FI3D widgets. The second strategy is to place the virtual tablet in a 1:1 mapping using the "Position" mode. The last one is to remotely control the virtual tablet using the "Tangible" mode to apply the current mapping. More research is needed to understand which strategies work best and under what scenarios.
Moreover, past research focusing on docking tasks for touch displays show that users generally decompose the task into a translation task and a rotation task, instead of doing everything simultaneously  [MCG12, BBC13]. This probably explains why some of our participants used the FI3D widgets instead of tangible operations to align the virtual tablet and the dataset correctly in the 2D condition, as the FI3D widgets facilitate such a decomposition. While most research so far highlighted that DoF integration and separation depended on the input technology [LJKM * 17], our findings suggest that the degree of immersion of the output modalities also matters, as we saw that 3D AR views seem to invite more DoF integration. These results are particularly important when considering how interaction paradigms such as tangible input and mid-air gestures are praised for their "naturalness"  [IU97].
At the same time, however, they should also distinguish visualizations where users need large overviews to those that need small overviews because spatially large representations combined with a relatively small mobile device would require constant focus switches. Except for the Galaxies, all our datasets are dense and require large overviews. A future study would shed light on different interaction designs for sparse datasets that require small overviews. Limitations. Despite our careful design, our experiments have limitations. Our measurement of task completion time is a major one. Through our observations of the videos, it seems that participants spent a lot of time in their first trial with a dataset attempting to understand it and trying different strategies, before sticking to a suitable one.
However, VR environments benefit from a better management of visual conflicts between the virtual datasets and the physical tablet. We then compared one remote adaptation in AR to the original Tangible Brush setup in a second experiment (Sect. 6). We found strong evidence that AR views, for 3D volumetric selections, lower significantly the users' workload compared to the 2D condition, while having similar accuracy and speed. Our results also show that users behave differently in AR environments compared to 2D ones, which should be further investigated in future work.
Extrusion with the AR Tangible Brush Next, we discuss the three AR mappings we described in Sect. 3 for the actual selection input using Brucker et al. 's model. shows the Tangible Brush interface (Fig. 5 ). "End Task" ends the current task. "Reset Geometry" replaces the dataset at its default position and orientation. The FI3D widgets surround the orthographic visualization of the dataset. In this screenshot, we show the Spring dataset as seen from its front face.
0 × 4. 7 × 2. 0m 3 (surface× height). We used the VRPN communication protocol to retrieve the tracking metrics of both the HoloLens and the multi-touch tablet in real time within the VICON coordinate system. We launch the server with the participant's ID as a parameter. The server loads the needed dataset depending on how far the participant is in the study. We use a homemade protocol on top of TCP/IP for the communication between all devices. We do not perceive any latency as we rely on a local network and do not buffer the data before sending it as Ethernet packets. We keep all relevant information on the server, to which the tablet and the HMD are paired when they connect, and they also are paired to each other via a dedicated IP entry step. The server maintains all relevant information and logs all events in JSON format, to allow us to extract metrics such as speed and accuracy using Python and R.
Datasets Users can move and rotate 3D visualizations on the tablet via FI3D [YSI * 10] (Fig. 21 ). In the AR conditions, the camera associated with the FI3D widgets faces the current visualization to always have it in sight. As the main view is the AR space and because the user can physically move around, parameterizing the camera based on the user' position would be unusable. In our replication of the original Tangible Brush, however, the camera associated with the FI3D widgets follows the virtual tablet position and orientation as this condition always relates to this virtual tablet that is frozen when users do not extrude anything. We start the application with the visualizations at a fixed position. Participants can relocate the visualization vertically (y-axis) for comfortable interaction, but we locked motions along the xand zaxes to maintain similar conditions across participants in AR.
In the condition that replicates the original implementation, however, we allowed participants to move the dataset along all axes because, in the AR conditions, users can move horizontally on the ground, which acts as xand z-translations that were impossible in the original. Rotations can only happen around the xand y-axes, with the x-axis being defined by the user's current position in the physical space in the AR conditions. While one can argue that a z-axis rotation can be useful, we note that the drawing of the lasso can be adjusted on the tablet, replacing a z-axis rotation. We made this choice to keep all the possible interactions (rotations, moving, scaling) simple in our overall system. Finally, we allow users to reset the default positions and orientations for both the 3D dataset and the virtual tablet.
For a dataset of 160K points (i. e. , the galaxy collision dataset in our study; Fig. 3 ), the resulting mask represents (160 • 10 3 + 7) / 8 = 20 kilobytes of data, which does not prohibit real-time interactions. When a user validates a selection, we consider all volumes and their associated Boolean operators. For each data point, we determine if it is inside or outside a given volume using raycasting by counting the intersections (even intersections for inside, odd intersections for outside) of a ray from the point to a random direction with triangles of the enclosing selection shape. We also optimize this process with a triangle lookup in a 16 × 16 × 16 grid. We then cast the ray from the point (which is at a position (i, j, k) in this 3D grid, and (x, y, z) in the 3D world), parallel to the x-axis.
Replicating the Original Tangible Brush Implementation As optical see-through displays cannot render black colors, we use a dark green background color to simulate the opaque virtual screen we used to replicate the original technique. We set the virtual tablet's position by default at the front-top-left corner of the 3D cube encapsulating the data, facing the center of the cube. With this choice, we do not align the mathematically generated datasets that have axis-aligned features with the default tablet's position and orientation (along x-, y-, and z-axis). As in the original, on the virtual screen's left side we render a perspective view from the point of view of the current virtual tablet. Its field-of-view (fov) depends on the current size of the virtual tablet: verticalFOV = atan ( 3 • sizeX ), with verticalFOV in radian and sizeX (the virtual tablet's width) in meter.