dataesr/scientific-paragraphs-categorization huggingface dataset:
In summary, the primary technical contributions of this article are as follows: • Decaf, the first learning-based MoCap approach for 3D hand and face interaction reconstruction with face surface deformations (Sec. 3). • A global fitting optimisation guided by the estimated contacts, learned interaction depth prior, and deformation model of the face to enable plausible 3D interactions (Sec. 3. 3). • The acquisition of the first markerless RGB-based 3D handface interaction dataset with surface deformations with consistent topology based on position-based dynamics (PBD). The reference 3D data for model training and evaluation are generated using a simple and effective non-uniform stiffness estimation approach for human head models, namely skullskin distance (SSD; Sec. 4). Our Decaf outperforms benchmark and existing related methods both qualitatively and quantitatively, with notable improvements in physical plausibility metrics (Sec.
Template Free Non-Rigid Surface Tracking Some methods in this category reconstruct non-rigid surfaces by acquiring first an explicit template mesh from RGB-D inputs  [Innmann et al. 2016]. Some use node graphs  [Lin et al. 2022]  or implicit SDF surface representations  [Slavcheva et al. 2017]  for non-rigid surface tracking. Guo et al. [2017]  propose a method that reconstructs the non-rigid surface along with the surface albedo and low-frequency lighting. Our approach differs from these works by considering the dynamics of the interactions between two different materials i. e. face and hand, and face surface stiffness values based on bone structure. Additionally, our dataset and method's output have consistent 3D mesh topologies that are very important for the supervision of network training in explicit surface space.
METHOD Our goal is to reconstruct hands interacting with a face in 3D, including non-rigid face deformations caused by the interaction, from a single monocular RGB video. Figure  2  provides an overview of the proposed framework. Our deformation and contact estimation network DefConNet, trained on our new dataset (Sec. 4), estimates face surface deformations and contact labels on both face and hand surfaces from an image sequence; the contact labels are crucial to achieve plausible and realistic interactions in 3D (Sec. 3. 2). The estimated deformations, contacts and 2D keypoints are subsequently sent to the global fitting optimisation stage (Sec. 3. 3), where we also utilise the interaction prior obtained from a conditional variational autoencoder  [Sohn et al. 2015 ] conditioned on the 2D key points for the improved interactions between the hands and face.
Interaction Estimation We introduce a learning-based approach that estimates plausible interactions in a scene, i. e. , the vertex-wise face deformations and contacts on the face and hand surfaces given only single-view RGB images. The approach is trained on our new dataset (Sec. 4). Our neural network accepts as input an image sequence I and outputs the deformation on the head model as per-vertex displacements in a camera frame p ∈ R 𝑀 ×3 , contact labels on the face c f ∈ {0, 1} 𝑀 and the hand c h ∈ {0, 1} 𝑁. The contact labels are binary signals i. e. 1 for contact, 0 otherwise. The network is trained to estimate the contact probability using the binary cross entropy (BCE): L labels = BCE(c f , ĉf ) + BCE(c h , ĉh ), (1) where ĉf and ĉh denote the ground-truth contact labels for the face and hand, respectively.
We also train the network to estimate the deformations using the ground-truth annotations p𝑚 : L def. = 1 𝑀 𝑀 ∑︁ 𝑚=1 (𝑤 𝑚 def p 𝑚 -p𝑚 2 2 + 𝑏 𝑚 def p 𝑚 ), (2) where 𝑤 𝑚 def = 0. 3, if || p𝑚 || = 0, 1. 0, otherwise, 𝑏 𝑚 def = 1, if ||p 𝑚 || > 𝜓, 0, otherwise. (3) The first term in Eq. (  2 ) allows the network to learn the 3D deformations in our dataset. The weight 𝑤 def helps to penalise the network predictions more on deforming vertices. We observe that this weighting strategy improves the network precision as the majority of the face vertices have no deformations. The second loss term in Eq. (  2 ) regularises the unnaturally large deformations on the face surface where 𝑏 def works as a binary label to penalise only the vertices with deformations greater than 𝜓 = 0. 1 [m].
DATASET In this work, we build a new markerless multi-view dataset for 3D hand-face interactions for method training and evaluation. It contains eight subjects-captured with 15 SONY DSC-RX0 cameras at 50 fps (i. e. , from 15 different viewpoints)-along with the corresponding reference 3D geometries of a right hand and head, including surface deformations of the head represented as per-vertex displacements. In total, the dataset contains 100K frames, see Table  1  for the details. Each actor performs seven different actions with three different facial expressions. For each captured view, the background masks are obtained using  [Sengupta et al. 2020 ]. The bounding boxes (for the hands and the faces) and 2D key points (for the faces), are obtained using  [Lugaresi et al. 2019 ] and  [Bulat and Tzimiropoulos 2017] , respectively.
In the remainder of this section, we elaborate on our dataset generation pipeline; see Fig. 5  for the overview. The first step of the pipeline, i. e. , multiview template fitting, is explained In Sec. 4. 1. Next, to obtain a reasonable stiffness value that considers the underlying skull structure of a human face, we introduce a simple but effective skull-skin distance (SSD) approach in Sec. 4. 2. The computed stiffness values are further utilised in the deformable object simulation relying on position based dynamics (PBD), and we obtain the final 3D geometry with plausible interactions arising from hand-face interactions (Sec. 4. 3).
EVALUATIONS We next evaluate our Decaf on our new dataset. As there are no existing methods that address the same problem we tackle, we compare our method to a most closely related approach, i. e. , a monocular full-body capture PIXIE  [Feng et al. 2021a ] and its variants that reconstruct only hands and face independently, denoted as PIXIE (hand+face). We also compare to our benchmark method that includes hand-only  [Lugaresi et al. 2019 ] and face-only  [Li et al. 2017]  trackers. Note that in this method variant, DefConNet and non-rigid collision handling (10) are deactivated. Our dataset contains separate training and testing sequences containing the same kinds of actions. We train our networks on the training sequences of 5 different subjects and conduct the quantitative evaluations on 3 different subjects unseen during the training.
Implementation and Training Details The neural networks were implemented in PyTorch  [Paszke et al. 2019]. The evaluations and network training were conducted on a computer with an NVIDIA QUADRO RTX 8000 graphics card and AMD EPYC 7502P 32 Core Processor. The training was continued until convergence using Adam optimiser [Kingma and Ba 2014] with a learning rate 3 • 10 -4. DefConNet models are trained until convergence which takes ≈12 hours. Since our dataset was captured with right-hand and face interactions, we flip the image and the corresponding 3D ground-truth annotations and contact labels horizontally to obtain the input and ground truth for the left hand. For the global fitting optimisation, we set the loss term weights of (5), 𝜆 𝜷 = 1 • 10 -5 , 𝜆 𝚿 = 1 • 10 -3 , 𝜆 V = 3 • 10 -4 , 𝜆 V = 3 • 10 -4.
Furthermore, the estimated contacts showcased in Fig. 10  faithfully mirror the contact regions evident in the input images. As a result, the final reconstructions show plausible hand-to-face interactions guided by the estimated contacts. To further assess the generalisability of our method on unseen actions, we train our networks excluding "poking a cheek (pointing hand)" and "punching a cheek" actions from the training dataset; the results for these actions are illustrated in Fig. 11. Our method produces satisfactory results for "poking a cheek (pointing hand)". On the other hand, the exclusion of "punching a cheek" from the training dataset is a highly challenging scenario as no other actions in the training data contain interactions between the back side of the hand and the face. Given that our approach is neural and learning-based, such a substantial deviation from the training set can lead to inaccurate interactions in the results.
Fig. 12  shows PVE plots for two test sequences from our dataset highlighting the stability of our results. Amount of occlusion stands for the per-frame ratio of face pixels occluded by hand pixels from the camera view and amount of deformations signifies the per-frame sums of deformations in the pseudo ground truth. We observe that the ablated versions of our method are starkly influenced by occlusions, which can be recognised with the help of peaks occurring at the frames with the (locally) largest deformations as well as the most significant occlusions. In contrast, our full model is affected by the occlusions substantially less and its curve has a smaller standard deviation of PVE, which verifies the importance of each loss term.
DISCUSSIONS AND LIMITATIONS Our Decaf captures plausible 3D deformations along with hand-face interactions solely from a monocular RGB video, effectively reducing unnatural collisions and non-touching artefacts. While our method is the first to address this problem set, it does have certain limitations. Our network learns from a newly created dataset computed using Position-Based Dynamics (PBD) with a skull-skin-distance (SSD) approach combined with the multi-view markerless motion capture setup. PBD is widely utilised in modern physics engines, ensuring that our pseudo-ground truth deformations are plausible. However, it may introduce some discrepancies between the actual deformations and calculated deformations as this PBD-based approach does not integrate visual information such as photometric loss.
CONCLUSIONS Decaf is the first monocular RGB-based approach for deformationaware 3D hand-face motion capture. Our method captures non-rigid face surface deformations arising from various hand-head interactions. It regards the human head anatomy (i. e. , skull-skin distance used to calculate non-uniform facial tissue stiffness), detects handhead contacts and is trained on a new dataset of facial performances. In the comprehensive experiments, Decaf demonstrates the highest 3D reconstruction (in terms of PVE) and plausibility metrics (in terms of F-score) among all compared methods. Especially significant are the advancement in terms of PVE compared to the most closely related previous method (roughly fourfold error reduction) and qualitative improvements in the estimated 3D geometry, which opens up many possibilities for downstream applications (e.
Additionally, we provide a walkthrough of the reconstruction of a superresolved dataset from acquired raw images using the Super-resolution Microscopy Analysis Platform (SMAP). Depending on the instrument configuration, the cost of components is in the range $80,000 -160,000, a fraction of the cost of a commercial instrument. A builder with some experience of optical systems is expected to require 3 -6 months from the start of system construction to attain high-quality 3D and multi-color biological images. localization microscopy, ((f)PALM) [1, 2]  or (direct) stochastic optical reconstruction microscopy ((d)STORM) [3,4]. On-off binding of fluorophores provides an alternative route, for example, in DNA-based point-accumulation in nanoscale topology microscopy (DNA-PAINT) [5] target structures are tagged with short single-stranded DNA, and fluorescently-labelled complementary strands stochastically bind and detach to elicit a blinking effect.
Parts List and Supplementary Note 1: Using the parts list and understanding part names), CAD models and associated toleranced drawings, software, and electronic schematics can be found on GitHub (https://github. com/ries-lab/3DSMLM). CAD models are included in SolidWorks (Dassault Systèmes) assembly and part file format for those wishing to modify the design. For those wishing to follow the protocol as written, the CAD assemblies are also provided in eDrawing format, which can be viewed using a freely available eDrawing viewer (https://www. edrawingsviewer. com/download-edrawings). The viewer also allows suppression of components, configuration of the assemblies, and cut away views, all of which are helpful to follow the protocol. Toleranced drawings for the custom mechanical parts are provided as SolidWorks drawings and as.
Control and synchronization: Control of the microscope including previewing and acquisition functions is carried out using the open-source µManager software  [39]  and a custom htSMLM  [26]  (high-throughput single molecule localization microscopy) graphical user interface (GUI). µManager allows common hardware functions to be mapped to specific core functionalities providing great flexibility to specific hardware choices. As long as the hardware chosen permits the required functionality, and a device adapter for the hardware is available, the user may deviate from the specific hardware choices outlined in the protocol (see Supplementary Notes 3 -6, Using nonspecified objective lenses/single-mode lasers and laser-engines/XY stages/cameras). Pre-compiled versions of the htSMLM GUI and device adapters, which are not available in the core µManager build, are provided as part of the protocol.
A guide to configuration of µManager and the htSMLM GUI is provided in the protocol. The microscope makes use of the open-source microFPGA project for synchronization of hardware components and for monitoring of signals related to the microscope focus lock system. Here, we provide microscope-specific configuration files for the FPGA, board files for reproduction and a detailed build up yielding the required functionality. A more generally applicable description of the highly-configurable microFPGA  [40]  has been provided elsewhere and may prove useful for particularly adventurous builders who want to further expand the microscope functionality or use microFPGA  [40]  for other projects. A full list of software controllable hardware elements used in the microscope is provided in Supplementary Table  4 : Micro-Manager Hardware Configuration.
Before any optical alignment can be performed, it is advisable that the various hardware elements can be controlled and configured from a single interface. To this end, the microFPGA, which allows for monitoring and analog/digital control of hardware, is constructed (unless specifically sourced in an assembled state, see Electronics rationale and sourcing) (steps 68 -103) and configured (steps 104 -107). Next, all software for acquisition is set up and configured, including µManager and the EMU htSMLM plugin, which together provide the user interfaces for the microscope (steps 108 -130). Having installed µManager, it is possible to fully test the microFPGA functionality (for self-assembled microFPGAs only) (steps 131 -137). Electrical connections between many of the hardware elements and the microFPGA are then made (steps 138 -142).
The microscope is fully functional at this stage and further steps require the installation of the SMLM data analysis software SMAP (steps 460 -465) for analysis of the data. As a first step towards 3D imaging a PSF calibration is performed using a fluorescent bead sample and the 3D imaging performance of the system is adjusted (steps 466 -473). Having completed the PSF calibration, the next steps detail the use of SMAP for 2D and 3D calibration based on a Gaussian or experimentally-derived model of the PSF respectively (steps 474 -487). The following section considers the rendering of the localized data including the merging, filtering and drift-correction of the individual localizations (steps 488 -494). At this juncture the necessary tools and knowledge have been established to make additional quality checks including checks of the focus lock systems' ability to correct axial drift, as well as assessment of lateral drift and vibrations (steps 495 -517).
At the core of the microFPGA is a low-cost hobby-grade FPGA (Alchitry Au+). In addition, various commercially available and custom boards are connected to the FPGA. These additional boards act to protect the FPGA, ensure proper connection with labelled SMB connectors, scale I/O voltages appropriately, and low-pass filter PWM signals to produce analog outputs. The data required for reproduction of all custom boards is included (see: https://github. com/ries-lab/3DSMLM), with original board files (in Altium), the Gerber files, bill of materials, schematics, and board files as. pdf. A list of all boards and parts required for the microFPGA is given in the parts list. Depending on the builder's expertise in electronics we suggest the following options. Firstly, a pre-assembled microFPGA can be sourced directly.
The former is not strictly required and can be omitted (if transmission imaging functionality is not required) or replaced by a suitable triggerable/directly modulatable (not pulse-width modulated) ring LED source. Doing so will require some modification of the CAD files to mount the device. Again, full data required for reproduction of the custom boards including the board-mounted LED ring is included (see https://github. com/ries-lab/3DSMLM). Conversely, the focus lock electronics are required and comprise a commercially available amplifier board (see parts list), which simply requires appropriate packaging including provision of appropriate power and making connections to various input/outputs, which can be handled by the builder or electronics workshop as necessary.
Data analysis and rendering During data acquisition the raw camera frames and metadata describing the state of the microscope are saved by µManager. This data requires extensive analysis steps to yield biological insights: fitting of single molecule positions, post-processing such as filtering and drift correction, rendering as a super-resolution image and quantitative evaluation. We developed the super-resolution microscopy software platform SMAP  [28]  and here detail the analysis steps using this solution, but there are numerous other software, notably Picasso  [56]  and ThunderStorm  [57]  that can be equally used. For an overview of the performance of different localization algorithms, we recommend users study the results of the SMLM software challenge  [58]. To analyze 3D or multi-color data in SMAP, first the PSF of the microscope and/or channel transformation are calibrated from image stacks of fluorescent beads.
Electronic components: The microscope uses many custom and commercially available electronic components. The custom electronics can be sourced as discussed in Electronics rationale and sourcing. Transmission illumination is provided by an LED ring source, which directly mounts to the microscope cover. A QPD and associated amplifier are used for focus locking and an optional singleelement photodiode (with electronics integrated into the FPGA and signal conversion unit) can be used for recording of experimental laser power. At the heart of the microscope control scheme lies a low-cost hobby-grade field-programmable gate array (FPGA, Alchitry Au) and a number of signal conversion boards (see https://github. com/mufpga/MicroFPGA  [40] ) to provide synchronization of hardware, readout of the QPD, analog/digital control of the single/multi-mode laser engine(s), and more generally to provide the builder with the flexibility to add and control additional hardware.
Microscope control computer: The microscope control computer can be configured as desired with a few points worth noting. If the same machine is to be used for analysis tasks, one should ensure that both sets of requirements are met. Smooth performance can be achieved by choosing a central processing unit (CPU) with at least 4 cores, for 8 threads in the tested 64-bit Windows 10/11 environment. Lesser configurations may result in µManager 2. 0 struggling to achieve a suitable image refresh rate (>10 Hz) during acquisitions, which is crucial to be able to assess conditions for blinking. We strongly recommend installing the operating system on an SSD and to reserve this disk for the operating system and required software and drivers. Other than these concerns, refer to the Computational infrastructure section for a discussion of data transfer and storage.
In this case, favor the motherboard-integrated GPU for connection of the monitor. At the time of publication, the following computer was used for image analysis: Dell Precision Tower 3650, Windows 10 Professional 64-bit, Intel Core i7-11700 Processor, 32 GB random-access memory (RAM), 1 TB SSD, Nvidia GeForce GTX 1650 GPU. The following guidelines provide a suggested configuration. Software for microscope control: The following software is required to control the microscope. Additional device adapters and a graphical user interface will be added to the base µManager installation during the protocol. • µManager 2. 0 latest official release • ImageJ (installed as part of µManager) Software for image analysis: The Super-resolution microscopy analysis platform (SMAP)  [28]  provides the basis for super-resolution image reconstruction in the protocol.
Control of the microscope and hardware Using µManager and the htSMLM GUI: Throughout the protocol a multitude of instructions will be given requiring interaction with software to control hardware and acquire data. The builder is expected to familiarize themselves with the software and controls prior to continuing with the protocol. For brevity, specific instructions regarding how to achieve a given command are provided. For example, the instruction: Image fluorescent beads, assumes that the user is familiar with previewing and acquiring images using the µManager GUI. Similarly, it is assumed that the user can set laser power, region of interest (ROI) and exposure time or select filters appropriately to achieve a desired temporal resolution and image signal to noise or exposure level.
Configuration of the microFPGA (Timing: 2 hours) Before the microFPGA can be used, it must be configured by uploading a configuration file to the device. 104. Install AlchitryLabs from: https://alchitry. com/alchitry-labs 105. Download the FPGA configuration binary (. bin) for the Alchitry Au from the release page of the microFPGA repository: https://github. com/mufpga/MicroFPGA/releases. 106. Connect the FPGA to the host PC. 107. Upload the configuration to the FPGA target using AlchitryLoader (installed as part of Alchitry labs). Setup and configuration of software and hardware for acquisition (Timing: 5 days) Certain hardware elements require non-factory configuration for operation. For example, the piezo flexure stage controller needs to be configured in its host-software such that it provides appropriate positional feedback on the basis of the analog signal provided by the focus lock system QPD.
The various piezo resonant stages used for illumination control (epi-/HILO/TIRF), back focal plane viewing and astigmatic lens positioning also require that a unique address is established for each in their host software. µManager, which ultimately provides the basis for hardware control must be installed. Communication with the various hardware devices from µManager requires that a hardware configuration is established, whereby communication with each device is defined via its device adapter (e. g. , COM port etc. for serial devices). All required device adapters for the hardware elements described are available in µManager with one exception, the objective lens piezo flexure stage uses a modified version of the PI_ZStage device adapter, PI_FocusLock, a pre-compiled version of which is available on our GitHub.