Reviewerly/exHarmony huggingface dataset:
From Users to Creators: Motivations, Implementation, and Impacts of Augmented and Virtual Reality in Science and Engineering Projects in K-12 Education

Current research on immersive technologies has widely described their application in K-12 settings, with evidence of their goals, approaches, and potential to enhance learning. However, there is a dearth of studies describing those technologies being used as authoring tools by students. This paper investigates how K-12 students apply virtual and augmented reality technologies as tools for designing experiments and solutions for their science and engineering projects. Study 1 analyzed 12 projects presented at a FEBRACE national science fair between 2015 and 2020 and Study 2 investigated, via an online questionnaire, the motivations and challenges from students who worked on those projects. Results include the context in which the projects were developed, motivation for using virtual and augmented reality technologies, fields in which they were applied, and types of software and hardware used. Findings inform how future research can build on those experiences to integrate immersive technologies in K-12 education.
Exploring the Effect of Virtual Reality with Haptics on Educational Research: A Meta-analysis From 2010 to 2020

Virtual reality (VR) not only expands the learning field and enriches the learning experience, but also provides an effective presentation for scarce educational resources. However, because VR mainly depends on image processing technology, its application in the field of education mostly stays at the level of knowledge observation, and many challenges are gradually exposed, such as untimely interaction and weak sense of existence. In recent years, haptics has gradually become the development frontier of human-computer interaction, breaking through the limitation of single visual stimulation of VR to a great extent. Some researchers began to explore the teaching mode of VR with haptics, in order to mobilize learners’ multi-sensory stimulus responses such as hearing, touch, force and movement, and enhance learners’ on-the-spot experience and flow experience. In order to deeply explore the effectiveness of VR with haptics in teaching and explore the regulatory variables affecting this teaching method, 1646 academic papers published from 2010 to 2020 were screened based on WOS and Scopus databases. After meta-analysis of 50 selected research data, it is found that: (1) from the perspective of overall effect, VR with haptics can significantly improve learners’ learning performance and efficiency; (2) Through the analysis of the effects of three regulatory variables, it is found that VR with touch has a positive impact on learning performance; VR with haptics has a positive impact on learning efficiency, especially for learners who have no previous learning experience and take practical skills as their learning goal for a long time. This discovery provides an important data reference for building an effective “virtual reality with haptics” teaching scene.
Canine Parvovirus Diagnosis Classification Utilizing Veterinary Free-Text Notes

This work demonstrates a NLP pipeline on classifying different canine parvovirus diagnosis from visit summaries. The preliminary results show promising efficacy in employing BERT based models for this task. This work also reveals a way to make use of the largely untapped data from veterinary free-text fields to help improve pet health.
Research Based on Affective Filter Theory: Is Social VR an Effective Tool for Learning a Second Language?

This study examines the affective filter theory in a social virtual reality (social VR) setting. Motivation, anxiety, and self-confidence were measured and compared in social VR and video conferencing. The research was conducted using mixed methods with qualitative and quantitative data. One-hour Korean language learning workshops were held to establish a teaching environment for this research. The research results indicated no significant differences in motivation, anxiety, or self-confidence between social VR and video conferencing as a learning tool. However, the debriefing interview following the workshop enabled us to develop a deeper understanding of participants’ motives to practice second language learning by leveraging social VR. Also, the interview data explained the strengths and weaknesses of using social VR as a learning tool, making our study meaningful in alternative language learning environments. This feasibility study revealed the potential of using social VR as a learning tool.
Systematic Motion Integration with Multiple Depth Cameras Allowing Sensor Movement for Stable Skeleton Tracking

In recent years, markerless motion capture using a depth camera or RGB camera without any restriction on the subject has been attracting attention. Especially, depth cameras such as Kinect and RealSense allow instantaneous motion capture even at home outside lab environment, which is attractive for rehabilitation usage. However, single depth camera can capture steadily skeleton only when the subject stands facing to camera for the limited range, thus it is hard to apply to track skeletons while walking. Multiple depth cameras setting may allow to expand the range, but it can involve non-practical calibration process and can affect instantaneous capture advantage of depth camera. In this study, we propose a systematic method to integrate the motion information of skeletal models obtained from multiple depth cameras. The proposed method can perform a quick calibration using skeletal models instead of external reference objects, and estimate the spatial relationship of the sensors that allows the depth camera to move. The result demonstrates stable skeleton tracking free from occlusion problem keeping instantaneous capture capability of depth cameras.
Doctoral Colloquium—ME++ Data Ethics of Biometrics Through Ballet and AR

This Doctoral Colloquium explores data ethics and biometric data through the development of an augmented reality (AR) classical ballet experience. This paper focuses on two areas: (i) The development and use of tool sets for motion capture and video extraction. (ii) On dancers’ understanding and experiences of the ‘body as a data artefact’ for the creation of motion graphic (3D rigged) characters of an AR experience. Questions: (1) Is there a difference in sense of self (identity) between the human and the virtual? (2) How does sharing your personal biometric data make you feel? (3) How can biometric and immersive development tools be used in the computing classroom and dance studios to raise awareness of data ethics?
Internet of Healthcare Things (IoHT): Towards a Digital Chain of Custody

Smart Internet of Healthcare Things (IoHT) have the potential to transform patient care dramatically at reduced cost. The reality, however, is that there are serious security and privacy concerns that prevent this goal from being accomplished. The vast amounts of data being generated need to be kept secure to prevent harm to patients' health and privacy. For example, a cyberattack on heart rates data could cause patients to be over- or under-prescribed, causing severe consequences, including death. In this new environment, not ensuring a proper digital chain of custody leads to digital forensics challenges that could impact a criminal or malpractice investigation. This project explores enhancements needed to ensure security and privacy when IoHT are to be used in healthcare. A model is proposed to ensure a secure digital chain of custody for IoHT using database auditing techniques. The current status of the proposed concept and future directions are also discussed.
Detection of Dementia Signals from Longitudinal Clinical Visits Using One-Class Classification

Dementia is one of the major health challenges in aging populations, with 50 million people diagnosed worldwide. However, dementia is often underdiagnosed or delayed resulting in missed opportunities for appropriate care plans. Identifying early signs of dementia is essential for better life quality of aging populations. Monitoring early signs of individual health changes could help clinicians diagnose dementia in its early stages with more effective treatment plans. However, rare data for dementia cases compared to the normal (i.e., imbalance class distribution) make it challenging to develop robust supervised learning models. In order to alleviate this issue, we investigated one-class classification (OCC) techniques, which use only majority class (i.e., normal cases) in model development to detect dementia signals from older adult clinical visits. The OCC models identify abnormality of older adults' longitudinal health conditions to predict incident dementia. The predictive performance of the OCC was compared with a recent streaming clustering-based technique and demonstrated higher predictive power. Our analysis showed that OCC has a promising potential to increase power in predicting dementia.
AGMC-TU Pap-Smear Cytological Image Dataset: Creation, Annotation, and Analysis towards Early Detection of Cervical Cancer

Automatic cervical cancer screening based on pap-smear images is a highly effective tool where the cells are categorized into normal and abnormal. However, success of most automation tool depends on the accurate extraction of features from the pap-smear images that represent some discriminative characteristics between these two categories of cells. In this paper, we described the designing protocols for creation of a new pap-smear image dataset entitled as AGMC-TU Pap-Smear Cytological Image Dataset. The dataset comprises of 50 normal and 50 abnormal pap-smear images belonging to ethnic and non-ethnic populations of low resource cervical cancer prone regions. Moreover, ground truths of suspicious nucleus regions are annotated in terms of pixel oriented binary masks are also provided with the dataset. Analysis of our dataset includes a conventional (i.e., shape features) and deep feature based study of pap-smear images by dividing them into two major groups: normal and abnormal. Outcome of the analysis clearly differentiates normal and abnormal pap-smear images.
Utilizing average symmetrical surface distance in active shape modeling for subcortical surface generation with slow-fast learning

In this paper, we propose and validate an automatic pipeline for subcortical surface generation by making use of the average symmetrical surface distance (ASSD) loss in active shape modeling (ASM). A group of template surfaces are first generated via large deformation diffeomorphic metric mapping based surface deformation. ASM is then employed to obtain the mean shape and shape variation parameters of the template surfaces. To obtain the optimal shape variation parameters which best fit the target structure after acting upon the mean shape, a recently proposed derivative-free optimization method (the slow-fast learning method) is adopted. The ASSD loss, in addition to the typically utilized Dice similarity coefficient loss, is employed during the learning process to help enhance the boundary accuracy. We successfully validate the importance of the ASSD loss through ablation analysis. In addition, we show the effectiveness of the slow-fast learning method by comparing it with other state-of-the-art derivative-free optimization algorithms. Our proposed pipeline is found to be capable of yielding subcortical surfaces with high accuracy, correct anatomical topology, and sufficient smoothness. Clinical Relevance- This work provides a useful tool for generating subcortical surfaces which are important biomarkers for a variety of brain disorders.
A Rendering Engine for Integral Imaging in Augmented Reality Guided Surgery

In the field of image-guided surgery, Augmented Reality wearable displays are a widely studied and documented technology for their ability to provide egocentric vision together with the overlap between real and virtual content. In particular, optical see-through (OST) displays have the advantage of maintaining visual perception of the real world. However, OST displays suffer from vergeance-accomodation conflict when virtual content is superimposed on real world. Furthermore, the calibration methods required to achieve geometric consistency between real and virtual are inherently error-prone. One of the solutions, already studied, to these problems is to use of integral imaging displays. In this paper we present an easy and straightforward real-time rendering strategy implemented in modern OpenGL to show the 3D image of a virtual object on a wearable OST display deploying the integral imaging approach. Clinical Relevance- The algorithm proposed open the way towards more effective AR surgical navigation in terms of comfort of the AR experience and accuracy of the AR guidance.
From 2D Ultrasound to Patient-Specific 3D Surface Models for Interventional Guidance

Ultrasound exam output of large organs like liver has traditionally been limited to still images or 2D cine loops of key structures, without 3D context. Having 3D context for follow up studies makes ultrasound scanning much easier and for interventional applications such as biopsy. 3D context will reduce wrong sample selection thereby increasing patient comfort. As of today, there is no existing solution which provides 3D anatomical context to users during scanning for large organs like liver. Even for routine measurements like liver volume, patients have to undergo CT or MR scan. In this paper, we propose a novel approach to build-patient specific 3D anatomical surface models from B-mode ultrasound images and tracking information from position sensors. The complexity of the problem stems from the fact that liver boundaries are often not very clear in ultrasound images, in addition to large variability in liver size and shape across patients. Our work uses state-of-the-art deep learning algorithms to detect surface landmarks of liver followed by registering a geometric model to surface point cloud to build patient specific 3D liver model. Further, the developed models will be used to guide users to right lesion locations during the interventional procedure. Our proposed semi -automated workflow ensures the accuracy of the developed models are within acceptable limits for the targeted problem.
A “smart” Imputation Approach for Effective Quality Control Across Complex Clinical Data Structures

The overwhelming need to improve the quality of complex data structures in healthcare is more important than ever. Although data quality has been the point of interest in many studies, none of them has focused on the development of quantitative and explainable methods for data imputation. In this work, we propose a "smart" imputation workflow to address missing data across complex data structures in the context of in silico clinical trials. AI algorithms were utilized to produce high-quality virtual patient profiles. A search algorithm was then developed to extract the best virtual patient profiles through the definition of a profile matching score (PMS). A case study was conducted, where the real dataset was randomly contaminated with multiple missing values (e.g., 10 to 50%). In total, 10000 virtual patient profiles with less than 0.02 Kullback-Leibler (KL) divergence were produced to estimate the PMS distribution. The best generator achieved the lowest average squared absolute difference (0.4) and average correlation difference (0.02) with the real dataset highlighting its increased effectiveness for data imputation across complex clinical data structures.
Image-Level Uncertainty in Pseudo-Label Selection for Semi-Supervised Segmentation

Advancements in deep learning techniques have proved useful in biomedical image segmentation. However, the large amount of unlabeled data inherent in biomedical imagery, particularly in digital pathology, creates a semi-supervised learning paradigm. Specifically, because of the time consuming nature of producing pixel-wise annotations and the high cost of having a pathologist dedicate time to labeling, there is a large amount of unlabeled data that we wish to utilize in training segmentation algorithms. Pseudo-labeling is one method to leverage the unlabeled data to increase overall model performance. We adapt a method used for image classification pseudo-labeling to select images for segmentation pseudo-labeling and apply it to 3 digital pathology datasets. To select images for pseudo-labeling, we create and explore different thresholds for confidence and uncertainty on an image level basis. Furthermore, we study the relationship between image-level uncertainty and confidence with model performance. We find that the certainty metrics do not consistently correlate with performance intuitively, and abnormal correlations serve as an indicator of a model's ability to produce pseudo-labels that are useful in training. Clinical relevance - The proposed approach adapts image-level confidence and uncertainty measures for segmentation pseudo-labeling on digital pathology datasets. Increased model performance enables better disease quantification for histopathology.
Designing Deep Neural Networks Robust to Sensor Failure in Mobile Health Environments

Missing data is a very common challenge in health monitoring systems and one reason for that is that they are largely dependent on different types of sensors. A critical characteristic of the sensor-based prediction systems is their dependency on hardware, which is prone to physical limitations that add another layer of complexity to the algorithmic component of the system. For instance, it might not be realistic to assume that the prediction model has access to all sensors at all times. This can happen in the real-world setup if one or more sensors on a device malfunction or temporarily have to be disabled due to power limitations. The consequence of such a scenario is that the model faces "missing input data" from those unavailable sensors at the deployment time, and as a result, the quality of prediction can degrade significantly. While the missing input data is a very well-known problem, to the best of our knowledge, no study has been done to efficiently minimize the performance drop when one or more sensors may be unavailable for a significant amount of time. The sensor failure problem investigated in this paper can be viewed as a spatial missing data problem, which has not been explored to date. In this work, we show that the naive known methods of dealing with missing input data such as zero-filling or mean-filling are not suitable for senors-based prediction and we propose an algorithm that can reconstruct the missing input data for unavailable sensors. Moreover, we show that on the MobiAct, MotionSense, and MHEALTH activity classification benchmarks, our proposed method can outperform the baselines by large accuracy margins of 8.2%, 15.1%, and 11.6%, respectively.
Detection of Osteoarthritis from Multimodal Hand Data

Osteoarthritis (OA) describes a degenerative joint disorder that is prevalent among older people and typically results in swollen and inflamed joints. The aim of this paper is to develop a method using images, videos and thermal data of 100 patients taken at Keio University Hospital to detect OA in hands. By using hand pose estimation on the video data, joint angles can be calculated and subsequently transformed into feature vectors. For the thermal and RGB images, hand keypoint detectors were trained to identify and crop the appropriate joints within the images. The resulting extracted features are combined and further trained on Support Vector Machines and Convolutional Neural Networks to obtain the final binary classification for each joint. While the proposed method generally shows favorable accuracy and F1-scores on the Proximal (PIP) and Distal Interphalangeal (DIP) joints, the performance on the Metacarpophalangeal (MCP) joints is limited by the low occurrence of affected joints in the dataset. We further compare the different modalities and found that, apart from the combined approach, using video data provides the best results. Clinical Relevance- The proposed method shows promising first results for the usage of visual and thermal data in combination with machine learning in order to detect OA.
A User-Centric approach for Personalization based on Human Activity Recognition

The objective of this work focuses on multiple independent user profiles that capture behavioral, emotional, medical, and physical patterns in the working and living environment resulting in one general user profile. Depending on the user's current activity (e.g. walking, eating, etc.), medical history, and other influential factors, the developed framework acts as a supplemental assistant to the user by providing not only the ability to enable supportive functionalities (e.g. image filtering, magnification, etc.) but also informative recommendations (e.g. diet, alcohol, etc.). The personalization of such a profile lies within the user's past preferences using human activity recognition as a base, and it is achieved through a statistical model, the Bayesian belief network. Training and real-time methodological pipelines are introduced and validated. The employed deep learning techniques for identifying human activities are presented and validated in publicly available and in-house datasets. The overall accuracy of human activity recognition reaches up to 86.96 %.
A Coarse-to-Fine Pathology Patch Selection for Improving Gene Mutation Prediction in Acute Myeloid Leukemia

Identifying gene mutation is essential to prognosis and therapeutic decisions for acute myeloid leukemia (AML) but the current gene analysis is inefficient and non-scalable. Pathological images are readily accessible and can be effectively modeled using deep learning. This work aims at predicting gene mutation directly by modeling bone marrow smear images. Traditionally, bone marrow smear slides are cropped into patches with manual segmentation for patch-level modeling. Slide-level modeling, such as multi-instance learning, could aggregate patches for holistic modeling, though suffer from excessive redundancy. In this study, we propose a discriminative multi-instance approach to select useful patches in a coarse-to-fine process. Specifically, we preprocess a slide into patches by using a trained pre-selector network. Then, we rule out low quality patches in the coarse selection with known prior knowledge, and refine the model using gene-discriminative patches in the fine selection. We evaluate the framework for CEBPA, FLT3, and NPM1 gene mutation prediction and obtain 71.67%, 56.26%, and 56.34% F1-score. Further analysis show the effect of different selection criteria on prediction gene mutations using pathological images.
Learning to Segment Fine Structures Under Image-Level Supervision With an Application to Nematode Segmentation

Image segmentation models trained only with image-level labels have become increasingly popular as they require significantly less annotation effort than models trained with scribble, bounding box or pixel-wise annotations. While methods utilizing image-level labels achieve promising performance for the segmentation of larger-scale objects, they perform less well for the fine structures frequently encountered in biological images. In order to address this performance gap, we propose a deep network architecture based on two key principles, Global Weighted Pooling (GWP) and segmentation refinement by low-level image cues, that, together, make segmentation of fine structures possible. We apply our segmentation method to image datasets containing such fine structures, nematodes (worms + eggs) and nematode cysts immersed in organic debris objects, which is an application scenario encountered in automated soil sample screening. Supervised only with image-level labels, our approach achieves Dice coefficients of 79.72% and 58.51 % for nematode and nematode cyst segmentation, respectively.