Reviewerly/exHarmony huggingface dataset:
Improving Visual Question Answering by Leveraging Depth and Adapting Explainability

During human-robot conversation, it is critical for robots to be able to answer users’ questions accurately and provide a suitable explanation for why they arrive at the answer they provide. Depth is a crucial component in producing more intelligent robots that can respond correctly as some questions might rely on spatial relations within the scene, for which 2D RGB data alone would be insufficient. Due to the lack of existing depth datasets for the task of VQA, we introduce a new dataset, VQA-SUNRGBD. When we compare our proposed model on this RGB-D dataset against the baseline VQN network on RGB data alone, we show that ours outperforms, particularly in questions relating to depth such as asking about the proximity of objects and relative positions of objects to one another. We also provide Grad-CAM activations to gain insight regarding the predictions on depth-related questions and find that our method produces better visual explanations compared to Grad-CAM on RGB data. To our knowledge, this work is the first of its kind to leverage depth and an explainability module to produce an explainable Visual Question Answering (VQA) system.
Human-Guided Goal Assignment to Effectively Manage Workload for a Smart Robotic Assistant

Managing robot workloads in human robot teams is critical for efficient team operation. If robots are overloaded with work, then they will miss deadlines and force humans to take on extra work. This paper presents a framework for a robot to assess its own workload based on an initial goal assignment. The robot does this by generating task and motion plans and computing the probability of missing deadlines due to the possibility of delays in task execution. A branch and bound based search is used to generate task and motion plans by minimizing task execution effort. The robot presents a diverse set of task and motion plans to the humans to offer multiple different options. Humans can either approve a plan or provide guidance to reduce the workload by either relaxing deadlines or removing goal(s) assigned to the robots.
Should AI Systems in Nuclear Facilities Explain Decisions the Way Humans Do? An Interview Study

There is a growing interest in the use of robotics and AI in the nuclear industry, however it is important to ensure these systems are ethically grounded, trustworthy and safe. An emerging technique to address these concerns is the use of explainability. In this paper we present the results of an interview study with nuclear industry experts to explore the use of explainable intelligent systems within the field. We interviewed 16 participants with varying backgrounds of expertise, and presented two potential use cases for evaluation; a navigation scenario and a task scheduling scenario. Through an inductive thematic analysis we identified the aspects of a deployment that experts want to know from explainable systems and we outline how these associate with the folk conceptual theory of explanation, a framework in which people explain behaviours. We established that an intelligent system should explain its reasons for an action, its expectations of itself, changes in the environment that impact decision making, probabilities and the elements within them, safety implications and mitigation strategies, robot health and component failures during decision making in nuclear deployments. We determine that these factors could be explained with cause, reason, and enabling factor explanations.
SM-EXO: Shape Memory alloy-based Hand EXOskeleton for Cobotic Application

The conventional smart gloves present a challenge regarding their portability as most work on gesture recognition techniques based on vision sensing and image processing. The multiple algorithms and signal filtering further make the overall process cumbersome. This work proposes a Shape Memory Alloy (SMA) integrated sensing mechanism in a smart glove for autonomous control. A novel hand gesture recognition technology is developed using kinaesthetic feedback from the finger joint movements. The paper presents a smart glove with an external SMA embedded tubing attachment for the thumb, index, and middle fingers. The motion of the SMA wires is constrained between a fixed end on the tip of the fingers, and the other end is connected to a linear position sensor with spring feedback. The SMA wires in this design exist in their Austenite phase at room temperature, thus exhibiting superelastic or pseudoelastic behavior. The tension in the SMA wire is observed and measured upon bending the fingers, corresponding to the mechanical travel in the linear position sensor. The individual and a combination of position sensor readings are then used as commands for actuating interactive toys. Using a three-finger approach, one can extract seven commands depending upon single or multiple finger movements. This data is further used to actuate the toys, and a use-case for cobotic application is proposed to help better understand interactive play, hand-eye coordination, and thus early cognitive development in children with Autism Spectrum Disorder (ASD). The discrete data output with binary data is independent of other devices or heavy data processing requirements, thus making the proposed novel SM-EXO a better alternative for non-portable and complex smart gloves.
A game-based approach for evaluating and customizing handwriting training using an autonomous social robot

Handwriting learning is a long and complex process that takes about ten years to be fully mastered. Nearly one-third of all children aged 4-12 experiences handwriting difficulties and, sadly, most of them are left to fight them on their own, due to the scarcity of tools for the detection and remediation of such difficulties. Building on state-of-the-art digital solutions for automated handwriting assessment and the training of specific handwriting-related skills, in this article we discuss requirements, rationale, and architecture of a system for handwriting training, which relies on a social robot as a mediator agent, offering personalized training and suggestions. The system is envisioned to operate autonomously and to support long-term interactions via personalization. Preliminary validation of the system in an experiment with 31 children showed its potential not only for autonomously guiding handwriting training sessions, but also for its inclusion in the teachers’ practice.
Context and Intention for 3D Human Motion Prediction: Experimentation and User study in Handover Tasks

In this work we present a novel attention deep learning model that uses context and human intention for 3D human body motion prediction in handover human-robot tasks. This model uses a multi-head attention architecture which incorporates as inputs the human motion, the robot end effector and the position of the obstacles. The outputs of the model are the predicted motion of the human body and the predicted human intention. We use this model to analyze a handover collaborative task with a robot where the robot is able to predict the future motion of the human and use this information in it’s planner. Several experiments are performed where human volunteers fill a standard poll to rate different features, taking into account when the robot uses the prediction versus when the robot doesn’t use the prediction.
Allocation of Multi-UAVs Timing-dependent Tasks based on Completion Time

UAVs have played an important role in many fields such as military, civilian and scientific research due to their strong advantages in information sharing and behavioral collaboration. However, it is quite difficult to provide a reasonable allocation algorithm for multiple UAVs timing-dependent tasks since the complexity of timing-dependent tasks and the heterogeneity among multiple UAVs. Aiming at the problem that multiple UAVs perform timing-dependent tasks, this paper firstly uses DAG to represent the timing-dependencies of tasks, secondly considers the priority of tasks, and finally combines the completion time of tasks to propose the task-completion-time based task allocation algorithm(TCT). The allocation results based on the randomly generated area scanning task set shows that, compared with the traditional solution, the assignment algorithm in this paper can improve the distance and time performance by up to 32% and 37% respectively.
Analyzing error sources and error propagation in an optical scanning 3D triangulation sensor system

Designing fast and high-resolution 3D measurement systems, which are key enabler for the production of the future, is a challenging task, since optical, electrical and mechatronic components with respective specifications need to be integrated. Particularly the resulting measurement uncertainty is of highest interest but can currently only be roughly determined in advance. This paper proposes an uncertainty framework for an optical scanning 3D triangulation sensor system to calculate the influence of the alignment and component specifications on the achievable system performance. This enables to calculate the required uncertainty of each component for given overall uncertainties in the lateral and axial direction. The default and best achievable specification for the manufacturing tolerances of each component, sensor noise and resolution of the detector, and angular resolution of the fast steering mirror used to manipulate the illumination path, are specified in advance. To keep the overall system cost low, the simulation of the optical path is initially performed with the default specifications. By comparing this simulation result with the ideal case, the overall uncertainties and contribution of each component can be determined. If the calculated uncertainties do not meet the requirements, the specification for the component, which contributes most to the uncertainty, can be gradually improved within the maximum specification. The procedure is repeated until the required levels of uncertainty are obtained or until it cannot be further improved since the maximum specifications have been exceeded. This ensures that only the specifications required to achieve the specified uncertainty are tuned.
Heterodyne interferometer microscopy for high precision surface measurement

Laser Doppler vibrometer (LDV) is typically used to detect and measure vibrations and dynamics of mechanical structures in a non- contacting and non-destructive way. Eckermann et al. proposed 1993 in a patent a technique to apply LDV to surface topography measurements. In this paper we explore the application of the technique to surface-topography analysis, providing the potential of axial resolution in the picometer regime for lateral resolutions defined by a confocal microscope. By moving the sample with high speed perpendicularly through the focused beam of the LDV, the topography of the surface will change the optical pathlength between LDV and specimen which the LDV measures with broad bandwidth. The advantage of this approach is that mechanical vibrations of the specimen and the stage can be filtered as they appear in a much lower frequency range as the topography information of the specimen. For the preliminary setup, we used a commercially available LDV (Polytec OVF-303s) focused on the sample with a 50x microscope objective. The sample is a chrome-covered Siemens Star Target with a step height of 140.8 nm. The high relative speed of the topography with up to 2.5 m/s is realized by rotating the sample using a speed-controlled servo motor. With this preliminary setup, we achieved a focus diameter of 1.4 µm and an axial resolution of 1.53 nm in the full bandwidth of 250 kHz.
Facial landmark feature fusion in transfer learning of child facial expressions

Automatic classification of child facial expressions is challenging due to the scarcity of image samples with annotations. Transfer learning of deep convolutional neural networks (CNNs), pretrained on adult facial expressions, can be effectively finetuned for child facial expression classification using limited facial images of children. Recent work inspired by facial age estimation and age-invariant face recognition proposes a fusion of facial landmark features with deep representation learning to augment facial expression classification performance. We hypothesize that deep transfer learning of child facial expressions may also benefit from fusing facial landmark features. Our proposed model architecture integrates two input branches: a CNN branch for image feature extraction and a fully connected branch for processing landmark-based features. The model-derived features of these two branches are concatenated into a latent feature vector for downstream expression classification. The architecture is trained on an adult facial expression classification task. Then, the trained model is finetuned to perform child facial expression classification. The combined feature fusion and transfer learning approach is compared against multiple models: training on adult expressions only (adult baseline), child expression only (child baseline), and transfer learning from adult to child data. We also evaluate the classification performance of feature fusion without transfer learning on model performance. Training on child data, we find that feature fusion improves the 10-fold cross validation mean accuracy from 80.32% to 83.72% with similar variance. Proposed fine-tuning with landmark feature fusion of child expressions yields the best mean accuracy of 85.14%, a more than 30% improvement over the adult baseline and nearly 5% improvement over the child baseline.
A hierarchical sparse Gaussian process for in situ inference in expensive physics simulations

High-fidelity physics simulations, such as the Energy Exascale Earth System Model (E3SM), are generating ever increasing quantities of data. In the near future, it will be infeasible to store the full computation after the simulation. To facilitate the training of complex machine learning models in the future, there is a glaring need for in-situ inference algorithms. In this work, we focus on the need for spatio-temporal inference models which are (i) highly scalable, (ii) distributed, (iii) able to capture small-scale, high-resolution structures, and (iv) longrange global structure. One possible approach is to leverage algorithms for fast approximate Gaussian process regression, such as the sparse variational Gaussian process (SVGP). In this work, we present a Hierarchical SVGP (HSVGP) model in which local models capture small-scale structures in-situ, while large-scale structures are captured and communicated by a global SVGP. We also consider the simpler approach in which independent, parallel sparse Gaussian processes are fit to each local region with no additional global information. Using both synthetic data and E3SM model output, we show the success of the HSVGP approach in some settings, but also the surprisingly good performance of the independent approach.
Enhancing SVT-AV1 with LCEVC to improve quality-cycles trade-offs and enhance sustainability of VOD transcoding

The trade-offs between compression performance and encoding complexity are key in software video encoding, even more so with increasing pressure on sustainability. Previous work "Towards much better SVT-AV1 quality-cycles tradeoffs for VOD applications" [1] described three approaches of evaluating compression efficiency vs cycles trade-offs within a convex-hull framework using the Dynamic Optimizer (DO) algorithm developed in [2] [3] for VOD applications. In parallel, the new video codec enhancer LCEVC (Low Complexity Enhancement Video Coding) [4], designed to provide gains in speed-quality trade-offs, has recently been standardized as MPEG-5 Part 2. The core idea of LCEVC is to use any video coding standard (such as AV1) as a base encoder at a lower resolution, and then reduce artifacts and reconstruct a full resolution output by combining the decoded low-resolution output with up to two low-complexity reconstruction enhancement sub-layers of the residual data. This paper starts by applying LCEVC to SVT-AV1 [5], as well as x264 [6] and x265 [7], while using two of the approaches presented in [1] to evaluate the resulting compression efficiency vs cycles trade-offs. The paper then discusses the benefits of LCEVC towards higher playback speed and lower battery power consumption when using AV1 software decoding. Results show that, with fast-encoding parameter selection using the discrete convex hull methodology, LCEVC improves the quality-cycles trade-offs for all the tested codecs and across the full complexity range. In the case of SVT-AV1, LCEVC yields a ~40% reduction in computations while achieving the same quality levels according to VMAF_NEG [8]. LCEVC also enlarges the set of mobile devices capable of playing HD as well as high-frame-rate content encoded with AV1 and extends mobile battery life by up to 50% with respect to state-of-the-art AV1 software decoding.
Language-Based Syllogistic Reasoning Using Deep Neural Networks

Abstract Syllogism is a common form of deductive reasoning that requires precisely two premises and one conclusion. It is considered as a logical method to arrive at new information. However, there has been limited research on language-based syllogistic reasoning that is not typically used in logic textbooks. In support of this new field of study, the authors created a dataset comprised of common-sense English pair sentences and named it Avicenna. The results of the binary classification task indicate that humans recognize the syllogism with 98.16% and the Avicenna-trained model with 89.19% accuracy. The present study demonstrates that aided with special datasets, deep neural networks can understand human inference to an acceptable degree. Further, these networks can be used in designing comprehensive systems for automatic decision-making based on textual resources with near human-level accuracy.
Decision tree boosted varying coefficient models

Abstract Varying coefficient models are a flexible extension of generic parametric models whose coefficients are functions of a set of effect-modifying covariates instead of fitted constants. They are capable of achieving higher model complexity while preserving the structure of the underlying parametric models, hence generating interpretable predictions. In this paper we study the use of gradient boosted decision trees as those coefficient-deciding functions in varying coefficient models with linearly structured outputs. In contrast to the traditional choices of splines or kernel smoothers, boosted trees are more flexible since they require no structural assumptions in the effect modifier space. We introduce our proposed method from the perspective of a localized version of gradient descent, prove its theoretical consistency under mild assumptions commonly adapted by decision tree research, and empirically demonstrate that the proposed tree boosted varying coefficient models achieve high performance qualified by their training speed, prediction accuracy and intelligibility as compared to several benchmark algorithms.
Usability and User Experience (UX) of Human-Automation Interaction: Concept and Measurement Method

Usability and UX are the concepts that represent the amount to what extent the interaction with the artifact, i.e. products and services, is meaningful and satisfying to the human user while the automation is one aspect of the human artifact interaction. The concept of usability was first defined by Shackel in 1991 as the extent of successfulness of the user's work, then was gradually formalized by Nielsen and ISO standardStandards. In 1998, Norman coined a new term “UX” (User eXperience) to represent the wider range of qualityQuality of human experience that could not be expressed by just the concept of usability. But, the lack of formal definition and methodologyMethodologies gave birth to various interpretations. Because of this confusing situation on the concept of UX, “UX White Paper” was published in 2010. But various interpretations still exist in terms of the concept of UX. For the purpose of solving this confusing situation, authors proposed (1) a conceptual model of UX that clarifies the difference between the usability and UX, (2) the process model (DECAGON model) of UX that is related to the lifecycleLifecycle model of the product or service, and (3) the measurementMeasurements method (ERM-AT) of UX that can be used to clarify how the automation and UX are related.
Deep learning‐based super‐resolution with feature coordinators preservation for vision‐based measurement

Vision-based displacement measurement is promising and emerging for structural monitoring. However, the accuracy of visual measurement is commonly limited by the resolution of the camera. The super-resolution (SR) technique can reconstruct high-resolution images from the corresponding low-resolution images within the constraints of prior knowledge. Existing SR algorithms mainly focus on improving the overall quality of the image. By contrast, the accurate extraction of the coordinates of feature points is the most important for the visual measurement. Besides, the SR network is usually trained by an artificial dataset whose low-resolution images are obtained by artificially degrading the corresponding high-resolution images, instead of those directly captured by cameras. However, this degradation usually is only a simple bicubic downsampling that cannot reflect the real degradation, which will provide inaccurate constraints to the network training. Therefore, this paper proposes a novel SR framework that can significantly preserve the feature coordinators for visual measurement (SRFCP). First, a deep learning-based SR network that focuses on feature preservation is proposed, which introduces both feature weighted branch and feature preserving loss. Second, an image degradation model is built based on the blur kernel and noise extracted from the images captured in real scene. Experiments on public datasets show that the proposed SRFCP performs well both in terms of the objective evaluation index and the subjective visual effect. Then, a binocular visual measurement platform is set up to measure the distance of adjacent feature points on a chessboard. Lastly, several SR algorithms are evaluated by the improvement they bring to the measurement accuracy. Experimental results show that the distance measurement performance can be significantly improved by the images reconstructed by the SRFCP. Therefore, the proposed SRFCP can accurately reconstruct the high-resolution images while preserving the features coordinates, which is crucial for the visual measurement in structural monitoring.