Reviewerly/exHarmony huggingface dataset:
Robot Musical Theater for Climate Change Education

The use of social robots has recently been investigated in various areas, including STEM (Science, Technology, Engineering, and Mathematics) education and artistic performances. To inform children of the seriousness of climate change and awareness that they can make change, we created the Robot Musical Theater performance. In this project, natural elements (wind, earth, fire, and water) were anthropomorphized and represented by humanoid robots (Pepper, Milo, and Nao). The robots were designed to motivate audience to participate in the action to prevent climate change. Because of COVID, only fourteen visitors as a single group were allowed to participate in real-time and posted to YouTube, where at the time of submission, 141 people have viewed the performance. The participants provided positive comments on the performance and showed their willingness to participate in the movement to prevent climate change, and expressed their further interest in STEM learning. This performance is expected to contribute to enhancing informal STEM and robotics learning, as well as advancing robotic arts.
E2Pose: Fully Convolutional Networks for End-to-End Multi-Person Pose Estimation

Highly accurate multi-person pose estimation at a high framerate is a fundamental problem in autonomous driving. Solving the problem could aid in preventing pedestrian-car accidents. The present study tackles this problem by proposing a new model composed of a feature pyramid and an original head to a general backbone. The original head is built using lightweight CNNs and directly estimates multi-person pose coordinates. This configuration avoids the complex post-processing and two-stage estimation adopted by other models and allows for a lightweight model. Our model can be trained end-to-end and performed in real-time on a resource-limited platform (low-cost edge device) during inference. Experimental results using the COCO and CrowdPose datasets showed that our model can achieve a higher framerate (approx. 20 frames/sec with NVIDIA Jetson AGX Xavier) than other state-of-the-art models while maintaining sufficient accuracy for practical use.
Pose-Aware Speech Driven Facial Landmark Animation Pipeline for Automated Dubbing

A novel neural pipeline allowing one to generate pose aware 3D animated facial landmarks synchronised to a target speech signal is proposed for the task of automatic dubbing. The goal is to automatically synchronize a target actors’ lips and facial motion to an unseen speech sequence, while maintaining the quality of the original performance. Given a 3D facial key point sequence extracted from any reference video, and a target audio clip, the neural pipeline learns how to generate head pose aware, identity aware landmarks and outputs accurate 3D lip motion directly at the inference stage. These generated landmarks can be used to render a photo-realistic video via an additional image to image conversion stage. In this paper, a novel data augmentation technique is introduced that increases the size of the training dataset from N audio/visual pairs up to NxN unique pairs for the task of automatic dubbing. The trained inference pipeline employs a LSTM-based network that takes Mel-coefficients as input from an unseen speech sequence, combined with head pose, and identity parameters extracted from a reference video to generate a new set of pose aware 3D landmarks that are synchronized with the unseen speech.
Heterogeneous Vulnerability Report Traceability Recovery by Vulnerability Aspect Matching

Security databases describe characteristics of discovered vulnerabilities in text for future studying and patching. However, due to different maintainers having different perspectives about vulnerabilities, they often describe the same vulnerability in different ways, creating obstacles for gathering comprehensive information about the vulnerabilities from different databases. To mitigate this problem, Common Vulnerability and Exposures (CVE) is established to identify each publicly disclosed vulnerability with a unique CVE id, and vulnerability databases by different vendors and organizations can reference the CVE ids in their vulnerability reports. In spite of the wide adoption of CVEs, traceability issues are still prevalent. Our empirical study on vulnerability traceability across four representative security databases (NVD, IBM X-Force, ExploitDB, Openwall) shows that there was a fast-increasing amount of CVE records, traceability delay, and missing issues become severe for the vulnerability databases. To address these issues, we develop an automatic traceability recovery method for recommending related external vulnerability reports to the reports in one database. As vulnerability reports from different databases differ in content details and length, our approach does not match the reports at the document level but extracts seven distinctive vulnerability key aspects that are widely present in vulnerability descriptions. As a proof of concept, we apply our methods to recommend the reports from IBM X-Force, ExploitDB and Openwall to the NVD report. We use NVD as the target because it is a de-facto standard vulnerability database that contains the most comprehensive list of vulnerabilities. Our experiments on a wide range of NLP methods show our aspect-level matching methods can achieve high MRR and accuracy for traceability recovery across heterogeneous vulnerability databases.
Semi-model-Based Reinforcement Learning in Organic Computing Systems

Reinforcement Learning (RL) can generally be distinguished into two main classes: model-based and model-free. While model-based approaches use some kind of model of the environment and exploit it for learning, model-free methods learn with the complete absence of a model. Interpolation-based RL, and more specifically Interpolated Experience Replay (IER), comes with some properties that fit very well into the domain of Organic Computing (OC). We demonstrate how an OC system can benefit from this concept and attempt to place IER into one of the two RL classes. To do so, we give a broad overview of how both of the terms (model-based and model-free) are defined and detail different model-based categorizations. It turns out that replay-based techniques are quite on the edge between both. Furthermore, even if interpolation based on stored samples could be classified as a kind of model, the general way of using the interpolated experiences remains replay-based. Here, the borders get blurry and the classes overlap. In conclusion, we define a third class: semi-model-based. Additionally, we show that some architectural approaches of the OC domain fit this new class very well and even encourage such methods.
Directed Ray Distance Functions for 3D Scene Reconstruction

We present an approach for full 3D scene reconstruction from a single unseen image. We trained on dataset of realistic non-watertight scans of scenes. Our approach uses a predicted distance function, since these have shown promise in handling complex topologies and large spaces. We identify and analyze two key challenges for predicting such image conditioned distance functions that have prevented their success on real 3D scene data. First, we show that predicting a conventional scene distance from an image requires reasoning over a large receptive field. Second, we analytically show that the optimal output of the network trained to predict these distance functions does not obey all the distance function properties. We propose an alternate distance function, the Directed Ray Distance Function (DRDF), that tackles both challenges. We show that a deep network trained to predict DRDFs outperforms all other methods quantitatively and qualitatively on 3D reconstruction from single image on Matterport3D, 3DFront, and ScanNet. (Project Page: https://nileshkulkarni.github.io/scene_drdf )
Design for Connecting People Through Digital Artworks with Personal Information

Nowadays, people often experience physical separation in daily life. The level of social connectedness between people is declining gradually, but social connectedness is much important for human well-being. Communication techniques for people connecting remotely are most aimed at verbal information communication. However, during physical interaction, there is still a lot of non-verbal personal information other than direct verbal information we can perceive that contributes to social connectedness. Therefore, we try to explore if combining non-verbal personal information into digital artworks in a daily context, would influence connectedness between people. In this paper, we present VizArt, a channel that helps people connect through digital artworks with non-verbal personal information. A functional prototype was implemented and a pilot experiment was conducted.
Learning with Recoverable Forgetting

Life-long learning aims at learning a sequence of tasks without forgetting the previously acquired knowledge. However, the involved training data may not be life-long legitimate due to privacy or copyright reasons. In practical scenarios, for instance, the model owner may wish to enable or disable the knowledge of specific tasks or specific samples from time to time. Such flexible control over knowledge transfer, unfortunately, has been largely overlooked in previous incremental or decremental learning methods, even at a problem-setup level. In this paper, we explore a novel learning scheme, termed as Learning wIth Recoverable Forgetting (LIRF), that explicitly handles the task- or sample-specific knowledge removal and recovery. Specifically, LIRF brings in two innovative schemes, namely knowledge deposit and withdrawal, which allow for isolating user-designated knowledge from a pre-trained network and injecting it back when necessary. During the knowledge deposit process, the specified knowledge is extracted from the target network and stored in a deposit module, while the insensitive or general knowledge of the target network is preserved and further augmented. During knowledge withdrawal, the taken-off knowledge is added back to the target network. The deposit and withdraw processes only demand for a few epochs of finetuning on the removal data, ensuring both data and time efficiency. We conduct experiments on several datasets, and demonstrate that the proposed LIRF strategy yields encouraging results with gratifying generalization capability.
Handwriting Recognition and Automatic Scoring for Descriptive Answers in Japanese Language Tests

This paper presents an experiment of automatically scoring handwritten descriptive answers in the trial tests for the new Japanese university entrance examination, which were made for about 120,000 examinees in 2017 and 2018. There are about 400,000 answers with more than 20 million characters. Although all answers have been scored by human examiners, handwritten characters are not labeled. We present our attempt to adapt deep neural network-based handwriting recognizers trained on a labeled handwriting dataset into this unlabeled answer set. Our proposed method combines different training strategies, ensembles multiple recognizers, and uses a language model built from a large general corpus to avoid overfitting into specific data. In our experiment, the proposed method records character accuracy of over 97% using about 2,000 verified labeled answers that account for less than 0.5% of the dataset. Then, the recognized answers are fed into a pre-trained automatic scoring system based on the BERT model without correcting misrecognized characters and providing rubric annotations. The automatic scoring system achieves from 0.84 to 0.98 of Quadratic Weighted Kappa (QWK). As QWK is over 0.8, it represents an acceptable similarity of scoring between the automatic scoring system and the human examiners. These results are promising for further research on end-to-end automatic scoring of descriptive answers.
Position Control of Tethered UAV with Onboard Inertial Sensors

In practice, tethered unmanned aerial vehicles (UAVs) are often used to address the problems faced by traditional UAVs, such as poor endurance, wind resistance, and potential safety hazards. Tethered UAVs usually work in rough or GPS-denied environments. Thus, precise landing control is critical for their recovery. In this paper, a new control strategy is proposed and the relative position of the tethered UAV can be controlled to the target while keeping the cable taut without GPS. A stability analysis of the proposed controller based on the Lyapunov function is given. Simulation and experimental results are provided to show the effectiveness and robustness of the control strategy.
How Do We Measure Trust in Visual Data Communication?

Trust is fundamental to effective visual data communication between the visualization designer and the reader. Although personal experience and preference influence readers’ trust in visualizations, visualization designers can leverage design techniques to create visualizations that evoke a "calibrated trust," at which readers arrive after critically evaluating the information presented. To systematically understand what drives readers to engage in "calibrated trust," we must first equip ourselves with reliable and valid methods for measuring trust. Computer science and data visualization researchers have not yet reached a consensus on a trust definition or metric, which are essential to building a comprehensive trust model in human-data interaction. On the other hand, social scientists and behavioral economists have developed and perfected metrics that can measure generalized and interpersonal trust, which the visualization community can reference, modify, and adapt for our needs. In this paper, we gather existing methods for evaluating trust from other disciplines and discuss how we might use them to measure, define, and model trust in data visualization research. Specifically, we discuss quantitative surveys from social sciences, trust games from behavioral economics, measuring trust through measuring belief updating, and measuring trust through perceptual methods. We assess the potential issues with these methods and consider how we can systematically apply them to visualization research.
AmsterTime: A Visual Place Recognition Benchmark Dataset for Severe Domain Shift

We introduce AmsterTime: a challenging dataset to benchmark visual place recognition (VPR) in presence of a severe domain shift. AmsterTime offers a collection of 2,500 well-curated images matching the same scene from a street view matched to historical archival image data from Amsterdam city. The image pairs capture the same place with different cameras, viewpoints, and appearances. Unlike existing benchmark datasets, AmsterTime is directly crowdsourced in a GIS navigation platform (Mapillary). We evaluate various baselines, including non-learning, supervised and self-supervised methods, pre-trained on different relevant datasets, for both verification and retrieval tasks. Our result credits the best accuracy to the ResNet-101 model pre-trained on the Landmarks dataset for both verification and retrieval tasks by 84% and 24%, respectively. Additionally, a subset of Amsterdam landmarks is collected for feature evaluation in a classification task. Classification labels are further used to extract the visual explanations using Grad-CAM for inspection of the learned similar visuals in a deep metric learning models.
On the co-occurrence of refactoring of test and source code

Refactoring is a widespread practice that aims to help improve the quality of a software system without altering its external behaviour. In practice, developers can perform refactoring operations on test and source code. However, while prior work showed that refactoring source code brings many benefits, few studies investigated test code refactoring and whether it co-occurred with source code. To examine the co-occurring refactorings, we conducted an empirical study of 60,465 commits spanning 77 open-source Java projects. First, we quantitatively analyzed the commits from those projects to identify co-occurring refactoring commits (i.e., commits contain refactorings performed on test and source code). Our results showed that on average 17.9% of refactoring commits are co-occurring refactoring commits, which is twice as much as test code-only refactoring commits. Also, we investigated the type of refactorings applied to test code in those co-occurring commits. We found Change Variable Type and Move Class are the most common applied refactorings. Second, we trained random forest classifiers to predict when refactoring test code should co-occur with refactoring source code using features extracted from the refactoring source code in ten selected projects. Our results showed that the classifier can accurately predict when test and source code refactoring co-occurs with AUC values between 0.67--0.92. Our analysis also showed that the most important features in our classifiers are related the refactoring size and developer refactoring experience.
Information Access Effort: Are Head Movements “Cheap” or Even “Free”?

Augmented reality head-mounted displays (AR-HMDs) can present information at the center field of view (FOV) to 360° around the user. Deciding where to place virtual content when using an AR-HMD could directly impact the effort required to access information for different tasks. The current paradigm investigated the cost of information access effort for two different tasks presented on a virtual display using an AR-HMD. Participants made comparison judgments for two types of tasks (focused attention and computation integration) based on information presented at increasing lateral distances from the left side of the virtual display. Results showed no loss in performance as predicted by the Information Access Effort function. However, results show that evoking head movements played a significant role in restoring and preserving accuracy at greater visual eccentricities without hindering response time.
Comparisons of Perceptions of Risk for Visualizations Using Animated Risk Trajectories Versus Cones of Uncertainty

Dynamic ensemble visualizations might effectively convey the future path of an approaching tropical storm, and they offer benefits over commonly used Cones of Uncertainty. This experiment examined perceptions of risk at coastal locations of varying eccentricity from the most likely forecast path of a hypothetical hurricane. Perceptions of high (need to evacuate) and moderate (need to prepare) risk from inbound storms dropped off dramatically for Cone of Uncertainty visualizations, consistent with the influence of a containment effect induced by the cone’s boundary line. In contrast, Animated Risk Trajectories offered a more continuous sense of declining risk with distance, and much higher rates of needing to prepare at larger eccentricities that while at lower risk would still be vulnerable to some potential future paths. The findings lend further support to the viability of Animated Risk
Adaptive Visual Cues for Guiding a Bimanual Unordered Task in Virtual Reality

Work on cueing performance in AR and VR has focused on sequential tasks in which each step must be completed in order before the user can proceed to the next. However, for unordered tasks such as putting books back on a library shelf, the user may be able to perform multiple steps concurrently without needing to follow a specific order. In such situations, giving the user multiple cues for potentially concurrent steps may improve performance time. To investigate this, we built a bimanual VR testbed in which the user needs to move objects to designated destinations, guided by different numbers of cues. The user can decide the order to perform the cued steps and, in some conditions, can affect which cues are shown.In a formal user study, we found that in most conditions, participants perform fastest with three cues. Dynamically updating the set of displayed cues based on hand proximity improves performance, and updating the set based on eye gaze improves performance even more. Finally, for both the hand-proximity and eye-gaze mechanisms, performance can be further improved by locking the cues for objects predicted to be moved next based on hand distance.
A Novel Anomaly Detection Method for Digital Twin Data Using Deconvolution Operation With Attention Mechanism

In recent years, industrial control systems have evolved toward stability and efficiency, increasing industrial control systems interconnected with the Internet, which means that industrial control systems are facing more serious cyber threats. Thus, it is critical for enterprises to consider issues related to data privacy and network security. Digital twin enables real-time synchronization and simulation of data from various physical components of industrial control systems. However, anomaly detection of twin data is still challenging because existing methods are usually multi-stage with tedious training and detection steps. Therefore, we propose a method called end-to-end anomaly detection with the aim to accomplish real-time anomaly detection quickly and accurately. In order to seek key features, multidimensional deconvolutional network and attention mechanism are applied to our model. The results of this study indicate that our method performs well on precision and F1 score in comparison to the state-of-art methods.
Distributed Multi-Agent Deep Reinforcement Learning for Robust Coordination against Noise

In multi-agent systems, noise reduction techniques are considerable for improving the overall system reliability as agents are required to rely on limited environmental information to develop cooperative and coordinated behaviors with the surrounding agents. However, previous studies have often applied centralized noise reduction methods to build robust and versatile coordination in noisy multi-agent environments, while distributed and decentralized autonomous agents are more plausible for real-world application. In this paper, we introduce a distributed attentional actor architecture model for a multi-agent system (DA3-X), using which we demonstrate that agents with DA3-X can selectively learn the noisy environment and behave cooperatively. We experimentally evaluate the effectiveness of DA3-X by comparing learning methods with and without DA3-X and show that agents with DA3-X can achieve better performance than baseline agents. Furthermore, we visualize heatmaps of attentional weights from the DA3-X to analyze how the decision-making process and coordinated behavior are influenced by noise.