Reviewerly/exHarmony huggingface dataset:
Weighted Node Mapping and Localisation on a Pixel Processor Array

This paper implements and demonstrates visual route mapping and localisation upon a Pixel Processor Array (PPA). The PPA sensor comprises of an array of Processing Elements (PEs), each of which can capture and process visual information directly. This provides significant parallel processing power allowing novel ways in which information can be processed on-sensor. Our method predicts the correct node within a topological map generated from an image sequence by measuring image similarities, spatial coherence, and exploiting the parallel nature of the PPA. Our implementation runs at +300Hz on large public datasets with +2K locations requiring 2.5W at 500 GOPS/W. We compare vs traditionally implemented methods demonstrating better F-1 performance even on simulation. As far as we are aware, we present the first on-sensor mapping and localisation system running entirely on-sensor.
Learning to Decode Contextual Information for Efficient Contour Detection

Contour detection plays an important role in both academic research and real-world applications. As the basic building block of many applications, its accuracy and efficiency highly influence the subsequent stages. In this work, we propose a novel lightweight system for contour detection that achieves state-of-the-art performance while keeps ultra-slim model size. The proposed method is built on an efficient encoder in a bottom-up/top-down fashion. Specially, we propose a novel decoder that compresses side features from an encoder and effectively decodes compact contextual information for high-accurate boundary localization. Besides, we propose a novel loss function that is able to assist a model to produce crisp object boundaries.
Synchrony within Triads using Virtual Reality

Synchrony, the natural time-dependence of behavior in human interaction, is a pervasive feature of communication. However, most studies of synchrony have focused on dyadic interaction. In the current work, we explore synchrony in three-person teams using immersive virtual reality. Participants spent about two hours collaborating on four separate design tasks. The tracking data from the VR system allowed precise measurement of head and hand movements, facilitating calculation of synchrony. Results replicated previous work that found nonverbal synchrony in dyads in immersive VR. Moreover, we manipulated the context of the task environment, an informal garage or a traditional conference room. The environment for the task influenced synchrony, with higher levels occurring in the conference room than the garage. We also explored different methods of extending synchrony from dyads to triads, and explore the relationship of synchrony to turn taking and gaze. This paper provides theoretical insights about nonverbal synchrony and how design work functions in triads and provides suggestions for designers of VR to support good collaboration.
Perceptual VVC quantization refinement with ensemble learning

Compressing videos while maintaining an acceptable level of Quality of Experience (QoE) is indispensable. To this aim, a feasible method is to further increase the Quantization Parameter (QP) of video stream to eliminate visual redundancy, simultaneously utilizing perceptual characteristics of Human Visual System (HVS) to impose a threshold constraint on the maximum QP. In this paper, we employ Just Noticeable Distortion (JND) to characterize the aforementioned threshold constraint, thereby avoiding perceptual loss during QP refinement process. We propose an effective JND-based algorithm for QP optimization, in which a video saliency detection is introduced to extract regions of interest, a refinement model based on a lightweight network is designed to predict QP value and an ensemble learning method to improve generalization performance. Theoretical analysis and experimental results demonstrate that the proposed algorithm has been successfully applied to Versatile Video Coding (VVC) to achieve significant bitrate reduction without sacrificing perceived quality.
Telematic Wearable Music

The author discusses telematic wearable music, recounting the design and evolution of improvised techniques and approaches in a remotely taught course offered to undergraduates. This new contribution to interactive interfaces for remote ensembles is musically motivated and inclusive for non-specialists who apply musical instincts they discover through participation. Students are introduced to wearable "Internet of Things" (IoT) computing, synthesis, and sound design, with the goal of developing rich, movement responsive, individually and/or collectively playable wearable instruments. The course facilitates practice-situated investigation of accessible, agile, and inexpensive modes of distributed creativity in musical interaction through experiential inquiry and tinkering. Telematic wearable music aspires to enact shared, situated spaces and less ocularcentric modes of learning through embodied sonic telepresence, emphasizing and enhancing embodied participation in synchronous remote learning.
Serverless Multi-Query Motion Planning for Fog Robotics

Robots in semi-structured environments such as homes and warehouses sporadically require computation of high-dimensional motion plans. Cloud and fog-based parallelization of motion planning can speed up planning. This can be further made efficient by the use of "serverless" on-demand computing as opposed to always-on high end computers. This paper explores parallelizing the computation of a sampling-based multi-query motion planner based on asymptotically-optimal Probabilistic Road Maps (PRM*) using the simultaneous execution of 100s of cloud-based serverless functions. We propose an algorithm to overcome the communication and bandwidth limitations of serverless computing and use different work-sharing techniques to further optimize the cost and run time. Additionally, we provide proofs of probabilistic completeness and asymptotic optimality. In experiments on synthetic benchmarks and on a physical Fetch robot performing a sequence of decluttering motions, we observe up to a 50x speedup relative to a 4 core edge computer with only a marginally higher cost.
End-to-End FPGA-based Object Detection Using Pipelined CNN and Non-Maximum Suppression

Object detection is an important computer vision task, with many applications in autonomous driving, smart surveillance, robotics, and other domains. Single-shot detectors (SSD) coupled with a convolutional neural network (CNN) for feature extraction can efficiently detect, classify and localize various objects in an input image with very high accuracy. In such systems, the convolution layers extract features and predict the bounding box locations for the detected objects as well as their confidence scores. Then, a non-maximum suppression (NMS) algorithm eliminates partially overlapping boxes and selects the bounding box with the highest score per class. However, these two components are strictly sequential; a conventional NMS algorithm needs to wait for all box predictions to be produced before processing them. This prohibits any overlap between the execution of the convolutional layers and NMS, resulting in significant latency overhead and throughput degradation. In this paper, we present a novel NMS algorithm that alleviates this bottleneck and enables a fully-pipelined hardware implementation. We also implement an end-to-end system for low-latency SSD-MobileNet-V1 object detection, which combines a state-of-the-art deeply-pipelined CNN accelerator with a custom hardware implementation of our novel NMS algorithm. As a result of our new algorithm, the NMS module adds a minimal latency overhead of only 0.13μ s to the SSD-MobileNet-V1 convolution layers. Our end-to-end object detection system implemented on an Intel Stratix 10 FPGA runs at a maximum operating frequency of 350 MHz, with a throughput of 609 frames-per-second and an end-to-end batch-1 latency of 2.4 ms. Our system achieves 1.5× higher throughput and 4.4× lower latency compared to the current state-of-the-art SSD-based object detection systems on FPGAs.
Order Matters: Generating Progressive Explanations for Planning Tasks in Human-Robot Teaming

Prior work on generating explanations in a planning context has focused on providing the rationale behind an AI agent’s decision-making. While these methods offer the right explanations, they fail to heed the cognitive requirement of understanding an explanation from the explainee or human’s perspective. In this work, we set out to address this issue by considering the order for communicating information in an explanation, or the progressiveness of making explanations. Progression is the notion of building complex concepts on simpler ones, which is known to benefit learning. In this work, we investigate a similar effect when an explanation is composed of multiple parts that are communicated sequentially. The challenge here lies in determining the order for receiving different parts of an explanation that would assist in understanding. Given the sequential nature, a formulation based on goal-based MDP is presented. The reward function of this MDP is learned via inverse reinforcement learning based on training data. We evaluated our approach in an escape-room domain to demonstrate its effectiveness. Upon analyzing the results, it revealed that the desired order arises strongly from both domain-dependent and independence features. This result confirmed our expectation that the process of understanding an explanation for planning tasks was progressive and context dependent. We also showed that the explanations generated using the learned rewards achieved better task performance and simultaneously reduced cognitive load. These results shed light on designing explainable robots across various domains.
Stream-Function Based 3D Obstacle Avoidance Mechanism for Mobile AUVs in the Internet of Underwater Things

The mobility manager in the Internet of Underwater Things (IoUT) plays a major role in data acquisition.In this paper, one novel obstacle avoidance method based on fluid mechanics is discussed for autonomous underwater vehicle (AUV) in 3D IoUT.The proposed method utilizes ocean current characteristics simplified as stream-function to design an optimal 3D trajectory with an obstacle avoidance function.3D stream-function is constructed for spherical and cylindrical obstructions in details.Theoretical analysis of 3D stream-function proves that traditional 2D stream-function can be extended to 3D IoUT space under specific conditions.One path deformation method based on virtual obstacle methodology is proposed to overcome the inherent hysteresis problem of the traditional stream-function based obstacle avoidance design.Moreover, we introduce an energy consumption model for AUV to prove that our obstacle avoidance algorithm can improve energy efficiency if actual ocean currents exist.Extensive simulation results verify that the proposed obstacle avoidance method has the characteristics of curve continuity and smoothness, enhanced obstacle avoidance effects, and high energy efficiency.Therefore, the proposed method meets the actual requirements of multi-obstacle avoidance path planning for AUV in IoUT.
A Geometric Folding Pattern for Robot Coverage Path Planning

Conventional coverage path planning algorithms are mainly based on the zigzag and spiral patterns or their combinations. The traversal order is limited by the linear or inside-outside manner. We propose a new set of coverage patterns induced from geometric folding operations, called the geometric folding pattern, to make coverage paths with more flexible traversal order. We study the modeling and parameterization of the geometric folding patterns. Then, a sampling operator is introduced. Based on the computational tools, we demonstrate the application of the proposed patterns in designing coverage paths. We show that the simple geometric folding patterns are flexible and controllable, which enables more choices for the coverage path planning problem.
Near-Optimal Multi-Robot Motion Planning with Finite Sampling

An underlying structure in several sampling-based methods for continuous multi-robot motion planning (MRMP) is the tensor roadmap (PR), which emerges from combining multiple PRM graphs constructed for the individual robots via a tensor product. We study the conditions under which the TR encodes a near-optimal solution for MRMP—satisfying these conditions implies near optimality for a variety of popular planners, including dRRT*, and the discrete methods M* and CBS when applied to the continuous domain. We develop the first finite-sample analysis of this kind, which specifies the number of samples, their deterministic distribution, and magnitude of the connection radii that should be used by each individual PRM graph, to guarantee near-optimality using the TR. This significantly improves upon a previous asymptotic analysis, wherein the number of samples tends to infinity. Our new finite sample-size analysis supports guaranteed high- quality solutions in practice within finite time. To achieve our new result, we first develop a sampling scheme, which we call the staggered grid, for finite-sample motion planning for individual robots, which requires significantly less samples than previous work. We then extend it to the much more involved MRMP setting which requires to account for interactions among multiple robots. Finally, we report on a few experiments that serve as a verification of our theoretical findings and raise interesting questions for further investigation.
Improving Unsupervised Domain Adaptive Re-Identification Via Source-Guided Selection of Pseudo-Labeling Hyperparameters

Unsupervised Domain Adaptation (UDA) for re-identification (re-ID) is a challenging task: to avoid a costly annotation of additional data, it aims at transferring knowledge from a domain with annotated data to a domain of interest with only unlabeled data. Pseudo-labeling approaches have proven to be effective for UDA re-ID. However, the effectiveness of these approaches heavily depends on the choice of some hyperparameters (HP) that affect the generation of pseudo-labels by clustering. The lack of annotation in the domain of interest makes this choice non-trivial. Current approaches simply reuse the same empirical value for all adaptation tasks and regardless of the target data representation that changes through pseudo-labeling training phases. As this simplistic choice may limit their performance, we aim at addressing this issue. We propose new theoretical grounds on HP selection for clustering UDA re-ID as well as method of automatic and cyclic HP tuning for pseudo-labeling UDA clustering: HyPASS. HyPASS consists in incorporating two modules in pseudo-labeling methods: (i) HP selection based on a labeled source validation set and (ii) conditional domain alignment of feature discriminativeness to improve HP selection based on source samples. Experiments on commonly used person re-ID and vehicle re-ID datasets show that our proposed HyPASS consistently improves the best state-of-the-art methods in re-ID compared to the commonly used empirical HP setting.
Constrained Image-Based Visual Servoing using Barrier Functions

This paper presents a novel constrained image-based visual servoing (IBVS) approach that guarantees target features to remain within the camera field of view (FOV) for the duration of the task. Barrier function (BF) is used to transform the feature dynamics on the image plane into equivalent dynamics with full state constraints. An IBVS controller is designed for the full state constrained system that regulates the original system. A stability analysis is provided to prove the local asymptotic convergence of the state error in the sense of Lyapunov. Simulation and robot implementation results show the efficiency of the proposed method to reach the desired features from any initial configuration while the visibility constraints of the feature points are being realized.
Omni-Sparsity DNN: Fast Sparsity Optimization for On-Device Streaming E2E ASR Via Supernet

From wearables to powerful smart devices, modern automatic speech recognition (ASR) models run on a variety of edge devices with different computational budgets. To navigate the Pareto front of model accuracy vs model size, researchers are trapped in a dilemma of optimizing model accuracy by training and fine-tuning models for each individual edge device while keeping the training GPU-hours tractable. In this paper, we propose Omni-sparsity DNN, where a single neural network can be pruned to generate optimized model for a large range of model sizes. We develop training strategies for Omni-sparsity DNN that allows it to find models along the Pareto front of word-error-rate (WER) vs model size while keeping the training GPU-hours to no more than that of training one singular model. We demonstrate the Omni-sparsity DNN with streaming E2E ASR models. Our results show great saving on training time and resources with similar or better accuracy on LibriSpeech compared to individually pruned sparse models: 2%-6.6% better WER on Test-other.
Development of a software module for recognizing the fingerspelling of the Russian Sign Language based on LSTM

Russian Sign Language is a natural language that serves as a means of communication for people with hearing impairments. Currently, it is necessary to provide communication paths between hearing and deaf people, which requires the solution of several specific problems, one of which is gesture recognition. The paper presents a system for recognizing isolated static and dynamic gestures of the dactyl alphabet of the Russian Sign Language. The system is based on machine learning methods and is an LSTM with Mediapipe Hands as feature extractor. The network showed an F-measure value of 91% on test data.
Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation

Model-based deep reinforcement learning has achieved success in various domains that require high sample efficiencies, such as Go and robotics. However, there are some remaining issues, such as planning efficient explorations to learn more accurate dynamic models, evaluating the uncertainty of the learned models, and more rational utilization of models. To mitigate these issues, we present MEEE, a model-ensemble method that consists of optimistic exploration and weighted exploitation. During exploration, unlike prior methods directly selecting the optimal action that maximizes the expected accumulative return, our agent first generates a set of action candidates and then seeks out the optimal action that takes both expected return and future observation novelty into account. During exploitation, different discounted weights are assigned to imagined transition tuples according to their model uncertainty respectively, which will prevent model predictive error propagation in agent training. Experiments on several challenging continuous control benchmark tasks demonstrated that our approach outperforms other model-free and model-based state-of-the-art methods, especially in sample complexity.
Architecture evaluation in continuous development

In automotive, stage-gate processes have previously been the norm, with architecture created mainly during an early phase and then used to guide subsequent development phases. Current iterative and Agile development methods, where the implementation evolves continuously, changes the role of architecture. We investigate how architecture evaluation can provide useful feedback during development of continuously evolving systems. Starting from the Architecture Tradeoff Analysis Method (ATAM), we performed architecture evaluation, both in a national research project led by an automotive Original Equipment Manufacturer (OEM), and at the OEM, in the context of continuous development. This allows us to include the experience of several architects from different organizations over several years. Using data produced during the evaluations we perform a post-hoc analysis to derive initial findings. We then validate and refine these findings through a series of focus groups with architects and industry experts. We propose principles of continuous evaluation and evolution of architecture, and based on these discuss a roadmap for future research. In iterative development settings, the needs are different from what typical architecture evaluation methods provide. Our principles show the importance of dedicated feedback-loops for continuous evolution of systems and their architecture.
Analytical Comparison of Clustering Techniques for the Recognition of Communication Patterns

Abstract The systematic processing of unstructured communication data as well as the milestone of pattern recognition in order to determine communication groups in negotiations bears many challenges in Machine Learning. In particular, the so-called curse of dimensionality makes the pattern recognition process demanding and requires further research in the negotiation environment. In this paper, various selected renowned clustering approaches are evaluated with regard to their pattern recognition potential based on high-dimensional negotiation communication data. A research approach is presented to evaluate the application potential of selected methods via a holistic framework including three main evaluation milestones: the determination of optimal number of clusters, the main clustering application, and the performance evaluation. Hence, quantified Term Document Matrices are initially pre-processed and afterwards used as underlying databases to investigate the pattern recognition potential of clustering techniques by considering the information regarding the optimal number of clusters and by measuring the respective internal as well as external performances. The overall research results show that certain cluster separations are recommended by internal and external performance measures by means of a holistic evaluation approach, whereas three of the clustering separations are eliminated based on the evaluation results.
Pareto Optimality for Fairness-constrained Collaborative Filtering

The well-known collaborative filtering (CF) models typically optimize a single objective summed over all historical user-item interactions. Due to inevitable imbalances and biases in real-world data, they may develop a policy that unfairly discriminates against certain subgroups with low sample frequencies. To balance overall recommendation performance and fairness, prevalent solutions apply fairness constraints or regularizations to enforce equality of certain performance across different subgroups. However, simply enforcing equality of performance may lead to large performance degradation of those advantaged subgroups. To address this issue, we formulate a constrained Multi-Objective Optimization (MOO) problem. In contrast to the single objective, we treat the performance of each subgroup equivalently as an objective. This ensures that the imbalanced subgroup sample frequency does not affect the gradient information. We further propose fairness constraints to limit the search space to obtain more balanced solutions. To solve the constrained MOO problem, a gradient-based constrained MOO algorithm is proposed to seek a proper Pareto optimal solution for the performance trade-off. Extensive experiments on synthetic and real-world datasets show that our approach could help improve the recommendation accuracy of disadvantaged groups, while not damaging the overall performance.