Reviewerly/exHarmony huggingface dataset:
Explain and improve: LRP-inference fine-tuning for image captioning models

This paper analyzes the predictions of image captioning models with attention mechanisms beyond visualizing the attention itself. We develop variants of Layer-wise Relevance Propagation (LRP) and gradient-based explanation methods, tailored to image captioning models with attention mechanisms. We compare the interpretability of attention heatmaps systematically against the explanations provided by explanation methods such as LRP, Grad-CAM, and Guided Grad-CAM. We show that explanation methods provide simultaneously pixel-wise image explanations (supporting and opposing pixels of the input image) and linguistic explanations (supporting and opposing words of the preceding sequence) for each word in the predicted captions. We demonstrate with extensive experiments that explanation methods (1) can reveal additional evidence used by the model to make decisions compared to attention; (2) correlate to object locations with high precision; (3) are helpful to "debug" the model, e.g. by analyzing the reasons for hallucinated object words. With the observed properties of explanations, we further design an LRP-inference fine-tuning strategy that reduces the issue of object hallucination in image captioning models, and meanwhile, maintains the sentence fluency. We conduct experiments with two widely used attention mechanisms: the adaptive attention mechanism calculated with the additive attention and the multi-head attention mechanism calculated with the scaled dot product.
Explainable Anomaly Detection Framework for Maritime Main Engine Sensor Data

In this study, we proposed a data-driven approach to the condition monitoring of the marine engine. Although several unsupervised methods in the maritime industry have existed, the common limitation was the interpretation of the anomaly; they do not explain why the model classifies specific data instances as an anomaly. This study combines explainable AI techniques with anomaly detection algorithm to overcome the limitation above. As an explainable AI method, this study adopts Shapley Additive exPlanations (SHAP), which is theoretically solid and compatible with any kind of machine learning algorithm. SHAP enables us to measure the marginal contribution of each sensor variable to an anomaly. Thus, one can easily specify which sensor is responsible for the specific anomaly. To illustrate our framework, the actual sensor stream obtained from the cargo vessel collected over 10 months was analyzed. In this analysis, we performed hierarchical clustering analysis with transformed SHAP values to interpret and group common anomaly patterns. We showed that anomaly interpretation and segmentation using SHAP value provides more useful interpretation compared to the case without using SHAP value.
Enhancing MR image segmentation with realistic adversarial data augmentation

The success of neural networks on medical image segmentation tasks typically relies on large labeled datasets for model training. However, acquiring and manually labeling a large medical image set is resource-intensive, expensive, and sometimes impractical due to data sharing and privacy issues. To address this challenge, we propose AdvChain, a generic adversarial data augmentation framework, aiming at improving both the diversity and effectiveness of training data for medical image segmentation tasks. AdvChain augments data with dynamic data augmentation, generating randomly chained photo-metric and geometric transformations to resemble realistic yet challenging imaging variations to expand training data. By jointly optimizing the data augmentation model and a segmentation network during training, challenging examples are generated to enhance network generalizability for the downstream task. The proposed adversarial data augmentation does not rely on generative networks and can be used as a plug-in module in general segmentation networks. It is computationally efficient and applicable for both low-shot supervised and semi-supervised learning. We analyze and evaluate the method on two MR image segmentation tasks: cardiac segmentation and prostate segmentation with limited labeled data. Results show that the proposed approach can alleviate the need for labeled data while improving model generalization ability, indicating its practical value in medical imaging applications.
Supporting dyadic learning of English for tourism purposes with scenery-based virtual reality

The field of language education has experienced a rise in using virtual reality (VR) to support interactive, contextualized, and collaborative language learning in recent years. The current study investigates the effects of auditory, visual, and textual input on speaking and writing in English for Tourism Purposes (ETP) through immersive, scenery-based virtual reality (SBVR) dyadic learning. Adopting a framework on ETP sociocultural, linguistic, and formulaic competence, the researchers designed relevant learning content using audios, images, and verbatim onscreen text on EduVenture VR to create an interactive, simulated tour. Thirty-eight English as a foreign language undergraduates participated in this single-group quasi-experiment. Various data sources included speaking and writing pre-, post- and delayed post-tests, a post-intervention survey, and reflections about the VR learning experience. The study reported significant speaking gains in language accuracy, destination promotion, knowledge retention, and replication of ETP language, as well as significant writing improvement in using sophisticated vocabulary, knowledge retention, and replication of ETP language. The study concludes that SBVR warrants positive impact on improving productive ETP skills. Possible factors, limitations, and pedagogical implications for using scenery-based VR in dyadic learning for ETP are discussed. Finally, the study suggests future research directions to combine technology-enhanced language learning theories with multimodal input and material design supported by immersive VR to enhance language production for specific purposes.
The Role of Trust in Proactive Conversational Assistants

Humans and machines harmoniously collaborating and benefiting from each other is a long lasting dream for researchers in robotics and artificial intelligence. An important feature of efficient and rewarding cooperation is the ability to assume possible problematic situations and act in advance to prevent negative outcomes. This concept of assistance is known under the term proactivity. In this article, we investigate the development and implementation of proactive dialogues for fostering a trustworthy human-computer relationship and providing adequate and timely assistance. Here, we make several contributions. A formalisation of proactive dialogue in conversational assistants is provided. The formalisation forms a framework for integrating proactive dialogue in conversational applications. Additionally, we present a study showing the relations between proactive dialogue actions and several aspects of the perceived trustworthiness of a system as well as effects on the user experience. The results of the experiments provide significant contributions to the line of proactive dialogue research. Particularly, we provide insights on the effects of proactive dialogue on the human-computer trust relationship and dependencies between proactive dialogue and user specific and situational characteristics.
Hierarchical Context Embedding for Region-Based Object Detection

State-of-the-art two-stage object detectors apply a classifier to a sparse set of object proposals, relying on region-wise features extracted by RoIPool or RoIAlign as inputs. The region-wise features, in spite of aligning well with the proposal locations, may still lack the crucial context information which is necessary for filtering out noisy background detections, as well as recognizing objects possessing no distinctive appearances. To address this issue, we present a simple but effective Hierarchical Context Embedding (HCE) framework, which can be applied as a plug-and-play component, to facilitate the classification ability of a series of region-based detectors by mining contextual cues. Specifically, to advance the recognition of context-dependent object categories, we propose an image-level categorical embedding module which leverages the holistic image-level context to learn object-level concepts. Then, novel RoI features are generated by exploiting hierarchically embedded context information beneath both whole images and interested regions, which are also complementary to conventional RoI features. Moreover, to make full use of our hierarchical contextual RoI features, we propose the early-and-late fusion strategies (i.e., feature fusion and confidence fusion), which can be combined to boost the classification accuracy of region-based detectors. Comprehensive experiments demonstrate that our HCE framework is flexible and generalizable, leading to significant and consistent improvements upon various region-based detectors, including FPN, Cascade R-CNN and Mask R-CNN.
Fine-Grained Data Selection for Improved Energy Efficiency of Federated Edge Learning

In Federated edge learning (FEEL), energy-constrained devices at the network edge consume significant energy when training and uploading their local machine learning models, leading to a decrease in their lifetime. This work proposes novel solutions for energy-efficient FEEL by jointly considering local training data, available computation, and communications resources, and deadline constraints of FEEL rounds to reduce energy consumption. This paper considers a system model where the edge server is equipped with multiple antennas employing beamforming techniques to communicate with the local users through orthogonal channels. Specifically, we consider a problem that aims to find the optimal user's resources, including the fine-grained selection of relevant training samples, bandwidth, transmission power, beamforming weights, and processing speed with the goal of minimizing the total energy consumption given a deadline constraint on the communication rounds of FEEL. Then, we devise tractable solutions by first proposing a novel fine-grained training algorithm that excludes less relevant training samples and effectively chooses only the samples that improve the model's performance. After that, we derive closed-form solutions, followed by a Golden-Section-based iterative algorithm to find the optimal computation and communication resources that minimize energy consumption. Experiments using MNIST and CIFAR-10 datasets demonstrate that our proposed algorithms considerably outperform the state-of-the-art solutions as energy consumption decreases by 79% for MNIST and 73% for CIFAR-10 datasets.
Dynamic Clustering in Federated Learning

In the resource management of wireless networks, Federated Learning has been used to predict handovers. However, non-independent and identically distributed data degrade the accuracy performance of such predictions. To overcome the problem, Federated Learning can leverage data clustering algorithms and build a machine learning model for each cluster. However, traditional data clustering algorithms, when applied to the handover prediction, exhibit three main limitations: the risk of data privacy breach, the fixed shape of clusters, and the non-adaptive number of clusters. To overcome these limitations, in this paper, we propose a three-phased data clustering algorithm, namely: generative adversarial network-based clustering, cluster calibration, and cluster division. We show that the generative adversarial network-based clustering preserves privacy. The cluster calibration deals with dynamic environments by modifying clusters. Moreover, the divisive clustering explores the different number of clusters by repeatedly selecting and dividing a cluster into multiple clusters. A baseline algorithm and our algorithm are tested on a time series forecasting task. We show that our algorithm improves the performance of forecasting models, including cellular network handover, by 43%.
Predicting the future from first person (egocentric) vision: A survey

Egocentric videos can bring a lot of information about how humans perceive the world and interact with the environment, which can be beneficial for the analysis of human behaviour. The research in egocentric video analysis is developing rapidly thanks to the increasing availability of wearable devices and the opportunities offered by new large-scale egocentric datasets. As computer vision techniques continue to develop at an increasing pace, the tasks related to the prediction of future are starting to evolve from the need of understanding the present. Predicting future human activities, trajectories and interactions with objects is crucial in applications such as human–robot interaction, assistive wearable technologies for both industrial and daily living scenarios, entertainment and virtual or augmented reality. This survey summarizes the evolution of studies in the context of future prediction from egocentric vision making an overview of applications, devices, existing problems, commonly used datasets, models and input modalities. Our analysis highlights that methods for future prediction from egocentric vision can have a significant impact in a range of applications and that further research efforts should be devoted to the standardization of tasks and the proposal of datasets considering real-world scenarios such as the ones with an industrial vocation.
Privacy-preserving Byzantine-robust federated learning

Robustness of federated learning has become one of the major concerns since some Byzantine adversaries, who may upload false data owning to unreliable communication channels, corrupted hardware or even malicious attacks, might be concealed in the group of the distributed worker. Meanwhile, it has been proved that membership attacks and reverse attacks against federated learning can lead to privacy leakage of the training data. To address the aforementioned challenges, we propose a privacy-preserving Byzantine-robust federated learning scheme (PBFL) which takes both the robustness of federated learning and the privacy of the workers into account. PBFL is constructed from an existing Byzantine-robust federated learning algorithm and combined with distributed Paillier encryption and zero-knowledge proof to guarantee privacy and filter out anomaly parameters from Byzantine adversaries. Finally, we prove that our scheme provides a higher level of privacy protection compared to the previous Byzantine-robust federated learning algorithms.
Vision–based framework for automatic interpretation of construction workers' hand gestures

Construction robots have been recently developed to improve construction productivity and safety. One of the critical steps to make the robots work with human workers as teams is to provide a user-friendly interface to support their mutual interactions on construction sites. Compared with existing interfaces, hand gestures are easy to use, natural, and intuitive. This paper proposed a novel vision-based framework to capture and interpret the worker's hand gestures as a human-robot interface in construction. The framework consists of three components: worker detection and tracking, recognition queues formulation, and hand gesture recognition. Its effectiveness on the hand gesture recognition was tested with field experiments and achieved the overall precision and recall of 87.0% and 66.7%. Also, a laboratory study was conducted to illustrate the use of the framework to interact with a robotic dump truck. Future work will integrate the proposed framework into robotic construction machines.
Neural-Swarm2: Planning and Control of Heterogeneous Multirotor Swarms Using Learned Interactions

We present <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Neural-Swarm2</i> , a learning-based method for motion planning and control that allows heterogeneous multirotors in a swarm to safely fly in close proximity. Such operation for drones is challenging due to complex aerodynamic interaction forces, such as downwash generated by nearby drones and ground effect. Conventional planning and control methods neglect capturing these interaction forces, resulting in sparse swarm configuration during flight. Our approach combines a physics-based nominal dynamics model with learned deep neural networks with strong Lipschitz properties. We make use of two techniques to accurately predict the aerodynamic interactions between heterogeneous multirotors: 1) Spectral normalization for stability and generalization guarantees of unseen data and 2) heterogeneous deep sets for supporting any number of heterogeneous neighbors in a permutation-invariant manner without reducing expressiveness. The learned residual dynamics benefit both the proposed interaction-aware multirobot motion planning and the nonlinear tracking control design because the learned interaction forces reduce the modelling errors. Experimental results demonstrate that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Neural-Swarm2</i> is able to generalize to larger swarms beyond training cases and significantly outperforms a baseline nonlinear tracking controller with up to three times reduction in worst-case tracking errors.
Analysis of Obstacle Avoidance Strategy for Dual-Arm Robot Based on Speed Field with Improved Artificial Potential Field Algorithm

In recent years, dual-arm robots have been favored in various industries due to their excellent coordinated operability. One of the focused areas of study on dual-arm robots is obstacle avoidance, namely path planning. Among the existing path planning methods, the artificial potential field (APF) algorithm is widely applied in obstacle avoidance for its simplicity, practicability, and good real-time performance over other planning methods. However, APF is firstly proposed to solve the obstacle avoidance problem of mobile robot in plane, and thus has some limitations such as being prone to fall into local minimum, not being applicable when dynamic obstacles are encountered. Therefore, an obstacle avoidance strategy for a dual-arm robot based on speed field with improved artificial potential field algorithm is proposed. In our method, the APF algorithm is used to establish the attraction and repulsion functions of the robotic manipulator, and then the concepts of attraction and repulsion speed are introduced. The attraction and repulsion functions are converted into the attraction and repulsion speed functions, which mapped to the joint space. By using the Jacobian matrix and its inverse to establish the differential velocity function of joint motion, as well as comparing it with the set collision distance threshold between two robotic manipulators of robot, the collision avoidance can be solved. Meanwhile, after introducing a new repulsion function and adding virtual constraint points to eliminate existing limitations, APF is also improved. The correctness and effectiveness of the proposed method in the self-collision avoidance problem of a dual-arm robot are validated in MATLAB and Adams simulation environment.
A multi-objective algorithm for crop pattern optimization in agriculture

The crop growth and crop yield in agriculture depend upon many factors such as weather conditions, soil type, and application of fertilizers. The crop net return can be increased by deciding suitable crops for a particular land depending upon its weather conditions. The application of an appropriate amount of fertilizers also promotes crop growth and yield. On the other hand, the use of fertilizers needs to be minimized to reduce the capital cost as well as to prevent its harmful effect on soil and the environment. This paper models the problem of increasing net crop benefit and reducing the application of fertilizers as multi-objective optimization functions. A hybrid CSA-PSO optimization algorithm is proposed for solving multi-objective problems by combining crow search algorithm (CSA) and particle swarm optimization (PSO). The performance of the proposed algorithm is evaluated against CEC 2009 benchmark functions. This algorithm is implemented for crop pattern optimization in India’s Telangana state, which depicts the proposed algorithm’s feasibility and effectiveness.
MimicNet

At-scale evaluation of new data center network innovations is becoming increasingly intractable. This is true for testbeds, where few, if any, can afford a dedicated, full-scale replica of a data center. It is also true for simulations, which while originally designed for precisely this purpose, have struggled to cope with the size of today's networks. This paper presents an approach for quickly obtaining accurate performance estimates for large data center networks. Our system,MimicNet, provides users with the familiar abstraction of a packet-level simulation for a portion of the network while leveraging redundancy and recent advances in machine learning to quickly and accurately approximate portions of the network that are not directly visible. MimicNet can provide over two orders of magnitude speedup compared to regular simulation for a data center with thousands of servers. Even at this scale, MimicNet estimates of the tail FCT, throughput, and RTT are within 5% of the true results.
Yog-Guru: Real-Time Yoga Pose Correction System Using Deep Learning Methods

Self-learning is an integral part in Yoga, but incorrect posture while performing yoga can lead to serious harm to muscles and ligaments of the body. Thus to prevent this we present an intuitive approach based on machine learning techniques to correct the practitioner's pose while performing various yoga asanas. The proposed system is aimed at providing concise feedback to the practitioner so they are able perform yoga poses correctly and assist them in identifying the incorrect poses and suggest a proper feedback for improvement in order to prevent injuries as well as increase their knowledge of a particular yoga pose. A data-set of Five Yoga pose (i.e. Natarajasana and Trikonasana, and Vrikshasana and Virbhadrasana 1 & 2 and Utkatasana) has been created from collecting images from the Internet as well as from different individuals that took part in development of this system.A deep learning model is proposed which uses convolutional neural networks (CNN) for yoga pose identification along with a human joints localization model followed by a process for identification of errors in the pose for developing the system. Using the proposed system we have been able to achieve a classification accuracy of 95% for pose identification. After obtaining all the information about the pose of the user the system gives feedback to improve or correct the posture of the user.
The robot-to-robot service encounter: an examination of the impact of inter-robot warmth

Purpose This study aims to examine humans’ reactions to service robots’ display of warmth in robot-to-robot interactions – a setting in which humans’ impressions of a service robot will not only be based on what this robot does in relation to humans, but also on what it does to other robots. Design/methodology/approach Service robot display of warmth was manipulated in an experimental setting in such a way that a service robot A expressed low versus high levels of warmth in relation to another service robot B. Findings The results indicate that a high level of warmth expressed by robot A vis-à-vis robot B boosted humans’ overall evaluations of A, and that this influence was mediated by the perceived humanness and the perceived happiness of A. Originality/value Numerous studies have examined humans’ reactions when they interact with a service robot or other synthetic agents that provide service. Future service encounters, however, will comprise also multi-robot systems, which means that there will be many opportunities for humans to be exposed to robot-to-robot interactions. Yet, this setting has hitherto rarely been examined in the service literature.