Reviewerly/exHarmony huggingface dataset:
Exploring Visual Pre-training for Robot Manipulation: Datasets, Models and Methods

Visual pre-training with large-scale real-world data has made great progress in recent years, showing great potential in robot learning with pixel observations. However, the recipes of visual pre-training for robot manipulation tasks are yet to be built. In this paper, we thoroughly investigate the effects of visual pre-training strategies on robot manipulation tasks from three fundamental perspectives: pre-training datasets, model architectures and training methods. Several significant experimental findings are provided that are beneficial for robot learning. Further, we propose a visual pre-training scheme for robot manipulation termed Vi-PRoM, which combines self-supervised learning and supervised learning. Concretely, the former employs contrastive learning to acquire underlying patterns from large-scale unlabeled data, while the latter aims learning visual semantics and temporal dynamics. Extensive experiments on robot manipulations in various simulation environments and the real robot demonstrate the superiority of the proposed scheme. Videos and more details can be found on https://explore-pretrain-robot.github.io.
Multimodal Diffusion Segmentation Model for Object Segmentation from Manipulation Instructions

In this study, we aim to develop a model that comprehends a natural language instruction (e.g., “Go to the living room and get the nearest pillow to the radio art on the wall”) and generates a segmentation mask for the target everyday object. The task is challenging because it requires (1) the understanding of the referring expressions for multiple objects in the instruction, (2) the prediction of the target phrase of the sentence among the multiple phrases, and (3) the generation of pixel-wise segmentation masks rather than bounding boxes. Studies have been conducted on language-based segmentation methods; however, they sometimes mask irrelevant regions for complex sentences. In this paper, we propose the Multimodal Diffusion Segmentation Model (MDSM), which generates a mask in the first stage and refines it in the second stage. We introduce a crossmodal parallel feature extraction mechanism and extend diffusion probabilistic models to handle crossmodal features. To validate our model, we built a new dataset based on the well-known Matterport3D and REVERIE datasets. This dataset consists of instructions with complex referring expressions accompanied by real indoor environmental images that feature various target objects, in addition to pixel-wise segmentation masks. The performance of MDSM surpassed that of the baseline method by a large margin of +10.13 mean IoU.
Methodology of Teaching Subjects Based on Mobile Technologies in Higher Education Institutions

In this scientific article, the content and functions of creating interactive mobile educational resources in the electronic educational environment and the development of teaching based on them, analyzing the methodology, management models and tools, and developing recommendations for the development of the system of creating interactive mobile educational resources are studied in detail. It has been scientifically proven that the development of teaching on the basis of interactive mobile learning technologies, the search for knowledge, its understanding, strengthening, the formation of special knowledge, and the generalization and systematization of knowledge have an important place in higher education institutions. The analysis carried out as part of this scientific research includes the following:analysis of literature on the method of creating and using interactive mobile educational resources in the electronic educational environment, development of models and software tools for creating interactive mobile educational resources;creation of mobile educational resources and levels of use in education, electronic resources for transfer of educational materials, content and functions of using interactive mobile educational resources.The technology of creating interactive mobile educational resources in the electronic educational environment, the method of using interactive educational resources, and the development of the model of teaching methods of subjects based on mobile technologies in higher education institutions will be scientifically and practically demonstrated.
Coordination of Bounded Rational Drones Through Informed Prior Policy

Biological agents, such as humans and animals, are capable of making decisions out of a very large number of choices in a limited time. They can do so because they use their prior knowledge to find a solution that is not necessarily optimal but good enough for the given task. In this work, we study the motion coordination of multiple drones under the above-mentioned paradigm, Bounded Rationality (BR), to achieve cooperative motion planning tasks. Specifically, we design a prior policy that provides useful goal-directed navigation heuristics in familiar environments and is adaptive in unfamiliar ones via Reinforcement Learning augmented with an environment-dependent exploration noise. Integrating this prior policy in the game-theoretic bounded rationality framework allows agents to quickly make decisions in a group considering other agents' computational constraints. Our investigation assures that agents with a well-informed prior policy increase the efficiency of the collective decision-making capability of the group. We have conducted rigorous experiments in simulation and in the real world to demonstrate that the ability of informed agents to navigate to the goal safely can guide the group to coordinate efficiently under the BR framework.
Separable Fuzzy Models

One of the directions of application of the theory of fuzzy models is model-theoretic formalization of quantum computing. The central concept of quantum computing is quantum entanglement. A quantum system is considered to be unentangled or separable if the results of measurements of some elements of the system do not depend on measurements of other elements.Fuzzy Model can be considered as a generalization of quantum system. The paper deals with the generalization of the separability property of the system to the case of arbitrary Fuzzy Model. The definition of Fuzzy Separable Model is given. It is proved that under given initial conditions (atomic diagram of the model) there exists not more than one Separable Fuzzy Model. The criterion, which the distribution function of a fuzzy model must have in order for this model to become separable, is described. The notion of the rank of the Separable Model is introduced, which can be regarded as a measure of entropy of fuzziness of the Separable Model.
DMCL: Robot Autonomous Navigation via Depth Image Masked Contrastive Learning

Achieving high performance in deep reinforcement learning relies heavily on the ability to obtain good state representations from pixel inputs. However, learning an observation-space-to-action-space mapping from high-dimensional inputs is challenging in reinforcement learning, particularly when dealing with consecutive depth images as input states. In addition, we observe that the consecutive inputs of depth images are highly correlated for the autonomous navigation of a mobile robot, which inspires us to capture temporal correlations between consecutive inputs and infer scene change relationships. To this end, we propose a novel end-to-end robot vision navigation method dubbed DMCL, which obtains good spatial-temporal state representation via Depth image Masked Contrastive Learning. It reconstructs the latent representation from consecutive depth images masked in both spatial and temporal dimensions, resulting in a complete environment state representation. To obtain the optimal navigation policy, we leverage the Soft Actor-Critic reinforcement learning in conjunction with the above representation learning. Extensive experiments demonstrate that the proposed DMCL outperforms representative state-of-the-art methods. The source code will be made publicly available.
EvCenterNet: Uncertainty Estimation for Object Detection Using Evidential Learning

Uncertainty estimation is crucial in safety-critical settings such as automated driving as it provides valuable information for several downstream tasks including high-level decision making and path planning. In this work, we propose EvCenterNet, a novel uncertainty-aware 2D object detection framework using evidential learning to directly estimate both classification and regression uncertainties. To employ evidential learning for object detection, we devise a combination of evidential and focal loss functions for the sparse heatmap inputs. We introduce class-balanced weighting for regression and heatmap prediction to tackle the class imbalance encountered by evidential learning. Moreover, we propose a learning scheme to actively utilize the predicted heatmap uncertainties to improve the detection performance by focusing on the most uncertain points. We train our model on the KITTI dataset and evaluate it on challenging out-of-distribution datasets including BDD100K and nuImages. Our experiments demonstrate that our approach improves the precision and minimizes the execution time loss in relation to the base model.
Multi-Robot Planning on Dynamic Topological Graphs Using Mixed- Integer Programming

Planning for multi-robot teams in complex environments is a challenging problem, especially when these teams must coordinate to accomplish a common objective. In general, optimal solutions to these planning problems are computationally intractable, since the decision space grows exponentially with the number of robots. In this paper, we present a novel approach for multi-robot planning on topological graphs using mixed-integer programming. Central to our approach is the notion of a dynamic topological graph, where edge weights vary dynamically based on the locations of the robots in the graph. We construct this graph using the critical features of the planning problem and the relationships between robots; we then leverage mixed-integer programming to minimize a shared cost that depends on the paths of all robots through the graph. To improve computational tractability, we formulated our optimization problem with a fully convex relaxation and designed our decision space around eliminating the exponential dependence on the number of robots. We test our approach on a multi-robot reconnaissance scenario, where robots must coordinate to minimize detectability and maximize safety while gathering information. We demonstrate that our approach is able to scale to a series of representative scenarios and is capable of computing optimal coordinated strategic behaviors for autonomous multi-robot teams in seconds.
User Interactions and Negative Examples to Improve the Learning of Semantic Rules in a Cognitive Exercise Scenario

Enabling a robot to perform new tasks is a complex endeavor, usually beyond the reach of non-technical users. For this reason, research efforts that aim at empowering end-users to teach robots new abilities using intuitive modes of interaction are valuable. In this article, we present INtuitive PROgramming 2 (INPRO2), a learning framework that allows inferring planning actions from demonstrations given by a human teacher. INPRO2 operates in an assistive scenario, in which the robot may learn from a healthcare professional (a therapist or caregiver) new cognitive exercises that can be later administered to patients with cognitive impairment. INPRO2 features significant improvements over previous work, namely: (1) exploitation of negative examples; (2) proactive interaction with the teacher to ask questions about the legality of certain movements; and (3) learning goals in addition to legal actions. Through simulations, we show the performance of different proactive strategies for gathering negative examples. Real-world experiments with human teachers and a TIAGo robot are also presented to qualitatively illustrate INPRO2.
Visualization of the Oscillatory Dynamics of an Island Power System

In this work, we discuss the design of visualizations for understanding the complex oscillatory dynamics of an island power system with renewable generation sources after the loss of a large oil power plant. As more renewable generation sources are added to power systems, the oscillatory dynamics will change, which requires new visualization techniques to determine causes and strategies to avoid unwanted behaviors in the future. Our approach integrates geographic views, time-series plots, and novel oscillatory-trajectory curves, providing unique insights into the interdependent oscillatory behaviors of multiple state variables and generators over time. By enabling multi-node and multivariate comparisons over time, users can qualitatively determine drivers of oscillations and differences in generator dynamics, which is not possible with other commonly used visualization techniques.
Active Multi-Modal Approach for Enhanced User Recognition in Social Robots

The field of Human-Robot Interaction (HRI) is swiftly expanding, driven by notable advancements in artificial intelligence (AI). Humanoid robots, now capable of being equipped with advanced AI models, are being considered for a wide array of applications due to their ability to perform complex tasks and interact with humans in both natural and intelligent ways. In the domain of social robotics, the capability to selectively interact with authorized users is crucial for ensuring security and providing personalized user experiences. Unimodal user recognition methods, such as audio-based user recognition, are commonly used in social robots. However, these methods can be susceptible to ambient noise and might exhibit reduced accuracy. Although face recognition modality is often employed to enhance accuracy, the majority of audio-visual person recognition methods are trained on datasets with only a single user in the frame.This paper introduces a method for audio-visual user recognition with multiple users in the frame, utilizing an additional sound localization modality. The proposed method is evaluated using a dataset created from interactions between the social robot Pepper and multiple users. The results demonstrated that the proposed method significantly outperformed unimodal user recognition methods.
Attention Guided Multi-attribute Architecture For Deepfake Detection

Deepfake content generated through visual and audio manipulations is speculated to become one of the most threatening artificial intelligence (AI) technologies capable of grave negative impact on society through identity fraud, reputation damage, and undermining of trust in large-scale political campaigns and criminal investigation procedures. A significant measure of the efforts towards solving this problem fails to generalize beyond the training data. We hypothesize that the primary reason behind the drawback is the dearth of efforts toward the exploitation of multiple channels of information from the data. Towards this goal, we investigate a multimodal paradigm exploiting an attention-informed feature generation procedure through a deep network. We further augment the supervisory signal using camera sensor fingerprints which contain additional corroboratory information for the decision-making process. We demonstrate that a non-linear transformation of the decision space augmented through multiple channels allows for a significant boost in the generalization capacity over unseen datasets or attacks. We illustrate the applicability of our framework on the widely known datasets of FaceForensics++ and CelebDF.
Vision Transformer-based Sinogram Enlarging for Reducing Artifacts in Sparse-view Micro-CT

Micro-computed tomography (micro-CT) provides three-dimensional (3D) morphological structures at the micrometer scale. Although this modality is expected to contribute to histopathology for analyzing the 3D microstructures of tissue specimens, micro-CT imaging for tissue specimens requires a long scan time. Sparse-view CT, which reduces the number of projections, has great promise for speeding up the scanning process; however, analytical reconstruction from fewer projections causes severe streak artifacts on tomographic images. Convolutional neural network (CNN) based artifact reduction methods have been proposed and outperformed conventional filter processing and compressed sensing approaches. However, CNNs are inefficient for capturing long-range characteristics. Vision Transformer, as an alternative to CNN, has recently appeared, and Swin Transformer achieved state-of-the-art performance on image classification tasks on benchmark datasets. SwinIR based on the Swin Transformer also outperformed the performance of CNN-based networks for image super-resolution tasks. In this paper, we proposed an artifact reduction method for sparse-view micro-CT. We developed a SwinIR-based deep learning network for vertically enlarging sparse-view sinograms to estimate full-view sinograms. Experimental results showed that the proposed method achieved the best performance with the lowest total error among the comparison methods.
RAMP: Hierarchical Reactive Motion Planning for Manipulation Tasks Using Implicit Signed Distance Functions

We introduce Reactive Action and Motion Planner (RAMP), which combines the strengths of sampling-based and reactive approaches for motion planning. In essence, RAMP is a hierarchical approach where a novel variant of a Model Predictive Path Integral (MPPI) controller is used to generate trajectories which are then followed asynchronously by a local vector field controller. We demonstrate, in the context of a table clearing application, that RAMP can rapidly find paths in the robot's configuration space, satisfy task and robot-specific constraints, and provide safety by reacting to static or dynamically moving obstacles. RAMP achieves superior performance through a number of key innovations: we use Signed Distance Function (SDF) representations directly from the robot configuration space, both for collision checking and reactive control. The use of SDFs allows for a smoother definition of collision cost when planning for a trajectory, and is critical in ensuring safety while following trajectories. In addition, we introduce a novel variant of MPPI which, combined with the safety guarantees of the vector field trajectory follower, performs incremental real-time global trajectory planning. Simulation results establish that our method can generate paths that are comparable to traditional and state-of-the-art approaches in terms of total trajectory length while being up to 30 times faster. Real-world experiments demonstrate the safety and effectiveness of our approach in challenging table clearing scenarios. Videos and code are available at: https://samsunglabs.github.io/RAMP-project-page/
Are Industrial ML Image Classifiers Robust to Data Affected by Network QoS Degradation?

In industrial applications, Machine Learning (ML) services are often deployed on cloud infrastructure and require a transfer of the input data over a network, which is susceptible to Quality of Service (QoS) degradation. In this paper we investigate the robustness of industrial ML classifiers towards varying Data Quality (DQ) due to degradation in network QoS. We define the robustness of an ML model as the ability to maintain a certain level of performance under variable levels of DQ at its input. We employ the classification accuracy as the performance metric for the ML classifiers studied. The POWDER testbed is utilized to create an experimental setup consisting of a real-world wireless network connecting two nodes. We transfer multiple video and image files between the two nodes under varying degrees of packet loss and varying buffer sizes to create degraded data. We then evaluate the performance of AWS Rekognition, a commercial ML tool for on-demand object detection, on corrupted video and image data. We also evaluate the performance of YOLOv7 to compare the performance of a commercial and an open-source model. As a result we demonstrate that even a slight degree of packet loss, 1% for images and 0.2% for videos, can have a drastic impact on the classification performance of the system. We discuss the possible ways to make industrial ML systems more robust to network QoS degradation.
Combining Lexicons and Neural Networks for Targeted Sentiment Analysis

In this paper we consider the targeted sentiment analysis problem for the Russian language. We compared several machine learning approaches based on transformer architecture: BERT, RuBERT, RuRoberta. The main focus of our studies was directed to the use of additional sources of knowledge stored in sentiment lexicons. Two Russian sentiment lexicons: RuSentiLex and RuSentiFrames were involved in our study. We use the lexicons in two different ways. The first technique uses the RuSentiLex for automatic labeling of additional text data to enhance training dataset. Two variants of the lexicon-based annotated texts were obtained: general and domain-specific collections. Another technique creates embeddings for word labels in sentiment lexicons and, in such a way, integrate the lexicon knowledge into the whole process of neural network training. The combination of two Russian vocabularies and the GRU-TSC architecture achieved state-of-the-art results on five evaluation datasets.
Utilizing Lexicographic Resources for Sentiment Classification in Uzbek Language

Nowadays it is seen the active development of text sentiment analysis tools, which help solve a huge range of problems, ranging from emotional tone or polarity to the nature of the analyzed text. Although significant advances have been made in sentiment analysis for languages with abundant computing resources, the problem still remains relevant for resource-poor languages, where Uzbek is one of them. The lack of complete linguistic databases and tools for this language represents a major obstacle for the scientific community.The present study aims to address this limitation by introducing a rule-based approach for sentiment analysis in the Uzbek language. In particular, the methodology uses three key vocabularies: the first includes a comprehensive list of more than 300 affixes used to generate word forms; the second focuses on cataloging exception words that defy standard morphological rules; and the third contains a list of word roots, some of the word roots are already marked with positive or negative sentiment polarity.When combined, morphological analysis with these lexical resources can effectively recognize tonal orientation in Uzbek sentences. This is especially important given the agglutinative nature of language and complex morphological structures that add layers of subtle meaning, influencing the interpretation of sentiments.
ML-based RF Fingerprinting for LoRaWAN Device Identification

Imperfections in the analog-front end of radio devices may be used for classification and clustering, providing an additional layer of protection to security and authentication protocols. In this paper, we present results on the use of a small two-layer convolutional neural network to identify and classify LoRaWAN devices. The classification process is done exclusively on the raw I/Q baseband samples recorded by a software-defined radio. Results show classification accuracy larger than 97%, as well as resilience against Gaussian noise. We also present initial results on the use of I/Q imbalance as an intentionally-introduced impairment to further improve classification accuracy.