Reviewerly/exHarmony huggingface dataset:
UVSS: Unified Video Stabilization and Stitching for Surround View of Tractor-Trailer Vehicles

Automotive surround-view camera systems have been commonly employed in automated driving to aid in near-field sensing and other perception tasks. Due to the large size of the body and the presence of multiple blind spots, panoramic surround-view systems are particularly crucial for tractor-trailer vehicles. However, the non-rigid body of tractor-trailer vehicles introduces pose changes between cameras, rendering traditional calibration-based methods inadequate. Additionally, cameras mounted separately on the tractor and the trailer will experience independent vibrations, resulting in undesirable shakiness in captured videos. In this paper, we propose a unified video stabilization and stitching method to address these challenges, which can smooth the unsteady frames and align the images from moving cameras. Delving into video stabilization techniques, we extend mesh-based motion model for unified stitching and leverage deep-learning based modules to handle complex real-world scenarios. Moreover, we design a new optimization framework to estimate the optimal displacements of mesh vertices, enabling simultaneous stabilization and stitching of frames. The experimental results, obtained by public datasets and videos captured from a model tractor-trailer vehicle, demonstrate that our approach outperforms previous methods and is highly effective in real-world applications.
Lightweight Semantic Segmentation Network for Semantic Scene Understanding on Low-Compute Devices

Semantic scene understanding is beneficial for mobile robots. Semantic information obtained through onboard cameras can improve robots' navigation performance. However, obtaining semantic information on small mobile robots with constrained power and computation resources is challenging. We propose a new lightweight convolution neural network comparable to previous semantic segmentation algorithms for mobile applications. Our network achieved 73.06% on the Cityscapes validation set and 71.8% on the Cityscapes test set. Our model runs at 116 fps with <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{1024\mathrm{x}2048}$</tex> , 172 fps with <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1024\mathrm{x}1024$</tex> , and 175 fps with <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$720\mathrm{x}960$</tex> on NVIDIA GTX 1080. We analyze a model size, which is defined as the summation of the number of floating operations and the number of parameters. The smaller model size enables tiny mobile robot systems that should operate multiple tasks simultaneously to work efficiently. Our model has the smallest model size compared to the real-time semantic segmentation convolution neural networks ranked on Cityscapes real-time benchmark and other high performing, lightweight convolution neural networks. On the Camvid test set, our model achieved a mIoU of 73.29% with Cityscapes pre-training, which outperformed the accuracy of other lightweight convolution neural networks. For mobile applicability, we measured frame-per-second on different low-compute devices. Our model operates 35 fps on Jetson Xavier AGX, 21 FPS on Jetson Xavier NX, and 14 FPS on a ROS ASUS gaming phone. <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1024\mathrm{x}2048$</tex> resolution is used for the Jetson devices, and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$512\mathrm{x}512$</tex> size is utilized for the measurement on the phone. Our network did not use extra datasets such as ImageNet, Coarse Cityscapes, and Mapillary. Additionally, we did not use TensorRT to achieve fast inference speed. Compared to other real-time and lightweight CNNs, our model achieved significantly more efficiency while balancing accuracy, inference speed, and model size.
Switching Head-Tail Funnel UNITER for Dual Referring Expression Comprehension with Fetch-and-Carry Tasks

This paper describes a domestic service robot (DSR) that fetches everyday objects and carries them to specified destinations according to free-form natural language instructions. Given an instruction such as “Move the bottle on the left side of the plate to the empty chair,” the DSR is expected to identify the bottle and the chair from multiple candidates in the environment and carry the target object to the destination. Most of the existing multimodal language understanding methods are impractical in terms of computational complexity because they require inferences for all combinations of target object candidates and destination candidates. We propose Switching Head-Tail Funnel UNITER, which solves the task by predicting the target object and the destination individually using a single model. Our method is validated on a dataset based on a standard dataset for Vision-and-Language Navigation with object manipulation tasks. The results show that our method outperforms the baseline method in terms of language comprehension accuracy. Furthermore, we conduct physical experiments in which a DSR delivers standardized everyday objects in a standardized domestic environment as requested by instructions with referring expressions. The experimental results show that the object grasping and placing actions are achieved with success rates of more than 90 %.
Multi-UAV Adaptive Path Planning Using Deep Reinforcement Learning

Efficient aerial data collection is important in many remote sensing applications. In large-scale monitoring scenarios, deploying a team of unmanned aerial vehicles (UAVs) offers improved spatial coverage and robustness against individual failures. However, a key challenge is cooperative path planning for the UAVs to efficiently achieve a joint mission goal. We propose a novel multi-agent informative path planning approach based on deep reinforcement learning for adaptive terrain monitoring scenarios using UAV teams. We introduce new network feature representations to effectively learn path planning in a 3D workspace. By leveraging a counterfactual baseline, our approach explicitly addresses credit assignment to learn cooperative behaviour. Our experimental evaluation shows improved planning performance, i.e. maps regions of interest more quickly, with respect to non-counterfactual variants. Results on synthetic and real-world data show that our approach has superior performance compared to state-of-the-art non-learning-based methods, while being transferable to varying team sizes and communication constraints.
Time-Optimal Control via Heaviside Step-Function Approximation

Least-squares programming is a popular tool in robotics due to its simplicity and availability of open-source solvers. However, certain problems like sparse programming in the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\ell_{0}$</tex> - or <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\ell_{0}-\mathbf{norm}$</tex> for time-optimal control are not equivalently solvable. In this work, we propose a non-linear hierarchical least-squares programming (NL-HLSP) for time-optimal control of non-linear discrete dynamic systems. We use a continuous approximation of the heaviside step function with an additional term that avoids vanishing gradients. We use a simple discretization method by keeping states and controls piece-wise constant between discretization steps. This way, we obtain a comparatively easily implementable NL-HLSP in contrast to direct transcription approaches of optimal control. We show that the NL-HLSP indeed recovers the discrete time-optimal control in the limit for resting goal points. We confirm the results in simulation for linear and non-linear control scenarios.
Algorithm for Finding a Descriptive Path for Delivering Cargo to Hard-to-Reach Areas

At the moment, due to the growing interest in unmanned aerial vehicles (UAV), as well as due to the rapid and constant development of the UAV market and high technologies in the field of artificial intelligence, this branch of robotics has begun to occupy more and more sectors of the economy. With the increasing popularity of fast and light vehicles such as UAVs, the demand for efficient flight path planning algorithms has also increased to achieve a variety of goals: overflight, obstacle detection and collision avoidance. One of these goals is achieved by our algorithm for flying around static obstacles in 3D space in a sparse weighted graph. A hybrid method has been developed for finding the safest and fastest route based on Dijkstra's algorithm, which works with satellite images of various terrain classifications. The novelty of the proposed algorithm lies in the combination of artificial neural networks for categorizing terrain classes and using these data when constructing a flight task. The idea is to build a route on top of the safest terrain so that in the event of an emergency fall or landing, the UAV will not be lost. It is more memory efficient because it does not require storing all the vertices in an open list, as do algorithms that perform a similar task. In addition to the software architecture, the most suitable hardware architecture for the intended purpose - delivering cargo to difficult terrain - is presented.
Real-Time Russian Sign Language Recognition

This paper describes our concept for recognizing human hand gestures, including sign language of the deaf, using Russian sign language as an example. The key problems that hinder the creation of a gesture recognition system are highlighted, namely: the difficulty of collecting a sufficient amount of data for training classical neural networks, the variability of movements between gestures in continuous signed speech, as well as the limited resources, both computing and hardware, for widespread use of the system. An architecture is proposed that partially solves the identified problems by gamifying data collection through the community of deaf people and optimizing performance by changing the number of significant parameters and distributing the computational load between system elements. We propose a recognition system structure consisting of a detector of the position and configuration of a person's hands, body and face using MediaPipe and step-by-step gesture classification based on data received from the detector. Step-by-step classification is used to increase accuracy with a low amount of training data and simplify the subsequent development of the system due to less retraining of models. The choice of architecture for the sign language recognition system is based on our experience testing previous prototypes. The results presented in this paper demonstrate the potential of the system being developed, but our prototypes are not the final product.
Evolving Physical Instinct for Morphology and Control Co-Adaption

The capability of a robot to perform tasks depends not only on precise motion control, but also on a well-suited body morphology. Adapting both morphology and control of robots to improve their task performance has been a widely studied and long-standing issue. While the bio-inspired bi-level optimization framework has gained popularity in recent years, it suffers from high computation complexity due to the time-consuming and inefficient learning process for each morphology. In fact, in nature, besides the adaptive morphology and the intelligent brain, animals also possess an important gift, which is physical instinct. These instincts allow animals to respond quickly to their surroundings in the neonatal period, facilitating skills acquisition. Inspired by this, we propose an evolvable instinct controller to enhance the morphology-control co-adaption. The instinct controller suggests rough motion inclinations, which require minimal domain knowledge and entail less sophisticated design. Its purpose is to assist the main controller in learning fine-grained and robust control efficiently. We implemented this idea in the context of legged locomotion and designed the instinct controller using phase-based FSMs. We propose the instinct-based co-adaption algorithm and construct GPU parallel simulation experiments on different morphology prototypes. The results indicate that combining the co-adaption process with instinct evolution leads to the development of superior morphologies and robust controllers compared with the conventional co-adaption approach, with minimal additional time cost.
Keypoints-Based Adaptive Visual Servoing for Control of Robotic Manipulators in Configuration Space

This paper presents a visual servoing method for controlling a robot in the configuration space by purely using its natural features. We first created a data collection pipeline that uses camera intrinsics, extrinsics, and forward kinematics to generate 2D projections of a robot's joint locations (keypoints) in image space. Using this pipeline, we are able to collect large sets of real-robot data, which we use to train realtime keypoint detectors. The inferred keypoints from the trained model are used as control features in an adaptive visual servoing scheme that estimates, in runtime, the Jacobian relating the changes of the keypoints and joint velocities. We compared the 2D configuration control performance of this method to the skeleton-based visual servoing method (the only other algorithm for purely vision-based configuration space visual servoing), and demonstrated that the keypoints provide more robust and less noisy features, which result in better transient response. We also demonstrate the first vision-based 3D configuration space control results in the literature, and discuss its limitations. Our data collection pipeline is available at https://github.com/JaniC-WPI/KPDataGenerator.git which can be utilized to collect image datasets and train realtime keypoint detectors for various robots and environments.
Recognizing Real-World Intentions using A Multimodal Deep Learning Approach with Spatial-Temporal Graph Convolutional Networks

Identifying intentions is a critical task for comprehending the actions of others, anticipating their future behavior, and making informed decisions. However, it is challenging to recognize intentions due to the uncertainty of future human activities and the complex influence factors. In this work, we explore the method of recognizing intentions alluded under human behaviors in the real world, aiming to boost intelligent systems' ability to recognize potential intentions and understand human behaviors. We collect data containing real-world human behaviors before using a hand dispenser and a temperature scanner at the building entrance. These data are processed and labeled into intention categories. A questionnaire is conducted to survey the human ability in inferring the intentions of others. Skeleton data and image features are extracted inspired by the answer to the questionnaire. For skeleton-based intention recognition, we propose a spatial-temporal graph convolutional network that performs graph convolutions on both part-based graphs and adaptive graphs, which achieves the best performance compared with baseline models in the same task. A deep-learning-based method using multimodal features is proposed to automatically infer intentions, which is demonstrated to accurately predict intentions based on past behaviors in the experiment, significantly outperforming humans.
Hierarchical Semi-Supervised Learning Framework for Surgical Gesture Segmentation and Recognition Based on Multi-Modality Data

Segmenting and recognizing surgical operation trajectories into distinct, meaningful gestures is a critical preliminary step in surgical workflow analysis for robot-assisted surgery. This step is necessary for facilitating learning from demonstrations for autonomous robotic surgery, evaluating surgical skills, and so on. In this work, we develop a hierarchical semi-supervised learning framework for surgical gesture segmentation using multi-modality data (i.e. kinematics and vision data). More specifically, surgical tasks are initially segmented based on distance characteristics-based profiles and variance characteristics-based profiles constructed using kinematics data. Subsequently, a Transformer-based network with a pre-trained ‘ResNet-18’ backbone is used to extract visual features from the surgical operation videos. By combining the potential segmentation points obtained from both modalities, we can determine the final segmentation points. Furthermore, gesture recognition can be implemented based on supervised learning. The proposed approach has been evaluated using data from the publicly available JIGSAWS database, including Suturing, Needle Passing, and Knot Tying tasks. The results reveal an average F1 score of 0.623 for segmentation and an accuracy of 0.856 for recognition. For more details about this paper, please visit our website: https://sites.google.com/view/surseg/home.
Design and Implementation of a Tool for Extracting Uzbek Syllables

The accurate syllabification of words plays a vital role in various Natural Language Processing applications. Syllabification is a versatile linguistic tool with applications in linguistic research, language technology, education, and various fields where understanding and processing language is essential. In this paper, we present a comprehensive approach to syllabification for the Uzbek language, including rule-based techniques and machine learning algorithms. Our rule-based approach utilizes advanced methods for dividing words into syllables, generating hyphenations for line breaks and count of syllables. Additionally, we collected a dataset for evaluating and training using machine learning algorithms comprising word-syllable mappings, hyphenations, and syllable counts to predict syllable counts as well as for the evaluation of the proposed model. Our results demonstrate the effectiveness and efficiency of both approaches in achieving accurate syllabification. The results of our experiments show that both approaches achieved a high level of accuracy, exceeding 99%. This study provides valuable insights and recommendations for future research on syllabification and related areas in not only the Uzbek language itself, but also in other closely-related Turkic languages with low-resource factor.
Developing Rule-Based and Gazetteer Lists for Named Entity Recognition in Uzbek Language: Geographical Names

Named entity recognition is one of the current problems in the field of natural language processing, especially for languages with limited resources, where the Uzbek language is one of them. In this scientific work, the authors conduct a review study and also propose a practical solution for recognizing named entities in the Uzbek language. The proposed approach is designed and operated on the basis of gazetteer rules and lists, with special emphasis on the identification of place names. The authors do not use machine learning methods due to limited linguistic resources, which makes the proposed approach easy to implement and fast to scale.The cornerstone of the proposed approach is two algorithms, where the first algorithm, based on morphological analysis, despite its simplicity, is somewhat limited in its capabilities. Conversely, the second algorithm is based on the syntactic analysis of texts in the Uzbek language and offers a more flexible approach to recognizing named entities.Both algorithms use a specific set of rules obtained as a result of a detailed study of the structure of the Uzbek language and its morphological and grammatical features.The purpose of this study is to enrich the relatively sparse literature on named entity recognition in the Uzbek language and to provide a solid basis for continued research in this area.
Sample-Efficient Real-Time Planning with Curiosity Cross-Entropy Method and Contrastive Learning

Model-based reinforcement learning (MBRL) with real-time planning has shown great potential in locomotion and manipulation control tasks. However, the existing planning methods, such as the Cross-Entropy Method (CEM), do not scale well to complex high-dimensional environments. One of the key reasons for underperformance is the lack of exploration, as these planning methods only aim to maximize the cumulative extrinsic reward over the planning horizon. Furthermore, planning inside the compact latent space in the absence of observations makes it challenging to use curiosity-based intrinsic motivation. We propose Curiosity CEM (CCEM), an improved version of the CEM algorithm for encouraging exploration via curiosity. Our proposed method maximizes the sum of state-action <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$Q$</tex> values over the planning horizon, in which these <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$Q$</tex> values estimate the future extrinsic and intrinsic reward, hence encouraging to reach novel observations. In addition, our model uses contrastive representation learning to efficiently learn latent representations. Experiments on image-based continuous control tasks from the DeepMind Control suite show that CCEM is by a large margin more sample-efficient than previous MBRL algorithms and compares favorably with the best model-free RL methods.
Multi-Source Soft Pseudo-Label Learning with Domain Similarity-based Weighting for Semantic Segmentation

This paper describes a method of domain adap-tive training for semantic segmentation using multiple source datasets that are not necessarily relevant to the target dataset. We propose a soft pseudo-label generation method by integrating predicted object probabilities from multiple source models. The prediction of each source model is weighted based on the estimated domain similarity between the source and the target datasets to emphasize contribution of a model trained on a source that is more similar to the target and generate reasonable pseudo-labels. We also propose a training method using the soft pseudo-labels considering their entropy to fully exploit information from the source datasets while suppressing the influence of possibly misclassified pixels. The experiments show comparative or better performance than our previous work and another existing multi-source domain adaptation method, and applicability to a variety of target environments.
Learning to Schedule in Multi-Agent Pathfinding

In this work, we consider the problem of allocating tasks and planning paths for multiple robots to operate cooper-atively. We formulate the problem as a bi-level optimization that involves optimizing the scheduling of robots and the collision-free path for each robot. To address the complexity of the environment with obstacles, we introduce a congestion-aware state representation technique with the aid of graph neural networks. We also derive a joint scheduling policy using multi-agent reinforcement learning, and we propose an additional auxiliary loss that promotes coordination among the robots. Through empirical evaluation, we demonstrate the effectiveness of our approach in solving the combined task allocation and path planning problem in a cooperative multi-robot system.