Reviewerly/exHarmony huggingface dataset:
Depth-Aware Vision-and-Language Navigation using Scene Query Attention Network

Vision-and-language navigation (VLN) has been an important task in the field of Robotics and Computer Vision. However, most existing vision-and-language navigation models only use features extracted from RGB observation as input, while robots can utilize depth sensors in the real world. Existing research has also shown that simply adding a depth stream to neural models could only provide a marginal improvement to the performance of the VLN task. Therefore, in our work, we develop a novel method for the VLN task using semantic map observations built from RGB-D input. We use vision-pretraining to efficiently encode the semantic map with CNN and scene query attention network by answering queries about semantic information of specific regions of a scene. The proposed method could be used with a simple model and does not require large-scale vision-language transformer pretraining, bringing a more than 10% increase in the success rate compared with a baseline model. When used together with the Speaker-Follower training technique, it achieves a success rate of 58 % on the test set for the R2R dataset in single-run setting, outperforming the previous RGB-D method and most existing RGB-only models that do not use large-scale vision-language transformers pretraining.
DKNAS: A Practical Deep Keypoint Extraction Framework Based on Neural Architecture Search

Keypoint extraction including both keypoint detection and description is a fundamental step in a wide range of geometric multimedia applications. In recent years, many learning-based approaches for keypoint extraction emerge and achieve promising results. However, they usually design network architectures empirically and lack of considerations about the comprehensive performance, which leads to limited applications. In this paper, we propose a practical framework based on Neural Architecture Search (NAS) technology, DKNAS, which can search architectures automatically and maintain efficiency and effectiveness, simultaneously. To the best of our knowledge, the proposed framework is the first NAS framework for keypoint extraction. The evaluation on HPatches dataset shows that our method achieves state-of-the-art results in the metrics of repeatability, localization error, homography accuracy and matching scores. Besides, our model is applied to a traditional Simultaneous Localization and Mapping (SLAM) system, ORB-SLAM2, to replace the handcrafted keypoints. Experimental results demonstrate that the system adopting our model outperforms ORB-SLAM2 and some other deep keypoints enhanced systems.
Meta-confidence estimation for stereo matching

We propose a novel framework to estimate the confidence of a disparity map taking into account, for the first time, the uncertainty affecting the confidence estimation process itself. Conversely to other tasks such as disparity estimation, the uncertainty of confidence directly hints that the confidence should be increased if initially low, but with high uncertainty, decreased otherwise. By modelling such a cue in the form of a second-level confidence, or meta-confidence, our solution allows for finding incorrect predictions inferred by confidence estimator and for learning a correction for them. Our strategy is suited for any state-of-the-art method known in literature, either implemented using random forest classifiers or deep neural networks. Especially, for deep neural networks-based models, we present a multi-headed confidence estimator followed by an uncertainty network, so as to predict mean confidence and meta-confidence within a single network without the cost of lower accuracy, a known limitation in literature for uncertainty estimation. Experimental results on a variety of stereo algorithms and confidence estimation models prove that the modeled meta-confidence is meaningful of the reliability of the estimated confidence and allows for refining it.
Receding Horizon Tracking of an Unknown Number of Mobile Targets using a Bearings-Only Sensor

Planning the motion of bearings-only sensors is critical for enabling accurate tracking of the positions of moving targets. In this paper, we demonstrate planning the observer's motion over horizons greater than one step for estimating an unknown and varying number of indistinguishable, maneuvering targets of interest using a probability hypothesis density (PHD) filter, with a Rériyi divergence reward for selecting actions. We describe approximations to make this approach computationally feasible, and we propose using Monte Carlo tree search (MCTS) to further reduce the cost. Finally, we present simulation results showing that longer planning horizons reduce the error in the estimates and that MCTS can reduce the cost of planning without sacrificing the quality of the estimates.
A Dual-Stream Architecture for Real-Time Morphological Analysis of Aneurysm in Robot-Assisted Minimally Invasive Surgery

Real-time and precise morphological analysis of intraoperative AAA is a significant pre-imperative for robot-assisted minimally invasive surgery (RMIS). However, this task is frequently accompanied by the difficulties of ambiguous boundaries and obscured surfaces of aneurysms. To remedy these problems, we propose a Light-Weight Dual-Stream Boundary-Aware Network (DSB-Net) and a novel diagnosis algorithm for real-time morphological analysis of AAA. In the network, the features at the boundaries are preserved by incorporating a boundary localization stream, while the interior segmentation accuracy is guaranteed with a mask prediction stream. Moreover, the diagnosis algorithm is developed to measure the exact size of AAA. Quantitative and qualitative assessments on two different types of datasets illustrate that (1) The presented DSB-Net remarkably outperforms the other previously proposed medical networks with the inference rate of 10.8 FPS, which meets the real-time clinical necessities. (2) The developed algorithm provides accurate size measurements for AAA, which indicates the proposed approach can be integrated into the robotic navigation framework for RMIS.
f-Cal: Aleatoric uncertainty quantification for robot perception via calibrated neural regression

While modern deep neural networks are performant perception modules, performance (accuracy) alone is insufficient, particularly for safety-critical robotic applications such as self-driving vehicles. Robot autonomy stacks also require these otherwise blackbox models to produce reliable and calibrated measures of confidence on their predictions. Existing approaches estimate uncertainty from these neural network perception stacks by modifying network architectures, inference procedure, or loss functions. However, in general, these methods lack calibration, meaning that the predictive uncertainties do not faithfully represent the true underlying uncertainties (process noise). Our key insight is that calibration is only achieved by imposing constraints across multiple examples, such as those in a mini-batch; as opposed to existing approaches which only impose constraints per-sample, often leading to overconfident (thus miscalibrated) uncertainty estimates. By enforcing the distribution of outputs of a neural network to resemble a target distribution by minimizing an <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$f$</tex> -divergence, we obtain significantly better-calibrated models compared to prior approaches. Our approach, f-Cal, outperforms existing uncertainty calibration approaches on robot perception tasks such as object detection and monocular depth estimation over multiple real-world benchmarks.
Off Environment Evaluation Using Convex Risk Minimization

Applying reinforcement learning (RL) methods on robots typically involves training a policy in simulation and deploying it on a robot in the real world. Because of the model mismatch between the real world and the simulator, RL agents deployed in this manner tend to perform suboptimally. To tackle this problem, researchers have developed robust policy learning algorithms that rely on synthetic noise disturbances. However, such methods do not guarantee performance in the target environment. We propose a convex risk minimization algorithm to estimate the model mismatch between the simulator and the target domain using trajectory data from both environments. We show that this estimator can be used along with the simulator to evaluate performance of an RL agents in the target domain, effectively bridging the gap between these two environments. We also show that the convergence rate of our estimator to be of the order of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$n^{-1/4}$</tex> , where <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$n$</tex> is the number of training samples. In simulation, we demonstrate how our method effectively approximates and evaluates performance on Gridworld, Cartpole, and Reacher environments on a range of policies. We also show that the our method is able to estimate performance of a 7 DOF robotic arm using the simulator and remotely collected data from the robot in the real world.
Message Passing Framework for Vision Prediction Stability in Human Robot Interaction

In Human Robot Interaction (HRI) scenarios, robot systems would benefit from an understanding of the user's state, actions and their effects on the environments to enable better interactions. While there are specialised vision algorithms for different perceptual channels, such as objects, scenes, human pose, and human actions, it is worth considering how their interaction can help improve each other's output. In computer vision, individual prediction modules for these perceptual channels frequently produce noisy outputs due to the limited datasets used for training and the compartmentalisation of the perceptual channels, often resulting in noisy or unstable prediction outcomes. To stabilise vision prediction results in HRI, this paper presents a novel message passing framework that uses the memory of individual modules to correct each other's outputs. The proposed framework is designed utilising common-sense rules of physics (such as the law of gravity) to reduce noise while introducing a pipeline that helps to effectively improve the output of each other's modules. The proposed framework aims to analyse primitive human activities such as grasping an object in a video captured from the perspective of a robot. Experimental results show that the proposed framework significantly reduces the output noise of individual modules compared to the case of running independently. This pipeline can be used to measure human reactions when interacting with a robot in various HRI scenarios.
Design and Evaluation of Object Classifiers for Probabilistic Decision-Making in Autonomous Systems

Object classification is a key element that enables effective decision-making in many autonomous systems. A more sophisticated system may also utilize the probability distribution over the classes instead of basing its decision only on the most likely class. This paper introduces new performance metrics: the absolute class error (ACE), expectation of absolute class error (EACE) and variance of absolute class error (VACE) for evaluating the accuracy of such probabilities. We test this metric using different neural network architectures and datasets. Furthermore, we present a new task-based neural network for object classification and compare its performance with a typical probabilistic classification model to show the improvement with threshold-based probabilistic decision-making.
DanceHAT: Generate Stable Dances for Humanoid Robots with Adversarial Training

Music to dance for humanoid robots is an interesting task. Robot dance generation is challenging when considering music pieces, human dancer motions, and robot stability simultaneously. Previous methods rely on human-designed motion library or stability constraints for robot postures. Hence, dance generation for humanoid robots requires expert design, which can be time-consuming across different humanoid platforms. In this work, we propose a novel method called DanceHAT, which generates stable humanoid dances by imitating human dancers with self-learning. DanceHAT is an adversarial training framework, which incorporates similarity loss and stability loss simultaneously. Furthermore, DanceHAT does not require human-designed features or robot model information. Experiments in the simulation environment and on the real robot demonstrate that our model can generate stable, diverse, and human-like dances for humanoid robots automatically. In addition, DanceHAT is a general training approach for robot imitation tasks with stability constraints, thus can be utilized in other humanoid tasks and will be researched in future works.
Practical Cognitive Speech Compression

This paper presents a new neural speech compression method that is practical in the sense that it operates at low bitrate, introduces a low latency, is compatible in computational complexity with current mobile devices, and provides a subjective quality that is comparable to that of standard mobile-telephony codecs. Other recently proposed neural vocoders also have the ability to operate at low bitrate. However, they do not produce the same level of subjective quality as standard codecs. On the other hand, standard codecs rely on objective and short-term metrics such as the segmental signal-to-noise ratio that correlate only weakly with perception. Furthermore, standard codecs are less efficient than unsupervised neural networks at capturing speech attributes, especially long-term ones. The proposed method combines a cognitive-coding encoder that extracts an interpretable unsupervised hierarchical representation with a multi stage decoder that has a GAN-based architecture. We observe that this method is very robust to the quantization of representation features. An AB test was conducted on a subset of the Harvard sentences that are commonly used to evaluate standard mobile-telephony codecs. The results show that the proposed method outperforms the standard AMR-WB codec in terms of delay, bitrate and subjective quality.
Segmentation and Shape Estimation of Multiple Deformed Cloths Using a CNN-Based Landmark Detector and Clustering

In this paper, we propose a method for segmentation and shape estimation of multiple deformed cloths stacked on a floor from an image using a CNN-based landmark detector and clustering. The proposed method first estimates landmark positions from the heatmaps generated from the landmark detector and then clusters them using their attributes also given from the landmark detector. The contributions of the proposed method are twofold: (1) it can perform segmentation and shape estimation of multiple cloths of the same type or different types simultaneously by modeling a cloth as a grid of landmarks and estimating their positions and attributes, (2) it can correctly segment severely occluded or seemingly disjointed cloths by estimating and classifying landmark attributes including grid index, cloth position, cloth orientation, and cloth type. We evaluate the proposed method on both synthetic and real image datasets and show that the proposed method outperforms three baseline methods. We also show that the proposed method enables our humanoid robot to pick up the desired type of cloth from a pile of cloths.
A Representation-Focused Training Algorithm for Deep Networks

We present an optimization algorithm for the training of deep networks that focuses the optimization on the parameters of the feature representation while eliminating the optimization on the parameters of the prediction map. The proposed algorithm can be analyzed for least-squares objectives for which the implicit minimization can be obtained analytically on mini batches. We illustrate the numerical performance of the optimization algorithm with convolutional networks and explore a variant using Adam as a wrapper algorithm.
Time-Series Models for Crime Prediction in India

The dark web is termed as the concealed content hidden behind interfaces offering secure chats or mailing applications on the WWW, and exploiting it for illegal practices is termed cybercrimes. As the dark web is enormously big compared to the surface web, cyber-crimes have become one of the biggest challenges to tackle. Predicting them before they actually happen is almost impossible. The work thus deals with predicting the number of cognizable crimes (physical and cybercrimes reported collectively) by National Crime Record Bureau (NCRB), the government-approved agency for maintaining crime records in India. Predicting crimes has become important as they advance to new levels of intelligence and diversification with the increased use of information and communication technology (ICT). This work presents the research based on evidences with ARIMA and LSTM time series models for predicting the cognizable crimes in India. The authors have used root mean squared error and mean absolute error as loss functions for error evaluation, and the results obtained from both the models are presented.
Structure Analysis of Tor Hidden Services Using DOM-Inspired Graphs

The dark web contains sensitive data that strategic organizations must identify well in advance to anticipate and handle threats. However, associates will prefer to automate classifying the dark web pages instead of opening them due to their disturbing visual images and dangerous links and attachments. Most research is focused on web page text analysis to infer dark web data. But no visible attempt is observed in the literature that classifies dark web content at the structure level. In the chapter, extended scope of work aims to predict the genre of the webpage without opening the web page. The work converts web pages to their respective DOM (document object model) graphs. DOM graphs essentially represent web page structure. A GNN (graph neural network) is trained with constructed DOM graphs to predict the page's genre. The various graph properties like a number of nodes, edges, etc. for web page DOM graphs are extracted. Unsupervised learning (i.e., k-means clustering) is performed on the dataset to group the web pages into clusters based on similarity in structure.
An Unsupervised Person Search Method for Video Surveillance

We propose an unsupervised person search method for video surveillance. This method considers both the spatial features of persons within each frame and the temporal relationship of the same person among different frames. Thus, the spatial features are extracted by region convolutional neural network, and the temporal relationship is organized by gate recurrent unit. The spatio-temporal features are generated by the following average pooling layer and indexed by locality sensitive hashing. A surveillance video database is constructed to evaluate the proposed method, and the experimental results demonstrate that our method improves the search accuracy by utilizing the spatio-temporal features.
Multi Strategy fusion pedestrian re-recognition method for community

With the increasing popularity of video surveillance equipment, pedestrian re-recognition has gradually become an important means of auxiliary screening. How to easily deploy high-precision models and effectively re-recognize pedestrians has become the focus of attention. In this paper, a community oriented Multi Strategy fusion pedestrian re-recognition method is proposed. And two semi-automatic annotation methods of pedestrian re-recognition datasets are implemented. It solved the problems of difficult deployment of pedestrian re-recognition projects and difficult production of datasets, improved the annotation speed of pedestrian re-recognition datasets and took into account the quality of pedestrian images. Finally, the pedestrian re-recognition task for 12 cameras in a community in Hebei Province is realized, and the camera ID and video frame information of a pedestrian are accurately located, which provides a reference for other scholars to deploy a community-oriented pedestrian re-recognition system. This paper states that the dataset is only used for scientific research and experiment, and will not be disseminated or used for any commercial purpose.
A Real-time UAV Object Detection System Design with FPGA Implementation

This paper proposes Target detection system design and FPGA implementation based on YOLOX algorithm, in order to realize offline real-time image detection in a UAV platform with limited resources and power consumption. First, this paper studied the algorithm of YOLOX convolutional neural network,image fusion mechanism is added to this network, designed and trained the neural network. Secondly, an embedded edge computing system is designed to further speed up the target detection speed at the hardware level.In this paper, the above scheme is implemented at the board level. The test results show that the average recognition speed is 50frame/s on the system, which basically achieves the design goal of real-time detection.
A Prototype-Based Few-Shot Named Entity Recognition

Few-shot Named Entity Recognition (NER) task focuses on identifying name entities on a small amount of supervised training data. The work based on prototype network shows strong adaptability on the few-shot NER task. We think that the core idea of these approaches is to learn how to aggregate the representation of token mappings in vector space around entity class. But, as far as we know, no such work has been investigated its effect. So, we propose the ClusLoss and the ProEuroLoss aiming to enhance the model's ability in terms of aggregating semantic information spatially, thus helping the model better distinguish entity types. Experimental results show that ProEuroLoss achieves state-of-the-art performance on the average F1 scores for both 1-shot and 5-shot NER tasks, while the ClusLoss has competitive performance on such tasks.
Exploring Joint Information of Multi-Scales for Vehicle Re-Identification

Vehicle re-identification (re-ID) is an essential component of intelligent video surveillance, which attempts to solve the problem of retrieving specific vehicle instances. The technical challenge is mainly the requirement that the algorithm be robust under different viewing angles, resolutions, occlusions, and lighting conditions, yet different conditions can arise where the appearance of the exact vehicle varies dramatically. In contrast, the appearance of two different vehicles is exceptionally similar. To overcome this enormous challenge: we adopt a novel multi-scales attention network architecture, which will feature representation for feature maps at different scales to enhance the recognition of intricate parts, and introduce an inter-coordinate attention module to build interdependencies between component positions, making it possible to build a more robust vehicle model feature representation, and finally this paper also proposes a camera-based vehicle re localization method that enables the model to calculate the vehicle occurrence probability using the potential information among road network cameras as a post-processing step to re-correct the confidence level of the re-identification ranking. Experimental results show that the method outperforms the reference method in all indexes even without post-processing. At the same time, the camera confidence correction can be used in overlay with the reordering clustering correction method, which can improve the results by four percentage points and MAP up to 81.4%.