Reviewerly/exHarmony huggingface dataset:
Unambiguous Pyramid Cost Volumes Fusion for Stereo Matching

Stereo matching is a challenging task in 3D vision. Only relying on single-scale cost aggregation provides deficient matching information. Prior works thus try to adopt pyramid cost volumes fusion to calculate the matching cost. However, the commonly used cost volume fusion process can not fully exploit the benefits of these multi-scale cost volumes. Motivated by the cross-scale feature discrepancy, we propose an Unambiguous Pyramid cost volumes Fusion Network terms as UPFNet, to reduce the ambiguity between pyramid cost volumes at different scales and boost the cross-scale information flow in the stereo matching framework based on 3D convolution. First, we propose a pyramid-cost progressive fusion (PPF) module, which adds consistent supervision for pre-fusion cost volumes to reduce feature semantic inconsistency and facilitates cross-scale interactions to narrow the detailed gap between different scales. The output disparity can be gradually refined in a coarse-to-fine manner. Furthermore, we design a residual disparity aggregation (RDA) module, introducing disparity dimension information to further exploit the local aggregation capability of 3D convolution by squeezing disparity and exciting channel response. Extensive experiments on the Scene Flow, KITTI and Middlebury benchmarks demonstrate the effectiveness of the proposed UPFNet. The results show that the proposed approach achieves state-of-the-art performance and is ranked first in the KITTI 2015 leaderboard when submission. Our codes are available at: https://github.com/Baboom-l/UPFNet.
Distributed-Proof-of-Sense: Blockchain Consensus Mechanisms for Detecting Spectrum Access Violations of the Radio Spectrum

The exponential growth in connected devices with Internet-of-Things (IoT) and next-generation wireless networks requires more advanced and dynamic spectrum access mechanisms. Blockchain-based approaches to Dynamic Spectrum Access (DSA) seem efficient and robust due to their inherited characteristics such as decentralization, immutability, and transparency. However, conventional consensus mechanisms used in blockchain networks are expensive to be used due to the cost, processing, and energy constraints. Moreover, addressing spectrum violations (i.e., unauthorized access to the spectrum) is not well-discussed in most blockchain-based DSA systems in the literature. In this work, we propose a newly tailored energy-efficient consensus mechanism called “Distributed-Proof-of-Sense (DPoS)” that is specially designed to enable DSA and detect spectrum violations. The proposed consensus algorithm motivates blockchain miners to perform spectrum sensing, which leads to the collection of a full spectrum of sensing data. An elliptic curve cryptography-based zero-knowledge proof is used as the core of the proposed mechanism. We use MATLAB simulations to analyze the performance of the consensus mechanism and implement several consensus algorithms in a microprocessor to highlight the benefits of adopting the proposed system.
Event Extraction by Associating Event Types and Argument Roles

Event extraction (EE), which acquires structural event knowledge from texts, can be divided into two sub-tasks: event type classification and element extraction (namely identifying triggers and arguments under different role patterns). As different event types always own distinct extraction schemas (i.e., role patterns), previous work on EE usually follows an isolated learning paradigm, performing element extraction independently for different event types. It ignores meaningful associations among event types and argument roles, leading to relatively poor performance for less frequent types/roles. This paper proposes a novel neural association framework for the EE task. Given a document, it first performs type classification via constructing a document-level event graph to associate sentence nodes of different types and adopting a document-awared graph attention network to learn sentence embeddings. Then, element extraction is achieved by building a new schema of argument roles, with a type-awared parameter inheritance mechanism to enhance role preference for extracted elements. As such, our model takes into account type and role associations during EE, enabling implicit information sharing among them. Experimental results show that our approach consistently outperforms most state-of-the-art EE methods in both sub-tasks, especially at least 2.51% and 1.12% improvement of the event trigger identification and argument role classification sub-tasks. Particularly, for types/roles with less training data, the performance is superior to the existing methods.
English Emotional Voice Conversion Using StarGAN Model

The StarGANv2-VC model is a many-to-many non-parallel generative adversarial network (GAN) voice conversion (VC) model that has proven effective in style conversion tasks. This study aimed to investigate the scalability and diversity of the model for English emotional voice conversion (EVC) across different speakers and emotions. We carried out five experiments using an Emotional Speech Database (ESD), comprising a single speaker-multi-emotion experiment, a multi-speakers-multi-emotions experiment (gender-dependent), and a multi-speakers-multi-emotions experiment (gender-independent). We also assessed the effect of training set size and compared the performance of the StarGANv2-VC model with a CycleGAN model. Our study found that the StarGANv2-VC model accurately converted the pitch of the voice across all four emotions (neutral, happy, sad, and angry). However, the model’s efficiency in converting multi-emotions to multi-speakers was not as high as its efficiency in voice conversion for multi-speakers. Further research is needed in this area. We objectively assessed the quality of the converted speech using Mel-frequency cepstral distortion (MCD) and root-mean-square error (RMSE) for spectrum and prosody, respectively. Additionally, we conducted cross-emotion recognition using a convolutional recurrent neural network (CRNN).
Sentiment Analysis in Turkish Question Answering Systems: An Application of Human-Robot Interaction

The use of the sentiment analysis technique, which aims to extract emotions and thoughts from texts, has become a remarkable research topic today, where the importance of human-robot interaction is gradually increasing. In this study, a new hybrid sentiment analysis model is proposed using machine learning algorithms to increase emotional performance for Turkish question and answer systems. In this context, as a first, we apply text preprocessing steps to the Turkish question-answer-emotion dataset. Subsequently, we convert the preprocessed question and answer texts into text vector form using Pretrained Turkish BERT Model and two different word representation methods, TF-IDF and word2vec. Additionally, we incorporate pre-determined polarity vectors containing the positive and negative scores of words into the question-answer text vector. As a result of this study, we propose a new hybrid sentiment analysis model. We separate vectorized and expanded question-answer text vectors into training and testing data and train and test them with machine learning algorithms. By employing this previously unused method in Turkish question-answering systems, we achieve an accuracy value of up to 91.05% in sentiment analysis. Consequently, this study contributes to making human-robot interactions in Turkish more realistic and sensitive.
Joint Learning of Semantic Segmentation and Height Estimation for Remote Sensing Image Leveraging Contrastive Learning

Semantic segmentation and height estimation are two critical tasks in remote sensing scene understanding that are highly correlated with each other. To address both tasks simultaneously, it is natural to consider designing a unified deep learning model that aims to improve performance by jointly learning complementary information among the associated tasks. In this paper, we learn the two tasks jointly under a deep multi-task learning framework and propose two novel objective functions, called cross-task contrastive loss and cross-pixel contrastive loss, respectively, to enhance multi-task learning performance through contrastive learning. Specifically, the cross-task contrastive loss is designed to maximize the mutual information of different task features and enforce the model to learn the consistency between semantic segmentation and height estimation. In addition, our method goes beyond previous approaches that only apply contrastive learning at the instance level. Instead, we design a pixel-wise contrastive loss function that pulls together pixel embeddings belonging to the same semantic class, while pushing apart pixel embeddings from different semantic classes. Furthermore, we find that this semantic-guided contrastive loss simultaneously improves the performance of the height estimation task. Our proposed approach is simple and effective and does not introduce any additional overhead to the model during the testing phase. We extensively evaluate our method on the Vaihingen and Potsdam datasets, and the experimental results demonstrate that our approach significantly outperforms the state-of-the-art methods in both height estimation and semantic segmentation.
Pyramidal Feature Adjustment Networks for High Dynamic Range Imaging of Dynamic Scenes

Multi-exposure high dynamic range (HDR) imaging aims to generate an HDR image from multiple low dynamic range (LDR) images. There are two main challenges in this task; camera and object motions causing ghost artifacts and over/under-exposure areas having no information. To tackle these challenges, we propose a deep neural network that aligns the multiple inputs through feature transformation and hallucinates the washed-out areas after fusion. Specifically, we propose a pyramidal feature adjustment network (PFAN) that adjusts the reference image features’ brightness and structure with respect to LDR inputs and regard them as the aligned features of the LDR inputs. Then, the features from the PFANs are integrated in a coarse-to-fine manner. They are first merged into an HDR image in a fusion network, and the washed-out regions are restored using valid information in a hallucination network. To deal with larger displacements in motion alignment, we also present a progressive training strategy, which begins the training with easy samples having little motion and then transfers the network with more dynamic samples. Extensive experiments demonstrate that our method achieves state-of-the-art performance on benchmark datasets. Our codes are available at https://github.com/haesoochung/PFAN.
A Novel Concatenation Decoding of Reed-Solomon Codes With SPC Product Codes

This letter introduces a novel concatenation decoding of Reed-Solomon (RS) codes with single parity-check product codes (SPC-PCs), including the concatenation scheme and the decoding algorithm. The proposed scheme sequentially arranges SPC-PCs to form <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$K$</tex-math></inline-formula> information symbols, then encodes them into <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> symbols using systematic RS code. A high-performance and low-complexity bit-iterative-update decoding (BIUD) algorithm is also proposed based on this scheme. We use the results of the parity-check to evaluate the reliability in each dimension and then iteratively update the reliability of each bit based on the structure of SPC-PCs. This way dramatically reduces the complexity of traditional reliability computation. Furthermore, the bit iterative update and the Berlekamp-Massey algorithm of RS codes can be well combined, resulting in more accurate results. The simulation and analysis results show that the proposed concatenation algorithm has significant performance improvement compared to the current RS-SPC concatenation decoding while having extremely low complexity.
IMDet: Injecting more supervision to CenterNet-like object detection

CenterNet-like object detectors are known for the concise post-processing method. But there are two bottlenecks that limit its further research and development. The first one is they just encode one positive sample for each object in the classification task, failing to make full use of supervision information. And simply encoding multiple positive samples is not as effective as expectation. The other one lies in that they perform predictions on one large-resolution feature map, not fully exploiting the insights of multi-level-feature object detection. The two shortcomings make their accuracy lag behind the detectors with one-to-many label assignment strategy and Feature Pyramids Networks (FPN). To solve the two issues, we first propose the Gaussian-based Target Assignment strategy (GTA) to effectively encode multiple positive samples. The GTA is supported by three rules and achieved cost-free. Second, we devise the Balanced Supervision Network (BSN) to ease the optimization of backbones. Through injecting more sufficient and balanced supervision, the proposed two methods improve CenterNet-like detectors to a competitive performance. The effectiveness of each method is fully verified on large-scale datasets with different data distributions, i.e., the MS-COCO and PASCAL VOC dataset.
Conflict-constrained Multi-agent Reinforcement Learning Method for Parking Trajectory Planning

Automated Valet Parking (AVP) has been exten-sively researched as an important application of autonomous driving. Considering the high dynamics and density of real parking lots, a system that considers multiple vehicles simultaneously is more robust and efficient than a single vehicle setting as in most studies. In this paper, we propose a dis-tributed Multi-agent Reinforcement Learning(MARL) method for coordinating multiple vehicles in the framework of an AVP system. This method utilizes traditional trajectory planning to accelerate the learning process and introduces collision conflict constraints for policy optimization to mitigate the path conflict problem. In contrast to other centralized multi-agent path finding methods, the proposed approach is scalable, distributed, and adapts to dynamic stochastic scenarios. We train the models in random scenarios and validate in several artificially designed complex parking scenarios where vehicles are always disturbed by dynamic and static obstacles. Experimental results show that our approach mitigates path conflicts and excels in terms of success rate and efficiency.
Stealthy Perception-based Attacks on Unmanned Aerial Vehicles

In this work, we study vulnerability of unmanned aerial vehicles (UAVs) to stealthy attacks on perception-based control. To guide our analysis, we consider two specific missions: ( <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$i$</tex> ) ground vehicle tracking (GVT), and (ii) vertical take-off and landing (VTOL) of a quadcopter on a moving ground vehicle. Specifically, we introduce a method to consistently attack both the sensors measurements and camera images over time, in order to cause control performance degradation (e.g., by failing the mission) while remaining stealthy (i.e., undetected by the deployed anomaly detector). Unlike existing attacks that mainly rely on vulnerability of deep neural networks to small input perturbations (e.g., by adding small patches and/or noise to the images), we show that stealthy yet effective attacks can be designed by changing images of the ground vehicle's landing markers as well as suitably falsifying sensing data. We illustrate the effectiveness of our attacks in Gazebo 3D robotics simulator.
A Reachability Tree-Based Algorithm for Robot Task and Motion Planning

This paper presents a novel algorithm for robot task and motion planning (TAMP) problems by utilizing a reachability tree. While tree-based algorithms are known for their speed and simplicity in motion planning (MP), they are not well-suited for TAMP problems that involve both abstracted and geometrical state variables. To address this challenge, we propose a hierarchical sampling strategy, which first generates an abstracted task plan using Monte Carlo tree search (MCTS) and then fills in the details with a geometrically feasible motion trajectory. Moreover, we show that the performance of the proposed method can be significantly enhanced by selecting an appropriate reward for MCTS and by using a pre-generated goal state that is guaranteed to be geometrically feasible. A comparative study using TAMP benchmark problems demonstrates the effectiveness of the proposed approach.
Frontier Semantic Exploration for Visual Target Navigation

This work focuses on the problem of visual target navigation, which is very important for autonomous robots as it is closely related to high-level tasks. To find a special object in unknown environments, classical and learning-based approaches are fundamental components of navigation that have been investigated thoroughly in the past. However, due to the difficulty in the representation of complicated scenes and the learning of the navigation policy, previous methods are still not adequate, especially for large unknown scenes. Hence, we propose a novel framework for visual target navigation using the frontier semantic policy. In this proposed framework, the semantic map and the frontier map are built from the current observation of the environment. Using the features of the maps and object category, deep reinforcement learning enables to learn a frontier semantic policy which can be used to select a frontier cell as a long-term goal to explore the environment efficiently. Experiments on Gibson and Habitat-Matterport 3D (HM3D) demonstrate that the proposed framework significantly outperforms existing map-based methods in terms of success rate and efficiency. Ablation analysis also indicates that the proposed approach learns a more efficient exploration policy based on the frontiers. A demonstration is provided to verify the applicability of applying our model to real-world transfer. The supplementary video and code can be accessed via the following link: https://sites.google.com/view/fsevn.
Failure-aware Policy Learning for Self-assessable Robotics Tasks

Self-assessment rules play an essential role in safe and effective real-world robotic applications, which verify the feasibility of the selected action before actual execution. But how to utilize the self-assessment results to re-choose actions remains a challenge. Previous methods eliminate the selected action evaluated as failed by the self-assessment rules, and re-choose one with the next-highest affordance (i.e. process-of-elimination strategy [1]), which ignores the dependency between the self-assessment results and the remaining untried actions. However, this dependency is important since the previous failures might help trim the remaining over-estimated actions. In this paper, we set to investigate this dependency by learning a failure-aware policy. We propose two architectures for the failure-aware policy by representing the self-assessment results of previous failures as the variable state, and leveraging recurrent neural networks to implicitly memorize the previous failures. Experiments conducted on three tasks demonstrate that our method can achieve better performances with higher task success rates by less trials. Moreover, when the actions are correlated, learning a failure-aware policy can achieve better performance than the process-of-elimination strategy.
CrossDTR: Cross-view and Depth-guided Transformers for 3D Object Detection

To achieve accurate 3D object detection at a low cost for autonomous driving, many multi-camera methods have been proposed and solved the occlusion problem of monocular approaches. However, due to the lack of accurate estimated depth, existing multi-camera methods often generate multiple bounding boxes along a ray of depth direction for difficult small objects such as pedestrians, resulting in an extremely low recall. Furthermore, directly applying depth prediction modules to existing multi-camera methods, generally composed of large network architectures, cannot meet the real-time requirements of self-driving applications. To address these issues, we propose Cross-view and Depth-guided Transformers for 3D Object Detection, CrossDTR. First, our lightweight depth predictor is designed to produce precise object-wise sparse depth maps and low-dimensional depth embeddings without extra depth datasets during supervision. Second, a cross-view depth-guided transformer is developed to fuse the depth embeddings as well as image features from cameras of different views and generate 3D bounding boxes. Extensive experiments demonstrated that our method hugely surpassed existing multi-camera methods by 10 percent in pedestrian detection and about 3 percent in overall mAP and NDS metrics. Also, computational analyses showed that our method is 5 times faster than prior approaches. Our codes will be made publicly available at https://github.com/sty61010/CrossDTR.
MonoPGC: Monocular 3D Object Detection with Pixel Geometry Contexts

Monocular 3D object detection reveals an economical but challenging task in autonomous driving. Recently center-based monocular methods have developed rapidly with a great trade-off between speed and accuracy, where they usually depend on the object center's depth estimation via 2D features. However, the visual semantic features without sufficient pixel geometry information, may affect the performance of clues for spatial 3D detection tasks. To alleviate this, we propose MonoPGC, a novel end-to-end Monocular 3D object detection framework with rich Pixel Geometry Contexts. We introduce the pixel depth estimation as our auxiliary task and design depth cross-attention pyramid module (DCPM) to inject local and global depth geometry knowledge into visual features. In addition, we present the depth-space-aware transformer (DSAT) to integrate 3D space position and depth-aware features efficiently. Besides, we design a novel depth-gradient positional encoding (DGPE) to bring more distinct pixel geometry contexts into the transformer for better object detection. Extensive experiments demonstrate that our method achieves the state-of-the-art performance on the KITTI dataset.
Image-to-Image Translation for Autonomous Driving from Coarsely-Aligned Image Pairs

A self-driving car must be able to reliably handle adverse weather conditions (e.g., snowy) to operate safely. In this paper, we investigate the idea of turning sensor inputs (i.e., images) captured in an adverse condition into a benign one (i.e., sunny), upon which the downstream tasks (e.g., semantic segmentation) can attain high accuracy. Prior work primarily formulates this as an unpaired image-to-image translation problem due to the lack of paired images captured under the exact same camera poses and semantic layouts. While perfectly-aligned images are not available, one can easily obtain coarsely-paired images. For instance, many people drive the same routes daily in both good and adverse weather; thus, images captured at close-by GPS locations can form a pair. Though data from repeated traversals are unlikely to capture the same foreground objects, we posit that they provide rich contextual information to supervise the image translation model. To this end, we propose a novel training objective leveraging coarsely-aligned image pairs. We show that our coarsely-aligned training scheme leads to a better image translation quality and improved downstream tasks, such as semantic segmentation, monocular depth estimation, and visual localization.