Reviewerly/exHarmony huggingface dataset:
Cross-Modal-Aware Representation Learning with Syntactic Hypergraph Convolutional Network for VideoQA

A key challenge in video question answering (VideoQA) is how to align textual concepts with the cross-modal visual regions accurately. Existing methods mostly rely on the alignment between individual words and relevant video regions, but individual words are generally not able to capture the complete information of a textual concept, which is often represented by the composition of several words. To address this issue, we propose to build a syntactic dependency tree for each question with an off-the-shelf tool and use it to extract meaningful word compositions (i.e., textual concept). By viewing the words and compositions as nodes and hyperedges, respectively, a hypergraph convolutional network (HCN) is built to learn the representations of textual concepts. Then, to enable cross-modal interaction of relevant concepts from different modalities, an optimal transport (OT) based alignment method is developed to establish the connection between textual concepts and their relevant visual regions. Experimental results on three benchmarks show that our method outperforms all competing baselines. Further analyses demonstrate the effectiveness of each component, and show that our model is good at modeling different levels of semantic compositions and filtering out irrelevant information.
The Fast Product Multi-Sensor Labeled Multi-Bernoulli Filter

The multi-sensor Labeled Multi-Bernoulli filter has the challenge of relying on the NP-hard multi-sensor update of the Generalized Labeled Multi-Bernoulli filter. This paper proposes the Fast Product Multi-Sensor Labeled Multi-Bernoulli filter, which is a filter for multi-sensor systems that solves this task by performing computationally simpler single-sensor Labeled Multi-Bernoulli filter updates based on a common prediction for each sensor. These single-sensor updates are then fused using a novel and efficient fusion strategy. Furthermore, the proposed filter is based on the Bayes parallel combination rule and can be seen as an efficient approximation of the multi-sensor Labeled Multi-Bernoulli filter. It enables full parallelization of the update step and benefits from sensor order independence compared to Iterated Corrector implementations. As a result, the robustness is increased, which is important for safety reasons, e.g., in autonomous driving. Our approach is evaluated on simulations, and the results are compared to an Iterated Corrector implementation of the Labeled Multi-Bernoulli filter.
Optimal Time-Efficient UAV Area Coverage Path Planning Based on Raster Map

In the original area coverage path planning algorithm, the generated paths are often slow and inefficient due to the diversity of task areas. In this paper, we propose an optimal time-efficient unmanned aerial vehicles (UAV) area coverage method based on raster maps, which enables it to perform better in practical application scenarios. Our work improves the original approach by replacing the previous angle-cost optimization strategy with optimal timing, and also conducts practical testing and modeling of UAVs, and finally experimental tests show that our approach is more time-efficient. On top of this, we also propose two pruning strategies, the first one is to discard those paths that are already far beyond the value during the calculation by continuously updating the minimum path consumption time, and the second one is to avoid erroneous path calculation by finding unreachable uncovered free grids in time during the calculation through the dead zone marker matrix. This paper also optimizes the path cost calculation to prevent the need to recalculate the previous path cost for each cost calculation. The above measures result in a reduction of the algorithm time computation by about 48.88%.
Enhancing Robustness of Deep Networks Against Noisy Labels Based on A Two-Phase Formulation of Their Learning Behavior

In this study we propose an explicit formulation of the learning behavior of deep neural networks (DNNs) trained with noisy labels in image classification. Specifically, we show theoretically and experimentally that the training process can be divided into two phases: a learning phase in which the outputs of DNNs converge to a hidden noisy label distribution; and a memorization phase in which DNNs start to overfit until the output for each sample converges to its corresponding noisy label. This two-phase formulation enables us to resolve a common pitfall of existing methods for robust training against noisy labels based on the small-loss assumption, namely clean samples have smaller losses than noisy samples in the early training phase. We show that these methods fail when the noise transition matrix is not column diagonally maximal and that this pitfall can be fixed by a simple modification of the small-loss assumption.
CHAN: Cross-Modal Hybrid Attention Network for Temporal Language Grounding in Videos

The goal of temporal language grounding (TLG) task is to temporally localize the most semantically matched video segment with respect to a given sentence query in an untrimmed video. How to effectively incorporate the cross-modal interactions between video and language is the key to improve grounding performance. Previous approaches focus on learning correlations by computing the attention matrix between each frame-word pair, while ignoring the global semantics conditioned on one modality for better associating the complex video contents and sentence query of the target modality. In this paper, we propose a novel Cross-modal Hybrid Attention Network, which integrates two parallel attention fusion modules to exploit the semantics of each modality and interactions in cross modalities. One is Intra-Modal Attention Fusion, which utilizes gated self-attention to capture the frame-by-frame and word-by-word relations conditioned on the other modality. The other is Inter-Modal Attention Fusion, which utilizes query and key features derived from different modalities to calculate the co-attention weights and further promote inter-modal fusion. Experimental results show that our CHAN significantly outperforms several existing state-of-the-arts on three challenging datasets (ActivityNet Captions, Charades-STA and TACOS), demonstrating the effectiveness of our proposed method.
Multitarget–Multidetection Tracking Using the Kernel SME Filter

With the growing availability of high-resolution sensors, processing more than one detection per target becomes increasingly critical when tracking multiple extended objects. However, contemporary sensors often generate spurious detections that need to be considered. Naively employing standard multitarget trackers may result in poor tracking performance for multitarget–multidetection tracking in cluttered environments, and the relevant extensions are nontrivial. This paper introduces a version of the kernel symmetric measurement equation (SME) filter that considers both multidetections and clutter. For a simulated scenario, our novel filter achieved a higher accuracy than the global nearest neighbor (GNN) and a fast variant of the joint probabilistic data association filter (JPDAF).
Deep Classifiers Evidential Fusion with Reliability

The majority of evidential fusion models presented in the literature is based on optimistic assumptions about the reliability of the models producing beliefs and assumes that they are equally reliable. At the same time, the belief models used in combination may have some limitations and may result in different reliabilities, which may decrease the performance of the combination. One way to confront this problem is to consider a discount rule utilizing reliability coefficients. One of the problems of using discounting is the way of modeling reliability coefficients. This paper proposes modeling reliability coefficients by considering a new effective measure of belief uncertainty. The new reliability coefficients are introduced in a multilayer decision fusion-based Convolutional Neural Network (CNN) architecture built within the Transferable Belief Model, as well as in a multimodal deep learning scenario. Case study results demonstrate the feasibility of representing reliability by the belief uncertainty measure considered.
Joint bias and target state estimation based on Doppler sensors

Target state estimation with Doppler-only sensors has attracted a lot of attention due to its wide potential applications in target localization and tracking. While existing Doppler-only tracking methods rely on the assumption that Doppler sensors have been correctly registered, in many practical cases there can be significant registration errors which imply measurement biases and thus performance degradation in target state estimation. Motivated by this issue, the present paper addresses the problem of jointly estimating target state and sensor biases based on Doppler-only measurements. The proposed method consists of two phases, i.e., (1) raw estimation of the target state without considering sensor biases, followed by (2) a bias compensation step that relies on linearization of the measurement function and joint estimation of target state-sensor biases via a least square method. The Cramer-Rao lower bound (CRLB) in estimating sensor biases is evaluated and the performance of the proposed method is also assessed via simulations.
A Gaussian Integral Filter with Multivariate Laplace Process Noise

This paper introduces the concept of the Gaussian integral filter (GIF), the limit of the Gaussian sum filter (GSF) for when the number of mixands tends to infinity. The GIF is obtained via a combination of GSF, quadrature, and interpolation. While it is a very general concept, in this paper the GIF is used to represent multiviariate Laplace (ML) distributions defining the process noise when tracking a maneuvering target. The filter is first applied to a linear three-dimensional toy problem, and then to a maneuvering target tracking problem in Earth orbit. For the more complex maneuvering target tracking problem, the filter requires only 1.4 times the computational resources of an unscented Kalman filter (UKF), while having errors up to 11 times smaller. For the same problem, the UKF slowly diverges.
Characterization of Emerging AI Workloads: Neural Logic Machines and Graph Convolutional Networks

The present renaissance of artificial intelligence has created new domains of AI models. Some of these domains include Neuro-Symbolic AI (NSAI) and Graph Neural Networks (GNN). NSAI and GNN models have already demonstrated the capability to outperform state-of-the-art deep learning models in domains such as image and video reasoning, and network classification respectively. They have also been shown to obtain high accuracy with significantly less training data than traditional deep neural network models. Due to the recency of the fields' emergence and relative sparsity of published results, the performance characteristics of these models are not well understood. In this paper, we describe and analyze two recent models in these domains. We find that the symbolic model has less potential parallelism than traditional neural models due to complex control flow and low-operational-intensity operations and high cost of data movement. Additionally in the GNN model, we find an abundance of sparse matrix multiplication. Dense MM has a high potential for parallelism through usage of tensor cores, meanwhile new techniques for increasing parallelism in sparse matrix multiplication will be of extreme importance for GNN models.
Automatic Estimation of Ship-Mounted Cameras’ Orientation by Hand-Eye Calibration

By developing a method for automatically calibrating the extrinsic parameters of ship-mounted cameras, this paper tests combining Structure from Motion-algorithms with Hand-Eye calibration solvers in a novel algorithm which demonstrates an ability to discern the orientation of cameras with accuracy comparable to- or better than current manual methods, proven through tests with both synthetic and real-world data.
Graph Convolutional GRU for Music-Oriented Dance Choreography Generation

Music-oriented dance choreography is a complex art form that requires consideration of various factors in music understanding and physical aesthetics. In addition, learning model to generate dance sequences automatically needs to overcome not only the challenges from choreography but also the complexity of human skeleton structure. In this paper, we propose a graph convolutional GRU (GraphGRU) model for music-oriented dance choreography by integrating the graph convolutional networks into the gated recurrent unit. The GraphGRU can model the spatial relationship among the human skeleton joints and the temporal relationship of the dancing sequences. Besides, we define multiple dance-driven kinematic graphs for GraphGRU and establish a path attention block to fuse the graph features. Moreover, we design a reverse-generation loss to ensure consistency between the generated dance and the music. Extensive experiments demonstrate that the proposed model GraphGRU can effectively solve the task of dance choreography.
Observability of Bias of Measurements in Bearings-Only Target Motion Analysis

The problem of biased measurements in bearings-only target motion analysis (BOTMA) is the topic of this article. More precisely, we thoroughly establish the observability condition of the trajectory of the target together with the bias (assumed constant during the whole scenarios).
The linear multitarget IPDA and its application on only a subset of the tracks

Track initiation in multi-object tracking for groups of objects traveling close to each other may require considering many unlikely tracks near each other. Limiting track numbers by not initiating within the gate, as is commonly done, does not work well in these scenarios. With many tracks, computing the exact marginal track to measurement probabilities in a joint integrated probabilistic data association (JIPDA) is computationally expensive.In addressing this problem, we consider the linear multitarget (LM) IPDA, which is a linear approximation of the data association in JIPDA. Here, we formulate it as a Poisson point process (PPP) approximation of the track measurement densities with a particular intensity function. Given our focus on track initialization, we devise how to use the LM approximation on only a subset of the tracks, while the other tracks can be treated as in JIPDA after that. This gives a novel new approximation of the track to measurement data association probabilities which we term LMS.Simulations show that using the LM technique is superior to the PHD filter in terms of posterior track existence probability. Further, it is seen that the PHD as an intensity function in LM performs worse than the original LM in terms of the data association probabilities. The LMS data association probabilities are also shown to typically have better worst-case errors than the original LM and loopy belief propagation (LBP). In terms of the GOSPA metric, initiating tracks on every measurement using LMS gives more reliable and faster track initialization for objects appearing close to already established tracks.
Improving CoatNet for Spatial and JPEG Domain Steganalysis

The covert communication technology represented by image steganography can realize the safe and reliable information transmission over an open network. Steganalysis is an important way to measure the security of steganography. The application of deep learning in the field of steganalysis has begun to emerge and achieved certain results. However, the existing steganalysis methods based on deep learning extract limited global features and the detection accuracy still needs to be improved. Therefore, this manuscript proposes an improving CoatNet for spatial and JPEG domain steganalysis (IMCoatNet). Firstly, the SKAttention structure is used to extract fine-grained features. Then, the convolution layer, Transformer layer and the pyramid squeezed attention is used to capture multi-scale spatial features. Finally, a weighted loss is designed to enhance the expression ability of the model. The results show that compared with the traditional Spatial Rich Model (SRM) method, the classical deep residual network for steganalysis (SRNet), and the latest siamese CNN for steganalysis (SiaStegNet), the detection accuracy of the proposed CoatNet method for WOW steganography is improved by 19.61%, 4.33% and 5.04%, respectively; The proposed IMCoatNet is also superior on JPEG domain detection.
Minimizing Computational Overhead while Retaining Gameplay Effectiveness in Starcraft Bots

Over the past decade, RTS games like Starcraft have emerged as testbeds for developing complex artificial intelligence. While several bots have been successful in their objective of beating opponents, no research has sought to minimize their computational resource usage. Reducing a models' computational complexity has the potential to expand the range of applications for such AI, which is the primary objective that VikingBot was designed to explore. By combining a heterogeneous agent model with a machine learning approach, VikingBot demonstrates that a Starcraft bot can minimize resource usage while retaining good gameplay effectiveness.
Uncertainty and Dependency-wise Requirements Prioritization

With the increase of complexity in software systems, optimizing the requirements management has become an important task in Software Engineering. The selection and prioritization of software requirements among plenty of requirements for each release of a software is a common problem. Several methods have been introduced to solve this common problem. Literature shows that the majority of existing requirement prioritization (RP) approaches ignore fundamental factors such as uncertainty in cost and interdependency between requirements. The objective of this paper is to present an approach to solve the RP problem that considers, with other commonly utilized factors, uncertainty and interdependences between requirements.
To Shuffle or Not To Shuffle: Mini-Batch Shuffling Strategies for Multi-class Imbalanced Classification

Mini-batch shuffling is important for the deep learning training process. Most people use the random shuffling method, which aims to produce a random permutation of the training dataset in every epoch. In this study, we explore mini-batch shuffling for multi-class imbalanced data classification by investigating several shuffling strategies. We find that different order of input data can significantly affect the results of deep learning models. The results show that our proposed strategies can improve the accuracy by around 2%, demonstrating that higher diversity and lower imbalance ratio in each mini-batch can lead to better results.
Inteplato: Generating Mappings of Heterogeneous Relational Schemas Using Unsupervised Learning

The growth of unstructured, unclean, and incomplete data poses quite a problem on attempts to integrate information from various disparate repositories. This paper describes Inteplato, a scalable web-based framework for domain-independent schema mappings between numerous heterogeneous databases. Starting with schema XML exports, the local schemas are visualized in a knowledge graph. A novel algorithm generates clusters based on similarity of local concepts by utilizing fuzzy string, synonym intersection, data type, and constraint similarities. Using these clusters, another algorithm generates mappings between global and local level concepts. The autonomy of each participating database system is maintained, and the mappings enable code automation for global-to-local query propagation. Accuracy was used as a metric to evaluate the generated mappings between local and global concepts.
Fairness-Aware Client Selection for Federated Learning

Federated learning (FL) has enabled multiple data owners (a.k.a. FL clients) to train machine learning models collaboratively without revealing private data. Since the FL server can only engage a limited number of clients in each training round, FL client selection has become an important research problem. Existing approaches generally focus on either enhancing FL model performance or enhancing the fair treatment of FL clients. The problem of balancing performance and fairness considerations when selecting FL clients remains open. To address this problem, we propose the Fairness-aware Federated Client Selection (FairFedCS) approach. Based on Lyapunov optimization, it dynamically adjusts FL clients’ selection probabilities by jointly considering their reputations, times of participation in FL tasks and contributions to the resulting model performance. By not using threshold-based reputation filtering, it provides FL clients with opportunities to redeem their reputations after a perceived poor performance, thereby further enhancing fair client treatment. Extensive experiments based on real-world multimedia datasets show that FairFedCS achieves 19.6% higher fairness and 0.73% higher test accuracy on average than the best-performing state-of-the-art approach.
Inclusion-Exclusion Knowledge Filtering Approach for Conversation-Based Preliminary Diagnosis

Using natural language processing (NLP) techniques, we conducted a preliminary diagnosis of the disease from the patient syndrome description. Because patients are not medical professionals, they cannot accurately describe all symptoms. To solve this issue, we build a medical knowledge graph (KG) by constructing symptom-disease relation triples for pre-processing the patient syndrome description. According to the medical KG, the descriptions were reconstructed into KG embedding representation. To avoid the knowledge noise issue, we investigate an inclusion-exclusion knowledge filtering approach (IKFA) for symptom-to-disease triples to load them to a pretrained language model (PLM), i.e., bidirectional encoder representations from Transformers (BERT). To train the IKFA, we built a medical diagnosis question-answer dataset (MDQA dataset), which contains large-scale and high-quality questions (patient symptom description) and answers (diagnosis) (Q&A) corpus with 1.63 million entries in the size of 213 MB. The KG was built based on 8,731 diseases with detailed syndrome descriptions in the size of 1.98 MB. The experimental results showed that the IKFA preliminarily diagnosed 8,731 different diseases based on the patient's initial symptom description with an accuracy of 0.9894.
Building Framework and Infrastructure to Assist a Minority Serving Institution (MSI) in Cybersecurity Program Development

The University of North Florida (UNF) offered to leverage resources generated as a result of its CAE-EDU designation to assist Edward Waters University (EWU), a local Minority-Service Institution (MSI), with establishing cyberse-curity educational opportunities for their STEM faculty and undergraduate students. This project, funded by NSA as part of their Cybersecurity Education Diversity Initiative (CEDI) program, designs the framework and cultivates the infrastructure by which EWU gains access to UNF instructional resources (e.g., courses, facilities), faculty expertise (i.e., curriculum, consulting) and student activities (e.g., clubs, advising) in cyber defense education. Project plans included (i) outreach activities with support from faculty of the Florida State College of Jacksonville (FSCJ), a local community college, also CAE-designated, as well as (ii) proof-of-concept assessments with support from faculty and students of Edward Waters University (EWU), an MSI with no existing cybersecurity programs. Some of the activities performed under this project and their outcomes are reported in this paper.