Reviewerly/exHarmony huggingface dataset:
Enhance Public Safety Surveillance in Smart Cities by Fusing Optical and Thermal Cameras

The recent advancements in the Internet of Video Things (IoVT) and Edge-Fog-Cloud Computing paradigm make smart public safety surveillance (SPSS) a realistic solution for an effective public safety service in smart cities. Typically, a fully functional SPSS system requires multiple sensory inputs for situational awareness (SAW). As an essential component in the context of highly complex, dynamic, and heterogeneous smart city operations, SPSS is expected to be environment-resilient. Personal safety is among the top concerns of the residents in smart cities, and correspondingly pedestrian detectors are critical. Contemporary pedestrian detectors use optical cameras, whose accuracy is diminished in low-light environments, and they are rendered ineffective when obstacles block the direct line of sight to the camera. Complementary imaging sensors such as infrared have shown promise. This paper presents a full-spectrum, environment-resilient surveillance platform as an ultimate solution, which consists of multiple imaging units to cover a wide sensing spectrum. The initial hybrid pedestrian detection (HYPE) scheme is based on the fusion of data obtained from an IoVT network equipped with optical and thermal cameras. We demonstrate that training the YOLOv5 object detection model on a dataset of infrared images improves its accuracy in the detection of humans present in thermal surveillance images. A 41% decrease in objectness loss is achieved after transfer learning is performed.
Co-embedding Multi-type Data for Information Fusion and Visual Analytics

This paper proposes a novel interactive system for exploratory document search in multi-type data sets that employs a data fusion approach. The system, designed to visualize different object types collectively and clearly display their semantic proximity, utilizes a co-embedding technique for knowledge fusion of multi-type data and projects the various types of objects onto a common lower-dimensional space. This produces a more informed representation and visualization that shows both in-type and across-type semantic proximity between objects. The system enables users’ exploration of multi-type document data by providing embedding-based scatter plots to visualize semantic relations within and across object types. In addition, based on user relevance feedback of displayed objects, the system offers recommendations of relevant objects. We have demonstrated the effectiveness of the proposed system and the underlying embedding method through comparison experiments with the proposed fusion-based approach.
Multi-Tier Client Selection for Mobile Federated Learning Networks

Federated learning (FL), which addresses data privacy issues by training models on resource-constrained mobile devices in a distributed manner, has attracted significant research attention. However, the problem of optimizing FL client selection in mobile federated learning networks (MFLNs), where devices move in and out of each others’ coverage and no FL server knows all the data owners, remains open. To bridge this gap, we propose a first-of-its-kind Socially-aware Federated Client Selection (SocFedCS) approach to minimize costs and train high-quality FL models. SocFedCS enriches the candidate FL client pool by enabling data owners to propagate FL task information through their local networks of trust, even as devices are moving into and out of each others’ coverage. Based on Lyapunov optimization, we first transform this time-coupled problem into a step-by-step optimization problem. Then, we design a method based on alternating minimization and self-adaptive global best harmony search to solve this mixed-integer optimization problem. Extensive experiments comparing SocFedCS against five state-of-the-art approaches based on four real-world multimedia datasets demonstrate that it achieves 2.06% higher test accuracy and 12.24% lower cost on average than the best-performing baseline.
Stone Soup: No Longer Just an Appetiser

This paper announces version 1.0 of Stone Soup: the open-source tracking and state estimation framework. We highlight key elements of the framework and outline example applications and community activities.Stone Soup is engineered with modularity and encapsulation at its heart. This means that its many components can be put together in any number of ways to build, compare, and assure almost any type of multi-target tracking and fusion algorithm. Since its inception in 2017, it has aimed to provide the target tracking and state estimation community with an open, easy-to-deploy framework to develop and assess the performance of different types of trackers. Now, through repeated application in many use cases, implementation of a wide variety of algorithms, multiple beta releases, and contributions from the community, the framework has reached a stable point.In announcing this release, we hope to encourage additional adoption and further contributions to the toolkit. We also acknowledge and express appreciation for the many contributions of time and expertise donated by the tracking and fusion community.
Deep-RSI: Deep learning for Radiographs Source Identification

In forensics, the authenticity of digital images is of the utmost importance, considering that modern technology makes it incredibly simple and quick to alter and generate fake but convincing images. As a result, digital image credibility has decreased, making it difficult to demonstrate the source of images. Prior studies have shown that magnetic resonance imaging (MRI) scans can be traced back to their sources, but radiographs have not. In this paper, we propose the Deep-RSI algorithm, an algorithm that identifies the source (manufacturer and model) of the device used to create radiographs. This is the first time that a medical forensics investigation of this kind will be accomplished to declare and confirm radiograph sources. Researchers in information forensics, security, and medical imaging can use this data to determine scientific fraud, like fake radiographs made from unreliable sources or cut-and-paste fakes. This proposed solution describes how non-content pixels in images enable us to discover the manufacturer and model of a radiographic machine. Since radiographs are obtained from different sites of the body, source recognition has to be sensitive and free of any content-specific information. This will prevent the convolutional neural networks (CNN) from detecting content-specific details and instead identify fingerprints that are unique to the source. CNNs start with low-level features and, in the convolutional blocks, generate high-level features to identify the radiographic machine sources. This proposed solution reports the source (manufacturer and model) of each image. We obtain the highest AUC of 0.97 and a prediction accuracy of 98.54% for radiographic machine manufacturer detection. Our results show that forensic assessments of radiographs can be done with a high level of certainty.
Classification of Warship Formations Using a Kohonen Network

Military systems require accurate technical solutions to support decision making. In naval warfare, an important problem to solve relies on a capability to detect higher level category artifacts, such as warship formations. This work aims to fill a gap observed in literature about this subject. In this paper, we proposed, implemented and tested a model to detect warship formations, capable of classifying the formation according to its type. We also investigate some published works related to this subject and point out the differences and gaps perceived. Using a Kohonen Network as classifier based on position and velocity of ships, this work describes the computational model used and the results obtained according to different number of ships and samples. Using synthetic data combined with real data, the results show, with different types of metrics, that the adopted solution has reliable and promising results and it is adequate to deal with a real-world problem.
E<sup>2</sup>: Entropy Discrimination and Energy Optimization for Source-free Universal Domain Adaptation

Universal domain adaptation (UniDA) transfers knowledge under both distribution and category shifts. Most UniDA methods accessible to source-domain data during model adaptation may result in privacy policy violation and source-data transfer inefficiency. To address this issue, we propose a novel source-free UniDA method coupling confidence-guided entropy discrimination and likelihood-induced energy optimization. The entropy-based separation of target-known and unknown classes is too conservative for known-class prediction. Thus, we derive the confidence-guided entropy by scaling the normalized prediction score with the known-class confidence, that more known-class samples are correctly predicted. Due to difficult estimation of the marginal distribution without source-domain data, we constrain the target-domain marginal distribution by maximizing (minimizing) the known (unknown)-class likelihood, which equals free energy optimization. Theoretically, the overall optimization amounts to decreasing and increasing internal energy of known and unknown classes in physics, respectively. Extensive experiments demonstrate the superiority of the proposed method.
Semantic Stage-Wise Learning for Knowledge Distillation

Knowledge distillation enhances the performance of the student model by transferring knowledge from the teacher model. Moreover, the attention mechanism has been introduced recently to enable each layer of the student to learn knowledge from all teacher layers, which brings about considerable optimization. However, noted that features from different layers, such as shallow and deep layers, might have a big semantic gap, and compulsively aligning one student layer to all teacher layers would mislead the learning process. To tackle this problem, an effective framework called Semantic Stage-Wise learning for Knowledge Distillation (SSWKD) is presented in this paper. We divide all layers into shallow and deep stages, and only allow feature alignment within the same stage to alleviate semantic mismatch. In addition, with the observation that the performance of deep networks relies more on some key features rather than evenly on all of them, a crucial feature enhancement method based on KL divergence is then proposed for SSWKD, forcing the student to pay more attention to critical features of the teacher. Extensive experiments and visualizations show that our SSWKD outperforms other distillation methods on CIFAR-100 and COCO2017 datasets for image classification, object detection, and instance segmentation tasks.
Probability Distributions in Coastal Regions for Association of Naval Vessels

A multi-target tracking method requires a data association algorithm to decide which measurements, from a set of many, to use for updating each track. In this paper we will consider a case where the position of multiple marine vessels is measured in a coastal region, and the time span between each measurement set is large. The vessels of interest are assumed to be sufficiently inert in the sense that the vessels intend to maintain a particular cruising speed while traveling to their destination. Moreover, by assuming that the vessels intend to travel the shortest route possible, we have a foundation to assign conditional probabilities to each measured position. We use the fast marching method (FMM) to compute a time-of-arrival (ToA) map which indicates where the vessel can be at a particular time, provided that the initial speed is known and remains constant. The ToA map is then evaluated with respect to a Gaussian distribution which provides a probability density function (pdf) used to evaluate the probability that a new measurement originate from the same vessel as a track.
Adversarial Speaker Disentanglement Using Unannotated External Data for Self-supervised Representation-based Voice Conversion

Nowadays, recognition-synthesis-based methods have been quite popular with voice conversion (VC). By introducing linguistics features with good disentangling characters extracted from an automatic speech recognition (ASR) model, the VC performance achieved considerable breakthroughs. Recently, self-supervised learning (SSL) methods trained with a large-scale unannotated speech corpus have been applied to downstream tasks focusing on the content information, which is suitable for VC tasks. However, a huge amount of speaker information in SSL representations degrades timbre similarity and the quality of converted speech significantly. To address this problem, we proposed a high-similarity any-to-one voice conversion method with the input of SSL representations. We incorporated adversarial training mechanisms in the synthesis module using external unannotated corpora. Two auxiliary discriminators were trained to distinguish whether a sequence of mel-spectrograms has been converted by the acoustic model and whether a sequence of content embeddings contains speaker information from external corpora. Experimental results show that our proposed method achieves comparable similarity and higher naturalness than the supervised method, which needs a huge amount of annotated corpora for training and is applicable to improve similarity for VC methods with other SSL representations as input.
DynaSlim: Dynamic Slimming for Vision Transformers

Vision transformers (ViTs) have achieved significant performance on various vision tasks. However, high computational and memory costs hinder their edge deployment. Existing compression methods employ static constraints between accuracy and efficiency during sparsification. The static constraints restrict the sparsification efficiency and their initialization relies heavily on human expertise. We propose a dynamic slimming strategy for ViT, DynaSlim, to achieve an adaptive accuracy-efficiency constraint during sparsification. We first equip fine-grained, adjustable sparsity weights, the scaling factor between accuracy and efficiency, for multiple dimensions, including input tokens, Multihead Self-Attention (MSA) and Multilayer Perceptron (MLP). We then employ the heuristic search for these non-differentiable factors and combine the search with regularization-based sparsification to obtain the optimal sparsed model. Finally, we compress and retrain the sparsed model under various budgets to get our resulting submodels. Experiments show that our DynaSlim outperforms previous state-of-the-art methods under different budgets. For example, we reduce both parameters and FLOPs of DeiT-B by 39% while increasing its accuracy by 1.9% on ImageNet-1K. Moreover, we demonstrate the transferability of our compressed models on several downstream datasets.
Structural Equivariance Self-Supervised Learning for Facial Pose Estimation

In this paper, we propose a self-supervised learning method for robust facial pose estimation. Conventional methods usually split the coherent head motion into discrete and finite outputs, likely leading to bias prediction because the performance of head pose estimation highly relies on structural facial appearance. To address this issue, our model achieves structural equivariance to poses through a self-supervised learning strategy from extrinsic attributes of face neighbors and underlying local associations. Specifically, we construct a complete neighbor graph to capture the extrinsic properties of face neighbors, where different latent semantic attributes are assigned to each subgraph. Accordingly, we design a set of proxy tasks based on different attribute subgraphs, where the model is encouraged to learn the underlying relation of local features under pose variation. Extensive experimental results on the challenging, widely evaluated datasets indicate the effectiveness of our model compared with the state of the arts.
Few-Shot Object Detection via Back Propagation and Dynamic Learning

Utilizing traditional object detectors to build a few-shot object detection (FSOD) model ignores the differences between classification and regression tasks and causes task conflict and class confusion, resulting in a decline in classification performance. In contrast, this paper focuses on the above shortcomings and utilizes the strategies of Back Propagation and Dynamic Learning to construct a model for addressing FSOD, named BPDL. Our BPDL has a two-fold main idea: a) it uses the optimized localization boxes to alleviate the task conflict and refine classification features by a correction loss, and b) it develops a dynamic learning strategy to filter the confusing features and mine more realistic prototype representations of the categories to calibrate classification. Extensive experiments on multiple benchmarks show that our BPDL model outperforms existing methods and advances the FSOD task’s state-of-the-art.
Exercise and Sedentary Activity Recognition Using Late Fusion: Building Adaptable Uncertain Models

Wearable smart devices are capable of capturing a variety of information from their users using a multitude of noninvasive sensing modalities. Using features from the raw measurements of wearable devices, sensor fusion enables us to obtain a holistic picture of the users’ context and monitor their activity state with increased accuracy. Human activity recognition using noninvasive sensors allows us to capture the natural behavior of users in their day-to-day lives. This in-the-wild activity recognition, however, poses several key challenges that must be addressed to create effective classification models. The main challenges are class imbalance, uncertainty in classifier decisions, and large feature spaces. To address them, this study further explores a probabilistic sensor fusion method called Naive Adaptive Probabilistic Sensor (NAPS) Fusion. In doing so, we establish the viability of NAPS Fusion for natural human activity recognition using noninvasive sensing modalities. NAPS Fusion handles dimensionality reduction by creating reduced feature sets and mitigates the class imbalance issue through the use of Synthetic Minority Oversampling Technique (SMOTE). Moreover, NAPS Fusion addresses uncertainty in the decisions of classifiers using a Dempster-Shafer theoretic late fusion framework. Our empirical evaluation demonstrates that NAPS Fusion has broad applications beyond its original design for cognitive state detection. It outperforms similar decision level sensor fusion methods (late fusion using averaging, LFA, and late fusion using learned weights, LFL) in the detection of exercise and sedentary activities such as walking, running, lying down, and sitting. We observe improvements of up to 56% in F1 score and up to 59% in precision with NAPS Fusion over the compared methods.
Adaptive Non-local Affinity Graph for Unsupervised Image Segmentation

In this paper, we present an adaptive superpixel-based graph construction method for unsupervised image segmentation. The empirical findings in this study provide a new understanding of applying non-local image patches to image segmentation. We demonstrate that high-level segmentation cues can be revealed by non-local similar patches, and based on it the similarity between superpixels overlapped with these similar patches can be adaptively constructed. The experimental result on the Berkeley Segmentation Database demonstrates the effectiveness and superiority of the proposed algorithm.
ERPG: Enhancing Entity Representations with Prompt Guidance for Complex Named Entity Recognition

Recently, sequence generation methods are widely used in complex named entity recognition. By selecting high-related tokens to generate complex named entities, these methods obtain several achievements. However, due to lack of guidance in learning output format and ignoring labels in obtaining features, sequence generation methods suffer invalid output and inaccurate recognition. To solve that, we propose an Enhancing Entity Representation method with Prompt Guidance (ERPG). Specifically, in order to reduce invalid output, we design the candidate entity generation module that generate candidate entities and their labels as expected. Besides, to accurately recognize candidate entities, we propose candidate entity refine module, which obtain distinguishable candidate entity representations and filter them accurately. Based on that, our method finally outperforms baselines by 1.20, 1.62 and 0.69 F1 scores in ACE2004, GENIA and CADEC corpora, which proves the effectiveness in complex named entity recognition.
Multimodal Aspect-Based Sentiment Classification with Knowledge-Injected Transformer

The multimodal aspect-based sentiment classification task (mABSC) aims to recognize the sentiment polarities of aspect entities according to the associated textual and visual resources. Inspired by the cross-modal alignment ability of Transformers, some recent mABSC methods have also proposed to use Transformers to discover relevance between aspect entity and cross-modal visual regions, based on which relevant visual regions can be identified and leveraged to help recognize the sentiment polarity of aspect entities. However, due to the limited training data in mABSC tasks, it is found that in the Transformers, the aspect entities are often attended to irrelevant visual information, which obviously will not benefit the sentiment polarity recognition. To address the issue, we seek help from external knowledge, including textual syntax and cross-modal relevancy knowledge. The basic idea is to cut off the irrelevant connections among textual or cross-modal modalities in the Transformer layer using a knowledge-induced matrix. To have the matrix going beyond simply capturing direct relations, a mechanism is developed to enable it to reflect multi-hop relations, followed by a discretization operation to filter out extreme relevancy. Extensive experiments on two public multimodal datasets show that our method outperforms all competing baselines. Further studies demonstrate the effectiveness of each component, and suggest that the introduced external knowledge can instruct the model to learn the correct relevance among textual or cross-modal features, thus benefiting the mABSC task.
FONT: Flow-guided One-shot Talking Head Generation with Natural Head Motions

One-shot talking head generation has received growing attention in recent years, with various creative and practical applications. An ideal natural and vivid generated talking head video should contain natural head pose changes. However, it is challenging to map head pose sequences from driving audio since there exists a natural gap between audio-visual modalities. In this work, we propose a Flow-guided One-shot model that achieves NaTural head motions(FONT) over generated talking heads. Specifically, we design a probabilistic CVAE-based model to predict head pose sequences from driving audio and source face. Then we develop a keypoint predictor that produces unsupervised keypoints describing the facial structure information from the source face, driving audio and pose sequences. Finally, a flow- guided occlusion-aware generator is employed to produce photo-realistic talking head videos from the estimated keypoints and source face. Extensive experimental results prove that FONT generates talking heads with natural head poses and synchronized mouth shapes, outperforming other compared methods.