Reviewerly/exHarmony huggingface dataset:
Learning Deep Multi-Level Similarity for Thermal Infrared Object Tracking

Existing deep Thermal InfraRed (TIR) trackers only use semantic features to represent the TIR object, which lack the sufficient discriminative capacity for handling distractors. This becomes worse when the feature extraction network is only trained on RGB images. To address this issue, we propose a multi-level similarity model under a Siamese framework for robust TIR object tracking. Specifically, we compute different pattern similarities using the proposed multi-level similarity network. One of them focuses on the global semantic similarity and the other computes the local structural similarity of the TIR object. These two similarities complement each other and hence enhance the discriminative capacity of the network for handling distractors. In addition, we design a simple while effective relative entropy based ensemble subnetwork to integrate the semantic and structural similarities. This subnetwork can adaptive learn the weights of the semantic and structural similarities at the training stage. To further enhance the discriminative capacity of the tracker, we propose a large-scale TIR video sequence dataset for training the proposed model. To the best of our knowledge, this is the first and the largest TIR object tracking training dataset to date. The proposed TIR dataset not only benefits the training for TIR object tracking but also can be applied to numerous TIR visual tasks. Extensive experimental results on three benchmarks demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods.
Smart and Practical Privacy-Preserving Data Aggregation for Fog-Based Smart Grids

With the increasingly powerful and extensive deployment of edge devices, edge/fog computing enables customers to manage and analyze data locally, and extends computing power and data analysis applications to network edges. Meanwhile, as the next generation of the power grid, the smart grid can achieve the goal of efficiency, economy, security, reliability, use safety and environmental friendliness for the power grid. However, privacy and secure issues in fog-based smart grid communications are challenging. Without proper protection, customers’ privacy will be readily violated. This article presents a smart and practical Privacy-preserving Data Aggregation (PDA) scheme with smart pricing and packing method for fog-based smart grids, which achieves diversified tariffs, multifunctional statistics and efficiency. Especially, we first propose a smart PDA scheme with Smart Pricing (PDA-SP). With PDA-SP, the Control Center (CC) can compute more complex and higher-order aggregation statistics to provide various services, provide diversiform pricing strategies and choose a double-winning strategy. Subsequently, we put forward a practical PDA scheme with Packing Method (PDA-PM), which is able to reduce the size of encrypted data and improve performance in performing various secure computations. Moreover, we extend our original packing method and present a more useful packing method, which can handle general vectors with large entries. The security analysis shows that our proposed scheme is secure against many threats. The performance evaluation reveals that the computation and communication overheads of our proposed scheme are effectively reduced by employing the Somewhat Homomorphic Encryption (SHE), and our packing method can further significantly reduce these overheads.
Low-dimensional dynamics for working memory and time encoding

Our decisions often depend on multiple sensory experiences separated by time delays. The brain can remember these experiences and, simultaneously, estimate the timing between events. To understand the mechanisms underlying working memory and time encoding, we analyze neural activity recorded during delays in four experiments on nonhuman primates. To disambiguate potential mechanisms, we propose two analyses, namely, decoding the passage of time from neural data and computing the cumulative dimensionality of the neural trajectory over time. Time can be decoded with high precision in tasks where timing information is relevant and with lower precision when irrelevant for performing the task. Neural trajectories are always observed to be low-dimensional. In addition, our results further constrain the mechanisms underlying time encoding as we find that the linear “ramping” component of each neuron’s firing rate strongly contributes to the slow timescale variations that make decoding time possible. These constraints rule out working memory models that rely on constant, sustained activity and neural networks with high-dimensional trajectories, like reservoir networks. Instead, recurrent networks trained with backpropagation capture the time-encoding properties and the dimensionality observed in the data.
Adversarial Attacks in Modulation Recognition With Convolutional Neural Networks

Deep learning (DL) models are vulnerable to adversarial attacks, by adding a subtle perturbation which is imperceptible to the human eye, a convolutional neural network (CNN) can lead to erroneous results, which greatly reduces the reliability and security of the DL tasks. Considering the wide application of modulation recognition in the communication field and the rapid development of DL, by adding a well-designed adversarial perturbation to the input signal, this article explores the performance of attack methods on modulation recognition, measures the effectiveness of adversarial attacks on signals, and provides the empirical evaluation of the reliabilities of CNNs. The results indicate that the accuracy of the target model reduce significantly by adversarial attacks, when the perturbation factor is 0.001, the accuracy of the model could drop by about 50% on average. Among them, iterative methods show greater attack performances than that of one-step method. In addition, the consistency of the waveform before and after the perturbation is examined, to consider whether the added adversarial examples are small enough (i.e., hard to distinguish by human eyes). This article also aims at inspiring researchers to further promote the CNNs reliabilities against adversarial attacks.
Unsupervised Domain Adaptation with Noise Resistible Mutual-Training for Person Re-identification

Unsupervised domain adaptation (UDA) in the task of person re-identification (re-ID) is highly challenging due to large domain divergence and no class overlap between domains. Pseudo-label based self-training is one of the representative techniques to address UDA. However, label noise caused by unsupervised clustering is always a trouble to self-training methods. To depress noises in pseudo-labels, this paper proposes a Noise Resistible Mutual-Training (NRMT) method, which maintains two networks during training to perform collaborative clustering and mutual instance selection. On one hand, collaborative clustering eases the fitting to noisy instances by allowing the two networks to use pseudo-labels provided by each other as an additional supervision. On the other hand, mutual instance selection further selects reliable and informative instances for training according to the peer-confidence and relationship disagreement of the networks. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art UDA methods for person re-ID.
Neural Unsupervised Domain Adaptation in NLP—A Survey

Deep neural networks excel at learning from labeled data and achieve state-of-the-art results on a wide array of Natural Language Processing tasks.In contrast, learning from unlabeled data, especially under domain shift, remains a challenge.Motivated by the latest advances, in this survey we review neural unsupervised domain adaptation techniques which do not require labeled target domain data.This is a more challenging yet a more widely applicable setup.We outline methods, from early traditional non-neural methods to pre-trained model transfer.We also revisit the notion of domain, and we uncover a bias in the type of Natural Language Processing tasks which received most attention.Lastly, we outline future directions, particularly the broader need for out-of-distribution generalization of future NLP. 1
Node Similarity Preserving Graph Convolutional Networks

Graph Neural Networks (GNNs) have achieved tremendous success in various real-world applications due to their strong ability in graph representation learning. GNNs explore the graph structure and node features by aggregating and transforming information within node neighborhoods. However, through theoretical and empirical analysis, we reveal that the aggregation process of GNNs tends to destroy node similarity in the original feature space. There are many scenarios where node similarity plays a crucial role. Thus, it has motivated the proposed framework SimP-GCN that can effectively and efficiently preserve node similarity while exploiting graph structure. Specifically, to balance information from graph structure and node features, we propose a feature similarity preserving aggregation which adaptively integrates graph structure and node features. Furthermore, we employ self-supervised learning to explicitly capture the complex feature similarity and dissimilarity relations between nodes. We validate the effectiveness of SimP-GCN on seven benchmark datasets including three assortative and four disassorative graphs. The results demonstrate that SimP-GCN outperforms representative baselines. Further probe shows various advantages of the proposed framework. The implementation of SimP-GCN is available at https://github.com/ChandlerBang/SimP-GCN.
SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets

In this paper, we present the results of the SemEval-2020 Task 9 on Sentiment Analysis of Code-Mixed Tweets (SentiMix 2020). 1 We also release and describe our Hinglish (Hindi-English) and Spanglish (Spanish-English) corpora annotated with word-level language identification and sentence-level sentiment labels.These corpora are comprised of 20K and 19K examples, respectively.The sentiment labels are -Positive, Negative, and Neutral.SentiMix attracted 89 submissions in total including 61 teams that participated in the Hinglish contest and 28 submitted systems to the Spanglish competition.The best performance achieved was 75.0%F1 score for Hinglish and 80.6% F1 for Spanglish.We observe that BERT-like models and ensemble methods are the most common and successful approaches among the participants.
Assessment of floating solar photovoltaics potential in existing hydropower reservoirs in Africa

Africa is characterised by a very high solar potential, with a yearly sum of solar irradiation exceeding 2000 kWh/m2. Many African countries are heavily dependent on hydropower, however, increasingly frequent droughts have been severely affecting hydropower generation in the last few decades. The installation of floating photovoltaics (FPV) in existing hydropower reservoirs, would provide solar electricity to help compensate hydropower production during dry periods and reduce evaporation losses while helping to sustainably satisfy the current and future energy needs of the fast-growing African population. This study provides a comprehensive analysis of the potential of FPV installation in Africa, by using highly accurate water surface data of the largest 146 hydropower reservoirs in the continent. In addition to the electricity production, evaporation savings and the potential extra hydroelectricity generated by these water savings are also estimated at reservoir level for four different cases and two types of floating structures. The results indicate that with a total coverage of less than 1%, the installed power capacity of existing hydropower plants can double and electricity output grow by 58%, producing an additional 46.04 TWh annually. In this case, the water savings could reach 743 million m3/year, increasing the annual hydroelectricity generation by 170.64 GWh.
Blockchain-based Lamport Merkle Digital Signature: Authentication tool in IoT healthcare

Internet of Things (IoT) includes medical devices, servers, and applications linked using computer networks to collect and share data. Recently there is a massive surge in IoT-enabled applications, especially in the medical field. Medical devices, like intravenous pumps, Magnetic Resonance Imaging (MRI) machines, infusion pumps, and ventilators, are connected with the hospital networks to enhance the quality and efficiency of delivering medical care. Integrated with the cloud server, IoT improves patients’ lifestyles and minimizes the time and cost by efficiently managing the medical resources. However, due to the presence of malicious users, sensitive patient data is often compromised. Motivated and driven by these factors, this work proposes a blockchain-assisted highly secure system for medical IoT devices using Lamport Merkle Digital Signature (LMDS). Initially, the Lamport Merkle Digital Signature Generation (LMDSG) model performs the task of authenticating IoT devices by constructing a tree in which the leaves symbolize sensitive patient medical data’s hash function. Further, a Centralized Healthcare Controller (CHC) performs the task of determining the root of the LMDSG by using Lamport Merkle Digital Signature Verification (LMDSV). In this verification process, when the hash of the public key ‘pbkey′ is equal to leaf ‘Pgn’, then it is the root of the tree, and the signature is valid. In this way, the proposed LMDS technique efficiently identifies malicious user behavior with minimum Computational Overhead (CO) and Computational Time (CT). The proposed LMDS technique’s performance is analyzed in terms of CO, CT, and authentication accuracy. The experimental analysis proves that the proposed LMDS technique ensures higher security and minimum CT and CO in medical IoT systems than other existing methods.
Structured Graph Learning for Scalable Subspace Clustering: From Single View to Multiview

Graph-based subspace clustering methods have exhibited promising performance. However, they still suffer some of these drawbacks: they encounter the expensive time overhead, they fail to explore the explicit clusters, and cannot generalize to unseen data points. In this work, we propose a scalable graph learning framework, seeking to address the above three challenges simultaneously. Specifically, it is based on the ideas of anchor points and bipartite graph. Rather than building an n×n graph, where n is the number of samples, we construct a bipartite graph to depict the relationship between samples and anchor points. Meanwhile, a connectivity constraint is employed to ensure that the connected components indicate clusters directly. We further establish the connection between our method and the K -means clustering. Moreover, a model to process multiview data is also proposed, which is linearly scaled with respect to n . Extensive experiments demonstrate the efficiency and effectiveness of our approach with respect to many state-of-the-art clustering methods.
RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words

Recent progress on visual question answering has explored the merits of grid features for vision language tasks. Meanwhile, transformer-based models have shown remarkable performance in various sequence prediction problems. However, the spatial information loss of grid features caused by flattening operation, as well as the defect of the transformer model in distinguishing visual words and non visual words, are still left unexplored. In this paper, we first propose Grid-Augmented (GA) module, in which relative geometry features between grids are incorporated to enhance visual representations. Then, we build a BERT-based language model to extract language context and propose Adaptive-Attention (AA) module on top of a transformer decoder to adaptively measure the contribution of visual and language cues before making decisions for word prediction. To prove the generality of our proposals, we apply the two modules to the vanilla transformer model to build our Relationship-Sensitive Transformer (RSTNet) for image captioning task. The proposed model is tested on the MSCOCO benchmark, where it achieves new state-of-art results on both the Karpathy test split and the online test server. Source code is available at GitHub <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Contrastive Embedding for Generalized Zero-Shot Learning

Generalized zero-shot learning (GZSL) aims to recognize objects from both seen and unseen classes, when only the labeled examples from seen classes are provided. Recent feature generation methods learn a generative model that can synthesize the missing visual features of unseen classes to mitigate the data-imbalance problem in GZSL. However, the original visual feature space is suboptimal for GZSL classification since it lacks discriminative information. To tackle this issue, we propose to integrate the generation model with the embedding model, yielding a hybrid GZSL framework. The hybrid GZSL approach maps both the real and the synthetic samples produced by the generation model into an embedding space, where we perform the final GZSL classification. Specifically, we propose a contrastive embedding (CE) for our hybrid GZSL framework. The proposed contrastive embedding can leverage not only the class-wise supervision but also the instance-wise supervision, where the latter is usually neglected by existing GZSL researches. We evaluate our proposed hybrid GZSL framework with contrastive embedding, named CE-GZSL, on five benchmark datasets. The results show that our CEGZSL method can outperform the state-of-the-arts by a significant margin on three datasets. Our codes are available on https://github.com/Hanzy1996/CE-GZSL.
Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection

Although current deep learning-based face forgery detectors achieve impressive performance in constrained scenarios, they are vulnerable to samples created by unseen manipulation methods. Some recent works show improvements in generalisation but rely on cues that are easily corrupted by common post-processing operations such as compression. In this paper, we propose LipForensics, a detection approach capable of both generalising to novel manipulations and withstanding various distortions. LipForensics targets high-level semantic irregularities in mouth movements, which are common in many generated videos. It consists in first pretraining a spatio-temporal network to perform visual speech recognition (lipreading), thus learning rich internal representations related to natural mouth motion. A temporal network is subsequently finetuned on fixed mouth embeddings of real and forged data in order to detect fake videos based on mouth movements without overfitting to low-level, manipulation-specific artefacts. Extensive experiments show that this simple approach significantly surpasses the state-of-the-art in terms of generalisation to unseen manipulations and robustness to perturbations, as well as shed light on the factors responsible for its performance.
(AF)<sup>2</sup>-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network

Autonomous robotic systems and self driving cars rely on accurate perception of their surroundings as the safety of the passengers and pedestrians is the top priority. Semantic segmentation is one of the essential components of road scene perception that provides semantic information of the surrounding environment. Recently, several methods have been introduced for 3D LiDAR semantic segmentation. While they can lead to improved performance, they are either afflicted by high computational complexity, therefore are inefficient, or they lack fine details of smaller object instances. To alleviate these problems, we propose (AF) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -S3Net, an end-to-end encoder-decoder CNN network for 3D LiDAR semantic segmentation. We present a novel multibranch attentive feature fusion module in the encoder and a unique adaptive feature selection module with feature map re-weighting in the decoder. Our (AF) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -S3Net fuses the voxel-based learning and point-based learning methods into a unified framework to effectively process the potentially large 3D scene. Our experimental results show that the proposed method outperforms the state-of-the-art approaches on the large-scale nuScenes-lidarseg and SemanticKITTI benchmark, ranking 1 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">st</sup> on both competitive public leaderboard competitions upon publication.
Realization of Real-Time Fault-Tolerant Quantum Error Correction

Correcting errors in real time is essential for reliable large-scale quantum computations. Realizing this high-level function requires a system capable of several low-level primitives, including single-qubit and two-qubit operations, midcircuit measurements of subsets of qubits, real-time processing of measurement outcomes, and the ability to condition subsequent gate operations on those measurements. In this work, we use a 10-qubit quantum charge-coupled device trapped-ion quantum computer to encode a single logical qubit using the [[7,1,3]] color code, first proposed by Steane [Phys. Rev. Lett. 77, 793 (1996)]. The logical qubit is initialized into the eigenstates of three mutually unbiased bases using an encoding circuit, and we measure an average logical state preparation and measurement (SPAM) error of 1.7(2)×10−3, compared to the average physical SPAM error 2.4(4)×10−3 of our qubits. We then perform multiple syndrome measurements on the encoded qubit, using a real-time decoder to determine any necessary corrections that are done either as software updates to the Pauli frame or as physically applied gates. Moreover, these procedures are done repeatedly while maintaining coherence, demonstrating a dynamically protected logical qubit memory. Additionally, we demonstrate non-Clifford qubit operations by encoding a T¯|+⟩L magic state with an error rate below the threshold required for magic state distillation. Finally, we present system-level simulations that allow us to identify key hardware upgrades that may enable the system to reach the pseudothreshold.15 MoreReceived 10 August 2021Revised 24 November 2021Accepted 7 December 2021Corrected 11 January 2022DOI:https://doi.org/10.1103/PhysRevX.11.041058Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.Published by the American Physical SocietyPhysics Subject Headings (PhySH)Research AreasAtom & ion coolingAtom & ion trapping & guidingQuantum computationQuantum error correctionPhysical SystemsTrapped ionsQuantum InformationAtomic, Molecular & Optical