OpenRaiser/Intern-Atlas huggingface dataset:
Human Gait Analysis using Gait Energy Image
Venue: arXiv cs.CV 2022
Year: 2022

Gait recognition is one of the most recent emerging techniques of human biometric which can be used for security based purposes having unobtrusive learning method. In comparison with other bio-metrics gait analysis has some special security features. Most of the biometric technique uses sequential template based component analysis for recognition. Comparing with those methods, we proposed a developed technique for gait identification using the feature Gait Energy Image (GEI). GEI representation of gait contains all information of each image in one gait cycle and requires less storage and low processing speed. As only one image is enough to store the necessary information in GEI feature recognition process is very easier than any other feature for gait recognition. Gait recognition has some limitations in recognition process like viewing angle variation, walking speed, clothes, carrying load etc. Our proposed method in the paper compares the recognition performance with template based feature extraction which needs to process each frame in the cycle. We use GEI which gives relatively all information about all the frames in the cycle and results in better performance than other feature of gait analysis.
Efficient Federated Learning on Knowledge Graphs via Privacy-preserving Relation Embedding Aggregation
Venue: arXiv cs.AI 2022
Year: 2022

Federated learning (FL) can be essential in knowledge representation, reasoning, and data mining applications over multi-source knowledge graphs (KGs). A recent study FedE first proposes an FL framework that shares entity embeddings of KGs across all clients. However, entity embedding sharing from FedE would incur a severe privacy leakage. Specifically, the known entity embedding can be used to infer whether a specific relation between two entities exists in a private client. In this paper, we introduce a novel attack method that aims to recover the original data based on the embedding information, which is further used to evaluate the vulnerabilities of FedE. Furthermore, we propose a Federated learning paradigm with privacy-preserving Relation embedding aggregation (FedR) to tackle the privacy issue in FedE. Besides, relation embedding sharing can significantly reduce the communication cost due to its smaller size of queries. We conduct extensive experiments to evaluate FedR with five different KG embedding models and three datasets. Compared to FedE, FedR achieves similar utility and significant improvements regarding privacy-preserving effect and communication efficiency on the link prediction task.
Strategic Maneuver and Disruption with Reinforcement Learning Approaches for Multi-Agent Coordination
Venue: arXiv cs.MA 2022
Year: 2022

Reinforcement learning (RL) approaches can illuminate emergent behaviors that facilitate coordination across teams of agents as part of a multi-agent system (MAS), which can provide windows of opportunity in various military tasks. Technologically advancing adversaries pose substantial risks to a friendly nation's interests and resources. Superior resources alone are not enough to defeat adversaries in modern complex environments because adversaries create standoff in multiple domains against predictable military doctrine-based maneuvers. Therefore, as part of a defense strategy, friendly forces must use strategic maneuvers and disruption to gain superiority in complex multi-faceted domains such as multi-domain operations (MDO). One promising avenue for implementing strategic maneuver and disruption to gain superiority over adversaries is through coordination of MAS in future military operations. In this paper, we present overviews of prominent works in the RL domain with their strengths and weaknesses for overcoming the challenges associated with performing autonomous strategic maneuver and disruption in military contexts.
Towards Responsible Natural Language Annotation for the Varieties of Arabic
Venue: arXiv cs.CL 2022
Year: 2022

When building NLP models, there is a tendency to aim for broader coverage, often overlooking cultural and (socio)linguistic nuance. In this position paper, we make the case for care and attention to such nuances, particularly in dataset annotation, as well as the inclusion of cultural and linguistic expertise in the process. We present a playbook for responsible dataset creation for polyglossic, multidialectal languages. This work is informed by a study on Arabic annotation of social media content.
DP-KB: Data Programming with Knowledge Bases Improves Transformer Fine Tuning for Answer Sentence Selection
Venue: arXiv cs.CL 2022
Year: 2022

While transformers demonstrate impressive performance on many knowledge intensive (KI) tasks, their ability to serve as implicit knowledge bases (KBs) remains limited, as shown on several slot-filling, question-answering (QA), fact verification, and entity-linking tasks. In this paper, we implement an efficient, data-programming technique that enriches training data with KB-derived context and improves transformer utilization of encoded knowledge when fine-tuning for a particular QA task, namely answer sentence selection (AS2). Our method outperforms state of the art transformer approach on WikiQA and TrecQA, two widely studied AS2 benchmarks, increasing by 2.0% p@1, 1.3% MAP, 1.1% MRR, and 4.4% p@1, 0.9% MAP, 2.4% MRR, respectively. To demonstrate our improvements in an industry setting, we additionally evaluate our approach on a proprietary dataset of Alexa QA pairs, and show increase of 2.3% F1 and 2.0% MAP. We additionally find that these improvements remain even when KB context is omitted at inference time, allowing for the use of our models within existing transformer workflows without additional latency or deployment costs.
Robotic Speech Synthesis: Perspectives on Interactions, Scenarios, and Ethics
Venue: arXiv cs.RO 2022
Year: 2022

In recent years, many works have investigated the feasibility of conversational robots for performing specific tasks, such as healthcare and interview. Along with this development comes a practical issue: how should we synthesize robotic voices to meet the needs of different situations? In this paper, we discuss this issue from three perspectives: 1) the difficulties of synthesizing non-verbal and interaction-oriented speech signals, particularly backchannels; 2) the scenario classification for robotic voice synthesis; 3) the ethical issues regarding the design of robot voice for its emotion and identity. We present the findings of relevant literature and our prior work, trying to bring the attention of human-robot interaction researchers to design better conversational robots in the future.
STICC: A multivariate spatial clustering method for repeated geographic pattern discovery with consideration of spatial contiguity
Venue: arXiv cs.LG 2022
Year: 2022

Spatial clustering has been widely used for spatial data mining and knowledge discovery. An ideal multivariate spatial clustering should consider both spatial contiguity and aspatial attributes. Existing spatial clustering approaches may face challenges for discovering repeated geographic patterns with spatial contiguity maintained. In this paper, we propose a Spatial Toeplitz Inverse Covariance-Based Clustering (STICC) method that considers both attributes and spatial relationships of geographic objects for multivariate spatial clustering. A subregion is created for each geographic object serving as the basic unit when performing clustering. A Markov random field is then constructed to characterize the attribute dependencies of subregions. Using a spatial consistency strategy, nearby objects are encouraged to belong to the same cluster. To test the performance of the proposed STICC algorithm, we apply it in two use cases. The comparison results with several baseline methods show that the STICC outperforms others significantly in terms of adjusted rand index and macro-F1 score. Join count statistics is also calculated and shows that the spatial contiguity is well preserved by STICC. Such a spatial clustering method may benefit various applications in the fields of geography, remote sensing, transportation, and urban planning, etc.
Emerging Artificial Intelligence Applications in Spatial Transcriptomics Analysis
Venue: arXiv cs.LG 2022
Year: 2022

Spatial transcriptomics (ST) has advanced significantly in the last few years. Such advancement comes with the urgent need for novel computational methods to handle the unique challenges of ST data analysis. Many artificial intelligence (AI) methods have been developed to utilize various machine learning and deep learning techniques for computational ST analysis. This review provides a comprehensive and up-to-date survey of current AI methods for ST analysis.
DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations
Venue: arXiv cs.CL 2022
Year: 2022

Automatic evaluation metrics are essential for the rapid development of open-domain dialogue systems as they facilitate hyper-parameter tuning and comparison between models. Although recently proposed trainable conversation-level metrics have shown encouraging results, the quality of the metrics is strongly dependent on the quality of training data. Prior works mainly resort to heuristic text-level manipulations (e.g. utterances shuffling) to bootstrap incoherent conversations (negative examples) from coherent dialogues (positive examples). Such approaches are insufficient to appropriately reflect the incoherence that occurs in interactions between advanced dialogue models and humans. To tackle this problem, we propose DEAM, a Dialogue coherence Evaluation metric that relies on Abstract Meaning Representation (AMR) to apply semantic-level Manipulations for incoherent (negative) data generation. AMRs naturally facilitate the injection of various types of incoherence sources, such as coreference inconsistency, irrelevancy, contradictions, and decrease engagement, at the semantic level, thus resulting in more natural incoherent samples. Our experiments show that DEAM achieves higher correlations with human judgments compared to baseline methods on several dialog datasets by significant margins. We also show that DEAM can distinguish between coherent and incoherent dialogues generated by baseline manipulations, whereas those baseline models cannot detect incoherent examples generated by DEAM. Our results demonstrate the potential of AMR-based semantic manipulations for natural negative example generation.
Carnot machine-based massive MIMO communication capacity modeling and performance analysis
Venue: arXiv cs.IT 2022
Year: 2022

Similar to the energy flowing process in traditional heat engines, information could be considered to flow in the communication systems with the form of energy and entropy. Combining the thermodynamic Carnot machine and the classical Shannon information theory, a generalized thermodynamic MIMO (multiple input multiple outputs) communication system is established to analyze the channel capacity using forward error correction codes. Based on the concepts of freedom and entropy in the communication system, the generalized channel capacity is proposed under the thermodynamic theory. Furthermore, the relationships between the proposed channel capacity and the noise freedom and coding overhead are derived and simulated. Simulation results verify the proposed channel capacity is coincident with the classical channel capacity.
Towards an AI-Driven Universal Anti-Jamming Solution with Convolutional Interference Cancellation Network
Venue: arXiv cs.NI 2022
Year: 2022

Wireless links are increasingly used to deliver critical services, while intentional interference (jamming) remains a very serious threat to such services. In this paper, we are concerned with the design and evaluation of a universal anti-jamming building block, that is agnostic to the specifics of the communication link and can therefore be combined with existing technologies. We believe that such a block should not require explicit probes, sounding, training sequences, channel estimation, or even the cooperation of the transmitter. To meet these requirements, we propose an approach that relies on advances in Machine Learning, and the promises of neural accelerators and software defined radios. We identify and address multiple challenges, resulting in a convolutional neural network architecture and models for a multi-antenna system to infer the existence of interference, the number of interfering emissions and their respective phases. This information is continuously fed into an algorithm that cancels the interfering signal. We develop a two-antenna prototype system and evaluate our jamming cancellation approach in various environment settings and modulation schemes using Software Defined Radio platforms. We demonstrate that the receiving node equipped with our approach can detect a jammer with over 99% of accuracy and achieve a Bit Error Rate (BER) as low as $10^{-6}$ even when the jammer power is nearly two orders of magnitude (18 dB) higher than the legitimate signal, and without requiring modifications to the link modulation. In non-adversarial settings, our approach can have other advantages such as detecting and mitigating collisions.
DEFORM: A Practical, Universal Deep Beamforming System
Venue: arXiv cs.NI 2022
Year: 2022

We introduce, design, and evaluate a set of universal receiver beamforming techniques. Our approach and system DEFORM, a Deep Learning (DL) based RX beamforming achieves significant gain for multi antenna RF receivers while being agnostic to the transmitted signal features (e.g., modulation or bandwidth). It is well known that combining coherent RF signals from multiple antennas results in a beamforming gain proportional to the number of receiving elements. However in practice, this approach heavily relies on explicit channel estimation techniques, which are link specific and require significant communication overhead to be transmitted to the receiver. DEFORM addresses this challenge by leveraging Convolutional Neural Network to estimate the channel characteristics in particular the relative phase to antenna elements. It is specifically designed to address the unique features of wireless signals complex samples, such as the ambiguous $2π$ phase discontinuity and the high sensitivity of the link Bit Error Rate. The channel prediction is subsequently used in the Maximum Ratio Combining algorithm to achieve an optimal combination of the received signals. While being trained on a fixed, basic RF settings, we show that DEFORM DL model is universal, achieving up to 3 dB of SNR gain for a two antenna receiver in extensive experiments demonstrating various settings of modulations, bandwidths, and channels. The universality of DEFORM is demonstrated through joint beamforming relaying of LoRa (Chirp Spread Spectrum modulation) and ZigBee signals, achieving significant improvements to Packet Loss/Delivery Rates relatively to conventional Amplify and Forward (LoRa PLR reduced by 23 times and ZigBee PDR increased by 8 times).
Blockchain for the Metaverse: A Review
Venue: arXiv cs.SI 2022
Year: 2022

Since Facebook officially changed its name to Metaverse in Oct. 2021, the metaverse has become a new norm of social networks and three-dimensional (3D) virtual worlds. The metaverse aims to bring 3D immersive and personalized experiences to users by leveraging many pertinent technologies. Despite great attention and benefits, a natural question in the metaverse is how to secure its users' digital content and data. In this regard, blockchain is a promising solution owing to its distinct features of decentralization, immutability, and transparency. To better understand the role of blockchain in the metaverse, we aim to provide an extensive survey on the applications of blockchain for the metaverse. We first present a preliminary to blockchain and the metaverse and highlight the motivations behind the use of blockchain for the metaverse. Next, we extensively discuss blockchain-based methods for the metaverse from technical perspectives, such as data acquisition, data storage, data sharing, data interoperability, and data privacy preservation. For each perspective, we first discuss the technical challenges of the metaverse and then highlight how blockchain can help. Moreover, we investigate the impact of blockchain on key-enabling technologies in the metaverse, including Internet-of-Things, digital twins, multi-sensory and immersive applications, artificial intelligence, and big data. We also present some major projects to showcase the role of blockchain in metaverse applications and services. Finally, we present some promising directions to drive further research innovations and developments towards the use of blockchain in the metaverse in the future.
GRS: Combining Generation and Revision in Unsupervised Sentence Simplification
Venue: arXiv cs.CL 2022
Year: 2022

We propose GRS: an unsupervised approach to sentence simplification that combines text generation and text revision. We start with an iterative framework in which an input sentence is revised using explicit edit operations, and add paraphrasing as a new edit operation. This allows us to combine the advantages of generative and revision-based approaches: paraphrasing captures complex edit operations, and the use of explicit edit operations in an iterative manner provides controllability and interpretability. We demonstrate these advantages of GRS compared to existing methods on the Newsela and ASSET datasets.
AutoAdversary: A Pixel Pruning Method for Sparse Adversarial Attack
Venue: arXiv cs.CV 2022
Year: 2022

Deep neural networks (DNNs) have been proven to be vulnerable to adversarial examples. A special branch of adversarial examples, namely sparse adversarial examples, can fool the target DNNs by perturbing only a few pixels. However, many existing sparse adversarial attacks use heuristic methods to select the pixels to be perturbed, and regard the pixel selection and the adversarial attack as two separate steps. From the perspective of neural network pruning, we propose a novel end-to-end sparse adversarial attack method, namely AutoAdversary, which can find the most important pixels automatically by integrating the pixel selection into the adversarial attack. Specifically, our method utilizes a trainable neural network to generate a binary mask for the pixel selection. After jointly optimizing the adversarial perturbation and the neural network, only the pixels corresponding to the value 1 in the mask are perturbed. Experiments demonstrate the superiority of our proposed method over several state-of-the-art methods. Furthermore, since AutoAdversary does not require a heuristic pixel selection process, it does not slow down excessively as other methods when the image size increases.
Speaker Embedding-aware Neural Diarization: an Efficient Framework for Overlapping Speech Diarization in Meeting Scenarios
Venue: arXiv cs.SD 2022
Year: 2022

Overlapping speech diarization has been traditionally treated as a multi-label classification problem. In this paper, we reformulate this task as a single-label prediction problem by encoding multiple binary labels into a single label with the power set, which represents the possible combinations of target speakers. This formulation has two benefits. First, the overlaps of target speakers are explicitly modeled. Second, threshold selection is no longer needed. Through this formulation, we propose the speaker embedding-aware neural diarization (SEND) framework, where a speech encoder, a speaker encoder, two similarity scorers, and a post-processing network are jointly optimized to predict the encoded labels according to the similarities between speech features and speaker embeddings. Experimental results show that SEND has a stable learning process and can be trained on highly overlapped data without extra initialization. More importantly, our method achieves the state-of-the-art performance in real meeting scenarios with fewer model parameters and lower computational complexity.
SWIPT-enabled NOMA in Distributed Antenna System with Imperfect Channel State Information for Max-Sum-Rate and Max-Min Fairness
Venue: arXiv cs.IT 2022
Year: 2022

Motivated by the fact that the data rate of non-orthogonal multiple access (NOMA) can be greatly increased with the help of the distributed antenna system (DAS), we presents a framework in which the DAS contributes not only to the data rate but also the energy harvesting of simultaneous wireless information and power transfer (SWIPT) enabled NOMA. This study considers the sum-rate maximization problem and the max-min fairness problem for SWIPT-enabled NOMA in DAS and proposes two different schemes of power splitting and power allocation for SWIPT and NOMA, respectively, with imperfect channel state information (CSI). Numerical results validate the theoretical findings and demonstrate that the proposed framework of using SWIPT-enabled NOMA in DAS achieves the higher data rates than the existing SWIPT-enabled NOMA while guaranteeing the minimum harvested energy.
Beyond a Video Frame Interpolator: A Space Decoupled Learning Approach to Continuous Image Transition
Venue: arXiv cs.CV 2022
Year: 2022

Video frame interpolation (VFI) aims to improve the temporal resolution of a video sequence. Most of the existing deep learning based VFI methods adopt off-the-shelf optical flow algorithms to estimate the bidirectional flows and interpolate the missing frames accordingly. Though having achieved a great success, these methods require much human experience to tune the bidirectional flows and often generate unpleasant results when the estimated flows are not accurate. In this work, we rethink the VFI problem and formulate it as a continuous image transition (CIT) task, whose key issue is to transition an image from one space to another space continuously. More specifically, we learn to implicitly decouple the images into a translatable flow space and a non-translatable feature space. The former depicts the translatable states between the given images, while the later aims to reconstruct the intermediate features that cannot be directly translated. In this way, we can easily perform image interpolation in the flow space and intermediate image synthesis in the feature space, obtaining a CIT model. The proposed space decoupled learning (SDL) approach is simple to implement, while it provides an effective framework to a variety of CIT problems beyond VFI, such as style transfer and image morphing. Our extensive experiments on a variety of CIT tasks demonstrate the superiority of SDL to existing methods. The source code and models can be found at \url{https://github.com/yangxy/SDL}.
Transferable Class-Modelling for Decentralized Source Attribution of GAN-Generated Images
Venue: arXiv cs.CV 2022
Year: 2022

GAN-generated deepfakes as a genre of digital images are gaining ground as both catalysts of artistic expression and malicious forms of deception, therefore demanding systems to enforce and accredit their ethical use. Existing techniques for the source attribution of synthetic images identify subtle intrinsic fingerprints using multiclass classification neural nets limited in functionality and scalability. Hence, we redefine the deepfake detection and source attribution problems as a series of related binary classification tasks. We leverage transfer learning to rapidly adapt forgery detection networks for multiple independent attribution problems, by proposing a semi-decentralized modular design to solve them simultaneously and efficiently. Class activation mapping is also demonstrated as an effective means of feature localization for model interpretation. Our models are determined via experimentation to be competitive with current benchmarks, and capable of decent performance on human portraits in ideal conditions. Decentralized fingerprint-based attribution is found to retain validity in the presence of novel sources, but is more susceptible to type II errors that intensify with image perturbations and attributive uncertainty. We describe both our conceptual framework and model prototypes for further enhancement when investigating the technical limits of reactive deepfake attribution.