Reviewerly/exHarmony huggingface dataset:
Cross-Modal Semantic Communications

It is well known that multi-modal services, including video, audio, and haptic signals, aim to provide immersive experience with low latency and high reliability. Although multi-modal signals have differences in structure, transmission delay, and jitter, there are potential cross-modal relations among these multi-modal signals that can be explored to realize multi-modal signal processing and streaming. As a revolutionary communication paradigm, semantic communications can not only provide human-oriented services by delivering the intended meanings, but also break the modality barriers through compact semantic information extracted from multi-modal signals, which can help to achieve cross-modal applications. However, due to the polysemy and ambiguity issues, how to guarantee high reliability for semantic communications is a tremendous challenge. To overcome this dilemma, this article designs a cross-modal semantic communication paradigm that involves three modules: The cross-modal knowledge graph (CKG) provides essential background knowledge and signal patches for encoding and decoding respectively; the cross-modal semantic encoder infers potential implicit semantics to reduce encoding polysemy; and the cross-modal semantic decoder ensures consistency between source signals and recovered signals in terms of bit level and semantic level in order to reduce decoding ambiguity. They collaboratively improve the reliability of the communication system, which is verified by the simulation result.
RTUNet: Residual transformer UNet specifically for pancreas segmentation

Accurate pancreas segmentation is crucial for the diagnostic assessment of pancreatic cancer. However, large position changes, high variability in shape and size, and the extremely blurred boundary make the task of pancreas segmentation challenging. To alleviate these challenges, we propose the residual transformer UNet (RTUNet) to fit the nature of the pancreas. Specifically, a residual transformer block is implemented to extract multi-scale features from a global perspective which captures high variabilities in the pancreas position. In addition, a dual convolutional down-sampling strategy is leveraged to obtain precise shape and size features of the pancreas in a large receptive field which prevents the loss of information. We finally propose a dice hausdorff distance loss that makes the network focus on the pancreas boundary. Through extensive experiments on the public NIH dataset, we achieved a dice similarity coefficient (DSC) of 86.25%, which outperforms the state-of-the-art DSC of 85.49%. In addition, our method surpasses the baselines by more than 3.0% on DSC and improves the min DSC by 2.93%. Furthermore, ablation studies are also performed to prove the effectiveness of each proposed module.
Memory consistent unsupervised off-the-shelf model adaptation for source-relaxed medical image segmentation

Unsupervised domain adaptation (UDA) has been a vital protocol for migrating information learned from a labeled source domain to facilitate the implementation in an unlabeled heterogeneous target domain. Although UDA is typically jointly trained on data from both domains, accessing the labeled source domain data is often restricted, due to concerns over patient data privacy or intellectual property. To sidestep this, we propose “off-the-shelf (OS)” UDA (OSUDA), aimed at image segmentation, by adapting an OS segmentor trained in a source domain to a target domain, in the absence of source domain data in adaptation. Toward this goal, we aim to develop a novel batch-wise normalization (BN) statistics adaptation framework. In particular, we gradually adapt the domain-specific low-order BN statistics, e.g., mean and variance, through an exponential momentum decay strategy, while explicitly enforcing the consistency of the domain shareable high-order BN statistics, e.g., scaling and shifting factors, via our optimization objective. We also adaptively quantify the channel-wise transferability to gauge the importance of each channel, via both low-order statistics divergence and a scaling factor. Furthermore, we incorporate unsupervised self-entropy minimization into our framework to boost performance alongside a novel queued, memory-consistent self-training strategy to utilize the reliable pseudo label for stable and efficient unsupervised adaptation. We evaluated our OSUDA-based framework on both cross-modality and cross-subtype brain tumor segmentation and cardiac MR to CT segmentation tasks. Our experimental results showed that our memory consistent OSUDA performs better than existing source-relaxed UDA methods and yields similar performance to UDA methods with source data.
SMURF: A Fully Autonomous Water Surface Cleaning Robot with A Novel Coverage Path Planning Method

In recent years, more attention has been paid to water surface environment protection. Current water surface waste cleaning mainly relies on manual operations, which are low-efficiency and dangerous. Therefore, in this paper, we design a fully autonomous water surface cleaning robot, SMURF, which achieves high-efficiency water surface cleaning without human operation and adapts to be used in various types of real-world water bodies. In addition, we propose a novel coverage path planning method on water surfaces and an improved nonlinear model predictive controller. The real-world experiment shows that SMURF works well in different kinds of water bodies and achieves much higher efficiency than traditional water surface cleaning methods.
Comparative Study of Real-Time Semantic Segmentation Networks in Aerial Images During Flooding Events

Real-time semantic segmentation of aerial imagery is essential for unmanned ariel vehicle applications, including military surveillance, land characterization, and disaster damage assessments. Recent real-time semantic segmentation neural networks promise low computation and inference time, appropriate for resource-limited platforms, such as edge devices. However, these methods are mainly trained on human-centric view datasets, such as Cityscapes and CamVid, unsuitable for aerial applications. Furthermore, we do not know the feasibility of these models under adversarial settings, such as flooding events. To solve these problems, we train the most recent real-time semantic segmentation architectures on the FloodNet dataset containing annotated aerial images captured after hurricane Harvey. This article comprehensively studies several lightweight architectures, including encoder–decoder and two-pathway architectures, evaluating their performance on aerial imagery datasets. Moreover, we benchmark the efficiency and accuracy of different models on the FloodNet dataset to examine the practicability of these models during emergency response for aerial image segmentation. Some lightweight models attain more than 60% test mIoU on the FloodNet dataset and yield qualitative results on images. This article highlights the strengths and weaknesses of current segmentation models for aerial imagery, requiring low computation and inference time. Our experiment has direct applications during catastrophic events, such as flooding events.
Characterization and Optimization of Coherent MZI-Based Nanophotonic Neural Networks Under Fabrication Non-Uniformity

Silicon-photonic neural networks (SPNNs) have emerged as promising successors to electronic artificial intelligence (AI) accelerators by offering orders of magnitude lower latency and higher energy efficiency. Nevertheless, the underlying silicon photonic devices in SPNNs are sensitive to inevitable fabrication-process variations (FPVs) stemming from optical lithography imperfections. Consequently, the inferencing accuracy in an SPNN can be highly impacted by FPVs -- e.g., can drop to below 10% -- the impact of which is yet to be fully studied. In this paper, we, for the first time, model and explore the impact of FPVs in the waveguide width and silicon-on-insulator (SOI) thickness in coherent SPNNs that use Mach-Zehnder Interferometers (MZIs). Leveraging such models, we propose a novel variation-aware, design-time optimization solution to improve MZI tolerance to different FPVs in SPNNs. Simulation results for two example SPNNs of different scales under realistic and correlated FPVs indicate that the optimized MZIs can improve the inferencing accuracy by up to 93.95% for the MNIST handwritten digit dataset -- considered as an example in this paper -- which corresponds to a <0.5% accuracy loss compared to the variation-free case. The proposed one-time optimization method imposes low area overhead, and hence is applicable even to resource-constrained designs
Decoupled Multi-task Learning with Cyclical Self-Regulation for Face Parsing

This paper probes intrinsic factors behind typical failure cases (e.g. spatial inconsistency and boundary confusion) produced by the existing state-of-the-art method in face parsing. To tackle these problems, we propose a novel Decoupled Multi-task Learning with Cyclical Self-Regulation (DML-CSR) for face parsing. Specifically, DML-CSR designs a multi-task model which comprises face parsing, binary edge, and category edge detection. These tasks only share low-level encoder weights without high-level interactions between each other, enabling to decouple auxiliary modules from the whole network at the inference stage. To address spatial inconsistency, we develop a dynamic dual graph convolutional network to capture global contextual information without using any extra pooling operation. To handle boundary confusion in both single and multiple face scenarios, we exploit binary and category edge detection to jointly obtain generic geometric structure and fine-grained semantic clues of human faces. Besides, to prevent noisy labels from degrading model generalization during training, cyclical self-regulation is proposed to self-ensemble several model instances to get a new model and the resulting model then is used to self-distill subsequent models, through alternating iterations. Experiments show that our method achieves the new state-of-the-art performance on the Helen, CelebAMask-HQ, and Lapa datasets. The source code is available at https://github.com/deepinsight/insightface/tree/master/parsing/dml_csr.
Hypergraph-Induced Semantic Tuplet Loss for Deep Metric Learning

In this paper, we propose Hypergraph-Induced Semantic Tuplet (HIST) loss for deep metric learning that leverages the multilateral semantic relations of multiple samples to multiple classes via hypergraph modeling. We formulate deep metric learning as a hypergraph node classification problem in which each sample in a mini-batch is regarded as a node and each hyperedge models class-specific semantic relations represented by a semantic tuplet. Unlike previous graph-based losses that only use a bundle of pairwise relations, our HIST loss takes advantage of the multilateral semantic relations provided by the semantic tuplets through hypergraph modeling. Notably, by leveraging the rich multilateral semantic relations, HIST loss guides the embedding model to learn class-discriminative visual semantics, contributing to better generalization performance and model robustness against input corruptions. Extensive experiments and ablations provide a strong motivation for the proposed method and show that our HIST loss leads to improved feature learning, achieving state-of-the-art results on three widely used benchmarks. Code is available at https://github.com/ljin0429/HIST.
IOTA Tangle 2.0: Toward a Scalable, Decentralized, Smart, and Autonomous IoT Ecosystem

IOTA Tangle is a distributed ledger technology (DLT), primarily designed for Internet-of-Things (IoT) networks and applications. IOTA Tangle utilizes a direct acyclic graph (DAG) structure for the ledger, with its protocol offering features attractive to the IoT domain, over most blockchain alternatives, such as feeless transactions, higher achievable transactions per second (TPS), and lower energy consumption. The original IOTA implementation relied on a bootstrap centralized coordinator solution for consensus which limited its degree of decentralization and scalability. This concern, alongside other limitations to its adoption, such as lack of smart contracts, are being addressed with the release of IOTA 2.0. This update brings with it significant changes in order to remove the coordinator and achieve a scalable decentralized solution. To this end, this paper provides a technical overview of the key features of IOTA 2.0 while discussing their relevance and benefits for the wider IoT ecosystem. The paper also provides performance insights and future research directions for IOTA 2.0.
An attention-based deep learning approach for inertial motion recognition and estimation in human-robot collaboration

In line with a human-centric smart manufacturing vision, human-robot collaboration is striving to combine robots’ high efficiency and quality with humans’ rapid adaptability and high flexibility. In particular, perception, recognition and estimation of human motion determine when and what robot to collaborate with humans. This work presents an attention-based deep learning approach for inertial motion recognition and estimation in order to infer when robotic assistance will be requested by the human and to allow the robot to perform partial human tasks. First, in the stage of motion perception and recognition, quaternion-based calibration and forward kinematic analysis methods enable the reconstruction of human motion based on data streaming from an inertial motion capture device. Then, in the stage of motion estimation, residual module and Bidirectional Long Short-Term Memory module are integrated with proposed attention mechanism for estimating arm motion trajectories further. Experimental results show the effectiveness of the proposed approach in achieving better recognition and estimation in comparison with traditional approaches and existing deep learning approaches. It is experimentally verified in a laboratory environment involving a collaborative robot employed in a small part assembly task.
Multi-view representation model based on graph autoencoder

Graph representation learning is a hot topic in non-Euclidean data in various domains, such as social networks, biological networks, etc. When some data labels are missing, graph autoencoder and graph variational autoencoder can perform outstanding abilities on node clustering or link prediction tasks. However, most existing graph representation learning ignores data's multi-modal features and takes the node features and graph structure features as one view. Besides, most graph autoencoders only reconstruct the node feature matrix or adjacency matrix, which does not fully use the hidden representation information. In this paper, we propose a multi-view representation model based on graph autoencoder, which can employ the global structure topology, latent local topology, and feature relative information. Meanwhile, we add another decoder to reconstruct the node feature matrix as an auxiliary task. In this way, the proposed framework can utilize the learned representation sufficiently. We validate the effectiveness of our framework on four datasets, and the experimental results demonstrate superior performance compared with other advanced frameworks.
Chatbots in teaching a foreign language: advantages and controversial issues

The current stage of technological development of the world community is characterized by the dynamic introduction of artificial intelligence technologies into various spheres of human life. Informatization of education contributed to the integration of new innovative technologies in the teaching of certain subjects, including a foreign language. Chatbots are one of the modern programs that operate on the basis of natural language processing and machine learning technologies, which can be used in the development of students’ foreign language speech skills. By chatbots from the standpoint of the methodology of teaching foreign languages, we propose to understand a dialog training program capable of developing foreign-language oral and written speech skills of a student based on the algorithms of human speech behavior embedded in it by maintaining a dialogue with him and imitating human speech. In this work, based on the analysis of methodological research, we highlight the advantages of chatbots in teaching a foreign language and the controversial issues of using this program in the educational process. The advantages include the following: 1) increasing students' motivation to learn a foreign language based on innovative artificial intelligence technology; 2) the opportunity for students to improve foreign language speech skills; 3) the availability of chatbots for the development of students' speech skills, regardless of their location and time; 4) reducing the level of anxiety of students when interacting with the machine. The controversial issues include: 1) students are not always ready to replace the teacher with a chatbot; 2) most of the conversational chatbots are aimed at students with a level of foreign language proficiency A2-B1; 3) stereotyped and limited set of phrases used by the chatbot.
Video description: A comprehensive survey of deep learning approaches

Abstract Video description refers to understanding visual content and transforming that acquired understanding into automatic textual narration. It bridges the key AI fields of computer vision and natural language processing in conjunction with real-time and practical applications. Deep learning-based approaches employed for video description have demonstrated enhanced results compared to conventional approaches. The current literature lacks a thorough interpretation of the recently developed and employed sequence to sequence techniques for video description. This paper fills that gap by focusing mainly on deep learning-enabled approaches to automatic caption generation. Sequence to sequence models follow an Encoder–Decoder architecture employing a specific composition of CNN, RNN, or the variants LSTM or GRU as an encoder and decoder block. This standard-architecture can be fused with an attention mechanism to focus on a specific distinctiveness, achieving high quality results. Reinforcement learning employed within the Encoder–Decoder structure can progressively deliver state-of-the-art captions by following exploration and exploitation strategies. The transformer mechanism is a modern and efficient transductive architecture for robust output. Free from recurrence, and solely based on self-attention, it allows parallelization along with training on a massive amount of data. It can fully utilize the available GPUs for most NLP tasks. Recently, with the emergence of several versions of transformers, long term dependency handling is not an issue anymore for researchers engaged in video processing for summarization and description, or for autonomous-vehicle, surveillance, and instructional purposes. They can get auspicious directions from this research.
A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

Non-autoregressive (NAR) generation, which is first proposed in neural machine translation (NMT) to speed up inference, has attracted much attention in both machine learning and natural language processing communities. While NAR generation can significantly accelerate inference speed for machine translation, the speedup comes at the cost of sacrificed translation accuracy compared to its counterpart, autoregressive (AR) generation. In recent years, many new models and algorithms have been designed/proposed to bridge the accuracy gap between NAR generation and AR generation. In this paper, we conduct a systematic survey with comparisons and discussions of various non-autoregressive translation (NAT) models from different aspects. Specifically, we categorize the efforts of NAT into several groups, including data manipulation, modeling methods, training criterion, decoding algorithms, and the benefit from pre-trained models. Furthermore, we briefly review other applications of NAR models beyond machine translation, such as grammatical error correction, text summarization, text style transfer, dialogue, semantic parsing, automatic speech recognition, and so on. In addition, we also discuss potential directions for future exploration, including releasing the dependency of KD, reasonable training objectives, pre-training for NAR, and wider applications, etc. We hope this survey can help researchers capture the latest progress in NAR generation, inspire the design of advanced NAR models and algorithms, and enable industry practitioners to choose appropriate solutions for their applications. The web page of this survey is at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/LitterBrother-Xiao/Overview-of-Non-autoregressive-Applications</uri> .
A comprehensive review on deep learning algorithms: Security and privacy issues

Machine Learning (ML) algorithms are used to train the machines to perform various complicated tasks that begin to modify and improve with experiences. It has become widely used for automated decisions. In particular, the applications which have a profound impact on society that rely on Deep Learning (DL) for autonomous decisions, such as Patient Health Record (PHR), Unmanned Aerial Vehicles (UAVs), etc. Such impacts have a vital concern about the potential vulnerabilities introduced by DL. Traditional attackers have powerful motives that can alter and modify DL algorithms to subvert the outcomes. In poisoning attacks, an attacker can consciously change training dataset, which is used to operate the outcomes of decision-based model. While in privacy and evasion attacks, an adversary can also misclassify new datasets to infer private information. Therefore, in this paper, we have provided a review of security and privacy issues of DL algorithms and analyzed their applications and challenges based on state-of-the-art literature. We have classified attacks, devised a taxonomy, and comprehensive analysis of defense techniques for the most common attacks such as poisoning, evasion, model extraction, and model inversion. We have also presented various privacy preserving techniques to ensure the privacy of dataset. We have proposed a secure cryptographic framework for dataset based on hash functions and Homomorphic Encryption (HE) scheme. Finally, we have provided recent research challenges and future studies concerning security and privacy issues. We believed that the highlighted limitations and weaknesses provide possible research questions and open matters for designing efficient future DL algorithms.