Reviewerly/exHarmony huggingface dataset:
All-optical silicon microring spiking neuron

With the rapid development of artificial intelligence and machine learning, brain-inspired neuromorphic photonics has emerged as an extremely attractive computing paradigm, promising orders-of-magnitude higher computing speed and energy efficiency compared to its electronic counterparts. Tremendous efforts have been devoted to photonic hardware implementations of mimicking the nonlinear neuron-like spiking response and the linear synapse-like weighting functionality. Here, we systematically characterize the spiking dynamics of a passive silicon microring neuron. The research of self-pulsation and excitability reveals that the silicon microring can function as an all-optical class II resonate-and-fire neuron. The typical refractory period has been successfully suppressed by configuring the pump power above the perturbation power, hence allowing the microring neuron to operate with a speed up to roughly sub-gigahertz. Additionally, temporal integration and controllable inhibition regimes are experimentally demonstrated for the first time, to the best of our knowledge. Our experimental verification is obtained with a commercial CMOS platform, hence offering great potential for large-scale neuromorphic photonics integration.
Memory-based Deep Reinforcement Learning for POMDPs

A promising characteristic of Deep Reinforcement Learning (DRL) is its capability to learn optimal policy in an end-to-end manner without relying on feature engineering. However, most approaches assume a fully observable state space, i.e. fully observable Markov Decision Processes (MDPs). In real-world robotics, this assumption is unpractical, because of issues such as sensor sensitivity limitations and sensor noise, and the lack of knowledge about whether the observation design is complete or not. These scenarios lead to Partially Observable MDPs (POMDPs). In this paper, we propose Long-Short-Term-Memory-based Twin Delayed Deep Deterministic Policy Gradient (LSTM-TD3) by introducing a memory component to TD3, and compare its performance with other DRL algorithms in both MDPs and POMDPs. Our results demonstrate the significant advantages of the memory component in addressing POMDPs, including the ability to handle missing and noisy observation data.
Tackling cyber-aggression: Identification and fine-grained categorization of aggressive texts on social media using weighted ensemble of transformers

The pervasiveness of aggressive content in social media has become a serious concern for government organizations and tech companies because of its pernicious societal effects. In recent years, social media has been repeatedly used as a tool to incite communal aggression, spread distorted propaganda, damage social harmony and demean the identity of individuals or a community in the public spaces. Therefore, restraining the proliferation of aggressive content and detecting them has become an urgent duty. Studies of the identification of aggressive content have mostly been done for English and other high-resource languages. Automatic systems developed for those languages can not accurately identify detrimental contents written in regional languages like Bengali. To compensate this insufficiency, this work presents a novel Bengali aggressive text dataset (called ‘BAD’) with two-level annotation. In level-A, 14158 texts are labeled as either aggressive or non-aggressive. While in level-B, 6807 aggressive texts are categorized into religious, political, verbal and gendered aggression classes each having 2217, 2085, 2043 and 462 texts respectively. This paper proposes a weighted ensemble technique including m-BERT, distil-BERT, Bangla-BERT and XLM-R as the base classifiers to identify and classify the aggressive texts in Bengali. The proposed model can readdress the softmax probabilities of the participating classifiers depending on their primary outcomes. This weighting technique has enabled the model to outdo the simple average ensemble and all other machine learning (ML), deep learning (DL) baselines. It has acquired the highest weighted f 1 -score of 93.43% in the identification task and 93.11% in the categorization task. Dataset developed as the part of this work is available at https://github.com/BAD-Bangla-Aggressive-Text-Dataset
An efficient multi-objective ant colony optimization for task allocation of heterogeneous unmanned aerial vehicles

Unmanned aerial vehicles (UAVs) have become powerful tools in modern military combat. How to properly allocate the tasks of heterogeneous UAVs in a combat is a fundamental and challenging problem. In this paper, we formulate the cooperative task allocation of heterogeneous UAVs as a constrained multi-objective optimization problem. To efficiently resolve the formulated problem, we further propose a multi-objective ant colony optimization (MOACO) algorithm with a new pheromone updating mechanism and four newly defined heuristic information. Simulation results on test cases with different scales and characteristics have shown that the proposed methods can perform better than several recently published algorithms, in terms of convergence speed, solution quality and solution diversity.
Integrated path planning and trajectory tracking control for quadrotor UAVs with obstacle avoidance in the presence of environmental and systematic uncertainties: Theory and experiment

This paper proposes an innovative integrated path planning and trajectory tracking control framework for a quadrotor unmanned aerial vehicle (UAV) in the presence of environmental and systematic uncertainties to achieve integrated guidance and control. Firstly, in order to perform real-time path planning, a computationally cost-effective planning algorithm is designed to find an optimal and smooth path while avoiding both static and dynamic obstacles. Then, by employing the pure-pursuit path following approach, the generated geometric path is converted to a trajectory profile related to time, which serves as the reference commands for the low-level trajectory tracking controller. Finally, a novel adaptive sliding mode trajectory tracking controller is proposed to compensate model uncertainties and maintain the desired tracking performance for the studied quadrotor UAV. With the proposed adaptive schemes, overestimation of uncertain parameters can be avoided, which further contributes to avoiding control chattering of the system. The performance of the proposed framework is validated through comparative simulation and experimental tests based on a quadrotor UAV subject to model uncertainties and environmental obstacles, which confirms the effectiveness and superiority of the proposed approach for practical applications.
Noise-trained deep neural networks effectively predict human vision and its neural responses to challenging images

Deep neural networks (DNNs) for object classification have been argued to provide the most promising model of the visual system, accompanied by claims that they have attained or even surpassed human-level performance. Here, we evaluated whether DNNs provide a viable model of human vision when tested with challenging noisy images of objects, sometimes presented at the very limits of visibility. We show that popular state-of-the-art DNNs perform in a qualitatively different manner than humans-they are unusually susceptible to spatially uncorrelated white noise and less impaired by spatially correlated noise. We implemented a noise training procedure to determine whether noise-trained DNNs exhibit more robust responses that better match human behavioral and neural performance. We found that noise-trained DNNs provide a better qualitative match to human performance; moreover, they reliably predict human recognition thresholds on an image-by-image basis. Functional neuroimaging revealed that noise-trained DNNs provide a better correspondence to the pattern-specific neural representations found in both early visual areas and high-level object areas. A layer-specific analysis of the DNNs indicated that noise training led to broad-ranging modifications throughout the network, with greater benefits of noise robustness accruing in progressively higher layers. Our findings demonstrate that noise-trained DNNs provide a viable model to account for human behavioral and neural responses to objects in challenging noisy viewing conditions. Further, they suggest that robustness to noise may be acquired through a process of visual learning.
Unsupervised deep hashing through learning soft pseudo label for remote sensing image retrieval

Unsupervised hashing is an important approach for large-scale content-based remote sensing (RS) image retrieval. Existing unsupervised hashing methods usually utilize data clustering to generate pseudo labels as supervised signals. In the RS domain, images of each class cannot be accurately grouped into individual clusters because image features are not well characterized by the pre-trained models learned on a nature image dataset. As a result, to preserve the similarity of images sharing at least one class, intra-cluster and inter-cluster similarities need to be carefully learned. However, existing pseudo-labels are hard pseudo-labels represented by scalar values, which cannot well reflect the semantic distance between inter-cluster images or the semantic distance between intra-cluster images. To address these problems, this paper proposes a soft-pseudo-label-based unsupervised deep hashing method for content-based RS image retrieval, called SPL-UDH. Soft pseudo-labels can accurately describe the global similarity between inter-cluster images by binarized vectors. Specifically, we design a deep auto-encoder network to learn soft pseudo-labels automatically and meanwhile to generate a local similarity matrix representing the proximity between intra-cluster images. Based on soft pseudo-labels and local similarity matrix, we propose a deep hashing network to simultaneously learn the inter-cluster similarity and the intra-cluster similarity between RS images. Moreover, we design a new objective function based on Bayesian theory so that the deep hashing network can be trained by jointly learning the soft pseudo-labels and the local similarity matrix. Extensive experiments on public RS image retrieval datasets demonstrate that SPL-UDH outperforms various state-of-the-art unsupervised hashing methods.
SpeeChin

This paper presents SpeeChin, a smart necklace that can recognize 54 English and 44 Chinese silent speech commands. A customized infrared (IR) imaging system is mounted on a necklace to capture images of the neck and face from under the chin. These images are first pre-processed and then deep learned by an end-to-end deep convolutional-recurrent-neural-network (CRNN) model to infer different silent speech commands. A user study with 20 participants (10 participants for each language) showed that SpeeChin could recognize 54 English and 44 Chinese silent speech commands with average cross-session accuracies of 90.5% and 91.6%, respectively. To further investigate the potential of SpeeChin in recognizing other silent speech commands, we conducted another study with 10 participants distinguishing between 72 one-syllable nonwords. Based on the results from the user studies, we further discuss the challenges and opportunities of deploying SpeeChin in real-world applications.
Deep hierarchical LSTM networks with attention for video summarization

This paper studies the video summarization task by formulating it as a sequential decision-making process, in which the input is a sequence of video frames and the output is a subset of the original frames. Long Short-Term Memory (LSTM) is a commonly used framework of prior video summarization methods due to great temporal dependencies modeling ability. However, the frame sequence in the video summarization task is relatively long, and LSTM can only handle short video clips of up to 80 frames in length. This paper proposes a novel deep summarization framework named Deep Hierarchical LSTM Networks with Attention for Video Summarization (DHAVS) that consists of delicate feature extraction, temporal dependencies modeling, and video summary generation. Specifically, we employ 3D CNN instead of 2D CNN to extract spatial–temporal features and design an attention-based hierarchical LSTM module to capture the temporal dependencies among video frames. Additionally, we treat video summarization as an imbalanced class distribution problem and design a cost-sensitive loss function. Experimental results show that the proposed method has 0.7% ∼ 21.3% and 4.5% ∼ 12.2% improved to the conventional methods on SumMe and TVSum datasets.
A Basic Framework for Privacy Protection in Personalized Information Retrieval

Personalized information retrieval is an effective tool to solve the problem of information overload. Along with the rapid development of emerging network technologies such as cloud computing, however, network servers are becoming more and more untrusted, resulting in a serious threat to user privacy of personalized information retrieval. In this paper, we propose a basic framework for the comprehensive protection of all kinds of user privacy in personalized information retrieval. Its basic idea is to construct and submit a group of well-designed dummy requests together with each user request to the server, to mix up the user requests and then cover up the user privacy behind the requests. Also, the framework includes a privacy model and its implementation algorithm. Finally, theoretical analysis and experimental evaluation demonstrate that the framework can comprehensively improve the security of all kinds of user privacy, without compromising the availability of personalized information retrieval.
Multi-attribute adaptive aggregation transformer for vehicle re-identification

With the continuous development of intelligent transportation systems, vehicle-related fields have emerged a research boom in detection, tracking, and retrieval. Vehicle re-identification aims to judge whether a specific vehicle appears in a video stream, which is a popular research direction. Previous researches have proven that the transformer is an efficient method in computer vision, which treats a visual image as a series of patch sequences. However, an efficient vehicle re-identification should consider the image feature and the attribute feature simultaneously. In this work, we propose a vehicle attribute transformer (VAT) for vehicle re-identification. First, we consider color and model as the most intuitive attributes of the vehicle, the vehicle color and model are relatively stable and easy to distinguish. Therefore, the color feature and the model feature are embedded in a transformer. Second, we consider that the shooting angle of each image may be different, so we encode the viewpoint of the vehicle image as another additional attribute. Besides, different attributes are supposed to have different importance. Based on this, we design a multi-attribute adaptive aggregation network, which can compare different attributes and assign different weights to the corresponding features. Finally, to optimize the proposed transformer network, we design a multi-sample dispersion triplet (MDT) loss. Not only the hardest samples based on hard mining strategy, but also some extra positive samples and negative samples are considered in this loss. The dispersion of multi-sample is utilized to dynamically adjust the loss, which can guide the network to learn more optimized division for feature space. Extensive experiments on popular vehicle re-identification datasets verify that the proposed method can achieve state-of-the-art performance.
Blockchain Interoperability: Towards a Sustainable Payment System

The highly fragmented blockchain and cryptocurrency ecosystem necessitates interoperability mechanisms as a requirement for blockchain-technology acceptance. The immediate implication of interchain interoperability is automatic swapping between cryptocurrencies. We performed a systematic review of the existing literature on Blockchain interoperability and atomic cross-chain transactions. We investigated different blockchain interoperability approaches, including industrial solutions, categorized them and identified the key mechanisms used, and list several example projects for each category. We focused on the atomic transactions between blockchain, a process also known as atomic swap. Furthermore, we studied recent implementations along with architectural approaches for atomic swap and deduced research issues and challenges in cross-chain interoperability and atomic swap. Atomic swap can instantly transfer tokens and significantly reduce the associated costs without using any centralized authority, and thus facilitates the development of a sustainable payment system for wider financial inclusion.
Accurate 3D reconstruction via fringe-to-phase network

Learning 3D shape representations from structured-light images for 3D reconstructions has become popular in many fields. This paper presents a new approach integrating a fringe-to-phase network with a fringe projection profilometry (FPP) technique to achieve 3D reconstructions with superior accuracy and speed performance. The proposed fringe-to-phase network has a UNet-like architecture, capable of retrieving three wrapped phase maps directly from a color image comprising three fringe patterns with designated frequencies. Because the phase maps contain the 3D shape representations of the measurement target, they serve as an intermediary to transform the single-shot fringe-pattern image into the 3D shapes of the target. The datasets with ground-truth phase labels are generated by using a tri-frequency FPP method. Unlike the existing techniques, the proposed approach yields both high-accuracy and fast-speed 3D reconstructions. Experiments have been accomplished to validate the proposed technique, which provides a promising tool for numerous scientific research and industrial applications.
Visual Sentiment Analysis Using Deep Learning Models with Social Media Data

Analyzing the sentiments of people from social media content through text, speech, and images is becoming vital in a variety of applications. Many existing research studies on sentiment analysis rely on textual data, and similar to the sharing of text, users of social media share more photographs and videos. Compared to text, images are said to exhibit the sentiments in a much better way. So, there is an urge to build a sentiment analysis model based on images from social media. In our work, we employed different transfer learning models, including the VGG-19, ResNet50V2, and DenseNet-121 models, to perform sentiment analysis based on images. They were fine-tuned by freezing and unfreezing some of the layers, and their performance was boosted by applying regularization techniques. We used the Twitter-based images available in the Crowdflower dataset, which contains URLs of images with their sentiment polarities. Our work also presents a comparative analysis of these pre-trained models in the prediction of image sentiments on our dataset. The accuracies of our fine-tuned transfer learning models involving VGG-19, ResNet50V2, and DenseNet-121 are 0.73, 0.75, and 0.89, respectively. When compared to previous attempts at visual sentiment analysis, which used a variety of machine and deep learning techniques, our model had an improved accuracy by about 5% to 10%. According to the findings, the fine-tuned DenseNet-121 model outperformed the VGG-19 and ResNet50V2 models in image sentiment prediction.
Artificial Intelligence (AI) for Screening Mammography, From the <i>AJR</i> Special Series on AI Applications

Artificial intelligence (AI) applications for screening mammography are being marketed for clinical use in the interpretative domains of lesion detection and diagnosis, triage, and breast density assessment and in the noninterpretive domains of breast cancer risk assessment, image quality control, image acquisition, and dose reduction. Evidence in support of these nascent applications, particularly for lesion detection and diagnosis, is largely based on multireader studies with cancer-enriched datasets rather than rigorous clinical evaluation aligned with the application's specific intended clinical use. This article reviews commercial AI algorithms for screening mammography that are currently available for clinical practice, their use, and evidence supporting their performance. Clinical implementation considerations, such as workflow integration, governance, and ethical issues, are also described. In addition, the future of AI for screening mammography is discussed, including the development of interpretive and noninterpretive AI applications and strategic priorities for research and development.
BDTwin: An Integrated Framework for Enhancing Security and Privacy in Cybertwin-Driven Automotive Industrial Internet of Things

The rapid development of the automotive Industrial Internet of Things requires secure networking infrastructure toward digitalization. Cybertwin (CT) is a next-generation networking architecture that serves as a communication, and digital asset owner, and can make the Vehicle-to-Everything (V2X) network flexible and secure. However, CT itself can publish end users’ digital assets to other entities as a service, making data security and privacy major obstacles in the realization of V2X applications. Motivated from the aforementioned discussion, this article presents BDTwin, a blockchain and deep-learning-based integrated framework to enhance security and privacy in CT-driven V2X applications. Specifically, a blockchain scheme is designed to ensure secure communication among vehicles, roadside units, CT-edge server, and cloud server using a smart contract-based enhance-Proof-of-Work (ePoW) and Zero Knowledge Proof (ZKP)-based verification process. Smart contracts are used to enforce rules and regulations that govern the behavior of V2X entities in a nondeniable and automated manner. In a deep-learning scheme, an autoregressive-deep variational autoencoder model is combined with attention-based bidirectional long short-term memory (A-BLSTM) for automatic feature extraction and attack detection by analyzing CT-edge servers data in a V2X environment. Security analysis and experimental results using two different sources, ToN-IoT and CICIDS-2017 show the superiority of the proposed BDTwin framework over some baseline and recent state-of-the-art techniques.
Cross-Modal Dynamic Networks for Video Moment Retrieval With Text Query

Video moment retrieval with text query aims to retrieve the most relevant segment from the whole video based on the given text query. It is a challenging cross-modal alignment task due to the huge gap between visual and linguistic modalities and the noise generated by manual labeling of time segments. Most of the existing works only use language information in the cross-modal fusion stage, neglecting that language information plays an important role in the retrieval stage. Besides, these works roughly compress the visual information in the video clips to reduce the computation cost which loses subtle video information in the long video. In this paper, we propose a novel model termed Cross-modal Dynamic Networks (CDN) which dynamically generates convolution kernel by visual and language features. In the feature extraction stage, we also propose a frame selection module to capture the subtle video information in the video segment. By this approach, the CDN can reduce the impact of the visual noise without significantly increasing the computation cost and leads to a better video moment retrieval result. The experiments on two challenge datasets, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , Charades-STA and TACoS, show that our proposed CDN method outperforms a bundle of state-of-the-art methods with more accurately retrieved moment video clips. The implementation code and extensive instruction of our proposed CDN method are provided at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/CFM-MSG/Code_CDN</uri> .