Reviewerly/exHarmony huggingface dataset:
Attention Feature Pyramid Network for Scene Text Detection

Scene text detection is an important step in the scene text reading system, which has witnessed rapid development with convolutional neural networks. Feature Pyramid Network (FPN)[1] is a key component in modern scene text detection frameworks. Detectors use pyramid feature representation to solve the challenge of scale variation in scene text detection. However, the inconsistency across different feature scales is the primary limitation for the detectors based on feature pyramid. Thus, we propose a novel strategy to solve this issue, called attention feature pyramid network (AFPN). It solves this problem by introducing feature enhancement module (FEM) and dual attention module (DAM). AFPN first performs feature enhancement on the backbone output, and then adaptively adjusts the features fusion by introducing an attention mechanism. In feature enhancement module, it learns the way to spatially filter conflictive information to suppress the inconsistency. In dual attention module, we perform two types of attention mechanism on the output of FEM and model the semantic interdependencies in spatial and channel dimensions respectively. AFPN optimizes the network through a self-attention mechanism, which is expected to greatly improve the detection results. By replacing FPN with AFPN in DBNet[2](DBNet is a real-time scene text detector with differentiable binarization), our models achieve 0.2 percent points higher F-Measure (F1) when using ResNet-50 as backbone and 1.4 percent points higher F-Measure when using ResNet-18 as backbone. Through experiments we found that AFPN can greatly improve the performance of lightweight backbone networks.
Automatic Modulation Classification Based on CNN, LSTM and Attention Mechanism

Automatic modulation recognition (AMR) is a key technology in the field of cognitive radio. To improve the performance, a new model for AMR, which incorporates an attention mechanism to extract the important features of the signal more effectively, is proposed in this paper. The simulation results show that our proposed method has a good recognition performance. In addition, our method can still achieve high accuracy, even if only a few samples are provided as the training set for training. We also visualize the output features of the model and show the positive effect of the attention mechanism on signal modulation category identification.
Detecting Appropriate and Inappropriate COVID-19 Face Mask Wear in Controlled Environments Using Transfer Learning-Based Convolutional Neural Network

The most recent epidemic of coronavirus disease (COVID-19) was brought on by a coronavirus subsequently found. The general population was still required to wear surgical face masks, sometimes known as medical masks, to safeguard against the coronavirus disease and monkeypox disease as well brought on by COVID-19 and monkeypox virus. In the majority of regulated conditions, it might be challenging to see if someone is wearing their mask properly. The researchers imply a COVID-19 detection of correct and improper wearing of surgical face masks in regulated areas as a way to help with the ongoing development of identification of facemask wearing to limit the spread of the virus. Models generated using deep learning to identify persons’ proper wearing of masks were used. The model with the lowest performance in this study’s model evaluation, Model 3, has an mAP of 0.0777. With an mAP of 0.9668 (96.68%) and 3.31 training loss, the model produced the best results in model 42. The said model obtained the highest mAP, which was used for testing and inferencing as a result. This study showed promising results and might be used to reliably identify appropriate mask wear in public by using proper detection of facemask technology.
Local Scale Adaptation to Hand Shape Model for Accurate and Robust Hand Tracking

Abstract The accuracy of hand tracking algorithms depends on how closely the geometry of the mesh model resembles the user's hand shape. Most existing methods rely on a learned shape space model; however, this fails to generalize to unseen hand shapes with significant deviations from the training set. We introduce local scale adaptation to augment this data‐driven shape model and thus enable modeling hands of substantially different sizes. We also present a framework to calibrate our proposed hand shape model by registering it to depth data and achieve accurate and robust tracking. We demonstrate the capability of our proposed adaptive shape model over the most widely used existing hand model by registering it to subjects from different demographics. We also validate the accuracy and robustness of our tracking framework on challenging public hand datasets where we improve over state‐of‐the‐art methods. Our adaptive hand shape model and tracking framework offer a significant boost towards generalizing the accuracy of hand tracking.
Forgery Detection Scheme of Deep Video Frame-rate Up-conversion Based on Dual-stream Multi-scale Spatial-temporal Representation

Video Frame-Rate Up-Conversion (FRUC) is originally designed to produce the high frame-rate video by periodically inserting new frames between two adjacent frames. However, it can also be utilized to synthesize the faked high frame-rate videos or spliced videos for malicious intents. The existing FRUC detection methods can efficiently identify its occurrence by exploring the blurring effects or deformed structures left over from the traditional FRUC methods. But the video FRUC has been substantially improved in the past years, especially in this deep learning era. Deep learning-based video FRUC (Deep FRUC) weakens the visual traces of the traditional ones such that it is challenging for the current FRUC detectors. In this paper, we propose a forensics algorithm based on dual-stream multi-scale spatial-temporal representation for Deep FRUC. Specifically, we develop a multi-scale receptive field strategy, and attention scheme to learn hidden tampered trace and spatial-temporal representation, respectively. Besides, frame residual and noise residual attention streams are complementary learning from the spatial-temporal dimension, in which the former can capture contrast differences and tampered abnormal boundaries, while the latter can explode noise inconsistencies between original and tampered frames. Experimental results show that the proposed algorithm can effectively achieve the best detection accuracy compared with the existing FRUC forensics works under the Deep FRUC video datasets.
Evaluating the Robustness of Deep Reinforcement Learning for Autonomous Policies in a Multi-Agent Urban Driving Environment

Background: Deep reinforcement learning is actively used for training autonomous car policies in a simulated driving environment. Due to the large availability of various reinforcement learning algorithms and the lack of their systematic comparison across different driving scenarios, we are unsure of which ones are more effective for training autonomous car software in single-agent as well as multi-agent driving environments. Aims: A benchmarking framework for the comparison of deep reinforcement learning in a vision-based autonomous driving will open up the possibilities for training better autonomous car driving policies. Method: To address these challenges, we provide an open and reusable benchmarking framework for systematic evaluation and comparative analysis of deep reinforcement learning algorithms for autonomous driving in a single- and multi-agent environment. Using the framework, we perform a comparative study of four discrete and two continuous action space deep reinforcement learning algorithms. We also propose a comprehensive multi-objective reward function designed for the evaluation of deep reinforcement learning-based autonomous driving agents. We run the experiments in a vision-only high-fidelity urban driving simulated environments. Results: The results indicate that only some of the deep reinforcement learning algorithms perform consistently better across single and multi-agent scenarios when trained in various multi-agent-only environment settings. For example, A3C- and TD3-based autonomous cars perform comparatively better in terms of more robust actions and minimal driving errors in both single and multi-agent scenarios. Conclusions: We conclude that different deep reinforcement learning algorithms exhibit different driving and testing performance in different scenarios, which underlines the need for their systematic comparative analysis. The benchmarking framework proposed in this paper facilitates such a comparison.
Redactable Blockchain with K-Time Controllable Editing

Considering the immutability of traditional blockchain restricts the governance and regulation of sensitive data on chain, this paper proposes a redactable blockchain scheme with k-time controllable cheating editing (CCE-RB for short). In the CCE-RB scheme, the regulator requires editors to generate vector commitments, so as to set the number of times each editor’s editing permissions can be used. The editor needs to open each component of the vector commitment successively before each editor prove that he/she has enough number of times of editing. The regulator has the right to revoke the editor’s editing permission when the editor’s editing times are exhausted or cheating editing behavior occurs, thus realizing the revocability of the editor’s editing permission. In a word, CCE-RB achieves regulation of whole editing process by block editability, controllability of editing times and revocability of editing permissions. By comparing and analyzing related schemes, the experiment results show that CCE-RB scheme has good efficiency.
CPDT: A copyright-preserving data trading scheme based on smart contracts and perceptual hashing

Data has become an integral part of the modern world, and its value has led to the creation of new markets for data trading. Protecting the copyright of trading data and ensuring the fairness of transactions present significant challenges in the current trading market. Therefore, we propose a copyright-preserving data trading scheme based on smart contracts and perceptual hashing, called CPDT, which use smart contracts, combined with perceptual hashing, to prevent malicious acts of illegal reselling and to ensure fair transactions. We have also designed a perceptual hashing algorithm for our trading scheme. Extensive experiments have been conducted to demonstrate the feasibility of the scheme.
MetaA: Multi-Dimensional Evaluation of Testing Ability via Adversarial Examples in Deep Learning

Deep learning (DL) has shown superior performance in many areas, making the quality assurance of DL-based software particularly important. Adversarial examples are generated by deliberately adding subtle perturbations in input samples and can easily attack less reliable DL models. Most existing works only utilize a single metric to evaluate the generated adversarial examples, such as attacking success rate or structure similarity measure. The problem is that they cannot avoid extreme testing situations and provide multifaceted evaluation results.This paper presents MetaA, a multi-dimensional evaluation framework for testing ability of adversarial examples in deep learning. Evaluating the testing ability represents measuring the testing performance to make improvements. Specifically, MetaA performs comprehensive validation on generating adversarial examples from two horizontal and five vertical dimensions. We design MetaA according to the definition of the adversarial examples and the issue mentioned in [1] that how to enrich the evaluation dimension rather than merely quantifying the improvement of DL and software.We conduct several analyses and comparative experiments vertically and horizontally to evaluate the reliability and effectiveness of MetaA. The experimental results show that MetaA can avoid speculation and reach agreement among different indicators when they reflect inconsistencies. The detailed and comprehensive analysis of evaluation results can further guide the optimization of adversarial examples and the quality assurance of DL-based software.
Self-Supervised Adversarial Example Detection by Disentangled Representation

Deep learning models are known to be vulnerable to adversarial examples that are elaborately designed for malicious purposes and are imperceptible to the human perceptual system. Autoencoder, when trained solely over benign examples, has been widely used for (self-supervised) adversarial detection based on the assumption that adversarial examples yield larger reconstruction errors. However, because lacking adversarial examples in its training and the too strong generalization ability of autoencoder, this assumption does not always hold true in practice. To alleviate this problem, we explore how to detect adversarial examples with disentangled label/semantic features under the autoencoder structure. Specifically, we propose Disentangled Representation-based Reconstruction (DRR). In DRR, we train an autoencoder over both correctly paired label/semantic features and incorrectly paired label/semantic features to reconstruct benign and counterexamples. This mimics the behavior of adversarial examples and can reduce the unnecessary generalization ability of autoencoder. We compare our method with the state-of-the-art self-supervised detection methods under different adversarial attacks and different victim models, and it exhibits better performance in various metrics (area under the ROC curve, true positive rate, and true negative rate) for most attack settings. Though DRR is initially designed for visual tasks only, we demonstrate that it can be easily extended for natural language tasks as well. Notably, different from other autoencoder-based detectors, our method can provide resistance to the adaptive adversary.
The Use of Pretrained Model for Matching App Reviews and Bug Reports

Matching APP reviews with bug reports can help APP developers to quickly identify new bugs from the users’ feedback. Existing solutions represent the semantics of APP reviews and bug reports via carefully designed features and models, the performance of which however depends heavily on the manually designed model and the training data set. Large-scale pretrained models can well capture the semantics of text and have demonstrated their success in many NLP tasks. Inspired by this, we explore the effect of various pretrained models on the matching accuracy of app review and bug report. We conduct a systematic study to analyze the factors of four major pretrained models (including T5, Sentence T5, Sentence MiniLM, Sentence BERT and so on) on the matching accuracy. We find that the accuracy of Sentence T5 and Sentence MiniLM in four open source applications is significantly greater than that of the state-of-the-art approach DeepMatcher. Based on the findings, we design a novel approach to match the APP reviews with bug reports based on the pretrained model Sentence T5 and Sentence MiniLM to calculate the sentence similarity. We test it on four open source applications and the results show that our method outperforms the existing solution. On average, the precision of Sentence T5 and Sentence MiniLM are increased by 17% and 13%, respectively, and the hit ratio are increased by 15% and 14%, respectively.
Secure Access Control for eHealth Data in Emergency Rescue Case based on Traceable Attribute-Based Encryption

With the development of cloud computing, patients can obtain efficient and high-quality medical services by uploading their eHealth data to the cloud for sharing among medical personnel. Because eHealth data contain lots of sensitive information, they are always encrypted before uploading to protect patients’ privacy. Ciphertext-policy attribute-based encryption (CP-ABE) is a commonly used cryptographic primitive since it achieves fine-grained and one-to-many access control on the encrypted data. However, encrypted eHealth data may become an obstacle for some healthcare scenarios, especially the emergency rescue scenes. In addition, the one-to-many access manner makes the traditional CP-ABE mechanism hard to track the identity of a traitor who sells the decryption privilege to others. In this paper, we propose an Emergency Access Control and Traceable (EmACT) attribute-based encryption scheme to address these issues. In addition, EmACT outsources the heavy bilinear computations of the decryption to the cloud, which means that EmACT is adaptable to the resource-constrained health-monitoring devices. The proposed scheme’s security is formally proven, and the experimental results demonstrate that EmACT is efficient and practicable.
IDROP: Intelligently detecting Return-Oriented Programming using real-time execution flow and LSTM

Return-Oriented Programming (ROP) has become one of the most widely used attack techniques for software vulnerability exploitation. Existing ROP detection methods fall into two types: hardware-based methods and software-based methods. The former is strongly dependent on specific hardware architectures and difficult to deploy. Although the latter can alleviate these problems, limited by the selection of features and thresholds, it cannot effectively discover neither variant ROP nor delayed ROP. In this work, we propose an intelligent detection method at runtime and implement the corresponding prototype system, IDROP, which uses real-time execution flow and LSTM to discovery ROP and its variants. Specifically, IDROP analyzes the differences between program execution flows that are independent of the ROP feature thresholds. Firstly, the Aspect Oriented Programming (AOP) is utilized to instrument the tested program, and the sliding window mechanism is applied to screen out suspicious program execution flow snapshots. Then, these suspicious execution flow snapshots are vectorized through data representation techniques. Finally, we build and train an LSTM model to discover ROP. Furthermore, we evaluate the performance of IDROP on a dataset consisting of 6000+ samples. The experimental results show that IDROP is effective in detecting ROP attacks, variant ROP and delayed ROP with an accuracy of 98%, 93% and 80%, respectively. In addition, IDROP has negligible space overhead and low performance overhead, which is similar to that of only using Pin for detection (about additional 2.5 times the program execution time before instrumentation).
GAN-Based Privacy-Preserving Unsupervised Domain Adaptation

In recent years, the rapid development of deep learning is attributed to the large amount of labeled data brought by the digital age. When there is no labeled data available in some application scenarios, domain adaptation can be used to transfer knowledge from the source domain with labeled data to the target domain without labeled data. In the process of domain adaptation, the target client requires direct access to the source data or model, which would lead to the risk of privacy leakage, e.g., Membership Inference Attacks (MIA). Attackers can collect the prediction vector of the model through black-box access to the source model, and then infer an individual’s membership in the source training dataset. To deal with this privacy issue, we propose a GAN-based Privacy-Preserving Unsupervised Domain Adaptation Framework. Specifically, the target client learns a conditional generator, sends the intermediate results perturbed by differential privacy to the source client, and the source client uses the source model to provide guidance for the generator so that the generator can generate the data corresponding to the input label that is the same as the data distribution in the target domain. We evaluate the performance of our proposed method on digital dataset and office-31dataset, which are popular domain adaptation benchmark datasets, and verify the security by the accuracy and F <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> -score of Membership Inference Attacks.
FFmpegSR: A General Framework Toward Real-Time 4K Super-Resolution

With the explosive growth of online video content, the demand for high-quality video is ever-rising. To take advantage of recent advances in deep learning, in this paper, we propose and implement a framework, FFmpegSR, that applies deep learning-based super-resolution into an FFmpeg filter to implement real-time 4K video super-resolution. FFmpegSR applies super-resolution to the Y channel only, allowing reduced inference time while maintaining good inference quality. To further improve the inference speed, we also develop a patch-based solution that uses saliency detection to select regions of interest on the video frame. This allows us to achieve faster inference on key patches only instead of full video frames. We used videos from a public dataset for evaluation. Results show that FFmpegSR can achieve real-time super-resolution to 4K with high visual quality.
Holmes: An Efficient and Lightweight Semantic Based Anomalous Email Detector

Email threat is a serious issue for enterprise security. The threat can be in various malicious forms, such as phishing, fraud, blackmail and malvertisement. The traditional anti-spam gateway often maintains a greylist to filter out unexpected emails based on suspicious vocabularies present in the email’s subject and contents. However, this type of signature-based approach cannot effectively discover novel and unknown suspicious emails that utilize various evolving malicious payloads. To address the problem, in this paper, we present Holmes, an efficient and lightweight semantic based engine for anomalous email detection. Holmes can convert each email event log into a sentence through word embedding and then identify abnormalities that deviate from a historical baseline based on those translated sentences. We have evaluated the performance of Holmes in a real-world enterprise environment, where around 5,000 emails are sent/received each day. In our experiments, Holmes shows a high capability to detect email threats, especially those that cannot be handled by the enterprise anti-spam gateway. It is also demonstrated through our experiment that Holmes can discover more concealed malicious emails that are immune from several commercial detection tools.
Federated Learning Scheme with Dual Security of Identity Authentication and Verification

Privacy protection in the era of big data has attracted attention, and privacy leakage may cause severe user losses. Federated learning (FL) is a distributed machine learning framework that stores data locally in participating nodes. FL is an effective way to protect data privacy under the current background of artificial intelligence. Blockchain is a distributed data structure with the advantages of decentralization, non-tampering, and non-counterfeiting. Traditional FL relies on centralized servers and is prone to the single point of failure, and the problem can be avoided by replacing the centralized server with blockchain technology. However, unauthorized participants may launch poison attacks, and untrusted nodes in the blockchain send faulty aggregation models to users, reducing the model’s availability. Given the above problems, this paper proposes an FL (DS-AVFL) framework with authentication and verification. Blockchain technology is used for dynamic identity management, only updates uploaded by authenticated local devices can be added to the blockchain, and local devices can verify the received global model. Experimental results show that the scheme is feasible and has reasonable computing costs.