Reviewerly/exHarmony huggingface dataset:
SNPF: Sensitiveness-Based Network Pruning Framework for Efficient Edge Computing

Convolutional neural networks (CNNs) are used comprehensively in the field of the Internet of Things (IoTs), such as mobile phones, surveillance, and satellite. However, the deployment of CNNs is difficult because the structure of hand-designed networks is complicated. Therefore, we propose a sensitiveness based network pruning framework (SNPF) to reduce the size of original networks to save computation resources. SNPF will evaluate the importance of each convolutional layer by the reconstruction of inference accuracy when we add extra noise to the original model, and then remove filters in terms of the degree of sensitiveness for each layer. Compared with previous weight-norm based pruning methods such as “l1-norm”“, BatchNorm-Pruning”, and “Taylor-Pruning”, SNPF is robust to the update of parameters, which can avoid the inconsistency of evaluation for filters if the parameters of the pre-trained model are not fully optimized. Namely, SNPF can prune the network at the early training stage to save computation resources. We test our method on three prevalent models of VGG-16, ResNet-18, ResNet-50 and a customized Conv-4 with 4 convolutional layers. They are then tested on CIFAR-10, CIFAR-100, ImageNet, and MNIST, respectively. Impressively, we observe that even when the VGG-16 is only trained with 50 epochs, we can get the same evaluation of layer importance as the results when the model is fully trained. Additionally, we can also achieve comparable pruning results to previous weight-oriented methods on the other three models.
AttAuth: An Implicit Authentication Framework for Smartphone Users Using Multimodality Data

Smartphones have become the most important devices for users to communicate and interact with different forms of media, and at the same time stored a large amount of sensitive and private data. The security and protection of such data has become increasingly critical. As the sensor technology rapid developed, the diversity of sensors on smartphones has greatly increased (e.g., motion sensor, touchscreen sensor), empowering the smartphones to provide continuous and implicit user authentication by capturing behavioral biometrics. Unfortunately, it remains a challenge to make full use of the data from the multi-modality sensors to provide accurate authentication. Towards this end, in this paper, we develop an implicit authentication framework AttAuth which explores organic integration of such multi-modality data to authenticate smartphone users through the usage session. Specifically, AttAuth first develops a series of data processing techniques to process the multi-channel motion sensor data and the discrete touchscreen data. Then, a temporal and channel-wise attention based recurrent neural network (TCA-RNN) is developed to build authentication model, which allows to jointly model continuously-monitored motion sensor data and irregularly-recorded discrete touchscreen data effectively. By generating a guidance vector based on touch events, TCA-RNN guides the temporal-wise attention mechanism on the processed multi-channel motion sensor data and outputs authentication results. We evaluate AttAuth on a real-world multi-modality smartphone usage dataset of 100 users. Extensive experiments demonstrate that AttAuth achieves the state-of-the-art authentication accuracy. Additional experiments are provided to examine the applicability of AttAuth in terms of running overheads and sensitivity to various application scenarios.
A Novel IoT-Enabled System for Real-Time Face Mask Recognition Based on Petri Nets

Due to Coronavirus Disease 2019 (COVID-19), many countries have formulated pandemic prevention regulations, requiring the masses to wear a face mask before entering public places and taking public transportation. However, if the entrances of some places are manually controlled to check whether people are wearing a face mask or not, it becomes not only labor intensive, but also time consuming. Therefore, this paper aims to develop a face mask recognition system based on an edge computing platform. The traditional manual inspection control method is replaced by artificial intelligence (AI) technology to achieve automatic recognition and control. As an edge computing platform, Jetson Nano is an embedded system equipped with an AI platform, which can be used for object detection and image classification. Developed by Ultralytics LLC, the YOLOv5 model with PyTorch framework runs on the edge computing platform, featuring high speed, high precision, and small size. According to the model training results, the average precision (AP) reaches 95.41%, while the mean average precision (mAP) reaches 94.42%. The average single-class running time is 0.016 seconds, and the file size of training model is 3.8MB. The recognition distance is up to 8m, and the maximum face rotation angle is 90. In addition, a Petri net software tool, Workflow Petri Net Designer (WoPeD), with graphical features based on mathematical theories, is used to verify the mask recognition system; and ensures the system has acceptable precision and recall values.
An Efficient FPGA-based Depthwise Separable Convolutional Neural Network Accelerator with Hardware Pruning

Convolutional neural networks (CNNs) have been widely deployed in computer vision tasks. However, the computation and resource intensive characteristics of CNN bring obstacles to its application on embedded systems. This article proposes an efficient inference accelerator on Field Programmable Gate Array (FPGA) for CNNs with depthwise separable convolutions. To improve the accelerator efficiency, we make four contributions: (1) an efficient convolution engine with multiple strategies for exploiting parallelism and a configurable adder tree are designed to support three types of convolution operations; (2) a dedicated architecture combined with input buffers is designed for the bottleneck network structure to reduce data transmission time; (3) a hardware padding scheme to eliminate invalid padding operations is proposed; and (4) a hardware-assisted pruning method is developed to support online tradeoff between model accuracy and power consumption. Experimental results show that for MobileNetV2 the accelerator achieves 10× and 6× energy efficiency improvement over the CPU and GPU implementation, and 302.3 frames per second and 181.8 GOPS performance that is the best among several existing single-engine accelerators on FPGAs. The proposed hardware-assisted pruning method can effectively reduce 59.7% power consumption at the accuracy loss within 5%.
Reference Tracking Optimization With Obstacle Avoidance via Task Prioritization for Automated Driving

Obstacle avoidance is a fundamental operation for automated driving and its formulation traditionally originates from robotics and decision making control fields. Given the high complexity required to compute an obstacle-free trajectory, this operation is usually demanded to a lower frequency planning layer that provides then a trajectory reference to be followed by a higher frequency control layer. As a result, whenever replanning is needed (for example, due to a new detected obstacle), the control layer must wait for a new planned trajectory to be generated. In this paper, we propose a novel methodology to approach obstacle avoidance already in the control layer, which allows a prompter response. In particular, we show how obstacle avoidance and reference tracking can be integrated, thus with no need to switch among different controllers, based on a null-space based behavioral control approach, implemented in a (possibly nonlinear) model predictive control scheme. We demonstrate practical implementation of the proposed methodology employing two different vehicle dynamic models and in four different (urban and highway) scenarios. Furthermore, we provide a sensitivity analysis to understand how parameters choice affects the automated vehicle behavior.
PBFL: Privacy-Preserving and Byzantine-Robust Federated-Learning-Empowered Industry 4.0

In Industry 4.0, Artificial Intelligence (AI) has been successfully applied in scenarios such as fault prediction, traffic analysis, and production decision-making. However, due to the sensitivity and security of data, privacy regulations prohibit the transfer and exchange of industrial data between entities, resulting in training data being fragmented into data silos that limit the accuracy of AI models. Federated learning (FL) can effectively break the data silo effect, but naive FL (FedAvg) is vulnerable to inference attacks from aggregators and Byzantine attacks from participants. To address these issues, we propose a privacy-preserving and Byzantine-robust federated learning scheme (PBFL) for Industry 4.0. Under the setting of an benign-majority participants, PBFL can always identify benign direction and magnitude of updates. Extensive experiments demonstrate that PBFL is more robust than state-of-the-art schemes, even with extreme proportion (49%) of malicious participants. Moreover, PBFL contains a series of well-optimized 2-Party computation (2PC) protocols, causing it reduces total runtime of the unoptimized implementation by around 3×~4× and 9×~10× for 32-bit and 64-bit circuits, respectively.
Adaptive Modulation and Retransmission Scheme for Semantic Communication Systems

Traditional adaptive modulation scheme aims to maximize the spectral efficiency by selecting the appropriate modulation scheme under the premise of perfect bit-level data transmission. However, in task-oriented semantic communication systems, imperfect transmission can still lead to a good inference performance of semantic tasks due to the error correction capability of neural networks. In this paper, we propose novel adaptive modulation and retransmission schemes to maximize the spectral efficiency while guaranteeing the performance of semantic tasks. Specifically, we first introduce the robustness verification problem in semantic communication systems to analyze the robustness of neural network inference. We then formulate and solve the modulation scheme selection problem constrained by a robustness probability threshold. Consequently, a novel adaptive modulation scheme is developed to maximize the spectral efficiency while guaranteeing the goal of semantic communication. We also develop a retransmission scheme using existing combining techniques to further increase the data rate under harsh channel conditions. Extensive simulations are performed in unencoded and encoded semantic communication systems to validate the effectiveness of the proposed schemes.
Labels Are Culprits: Defending Gradient Attack on Privacy

Federated learning (FL) is widely studied for local privacy protection, and it involves exchanging model parameters rather than raw data among clients. However, gradient attacks (GAs) make a malicious client or parameter server of FL infer the local data of other clients only based on the model parameters exchanged. In FL frameworks and processes, it is important to understand the features that provide heuristic information for inferring raw data, as well as how best to defend against GAs. The academic community is currently investigating this problem. In this study, we demonstrate that the labels of input samples play a key role in the success of GAs. We analyze the rank of the coefficient matrix of the non-homogeneous linear equation of gradients and input samples and propose an approach that performs special operations on the repetition and order of labels. The approach achieves a better defense effect against GAs without using a differential privacy (DP) framework. Our experimental results show that GAs fail (i.e., without leaking any valid information about local data) during the entire training process of a deep convolutional network in FL, and the accuracy of the network is less affected than that of DP. The code is available at https://github.com/zhaohuali/Label-based-Defense.
Time Estimation for a New Block Generation in Blockchain-Enabled Internet of Things

The Internet of Things (IoT) has emerged with Distributed Ledger Technology (DLT) to address existing scalability challenges and improve the trustworthiness of machine-tomachine communication. Among the numerous potential benefits of combining IoT and DLT, Blockchain, a subset of DLT, is a crucial enabler to accelerate secure IoT adoption. Appending a new block to a blockchain, especially in a blockchain-based IoT ecosystem, requires more delay than expected. This delay is one of several issues limiting the broader adoption of blockchain within the IoT domain. To assess this delay, we develop a new comprehensive model to estimate the time required to generate a new block in a blockchain-enabled IoT system. To this end, we develop sub-computation models and compare time consumption associated with the block generation process by conducting an extensive analysis of the following selected IoT layers: device layer, cluster head layer, fog/edge layer, and cloud layer. Our study identifies potential time-consuming steps in adding a new block to a network. Our results demonstrate that the type of blockchain framework and data encryption algorithms could affect the block generation time and that Avalanche, Conflux, Algorand, Polkadot Hyperledger Fabric outperforms Ethereum in terms of block generation time in IoT networks. On the other hand, the blockchain framework does not play a significant role in block generation time for smaller data packets. We also observed the benefit of using 256-bit ECC (elliptic curve cryptography) encryption and the fog layer in IoT networks to enhance the scalability of the block generation process. All in all, our results indicate that the total block generation time varies depending on the selected IoT framework, data encryption algorithm, blockchain type, and key functions of the layers. However, we found that time delays associated with queuing or block size are negligible relative to the other key components of block generation time.
Conversational recommender based on graph sparsification and multi-hop attention

Conversational recommender systems provide users with item recommendations via interactive dialogues. Existing methods using graph neural networks have been proven to be an adequate representation of the learning framework for knowledge graphs. However, the knowledge graph involved in the dialogue context is vast and noisy, especially the noise graph nodes, which restrict the primary node’s aggregation to neighbor nodes. In addition, although the recurrent neural network can encode the local structure of word sequences in a dialogue context, it may still be challenging to remember long-term dependencies. To tackle these problems, we propose a sparse multi-hop conversational recommender model named SMCR, which accurately identifies important edges through matching items, thus reducing the computational complexity of sparse graphs. Specifically, we design a multi-hop attention network to encode dialogue context, which can quickly encode the long dialogue sequences to capture the long-term dependencies. Furthermore, we utilize a variational auto-encoder to learn topic information for capturing syntactic dependencies. Extensive experiments on the travel dialogue dataset show significant improvements in our proposed model over the state-of-the-art methods in evaluating recommendation and dialogue generation.
Reparameterizing and dynamically quantizing image features for image generation

For autoregressive image generation, vector-quantized VAEs (VQ-VAEs) quantize image features with discrete codebook entries and reconstruct images from quantized features. However, they treat each codebook entry separately, which causes losses of image details. In this paper, we propose to reparameterize image features with weight vectors to treat all codebook entries as an entity, and present a novel dynamically vector quantized VAE (DVQ-VAE) to quantize reparameterized image features. Specifically, each image feature corresponds to a weight vector and we sum weighted codebook entries to obtain values of image features. In this way, image features can incorporate information from different codebook entries. Additionally, a novel continuous weight regularization loss is proposed to improve the reconstruction of image details. Our method achieves competitive results with prior state-of-the-art works for image generation and extensive experiments are conducted to take a deep insight into our DVQ-VAE.
Facial expression-enhanced recommendation for virtual fitting rooms

With the development of Augmented Reality (AR) technology in the retail industry, virtual fitting room (VFR) are considered promising enhancement of e-commerce by providing users with an immersive environment to try on new products, especially fashion products. While allowing users having more vivid impression of products, virtual fitting rooms also offer sellers more channels to collect information on user preferences, which can be used to enhance recommender systems. This study proposes to leverage facial expression recognition technology together with fine-grained human-computer interactions in virtual fitting rooms to personalize product recommendations. This paper proposes a recommendation algorithm based on confidence setting, negative feedback sampling, and matrix factorization to model user behaviors in virtual fitting rooms. We conduct an experiment on 81 subjects to evaluate the proposed method. Experimental results show the proposed method outperforms existing methods using traditional behavior information. Our study provides a strong support to the value of AR in enhancing e-commerce.
A High-Performance, Conflict-Free Memory-Access Architecture for Modular Polynomial Multiplication

In this paper, we present the HiCoP architecture, a high-performance, conflict-free memory access, modular polynomial multiplication design that accelerates the number theoretic transform (NTT), inverse NTT (INTT), and modular polynomial multiplications. To optimize hardware costs, the HiCoP architecture utilizes a high-radix reconfigurable butterfly unit (RBU) that can be dynamically configured to perform NTT, INTT, and point-wise multiplications, alongside an area-efficient Montgomery modular multiplier (MMM) tailored for NTT-friendly modulus. Moreover, by integrating pre-processing, post-processing, and Montgomery domain transformations into NTT and INTT operations, we effectively minimize the cycle count for modular polynomial multiplication. Additionally, we propose a novel conflict-free memory access algorithm that simplifies the control logic and eliminates the need for ping-pong memory in the HiCoP architecture. Experimental results of modular polynomial multiplications demonstrate significant performance gains for the HiCoP architecture implemented on the Xilinx Virtex-7 field-programmable gate array (FPGA) platform, with up to 8.75×, 4.15×, 10.57×, and 8.50× improvements in throughput-to-hardware-cost ratio for LUT count, FF count, BRAM count, and DSP count, respectively.
Augmented Reality Navigation System for Biliary Interventional Procedures With Dynamic Respiratory Motion Correction

Biliary interventional procedures require physicians to track the interventional instrument tip (Tip) precisely with X-ray image. However, Tip positioning relies heavily on the physicians' experience due to the limitations of X-ray imaging and the respiratory interference, which leads to biliary damage, prolonged operation time, and increased X-ray radiation.We construct an augmented reality (AR) navigation system for biliary interventional procedures. It includes system calibration, respiratory motion correction and fusion navigation. Firstly, the magnetic and 3D computed tomography (CT) coordinates are aligned through system calibration. Secondly, a respiratory motion correction method based on manifold regularization is proposed to correct the misalignment of the two coordinates caused by respiratory motion. Thirdly, the virtual biliary, liver and Tip from CT are overlapped to the corresponding position of the patient for dynamic virtual-real fusion.Our system is respectively evaluated and achieved an average alignment error of 0.75 ± 0.17 mm and 2.79 ± 0.46 mm on phantoms and patients. The navigation experiments conducted on phantoms achieve an average Tip positioning error of 0.98 ± 0.15 mm and an average fusion error of 1.67 ± 0.34 mm after correction.Our system can automatically register the Tip to the corresponding location in CT, and dynamically overlap the 3D virtual model onto patients to provide accurate and intuitive AR navigation.This study demonstrates the clinical potential of our system by assisting physicians during biliary interventional procedures. Our system enables dynamic visualization of virtual model on patients, reducing the reliance on contrast agents and X-ray usage.
Multi-view fusion for universal translation quality estimation

Machine translation quality estimation (QE) aims to evaluate the result of translation without reference. Despite the progress it has made, state-of-the-art QE models are proven to be biased. More specifically, they over-rely on spurious statistical features while ignoring the bilingual semantic adequacy, leading to performance degradation. Besides, existing approaches require large amounts of annotation data, restricting their applications in new domains and languages. In this work, we propose a universal framework for quality estimation based on multi-view fusion. We first introduce noise to the target side of the parallel sentence pair, either by pre-trained language model or by large language model. After that, with the clean parallel pairs and the noised pairs as different views, the QE model is trained to distinguish the clean pairs from the noised ones. Our method can improve the accuracy and generalizability in supervised scenario, and can solely perform estimation in zero-shot scenario. We perform experiments on WMT QE evaluation datasets under different scenarios, verifying the effectiveness of our method. We also make an in-depth investigation of the bias of QE model.
SCH: Symmetric Consistent Hashing for cross-modal retrieval

When handling large-scale multimodal data, hashing-based retrieval methods have gained significant attention due to their advantages in terms of low storage consumption and quick query speed. Despite the good performance of supervised hashing techniques, there are still certain limitations that need to be addressed. Firstly, they ignore valuable information such as reconstruction residuals when employing matrix factorization to learn the low-dimensional space. Secondly, most methods map heterogeneous data to the same feature space, which is unreasonable because different modalities possess distinct dimensions and distributions. Lastly, some methods relax constraints in order to optimize discrete hash codes, leading to considerable quantization errors. To solve the above problems, a novel two-step hashing-based technology called Symmetric Consistent Hashing (SCH) is proposed. Specifically, in the first step, two separate latent semantic spaces are learned by leveraging Principal Component Analysis to mitigate principal energy loss and Locality Preserving Projections to preserve local similarity. Then, the latent spaces of different modalities are aligned to facilitate the sharing of high-level semantic information by exploiting modality consistency. Next, hash codes are learned from distinct latent spaces, which are embedded in the second step for learning hash functions through Kernel Logistic Regression. To validate the effectiveness of SCH, extensive experiments on three publicly available benchmark datasets are accomplished. The results demonstrate the superiority of SCH over state-of-the-art hashing baselines, thus confirming the potency of the proposed approach.