Reviewerly/exHarmony huggingface dataset:
SHIELD: Defending Textual Neural Networks against Multiple Black-Box Adversarial Attacks with Stochastic Multi-Expert Patcher

Even though several methods have proposed to defend textual neural network (NN) models against black-box adversarial attacks, they often defend against a specific text perturbation strategy and/or require re-training the models from scratch.This leads to a lack of generalization in practice and redundant computation.In particular, the state-of-the-art transformer models (e.g., BERT, RoBERTa) require great time and computation resources.By borrowing an idea from software engineering, in order to address these limitations, we propose a novel algorithm, SHIELD, which modifies and re-trains only the last layer of a textual NN, and thus it "patches" and "transforms" the NN into a stochastic weighted ensemble of multi-expert prediction heads.Considering that most of current black-box attacks rely on iterative search mechanisms to optimize their adversarial perturbations, SHIELD confuses the attackers by automatically utilizing different weighted ensembles of predictors depending on the input.In other words, SHIELD breaks a fundamental assumption of the attack, which is a victim NN model remains constant during an attack.By conducting comprehensive experiments, we demonstrate that all of CNN, RNN, BERT, and RoBERTa-based textual NNs, once patched by SHIELD, exhibit a relative enhancement of 15%-70% in accuracy on average against 14 different black-box attacks, outperforming 6 defensive baselines across 3 public datasets.
Crowd Decision Making: Sparse Representation Guided by Sentiment Analysis for Leveraging the Wisdom of the Crowd

The “wisdom of the crowd” theory states that a nonexpert crowd makes smarter decisions than a reduced set of experts. Social network platforms are a source of evaluations in the natural language of any topic, which may be considered as the evaluations of a nonexpert crowd. Decision-making (DM) models are constrained by their inability of processing large amounts of evaluations in natural language, as those ones from social networks. We claim that evaluations from social networks can enhance the quality of multiperson multicriteria DM models. Accordingly, we propose a crowd DM model guided by sentiment analysis (SA), which solves decision situations leveraging the wisdom of the crowd available in social networks. The model uses several deep-learning SA classification models through opinion triplets to incorporate all the evaluation shades in the DM model. Likewise, the likely lack of information stemmed from the consideration of a large set of users is tackled with a sparse representation of the evaluations. We annotate and release the TripR-2020Large dataset, and we use it to evaluate the model in the use case of restaurant choice. The results show that the integration of the wisdom of the crowd and the different shades of the evaluations enhances the quality of the decision.
Deep Image Inpainting With Enhanced Normalization and Contextual Attention

Deep learning-based image inpainting has been widely studied, leading to great success. However, many methods adopt convolution and normalization operations, which will bring up some issues to affect the performance. The vanilla normalization cannot distinguish the pixels in corrupted regions from the other valid pixels, resulting in the mean and variance shifts. In addition, the limited receptive field of convolution makes it unable to capture long-range valid information directly. In order to tackle these challenges, we propose a novel deep generative model for image inpainting with two key modules, namely, the channel and spatially adaptive batch normalization (CSA-BN) module, and the selective latent-space-mapping-based contextual attention (SLSM-CA) layer. We replace the vanilla normalization with the CSA-BN module. By channel and spatially adaptive denormalization, the CSA-BN module can mitigate the spatial mean and variance shifts in each channel in a targeted way. In addition, we also integrate the SLSM-CA layer into our model to capture the long-range correlations explicitly. By introducing dual-branch attention and a feature selection module, the SLSM-CA layer can selectively utilize the multi-scale background information to improve prediction quality. What’s more, it introduces the latent spaces to achieve the low-rank approximations of attention matrices and to reduce computational costs. Extensive quantitative and qualitative evaluations demonstrate the superiority of the proposed method compared with state-of-the-art methods.
Client-Side Optimization Strategies for Communication-Efficient Federated Learning

Federated learning (FL) is a swiftly evolving field within machine learning for collaboratively training models at the network edge in a privacy-preserving fashion, without training data leaving the devices where it was generated. The privacy-preserving nature of FL shows great promise for applications with sensitive data such as healthcare, finance, and social media. However, there are barriers to real-world FL at the wireless network edge, stemming from massive wireless parallelism and the high communication costs of model transmission. The communication cost of FL is heavily impacted by the heterogeneous distribution of data across clients, and some cutting-edge works attempt to address this problem using novel client-side optimization strategies. In this article, we provide a tutorial on model training in FL, and survey the recent developments in client-side optimization and how they relate to the communication properties of FL. We then perform a set of comparison experiments on a representative subset of these strategies, gaining insights into their communication-convergence trade-offs. Finally, we highlight challenges to client-side optimization and provide suggestions for future developments in FL at the wireless edge.
AnonymousFox: An Efficient and Scalable Blockchain Consensus Algorithm

Blockchain is an innovative application of distributed storage, consensus algorithm, encryption algorithm, and other computer technologies. The consensus algorithm is the key to keep consistent among blockchain nodes. In most existing consensus algorithms, the leader node is responsible for proposing new block and communicating with other nodes. The leader node is easy to be the target of malicious attackers. With the increase of the number of nodes, the throughput and scalability of the blockchain system are also unsatisfactory. To address such issues, we propose the AnonymousFox consensus algorithm, which is suitable for the consortium blockchain and private blockchain. First, we design an anonymous leader node sorting algorithm, which hides the identity of the leader node through a variety of encryption algorithms. It periodically changes the ordered leader list to hide the target of malicious attackers. In addition, we design a consensus algorithm based on the anonymous identity of the leader node. Through one-to-many message communication, the amount of messages is greatly reduced. The complexity of the algorithm is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$O(n)$ </tex-math></inline-formula> . It solves the problem of ordered replication of state machines when the leader node is anonymous. We analyze the algorithm, it ensures safety and liveness when the fault nodes are less than one-third of the total. We evaluate the throughput, latency, scalability, resource consumption, exception processing, smart contract, and blockchain network through experiments. The throughput of the proposed algorithm is 49.3% higher than that of the practical Byzantine fault tolerance (PBFT) algorithm. The experimental results show that the proposed algorithm has high performance and scalability.
Multi-Object Tracking and Segmentation With Embedding Mask-Based Affinity Fusion in Hierarchical Data Association

In this paper, we propose a highly feasible fully online multi-object tracking and segmentation (MOTS) method that uses instance segmentation results as an input. The proposed method is based on the Gaussian mixture probability hypothesis density (GMPHD) filter, a hierarchical data association (HDA), and a mask-based affinity fusion (MAF) model to achieve high-performance online tracking. The HDA consists of two associations: segment-to-track and track-to-track associations. One affinity, for position and motion, is computed by using the GMPHD filter, and the other affinity, for appearance is computed by using the responses from single object trackers such as kernalized correlation filter, SiamRPN, and DaSiamRPN. These two affinities are simply fused by using a score-level fusion method such as min-max normalization referred to as MAF. In addition, to reduce the number of false positive segments, we adopt mask IoU-based merging (mask merging). The proposed MOTS framework with the key modules: HDA, MAF, and mask merging, is easily extensible to simultaneously track multiple types of objects with CPU-only execution in parallel processing. In addition, the developed framework only requires simple parameter tuning unlike many existing MOTS methods that need intensive hyperparameter optimization. In the experiments on the two popular MOTS datasets, the key modules show some improvements. For instance, ID-switch decreases by more than half compared to a baseline method in the training sets. In conclusion, our tracker achieves state-of-the-art MOTS performance in the test sets.
Robust Image-Based Landing Control of a Quadrotor on an Unpredictable Moving Vehicle Using Circle Features

This paper addresses the landing problem of a quadrotor on an unpredictable moving vehicle, using a robust image-based visual servoing (IBVS) method. The circle-based image moments are defined to construct image dynamics, and the passivity-like property of the circle features is preserved by reprojecting to the virtual image plane. The landing control system is decoupled into translation and rotation modules due to the rotation invariance of the proposed circle features. First, by exploiting the error transformation in the image space, a robust IBVS controller can overcome the lack of both the desired depth information of target features and the velocity feedback of the target. Next, an adaptive geometric attitude controller is developed directly using rotation matrices to avoid the singularities of Euler-angles and the ambiguity of quaternions. One benefit of the proposed scheme is that it can potentially improve the camera visibility, guarantee the transient and steady-state behaviors in image space, and be efficiently implemented on the low-cost quadrotor. Finally, The stability analysis is presented using Lyapunov stability theory on cascaded systems, and the effectiveness of the proposed control strategy is demonstrated through simulations and experiments. Note to Practitioners—The motivation of this paper is to investigate a practical control strategy for the image-based landing control of underactuated quadrotors on an unpredictable moving vehicle. In most of the existing image-based landing control schemes for underactuated quadrotors, having the prior predictive model of the moving landing vehicle to provide a feed-forward compensation during the landing maneuver. However, due to the fact that the landing environment and vehicle are primarily stochastic, resulting in no predictive models are valid in practice. Therefore, this paper suggests a robust image-based landing control strategy without the model or state of the moving landing vehicle. In particular, a novel virtual circle feature, possessing the characteristic of rotation invariance, is designed for the landing of underactuated quadrotors, which decouples the landing system and simplifies the control design. Moreover, the image feature errors are directly retained within prescribed performance funnels in the image space. As a result, the transient and steady-state landing behaviors can be implicitly guaranteed in Cartesian space. The stability and convergence of the system are analyzed mathematically and the experiment using quadrotors provides promising results. In ongoing research, we are addressing the issues of collision avoidances and unknown disturbances to provide a more realistic setup for the autonomous deployment and recovery of underactuated quadrotors in GPS-denied environments.
A Phase-Based Approach for ENF Signal Extraction From Rolling Shutter Videos

Electric Network Frequency (ENF) analysis has been an intriguing tool for multimedia forensics as former studies have paved the way for estimating ENF signals from digital audio, video, or even image files. However, for ENF signals to be widely used in extensive applications, supplementary research is needed so that ENF signals can be stably extracted without restrictions. In this letter, we propose a new phase-based approach for extracting ENF signals from CMOS sensor recordings. It uses phase differences between row signals from two consecutive frames, such that problems due to missing sample points during the idle periods are circumvented. The proposed method has substantial advantages in that it is applicable without a predefined read-out time and when the length of given videos is too short. Extensive experiments conducted with numerous devices demonstrate that the proposed method can take precedence over state-of-the-art methods because it robustly produces accurate ENF estimates in terms of alias frequency on the frame-level. The coding framework used for this letter is available at: <uri>https://github.com/hyekyunghan/Phase-based-ENF-extraction-method</uri>.
Privacy-Preserving Collaborative Data Collection and Analysis with Many Missing Values

Privacy-preserving data mining techniques are useful for analyzing various information, such as Internet of Things data and COVID-19-related patient data.However, collecting a large amount of sensitive personal information is a challenging task.In addition, this information may have missing values, which are not considered in the existing methods for collecting personal information while ensuring data privacy.Failure to account for missing values reduces the accuracy of the data analysis.In this paper, we propose a method for privacy-preserving data collection that considers many missing values.The patient data are anonymized and sent to a data collection server.The data collection server creates a generative model and a contingency table suitable for multi-attribute analysis based on expectationmaximization and Gaussian copula methods.Using differential privacy (the de facto standard) as a privacy metric, we conduct experiments on synthetic and real data, including COVID-19related data.The results are 50-80% more accurate than those of existing methods that do not consider missing values.
A Privacy-Enhanced Multiarea Task Allocation Strategy for Healthcare 4.0

The continuous development of Healthcare 4.0 has brought great convenience to people. Through the Internet of Things technology, doctors can analyze patients’ health data and make timely diagnosis. However, behind the high efficiency, the mobile crowdsensing technology used for data transmission still has the risk of leaking the privacy of task and patient information. To this end, this article proposes a privacy-enhanced multi-area task assignment strategy, named PMTA. Specifically, we use deep differential privacy to add noise to patient data, and then put the noise-added dataset into a deep Q-network for training, combined with a spectral clustering algorithm, to obtain an optimal classification strategy. Further, in order to address the problem of data silos, we adopt federated learning to jointly train the classification models of different hospitals to obtain a global model and realize data sharing among different hospitals. Finally, we use the optimal classification of patients for task deployment on the blockchain, and limit patients to only apply for tasks of the corresponding level through the smart contract technology, so as to protect task privacy. Experimental results show that our strategy can not only effectively protect task and patient privacy, but also achieve better system performance.
Deepq: Residue analysis of localization images in large scale solid state physical environments

Deep Learning is the process to led machine learning, natural language processing and neural networks. The various deep learning models, computer vision systems and artificial intelligence services are used to study of various real time applications. Due to lack of computing resource the conventional neural network are produces delay in progress and reduce the GPUs performance and throughput. In this paper we review difference deep learning approaches with increases GPUs performance and apply various image processing classification and localization techniques. The high availability and GPUs performance can be verified by state-of-arts results using conventional deep learning methods.
Regularizing Visual Semantic Embedding With Contrastive Learning for Image-Text Matching

Learning visual semantic embedding for image-text matching has achieved high success by using triplet loss to pull positive image-text pairs which share similar semantic meaning and to push negative image-text pairs which share different semantic meaning. Without modeling constraints from image-image or text-text pairs, the generated visual semantic embedding inevitably faces the problem of semantic misalignments among similar images or among similar texts. To solve this problem, we present a contrastive visual semantic embedding framework, named as ConVSE, which achieves intra-modal semantic alignment by contrastive learning from augmented image-image (or text-text) pairs and achieves inter-modal semantic alignment by applying hardest-negative-enhanced triplet loss on image-text pairs. To the best of our knowledge, we are the first to find that contrastive learning benefits visual semantic embedding. Extensive experiments on large scale MSCOCO and Flickr30K datasets verify the effectiveness of our proposed ConVSE by outperforming visual semantic embedding-based methods and achieving new state-of-the-arts. Our code and pretrained model are publicly available at: \url{https://github.com/liuyyy111/ConVSE}.
EC²Detect: Real-Time Online Video Object Detection in Edge-Cloud Collaborative IoT

Video object detection is a fundamental technology of intelligent video analytics for Internet of Things (IoT) applications. However, even with extraordinary detection accuracy, predominating solutions based on deep convolutional neural networks (DCNNs) cannot achieve real-time online object detection on video streams with a low end-to-end (E2E) response latency and therefore cannot be applied to proliferating latency-sensitive IoT applications like autonomous driving requiring large-scale intelligent video analytics. To address this issue, we present EC2Detect, an edge-cloud collaborative real-time online video object detection method. Specifically, we propose a tracking-assisted object detection architecture based on edge-cloud collaboration with keyframe selection, where the accurate but heavy object detection is conducted by the Cloud on sparse keyframes adaptively selected according to their semantic variation, and the lightweight object tracking is used to localize and identify objects in other frames at edge devices. Extensive experiments of our real-world prototype demonstrate that, EC2Detect significantly outperforms state-of-the-art methods in terms of processing speed (up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4.77\times $ </tex-math></inline-formula> faster), E2E latency (up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8.12 \times $ </tex-math></inline-formula> lower), and edge-cloud bandwidth occupation ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$17 \times $ </tex-math></inline-formula> lower) with an acceptable mAP, which can effectively support large-scale intelligent video analytics in practice. Source code of EC2Detect is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ECCDetect/ECCDetect</uri> .
Synergistic Digital Twin and Holographic Augmented-Reality-Guided Percutaneous Puncture of Respiratory Liver Tumor

Thermal ablation is an exciting new minimally invasive treatment that destroys liver tumors without removing them. It uses image guidance to place a needle through the skin into a liver tumor, which is highly dependent on surgeons’ experience. With the development of digital medicine, augmented reality (AR) has become a more intuitive and safer way to achieve real-time navigation. However, the technology is still in its infancy due to its limited accuracy and real-time performance. To address these problems, we syncretized the holographic AR with the digital twin technique to track the dynamic surgical scene and provide the 3-D navigation of heterogeneous target regions via internal motion prediction. To tackle the dilemma of real-time performance and precise internal motion estimation, a dynamic adaptation scheme is proposed to compensate for the time cost induced by the external/internal correlation model and data transmission. We carried out a series of experiments to validate our methods. With the proposed external/internal correlation model, the average estimation errors of the tumor and vessels are 2.18 and 2.79 mm, respectively. Besides, we performed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in vivo</i> experiments on two beagle dogs with an artificial lesion in their liver, respectively, and the puncture accuracy of our method are 2.5 and 2.17 mm. The results show that on one hand, our method can fulfill the real-time requirement of AR via using the intraoperative data, which is also more precise than that with preoperative data. On the other hand, our method can provide more 3-D information for surgeons, such as vessels, which can well ensure the safety of operation.
Communication-Efficient Federated Learning: A Second Order Newton-Type Method With Analog Over-the-Air Aggregation

Owing to their fast convergence, second-order Newton-type learning methods have recently received attention in the federated learning (FL) setting. However, current solutions are based on communicating the Hessian matrices from the devices to the parameter server, at every iteration, incurring a large number of communication rounds; calling for novel communication-efficient Newton-type learning methods. In this article, we propose a novel second-order Newton-type method that, similarly to its first-order counterpart, requires every device to share only a model-sized vector at each iteration while hiding the gradient and Hessian information. In doing so, the proposed approach is significantly more communication-efficient and privacy-preserving. Furthermore, by leveraging the over-the-air aggregation principle, our method inherits privacy guarantees and obtains much higher communication efficiency gains. In particular, we formulate the problem of learning the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">inverse Hessian-gradient product</i> as a quadratic problem that is solved in a distributed way. The framework alternates between updating the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">inverse Hessian-gradient product</i> using a few alternating direction method of multipliers (ADMM) steps, and updating the global model using Newton’s method. Numerical results show that our proposed approach is more communication-efficient and scalable under noisy channels for different scenarios and across multiple datasets.