Reviewerly/exHarmony huggingface dataset:
How Does Data Augmentation Affect Privacy in Machine Learning?

It is observed in the literature that data augmentation can significantly mitigate membership inference (MI) attack. However, in this work, we challenge this observation by proposing new MI attacks to utilize the information of augmented data. MI attack is widely used to measure the model's information leakage of the training set. We establish the optimal membership inference when the model is trained with augmented data, which inspires us to formulate the MI attack as a set classification problem, i.e., classifying a set of augmented instances instead of a single data point, and design input permutation invariant features. Empirically, we demonstrate that the proposed approach universally outperforms original methods when the model is trained with data augmentation. Even further, we show that the proposed approach can achieve higher MI attack success rates on models trained with some data augmentation than the existing methods on models trained without data augmentation. Notably, we achieve a 70.1\% MI attack success rate on CIFAR10 against a wide residual network while the previous best approach only attains 61.9\%. This suggests the privacy risk of models trained with data augmentation could be largely underestimated.
Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification*

Abstract We analyze in a closed form the learning dynamics of the stochastic gradient descent (SGD) for a single-layer neural network classifying a high-dimensional Gaussian mixture where each cluster is assigned one of two labels. This problem provides a prototype of a non-convex loss landscape with interpolating regimes and a large generalization gap. We define a particular stochastic process for which SGD can be extended to a continuous-time limit that we call stochastic gradient flow. In the full-batch limit, we recover the standard gradient flow. We apply dynamical mean-field theory from statistical physics to track the dynamics of the algorithm in the high-dimensional limit via a self-consistent stochastic process. We explore the performance of the algorithm as a function of the control parameters shedding light on how it navigates the loss landscape.
HABERTOR: An Efficient and Effective Deep Hatespeech Detector

We present our HABERTOR model for detecting hatespeech in large scale user-generated content.Inspired by the recent success of the BERT model, we propose several modifications to BERT to enhance the performance on the downstream hatespeech classification task.HABERTOR inherits BERT's architecture, but is different in four aspects: (i) it generates its own vocabularies and is pre-trained from the scratch using the largest scale hatespeech dataset; (ii) it consists of Quaternionbased factorized components, resulting in a much smaller number of parameters, faster training and inferencing, as well as less memory usage; (iii) it uses our proposed multisource ensemble heads with a pooling layer for separate input sources, to further enhance its effectiveness; and (iv) it uses a regularized adversarial training with our proposed finegrained and adaptive noise magnitude to enhance its robustness.Through experiments on the large-scale real-world hatespeech dataset with 1.4M annotated comments, we show that HABERTOR works better than 15 state-ofthe-art hatespeech detection methods, including fine-tuning Language Models.In particular, comparing with BERT, our HABERTOR is 4∼5 times faster in the training/inferencing phase, uses less than 1/3 of the memory, and has better performance, even though we pretrain it by using less than 1% of the number of words.Our generalizability analysis shows that HABERTOR transfers well to other unseen hatespeech datasets and is a more efficient and effective alternative to BERT for the hatespeech classification.
Generating Fact Checking Briefs

Fact checking at scale is difficult-while the number of active fact checking websites is growing, it remains too small for the needs of the contemporary media ecosystem.However, despite good intentions, contributions from volunteers are often error-prone, and thus in practice restricted to claim detection.We investigate how to increase the accuracy and efficiency of fact checking by providing information about the claim before performing the check, in the form of natural language briefs.We investigate passage-based briefs, containing a relevant passage from Wikipedia, entitycentric ones consisting of Wikipedia pages of mentioned entities, and Question-Answering Briefs, with questions decomposing the claim, and their answers.To produce QABriefs, we develop QABRIEFER, a model that generates a set of questions conditioned on the claim, searches the web for evidence, and generates answers.To train its components, we introduce QABRIEFDATASET which we collected via crowdsourcing.We show that fact checking with briefs -in particular QABriefs -increases the accuracy of crowdworkers by 10% while slightly decreasing the time taken.For volunteer (unpaid) fact checkers, QABriefs slightly increase accuracy and reduce the time required by around 20%.
Stateless Cloud Auditing Scheme for Non-Manager Dynamic Group Data With Privacy Preservation

As one of the core services of cloud computing, cloud storage could satisfy various storage and management requirements caused by the growth of data. Considering the complexity and uncontrollability of the cloud storage environment, many cloud auditing schemes were presented to assure the integrity of data in the cloud. However, most existing schemes have security risks, such as identity privacy and data privacy disclosure, authority abuse of group managers and collusion attacks during user revocation. To solve these problems, we propose a stateless cloud auditing scheme for non-manager dynamic group data with privacy preservation. The proposed scheme not only realizes user identity privacy preservation but also preserves data privacy security with the random masking technique. Unlike other solutions, our scheme allows t group users to trace the user's identity cooperatively without group managers, which eliminates authority abuse of group managers and provides non-frameability. Meanwhile, utilizing the concept of Shamir Secret Sharing, our scheme divides the re-signing process into several parts to resist collusion attacks during user revocation. By the designed binary tree, group users could trace dynamic data changes and recover the latest data when existing data are damaged. Besides, both users and the third-party auditor (TPA) are stateless in our scheme; that is, they no need to maintain data index information during cloud auditing. Our scheme also achieves mutual supervision between users and cloud service providers (CSPs), which ensures data are non-repudiation on both parties. Furthermore, we construct an efficient incentive for data visitors by using the blockchain technology and design a secure data sharing model to guarantee that data owners control their data ownership. Certificateless cryptography assures that the proposed scheme avoids certificate management and key escrow problems. Finally, security analysis and performance evaluation show that our scheme is secure and efficient.
LevelSet R-CNN: A Deep Variational Method for Instance Segmentation

Obtaining precise instance segmentation masks is of high importance in many modern applications such as robotic manipulation and autonomous driving. Currently, many state of the art models are based on the Mask R-CNN framework which, while very powerful, outputs masks at low resolutions which could result in imprecise boundaries. On the other hand, classic variational methods for segmentation impose desirable global and local data and geometry constraints on the masks by optimizing an energy functional. While mathematically elegant, their direct dependence on good initialization, non-robust image cues and manual setting of hyperparameters renders them unsuitable for modern applications. We propose LevelSet R-CNN, which combines the best of both worlds by obtaining powerful feature representations that are combined in an end-to-end manner with a variational segmentation framework. We demonstrate the effectiveness of our approach on COCO and Cityscapes datasets.
Controlling Neural Learning Network with Multiple Scales for Image Splicing Forgery Detection

The guarantee of social stability comes from many aspects of life, and image information security as one of them is being subjected to various malicious attacks. As a means of information attack, image splicing forgery refers to copying some areas of an image to another image to hide the traces of the original information and leads to grave consequences. Image splicing forgery is extremely complex since the attributes of the two images subjected to the pasting and copying operations are greatly different. In order to solve the issue mentioned above, we propose a method by applying a neural learning network controlled by multiple scales (MCNL-Net) based on U-Net to identify whether an image has been tampered and to locate the tampered regions. Firstly, the learning capacity of MCNL-Net is enhanced by the combination of a residual propagation module and a residual feedback module. An ingenious strategy is designed to control the size of local receptive field in each building block of MCNL-Net. The strategy makes MCNL-Net able to achieve properties and superiorities of multi-scale structure and learn specified features. For further improving the detection performance of MCNL-Net, a block attention mechanism is proposed to control the advanced degree of the input information in each building block. In addition, a MaxBlurPool method is applied into image splicing forgery detection for the first time, preserving the shift-equivariance of a convolutional neural network. Through experiments, we demonstrate that MCNL-Net can achieve more promising results and offer stronger robustness than the state-of-the-art splicing forgery detection methods.
Hyperparameter Optimization and Regularization on Fashion-MNIST Classification

Nowadays the most exciting technology breakthrough has been the rise of the deep learning. In computer vision Convolutional Neural Networks (CNN or ConvNet) are the default deep learning model used for image classification problems. In these deep network models, feature extraction is figure out by itself and these models tend to perform well with huge amount of samples. Herein we explore the impact of various Hyper-Parameter Optimization (HPO) methods and regularization techniques with deep neural networks on Fashion-MNIST (F-MNIST) dataset which is proposed by Zalando Research. We have proposed deep ConvNet architectures with Data Augmentation and explore the impact of this by configuring the hyperparameters and regularization methods. As deep learning requires a lots of data, the insufficiency of image samples can be expand through various data augmentation methods like Cropping, Rotation, Flipping, and Shifting. The experimental results show impressive results on this new benchmarking dataset F-MNIST.
Things in the air: tagging wearable IoT information on drone videos

Abstract Drones have been applied to a wide range of security and surveillance applications recently. With drones, Internet of Things are extending to 3D space. An interesting question is: Can we conduct person identification (PID) in a drone view? Traditional PID technologies such as RFID and fingerprint/iris/face recognition have their limitations or require close contact to specific devices. Hence, these traditional technologies can not be easily deployed to drones due to dynamic change of view angle and height. In this work, we demonstrate how to retrieve IoT data from users’ wearables and correctly tag them on the human objects captured by a drone camera to identify and track ground human objects. First, we retrieve human objects from videos and conduct coordination transformation to handle the change of drone positions. Second, a fusion algorithm is applied to measure the correlation of video data and inertial data based on the extracted human motion features. Finally, we can couple human objects with their wearable IoT devices, achieving our goal of tagging wearable device data (such as personal profiles) on human objects in a drone view. Our experimental evaluation shows a recognition rate of 99.5% for varying walking paths, and 98.6% when the drone’s camera angle is within 37°. To the best of our knowledge, this is the first work integrating videos from drone cameras and IoT data from inertial sensors.
Hand-in-Air (HiA) and Hand-on-Target (HoT) Style Gesture Cues for Mixed Reality Collaboration

In collaboration systems, collaborators often use hand gestures for showing information relating to a distant object.However, in viewer's perspective, it is sometimes difficult to know where the conventional gesture cue (Hands-in-Air style: HiA) refers to because the conventional HiA gesture appears away from the distant object.In this paper, we investigate how two factors, distance to the object and view angle difference between collaborators, influence the understanding of HiA gesture by comparing the use of it at 25 positions with 5 distances and 5 view angles.In a user study, we found that the distance to the target object and view angle difference negatively influenced HiA gesture communication.The influence of the distance and view angle was more serious between smaller angles (0 ~ 30 degree) than larger angles (30 ~ 60 degree), and between shorter distances (1m ~ 1.5m) than between longer distance (1.5m ~ 2m).As a solution, we propose the Hands-on-Target (HoT) style which positions the hand gesture cue on the surface of the target object.The HoT style gesture cue dramatically reduces the negative effect of the distance and view angle difference.Participants completed the task 29.3 percent faster, selected the correct object 2.75 times more, and felt 63.1 percent less mental effort.For further investigation, we discuss on the extensibility of the HoT interface that it can be used not only for object selection but also for diverse type of gesture communication including gesture for object manipulation because the HoT interface can support all possible real world hand gestures.
In Search of Patient Zero: Visual Analytics of Pathogen Transmission Pathways in Hospitals

Pathogen outbreaks (i.e., outbreaks of bacteria and viruses) in hospitals can cause high mortality rates and increase costs for hospitals significantly. An outbreak is generally noticed when the number of infected patients rises above an endemic level or the usual prevalence of a pathogen in a defined population. Reconstructing transmission pathways back to the source of an outbreak - the patient zero or index patient - requires the analysis of microbiological data and patient contacts. This is often manually completed by infection control experts. We present a novel visual analytics approach to support the analysis of transmission pathways, patient contacts, the progression of the outbreak, and patient timelines during hospitalization. Infection control experts applied our solution to a real outbreak of Klebsiella pneumoniae in a large German hospital. Using our system, our experts were able to scale the analysis of transmission pathways to longer time intervals (i.e., several years of data instead of days) and across a larger number of wards. Also, the system is able to reduce the analysis time from days to hours. In our final study, feedback from twenty-five experts from seven German hospitals provides evidence that our solution brings significant benefits for analyzing outbreaks.
WITHDRAWN: Deep learning based image processing approaches for image deblurring

Mobile phone imagery has evolved substantially over the past two decades. The camera is one of the primary features of a new cell phone. A lot of research in this field has been done to enhance the quality of the image. A small phone structure limits the camera module mounted in cell phones, which means that there can be no mounted thick lenses or large image sensors on the phones. This affects the volume and the picture quality of the captured light. With less light, phones perform worse than digital Single-Lens Reflex (DSLR) cameras as opposed to image quality tests. Post processing is then used to enhance the quality of the image. Dark conditions and fast movement in mobile imaging are difficult. In dark conditions, longer exposure period collects more light, which can cause movement blurred objects. Motion blur artefact, e.g. when photo to graphing a racing car, and may also be caused by fast moving object at daylight. The movement blur causes sharp information to be lost and thus poor picture quality. A deblur ring is the tool used to eliminate flutter from photographs and make them appear clearer. In the field of signal processing deep learning-based approaches have recently become popular. The results have been promising since profound learning algorithms can learning and model nonlinear and complex connections. Deep learning algorithms were also used for several tasks in image restore work, as was done here.
RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER

Multimodal named entity recognition (MNER) for tweets has received increasing attention recently.Most of the multimodal methods used attention mechanisms to capture the text-related visual information.However, unrelated or weakly related text-image pairs account for a large proportion in tweets.Visual clues unrelated to the text would incur uncertain or even negative effects for multimodal model learning.In this paper, we propose a novel pre-trained multimodal model based on Relationship Inference and Visual Attention (RIVA) for tweets.The RIVA model controls the attention-based visual clues with a gate regarding the role of image to the semantics of text.We use a teacher-student semi-supervised paradigm to leverage a large unlabeled multimodal tweet corpus with a labeled data set for text-image relation classification.In the multimodal NER task, the experimental results show the significance of text-related visual features for the visual-linguistic model and our approach achieves SOTA performance on the MNER datasets.
Real-Time Uncertainty Estimation in Computer Vision via Uncertainty-Aware Distribution Distillation

Calibrated estimates of uncertainty are critical for many real-world computer vision applications of deep learning. While there are several widely-used uncertainty estimation methods, dropout inference [11] stands out for its simplicity and efficacy. This technique, however, requires multiple forward passes through the network during inference and therefore can be too resource-intensive to be deployed in real-time applications. We propose a simple, easy-to-optimize distillation method for learning the conditional predictive distribution of a pre-trained dropout model for fast, sample-free uncertainty estimation in computer vision tasks. We empirically test the effectiveness of the proposed method on both semantic segmentation and depth estimation tasks, and demonstrate our method can significantly reduce the inference time, enabling real-time uncertainty quantification, while achieving improved quality of both the uncertainty estimates and predictive performance over the regular dropout model.
VisDrone-DET2020: The Vision Meets Drone Object Detection in Image Challenge Results

The Vision Meets Drone Object Detection in Image Challenge (VisDrone-DET 2020) is the third annual object detector benchmarking activity. Compared with the previous VisDrone-DET 2018 and VisDrone-DET 2019 challenges, many submitted object detectors exceed the recent state-of-the-art detectors. Based on the selected 29 robust detection methods, we discuss the experimental results comprehensively, which shows the effectiveness of ensemble learning and data augmentation in drone captured object detection. The full challenge results are publicly available at the website http://aiskyeye.com/leaderboard/ .
Efficient Real-Time Human Detection Using Unmanned Aerial Vehicles Optical Imagery

Unmanned Aerial Vehicles (UAVs) are promising technologies within many different application scenarios including human detection in search and rescue and surveillance use cases, which have received considerable attention worldwide. However, adverse conditions, such as varying altitude, overhead camera placement, changing illumination and moving platform, impose challenges for high-performance yet cost-efficient human detection. To overcome these challenges, we propose a novel combination of dilated convolutions with Path Aggregation Network (PAN) as a new deep neural network-based human detection algorithm in real time. Furthermore, we establish a comprehensive human detection dataset with varying backgrounds, illuminations, and contrast and train the proposed machine-learning model on the collected dataset. Our approach achieves both high precision (88.0% mean Average Precision (mAP)) and real time (67.0 Frames Per Second (FPS)) on a commercial off-the-shelf PC platform. In terms of accuracy, the result is comparable to the standard You Only Look Once v3 (YOLOv3). However, the speed is twice as that of the standard YOLOv3. YOLOv4 is slightly more accurate (89.8%) than our approach. However, it is slower (38.0 versus 67.0 FPS) and has more Billion Floating-Point Operations (BFLOPS). The proposed algorithm has also trained with the VisDrone2019 dataset and compared with seven studies using this dataset. The results have further validated the effectiveness of the proposed approach. Moreover, the algorithm has been evaluated on an embedded system (Jetson AGX Xavier), which demonstrates the usefulness of this method on power-constrained devices. The proposed algorithm is fast, memory efficient, and computationally less expensive to achieve high detection performance. It is expected to contribute significantly to the wider use of UAV applications including search and rescue missions to locate missing people, and surveillance particularly for applications running on resource-constrained platforms, like smartphones or tablets. This proposed system is now being used in aerial drone system of Police of Scotland to help them locate and find missing and vulnerable people. The results of the project were broadcasted by BBC Scotland.
Multiple Importance Sampling for Efficient Symbol Error Rate Estimation

Digital constellations formed by hexagonal or other non-square two-dimensional lattices are often used in advanced digital communication systems. The integrals required to evaluate the symbol error rate (SER) of these constellations in the presence of Gaussian noise are in general difficult to compute in closed-form, and therefore Monte Carlo simulation is typically used to estimate the SER. However, naive Monte Carlo simulation can be very inefficient and requires very long simulation runs, especially at high signal-to-noise ratios. In this letter, we adapt a recently proposed multiple importance sampling (MIS) technique, called ALOE (for `At Least One rare Event'), to this problem. Conditioned to a transmitted symbol, an error (or rare event) occurs when the observation falls in a union of half-spaces or, equivalently, outside a given polytope. The proposal distribution for ALOE samples the system conditionally on an error taking place, which makes it more efficient than other importance sampling techniques ALOE provides unbiased SER estimates with simulation times orders of magnitude shorter than conventional Monte Carlo.