Reviewerly/exHarmony huggingface dataset:
Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation.

Supervised learning in large discriminative models is a mainstay for modern computer vision. Such an approach necessitates investing in large-scale human-annotated datasets for achieving state-of-the-art results. In turn, the efficacy of supervised learning may be limited by the size of the human annotated dataset. This limitation is particularly notable for image segmentation tasks, where the expense of human annotation is especially large, yet large amounts of unlabeled data may exist. In this work, we ask if we may leverage semi-supervised learning in unlabeled video sequences and extra images to improve the performance on urban scene segmentation, simultaneously tackling semantic, instance, and panoptic segmentation. The goal of this work is to avoid the construction of sophisticated, learned architectures specific to label propagation (e.g., patch matching and optical flow). Instead, we simply predict pseudo-labels for the unlabeled data and train subsequent models with both human-annotated and pseudo-labeled data. The procedure is iterated for several times. As a result, our Naive-Student model, trained with such simple yet effective iterative semi-supervised learning, attains state-of-the-art results at all three Cityscapes benchmarks, reaching the performance of 67.8% PQ, 42.6% AP, and 85.2% mIOU on the test set. We view this work as a notable step towards building a simple procedure to harness unlabeled video sequences and extra images to surpass state-of-the-art performance on core computer vision tasks.
GazeMetrics: An Open-Source Tool for Measuring the Data Quality of HMD-based Eye Trackers

As virtual reality (VR) garners more attention for eye tracking research, knowledge of accuracy and precision of head-mounted display (HMD) based eye trackers becomes increasingly necessary. It is tempting to rely on manufacturer-provided information about the accuracy and precision of an eye tracker. However, unless data is collected under ideal conditions, these values seldom align with on-site metrics. Therefore, best practices dictate that accuracy and precision should be measured and reported for each study. To address this issue, we provide a novel open-source suite for rigorously measuring accuracy and precision for use with a variety of HMD-based eye trackers. This tool is customizable without having to alter the source code, but changes to the code allow for further alteration. The outputs are available in real time and easy to interpret, making eye tracking with VR more approachable for all users.
Bayesian Image Classification with Deep Convolutional Gaussian Processes

In decision-making systems, it is important to have classifiers that have calibrated uncertainties, with an optimisation objective that can be used for automated model selection and training. Gaussian processes (GPs) provide uncertainty estimates and a marginal likelihood objective, but their weak inductive biases lead to inferior accuracy. This has limited their applicability in certain tasks (e.g. image classification). We propose a translation-insensitive convolutional kernel, which relaxes the translation invariance constraint imposed by previous convolutional GPs. We show how we can use the marginal likelihood to learn the degree of insensitivity. We also reformulate GP image-to-image convolutional mappings as multi-output GPs, leading to deep convolutional GPs. We show experimentally that our new kernel improves performance in both single-layer and deep models. We also demonstrate that our fully Bayesian approach improves on dropout-based Bayesian deep learning methods in terms of uncertainty and marginal likelihood estimates.
Stochastic Nested Variance Reduction for Nonconvex Optimization

We study finite-sum nonconvex optimization problems, where the objective function is an average of $n$ nonconvex functions. We propose a new stochastic gradient descent algorithm based on nested variance reduction. Compared with conventional stochastic variance reduced gradient (SVRG) algorithm that uses two reference points to construct a semi-stochastic gradient with diminishing variance in each epoch, our algorithm uses $K+1$ nested reference points to build an semi-stochastic gradient to further reduce its variance in each epoch. For smooth functions, the proposed algorithm converges to an approximate first order stationary point (i.e., $\|\nabla F(\xb)\|_2\leq \epsilon$) within $\tO(n\land \epsilon^{-2}+\epsilon^{-3}\land n^{1/2}\epsilon^{-2})$\footnote{$\tO(\cdot)$ hides the logarithmic factors} number of stochastic gradient evaluations, where $n$ is the number of component functions, and $\epsilon$ is the optimization error. This improves the best known gradient complexity of SVRG $O(n+n^{2/3}\epsilon^{-2})$ and the best gradient complexity of SCSG $O(\epsilon^{-5/3}\land n^{2/3}\epsilon^{-2})$. For gradient dominated functions, our algorithm achieves $\tO(n\land \tau\epsilon^{-1}+\tau\cdot (n^{1/2}\land (\tau\epsilon^{-1})^{1/2})$ gradient complexity, which again beats the existing best gradient complexity $\tO(n\land \tau\epsilon^{-1}+\tau\cdot (n^{1/2}\land (\tau\epsilon^{-1})^{2/3})$ achieved by SCSG. Thorough experimental results on different nonconvex optimization problems back up our theory.
Exploring Relations in Untrimmed Videos for Self-Supervised Learning

Existing video self-supervised learning methods mainly rely on trimmed videos for model training. They apply their methods and verify the effectiveness on trimmed video datasets including UCF101 and Kinetics-400, among others. However, trimmed datasets are manually annotated from untrimmed videos. In this sense, these methods are not truly unsupervised. In this article, we propose a novel self-supervised method, referred to as Exploring Relations in Untrimmed Videos (ERUV), which can be straightforwardly applied to untrimmed videos (real unlabeled) to learn spatio-temporal features. ERUV first generates single-shot videos by shot change detection. After that, some designed sampling strategies are used to model relations for video clips. The strategies are saved as our self-supervision signals. Finally, the network learns representations by predicting the category of relations between the video clips. ERUV is able to compare the differences and similarities of video clips, which is also an essential procedure for video-related tasks. We validate our learned models with action recognition, video retrieval, and action similarity labeling tasks with four kinds of 3D convolutional neural networks. Experimental results show that ERUV is able to learn richer representations with untrimmed videos, and it outperforms state-of-the-art self-supervised methods with significant margins.
Image denoising by transfer learning of generative adversarial network for dental CT

The successful development of the image denoising techniques for low-dose computed tomography (LDCT) was largely owing to the public-domain availability of spatially-aligned high- and low-dose CT image pairs. Even though low-dose CT scans are also highly desired in dental imaging, public-domain databases of dental CT image pairs have not been established yet. In this paper, we propose a dental CT image denoising method based on the transfer learning of a generative adversarial network (GAN) from the public-domain CT images. We trained a generative adversarial network with the Wasserstein loss function (WGAN) using 5,100 high- and low-dose medical CT image pairs of human chest and abdomen. For the generative network of GAN, we used the U-net structure of five stages to exploit its high computational efficiency. After training the proposed network, named U-WGAN, we fine-tuned the network with 3,006 dental CT image pairs of two different human skull phantoms. For the high- and low-dose scans of the phantoms, we set the tube current of the dental CT to 10 mA and 4 mA, respectively, with the tube voltage set to 90 kVp in both scans. We applied the trained network to denoising of low-dose dental CT images of dental phantoms and adult humans. The U-net processed images showed over-smoothing effects even though U-net had a good performance in the quantitative metrics. U-WGAN showed similar denoising performance to WGAN, but it reduced the computation time of WGAN by a factor of 10. The fine-tuning procedure in the transfer learning scheme enhanced the network performance in terms of the quantitative metrics, and it also improved visual appearance of the processed images. Even though the number of fine-tuning images was very limited in this study, we think the transfer learning scheme can be a good option for developing deep learning networks for dental CT image denoising.
An Efficient Privacy-Preserving Multi-Keyword Query Scheme in Location Based Services

With the proliferation of location-aware mobile devices and the prevalence of wireless communications, location-based services (LBS) have attracted much particular attention in recent years.For flexibility and cost savings, the LBS provider outsources the LBS data to the cloud in order to serve the increasing number of mobile users.To guarantee users' privacy and data confidentiality, some excellent works have been proposed which focus on secure query over the location server.However, these existing works have two limitations.On the one hand, they cannot preserve users' location and query content privacy simultaneously.On the other hand, they fail to support multi-keyword queries.In this article, aiming at a multi-keywords query in LBS, we propose a novel efficient and privacy-preserving multi-keyword query scheme (PPMQ) over the outsourced cloud, which satisfies the requirements of the location and query content privacy protection, query efficiency, the confidentiality of LBS data and scalability regarding the data users.To improve the efficiency of our proposed scheme, we utilize the linear quad-tree technique to build a grid system to represent the location information in the query condition as well as a searchable index.To protect the location privacy, we combine decimal Morton code and public-key cryptography techniques to build a searchable index or to generate a trapdoor.To enable the cloud server to perform a secure multi-keyword query, we systematically construct a privacy-preserving query scheme with bilinear pairing-based cryptography.In particular, our proposed scheme is scalable and very suitable for multi-user environments due to the flexible user registration and revocation mechanisms.Furthermore, a detailed security analysis shows that the proposed scheme can ensure the confidentiality of LBS data, and protect the location and query content privacy.Extensive experiments are conducted on a real LBS dataset, and the simulation results confirm the security and efficiency of our scheme.
Lightning-fast and privacy-preserving outsourced computation in the cloud

Abstract In this paper, we propose a framework for lightning-fast privacy-preserving outsourced computation framework in the cloud, which we refer to as LightCom. Using LightCom, a user can securely achieve the outsource data storage and fast, secure data processing in a single cloud server different from the existing multi-server outsourced computation model. Specifically, we first present a general secure computation framework for LightCom under the cloud server equipped with multiple Trusted Processing Units (TPUs), which face the side-channel attack. Under the LightCom, we design two specified fast processing toolkits, which allow the user to achieve the commonly-used secure integer computation and secure floating-point computation against the side-channel information leakage of TPUs, respectively. Furthermore, our LightCom can also guarantee access pattern protection during the data processing and achieve private user information retrieve after the computation. We prove that the proposed LightCom can successfully achieve the goal of single cloud outsourced data processing to avoid the extra computation server and trusted computation server, and demonstrate the utility and the efficiency of LightCom using simulations.
Human 3D pose estimation in a lying position by RGB-D images for medical diagnosis and rehabilitation

Posture recognition in the human lying position is of great significance for the rehabilitation evaluation of lying patients and the diagnosis of infants with early cerebral palsy. In this paper, we proposed a novel method for human 3D pose estimation in a lying position with the RGB image and corresponding depth information. Firstly, we employ current pose estimation method on RGB images to achieve the human full body 2D keypoints. By combining the depth information and coordinate transformation, the 3D movement of human in lying position can be obtained. We validate our method with two public datasets. The results show that the accuracy can reach the state-of-the-art.
On the Cryptographic Deniability of the Signal Protocol

Offline deniability is the ability to a posteriori deny having participated in a particular communication session. This property has been widely assumed for the Signal messaging application, yet no formal proof has appeared in the literature. In this paper, we present what we believe is the first formal study of the offline deniability of the Signal protocol. Our analysis shows that building a deniability proof for Signal is non-trivial and requires very strong assumptions on the underlying mathematical groups where the protocol is run.
Activity recognition for locomotion and transportation dataset using deep learning

Team "DL_Lock": The Sussex-Huawei Locomotion-Transportation (SHL) recognition challenge 2020 poses a unique opportunity to work on a broad, real-life dataset to classify transport-related activities in a user and location-independent manner. Since deep learning architectures have now received great attention on achieving promising results on time series classification tasks, we focused our experiments on some recent state-of-the-art deep learning architectures such as CNN, Resnet, and InceptionTime. A considerable amount of time was spent on the preprocessing pipeline, which turned out to be a critical phase that impacted most of the results. At the end and after many experiments and hyperparameter tuning, we were able to achieve a 79% F1 score on the validation dataset using InceptionTime architecture. The objective of this paper is to present the technical description of the Machine Learning processing pipeline, the algorithms used, and the results achieved during the development/training phase.
RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms

Pre-trained language models (PTLMs) have achieved impressive performance on commonsense inference benchmarks, but their ability to employ commonsense to make robust inferences, which is crucial for effective communications with humans, is debated.In the pursuit of advancing fluid human-AI communication, we propose a new challenge, RICA: Robust Inference using Commonsense Axioms, that evaluates robust commonsense inference despite textual perturbations.To generate data for this challenge, we develop a systematic and scalable procedure using commonsense knowledge bases and probe PTLMs across two different evaluation settings.Extensive experiments on our generated probe sets with more than 10k statements show that PTLMs perform no better than random guessing on the zero-shot setting, are heavily impacted by statistical biases, and are not robust to perturbation attacks.We also find that fine-tuning on similar statements offer limited gains, as PTLMs still fail to generalize to unseen inferences.Our new large-scale benchmark exposes a significant gap between PTLMs and human-level language understanding and offers a new challenge for PTLMs to demonstrate commonsense. 1 Logical Template
SPATM: A Social Period-Aware Topic Model for Personalized Venue Recommendation

Personalized venues recommendation is essential to help people find attractive venue to visit as growth of location-based social networks. Existing approaches never distinguish user individual interests from her social preferences, which leads to a bottleneck of modeling user check-in behaviors accurately. In this paper, we find the differences between user interests and her social preferences clearly and investigate the time law of user check-in behaviors in depth. Consequently, we propose a social-period-aware topic model (SPATM) to learn the influence weights of both user interests and her social preferences on making-decision for each check-in time automatically. Especially, we model latent topic by leveraging smaller size of dynamic activities instead of static categories, which can alleviate the data sparsity problem by using more co-occurrent activities information. Moreover, our approach can automatically judge whether a user’s social preference is periodic or aperiodic and learn the periodicity of periodic one. Furthermore, the Alias Sampling based training approach is introduced to improve sampling efficiency. The results demonstrate our proposed model is effective and outperforms the state-of-the-art approaches in terms of effectiveness and efficiency. Besides, SPATM can learn semantically coherent latent topics and geographically dispersed latent social topics which are useful to explain recommendation.
An Empirical Study on Clarifying Question-Based Systems

Search and recommender systems that take the initiative to ask clarifying questions to better understand users' information needs are receiving increasing attention from the research community. However, to the best of our knowledge, there is no empirical study to quantify whether and to what extent users are willing or able to answer these questions. In this work, we conduct an online experiment by deploying an experimental system, which interacts with users by asking clarifying questions against a product repository. We collect both implicit interaction behavior data and explicit feedback from users showing that: (a) users are willing to answer a good number of clarifying questions (11 on average), but not many more than that; (b) most users answer questions until they reach the target product, but also a fraction of them stops due to fatigue or due to receiving irrelevant questions; (c) part of the users' answers (17%) are actually opposite to the description of the target product; while (d) most of the users (84%) find the question-based system helpful towards completing their tasks. Some of the findings of the study contradict current assumptions on simulated evaluations in the field, while they point towards improvements in the evaluation framework and can inspire future interactive search/recommender system designs.
Ultrasound-Based Articulatory-to-Acoustic Mapping with WaveGlow Speech Synthesis

For articulatory-to-acoustic mapping using deep neural networks, typically spectral and excitation parameters of vocoders have been used as the training targets.However, vocoding often results in buzzy and muffled final speech quality.Therefore, in this paper on ultrasound-based articulatory-to-acoustic conversion, we use a flow-based neural vocoder (WaveGlow) pre-trained on a large amount of English and Hungarian speech data.The inputs of the convolutional neural network are ultrasound tongue images.The training target is the 80-dimensional mel-spectrogram, which results in a finer detailed spectral representation than the previously used 25-dimensional Mel-Generalized Cepstrum.From the output of the ultrasoundto-mel-spectrogram prediction, WaveGlow inference results in synthesized speech.We compare the proposed WaveGlowbased system with a continuous vocoder which does not use strict voiced/unvoiced decision when predicting F0.The results demonstrate that during the articulatory-to-acoustic mapping experiments, the WaveGlow neural vocoder produces significantly more natural synthesized speech than the baseline system.Besides, the advantage of WaveGlow is that F0 is included in the mel-spectrogram representation, and it is not necessary to predict the excitation separately.
Quantum process tomography of a high-dimensional quantum communication channel

The characterization of quantum processes, e.g. communication channels, is an essential ingredient for establishing quantum information systems. For quantum key distribution protocols, the amount of overall noise in the channel determines the rate at which secret bits are distributed between authorized partners. In particular, tomographic protocols allow for the full reconstruction, and thus characterization, of the channel. Here, we perform quantum process tomography of high-dimensional quantum communication channels with dimensions ranging from 2 to 5. We can thus explicitly demonstrate the effect of an eavesdropper performing an optimal cloning attack or an intercept-resend attack during a quantum cryptographic protocol. Moreover, our study shows that quantum process tomography enables a more detailed understanding of the channel conditions compared to a coarse-grained measure, such as quantum bit error rates. This full characterization technique allows us to optimize the performance of quantum key distribution under asymmetric experimental conditions, which is particularly useful when considering high-dimensional encoding schemes.
The Audio Auditor: User-Level Membership Inference in Internet of Things Voice Services

Abstract With the rapid development of deep learning techniques, the popularity of voice services implemented on various Internet of Things (IoT) devices is ever increasing. In this paper, we examine user-level membership inference in the problem space of voice services, by designing an audio auditor to verify whether a specific user had unwillingly contributed audio used to train an automatic speech recognition (ASR) model under strict black-box access. With user representation of the input audio data and their corresponding translated text, our trained auditor is effective in user-level audit. We also observe that the auditor trained on specific data can be generalized well regardless of the ASR model architecture. We validate the auditor on ASR models trained with LSTM, RNNs, and GRU algorithms on two state-of-the-art pipelines, the hybrid ASR system and the end-to-end ASR system. Finally, we conduct a real-world trial of our auditor on iPhone Siri, achieving an overall accuracy exceeding 80%. We hope the methodology developed in this paper and findings can inform privacy advocates to overhaul IoT privacy.
Detecting Covert Cryptomining Using HPC

Cybercriminals have been exploiting cryptocurrencies to commit various unique financial frauds. Covert cryptomining - which is defined as an unauthorized harnessing of victims’ computational resources to mine cryptocurrencies - is one of the prevalent ways nowadays used by cybercriminals to earn financial benefits. Such exploitation of resources causes financial losses to the victims. In this paper, we present our efficient approach to detect covert cryptomining on users’ machine. Our solution is a generic solution that, unlike currently available solutions to detect covert cryptomining, is not tailored to a specific cryptocurrency or a particular form of cryptomining. In particular, we focus on the core mining algorithms and utilize Hardware Performance Counters (HPC) to create clean signatures that grasp the execution pattern of these algorithms on a processor. We built a complete implementation of our solution employing advanced machine learning techniques. We evaluated our methodology on two different processors through an exhaustive set of experiments. In our experiments, we considered all the cryptocurrencies mined by the top-10 mining pools, which collectively represent the largest share of the cryptomining market. Our results show that our classifier can achieve a near-perfect classification with samples of length as low as five seconds. Due to its robust and practical design, our solution can even adapt to zero-day cryptocurrencies. Finally, we believe our solution is scalable and can be deployed to tackle the uprising problem of covert cryptomining.