Reviewerly/exHarmony huggingface dataset:
Simulatable Verifiable Random Function from the Lwe Assumption

A verifiable random function (VRF) is a pseudorandom function F that can be publicly verified. A simulatable VRF (sVRF) is an important variant of a VRF, which additionally provides simulatability. Informally, the simulatability of a VRF depicts the ability to simulate a valid proof \pi that y = F(sk, x) for any input x and any output value y. A (simulatable) VRF can be used in the E-Cash, E-Lottery, blockchain and constructing the multi-theorem non-interactive zero-knowledge (NIZK) proof. However, up to now, the existing constructions of an sVRF either rely on non-standard assumptions (e.g., the Q-type ones), or are built in the random oracle model, or resort to time-consuming techniques like the Cook-Levin reduction.In this paper, we design the first sVRF from the LWE assumption in the standard model (free of a random oracle) without using a Cook-Levin reduction. In our construction of an sVRF, we take as building blocks a pseudorandom function, a trapdoor fully homomorphic commitment (FHC) scheme, and a NIZK proof system for a language specified by FHC. Our trapdoor FHC is the key technical tool, which helps the simplification of the underlying NIZK language, thus making possible an instantiation of a NIZK proof from LWE without a Cook-Levin reduction. Together with an LWE-based PRF, we obtain an sVRF scheme from LWE.
ANT: Adapt Network Across Time for Efficient Video Processing

Abundant redundancies exist in video streams, thereby pointing to opportunities to save computations. Towards this end, we propose the Adaptive Network across Time (ANT) framework to harness these redundancies for reducing the computational cost of video processing. Unlike most dynamic networks that adapt their structures to different static inputs, our method adapts networks along the temporal dimension. By inspecting the semantic differences between frames, the proposed ANT chooses a purpose-fit network at test time to reduce overall computation, i.e., switching to a smaller network when observing mild differences. The proposed ANT adapts the structured networks within a supernet, making it hardware-friendly and there-fore achieves actual acceleration in real-world scenarios. The proposed ANT is powered by (1) a fusion module that utilizes the past features and (2) a dynamic gate to adjust the network in a predictive fashion with negligible extra cost. To ensure the generality of each subnet and the gate’s fairness, we propose a two-stage training scheme. We first train a weight-sharing supernet and then jointly train fusion modules and gates. Evaluation of the video detection task with the modern EfficientDet reveals the effectiveness of our approach.
Neural Face Video Compression using Multiple Views

Recent advances in deep generative models led to the development of neural face video compression codecs that use an order of magnitude less bandwidth than engineered codecs. These neural codecs reconstruct the current frame by warping a source frame and using a generative model to compensate for imperfections in the warped source frame. Thereby, the warp is encoded and transmitted using a small number of keypoints rather than a dense flow field, which leads to massive savings compared to traditional codecs. However, by relying on a single source frame only, these methods lead to inaccurate reconstructions (e.g. one side of the head becomes unoccluded when turning the head and has to be synthesized). Here, we aim to tackle this issue by relying on multiple source frames (views of the face) and present encouraging results.
Semi-Supervised Few-Shot Learning from A Dependency-Discriminant Perspective

We study the few-shot learning (FSL) problem, where a model learns to recognize new objects with extremely few labeled training data per category. Most of previous FSL approaches resort to the meta-learning paradigm, where the model accumulates inductive bias through learning from many training tasks, in order to solve new unseen few-shot tasks. In contrast, we propose a simple semi-supervised FSL approach to exploit unlabeled data accompanying the few-shot task to improve FSL performance. More exactly, to train a classifier, we propose a Dependency Maximization loss based on the Hilbert-Schmidt norm of the cross-covariance operator, which maximizes the statistical dependency between the embedded feature of the unlabeled data and their label predictions, together with the supervised loss over the support set. The obtained classifier is used to infer the pseudo-labels of the unlabeled data. Furthermore, we propose an Instance Discriminant Analysis to evaluate the credibility of the pseudo-labeled examples and select the faithful ones into an augmented support set, which is used to retrain the classifier. We iterate the process until the pseudo-labels of the unlabeled data becomes stable. Through extensive experiments on four widely used few-shot classification benchmarks, including mini-ImageNet, tiered-ImageNet, CUB, and CIFARFS, the proposed method outperforms previous state-of-the-art FSL methods.
Do What You Can, With What You Have: Scale-aware and High Quality Monocular Depth Estimation Without Real World Labels

Learning robust and scale-aware monocular depth estimation (MDE) requires expensive data annotation efforts. Self-supervised approaches use unlabelled videos but, due to ambiguous photometric reprojection loss and no labelled supervision, produce inferior quality relative (scale ambiguous) depth maps with over-smoothed object boundaries. Approaches using synthetic training data suffer from the non-trivial domain adaptation problem; despite complicated unsupervised domain adaptation (UDA) techniques, these methods still do not generalize well to real datasets.This work presents a novel and effective training methodology to combine self-supervision from unlabelled monocular videos and dense supervision from the synthetic dataset synergistically without complicated UDA techniques. With our method, geometry and semantics are learned from monocular videos, whereas scale-awareness and qualitative attributes, e.g., sharp and smooth depth variations, that are crucial for practical use cases are learned from the synthetic dataset. Our method outperforms self-supervised, semi-supervised, and all the domain adaptation methods on standard benchmark datasets while being competitive with fully supervised methods.Furthermore, our method leads to qualitatively superior depth maps, which increases its practical utility compared to existing methods. We demonstrate this by applying our method to develop an MDE model for a real life application—DSLR-like shallow depth-of-field effect on smartphones. The new high quality synthetic depth dataset that we generate for this task will be available to the community.
CDAD: A Common Daily Action Dataset with Collected Hard Negative Samples

The research on action understanding has achieved significant progress with the establishment of various benchmark datasets. However, the results of action understanding are far from satisfactory in practice. One reason is that the existing action datasets ignore the existence of many hard negative samples in real-world scenarios, which are usually undefined confusion actions, e.g., holding a pen near the mouth vs. smoking. In this work, we focus on the common actions in our daily life and present a novel Common Daily Action Dataset (CDAD), which consists of 57,824 video clips of 23 well-defined common daily actions with rich manual annotations. Particularly, for each daily action, we collect not only diverse positive samples but also various hard negative samples that have minor differences (share similarities) in action with the positive ones. The established CDAD dataset could not only serve as a benchmark for several important daily action understanding tasks, including multi-label action recognition, temporal action localization, and spatial-temporal action detection, but also provide a testbed for researchers to investigate the influence of highly similar negative samples in learning action understanding models. Datasets and codes are available: https://github.com/MartinXM/CDAD.
Unstructured Object Matching using Co-Salient Region Segmentation

Unstructured object matching is a less-explored and very challenging topic in the scientific literature. This includes matching scenarios where the context, appearance and the geometrical integrity of the objects to be matched changes drastically from one image to another (e.g. a pair of pyjamas which in one image is folded and in the other is worn by a person), making it impossible to determine a transformation which aligns the matched regions. Traditional approaches like keypoint-based feature matching perform poorly on this use case due to the high complexity in terms of viewpoint, scene context variety, background variations or high degrees of freedom concerning structural configurations. In this paper we propose a deep learning framework consisting of a twins based matching approach leveraging a co-salient region segmentation task and a cosine-similarity based region descriptor pairing technique. The importance of our proposed framework is demonstrated on a novel use case consisting of image pairs with various objects used by children. Additionally, we evaluate on Human3.6M and Market-1501, two datasets with humans depicting various appearances and kinematic configurations captured under different backgrounds.
CSG0: Continual Urban Scene Generation with Zero Forgetting

With the rapid advances in generative adversarial networks (GANs), the visual quality of synthesised scenes keeps improving, including for complex urban scenes with applications to automated driving. We address in this work a continual scene generation setup in which GANs are trained on a stream of distinct domains; ideally, the learned models should eventually be able to generate new scenes in all seen domains. This setup reflects the real-life scenario where data are continuously acquired in different places at different times. In such a continual setup, we aim for learning with zero forgetting, i.e., with no degradation in synthesis quality over earlier domains due to catastrophic forgetting. To this end, we introduce a novel framework that not only (i) enables seamless knowledge transfer in continual training but also (ii) guarantees zero forgetting with a small overhead cost. While being more memory efficient, thanks to continual learning, our model obtains better synthesis quality as compared against the brute-force solution that trains one full model for each domain. Especially, under extreme low-data regimes, our approach outperforms the brute-force one by a large margin.
Object Prior Embedded Network for Query-Agnostic Image Retrieval

The Text-to-Image retrieval task plays an important role in bridging the gap between vision and language modalities. This task is challenging and far from being solved, because of the large visual-semantic discrepancy between language and vision. Recent studies on vision-language contrastive learning have shown that it can effectively learn good representations from massive image-text pairs. However, most existing methods simply concatenate image and text features as input and resort to the deep network to learn the visual-semantic relationship between image and text in a brute force manner. The insufficient alignments information pose a challenging weakly-supervised learning task, and results in only limited accuracy in previous methods. Motivated by the observation that the salient objects in an image can be accurately detected and are often mentioned in the paired text, in this paper, we propose a novel cross-attention transformer that uses objects detected in image as anchor points and prior to significantly ease the learning of image-text alignments, and thus boost the text-to-image search accuracy. In addition, unlike the query-dependent architectures adopted by most previous methods, our proposed method is query-agnostic and is thus significantly faster in the inference process. The extensive experiments on Flickr30K and MSCOCO captions datasets demonstrate that our proposed method can outperform the SOTA method while preserving the inference efficiency.
Towards Detailed Characteristic-Preserving Virtual Try-On

While virtual try-on has rapidly progressed recently, existing virtual try-on methods still struggle to faithfully represent various details of the clothes when worn. In this paper, we propose a simple yet effective method to better preserve details of the clothing and person by introducing an additional fitting step after geometric warping. This minimal modification enables disentangling representations of the clothing from the wearer, hence we are able to preserve the wearer-agnostic structure and details of the clothing, to fit a garment naturally to a variety of poses and body shapes. Moreover, we propose a novel evaluation framework applicable to any metric, to better reflect the semantics of clothes fitting. From extensive experiments, we empirically verify that the proposed method not only learns to disentangle clothing from the wearer, but also preserves details of the clothing on the try-on results.
Active Object Detection with Epistemic Uncertainty and Hierarchical Information Aggregation

Despite the huge success of object detection, the training process still requires an immense amount of labeled data. Active learning has been proposed as a practical solution, but existing works on active object detection do not utilize the concept of epistemic uncertainty, which is an important metric for capturing the usefulness of the sample. Previous works also pay little attention to the relation between bounding boxes when computing the informativeness of an image. In this paper, we propose a new active object detection strategy that improves these two shortcomings of existing methods. We specifically consider a Bayesian framework and propose a new module termed model evidence head (MEH), to take advantage of epistemic uncertainty in object detection. We also propose hierarchical uncertainty aggregation (HUA), which realigns all bounding boxes into multiple levels and aggregates uncertainties in a bottom-up order, to compute the informativeness of an image. Experimental results show that our method outperforms existing state-of-the-art methods by a considerable margin.
Is Neuron Coverage Needed to Make Person Detection More Robust?

The growing use of deep neural networks (DNNs) in safety- and security-critical areas like autonomous driving raises the need for their systematic testing. Coverage-guided testing (CGT) is an approach that applies mutation or fuzzing according to a predefined coverage metric to find inputs that cause misbehavior. With the introduction of a neuron coverage metric, CGT has also recently been applied to DNNs. In this work, we apply CGT to the task of person detection in crowded scenes. The proposed pipeline uses YOLOv3 for person detection and includes finding DNN bugs via sampling and mutation, and subsequent DNN retraining on the updated training set. To be a bug, we require a mutated image to cause a significant performance drop compared to a clean input. In accordance with the CGT, we also consider an additional requirement of increased coverage in the bug definition. In order to explore several types of robustness, our approach includes natural image transformations, corruptions, and adversarial examples generated with the Daedalus attack. The proposed framework has uncovered several thousand cases of incorrect DNN behavior. The relative change in mAP performance of the retrained models reached on average between 26.21% and 64.24% for different robustness types. However, we have found no evidence that the investigated coverage metrics can be advantageously used to improve robustness.
Robustness and Adaptation to Hidden Factors of Variation

We tackle here a specific, still not widely addressed aspect, of AI robustness, which consists of seeking invariance / insensitivity of model performance to hidden factors of variations in the data. Towards this end, we employ a two step strategy that a) does unsupervised discovery, via generative models, of sensitive factors that cause models to under-perform, and b) intervenes models to make their performance invariant to these sensitive factors’ influence. We consider 3 separate interventions for robustness, including: data augmentation, semantic consistency, and adversarial alignment. We evaluate our method using metrics that measure trade offs between invariance (insensitivity) and overall performance (utility) and show the benefits of our method for 3 settings (unsupervised, semi-supervised and generalization).
Perceptual in-Loop Filter for Image and Video Compression

In this paper, we introduce our hybrid image and video compression scheme enhanced by CNN-optimized in-loop filter. Specifically, a Structure Preserving in-Loop Filter (SPiLF) is incorporated in the hybrid video codec Enhanced Compression Model (ECM), where two branches, i.e., gradient branch and pixel branch, are developed based on the dense residual unit (DRU). To provide pleasant visual quality, the Generative adversarial networks (GAN) loss and LPIPS loss are further considered. Therefore, the proposal is mainly focusing on perceptual-friendly image compression for human vision, whilst video compression could be further investigated. The experiments show that the proposed method achieves advanced visual quality when compared to the traditional methods.
PhoneDepth: A Dataset for Monocular Depth Estimation on Mobile Devices

Monocular depth estimation has been studied as a classic and learning based computer vision problem for decades. However, little attention received the efficiency and the deployment of methods on mobile hardware. All publicly available datasets have severe limitations related to their applicability to camera data captured with real mobile devices. For instance, the main issues with current datasets include (but not exhaustively) low quality of images due the cameras or collection methods, domain specifically generated datasets as is the case for autonomous driving, small number of samples, sparse depthmaps, etc. In response, we introduce PhoneDepth, a novel dataset that aims to take advantage of modern phones hardware and professional stereo cameras. Depthmaps are acquired from three sources: Time of Flight sensor, Dual Pixel sensor and stereo camera; while the images correspond to mobile phone photos. We prove its high value by training neural networks with multiple depth supervision, fine-tuning on other datasets and for depth refinement. Along with the dataset we present benchmark models and a toolbox to facilitate the dataset usage.
PEA: Improving the Performance of ReLU Networks for Free by Using Progressive Ensemble Activations

In recent years novel activation functions have been proposed to improve the performance of neural networks, and they show superior performance compared to the ReLU counterpart. However, there are environments, where the availability of complex activations is limited, and usually only the ReLU is supported. In this paper we propose methods that can be used to improve the performance of ReLU networks by using these efficient novel activations during model training. More specifically, we propose ensemble activations that are composed of the ReLU and one of these novel activations. Furthermore, the coefficients of the ensemble are neither fixed nor learned, but are progressively updated during the training process in a way that by the end of the training only the ReLU activations remain active in the network and the other activations can be removed. This means that in inference time the network contains ReLU activations only. We perform extensive evaluations on the ImageNet classification task using various compact network architectures and various novel activation functions. Results show 0.2–0.8% top-1 accuracy gain, which confirms the applicability of the proposed methods. Furthermore, we demonstrate the proposed methods on semantic segmentation and we boost the performance of a compact segmentation network by 0.34% mIOU on the Cityscapes dataset.
Why Object Detectors Fail: Investigating the Influence of the Dataset

A false negative in object detection describes an object that was not correctly localised and classified by a detector. In prior work, we introduced five ‘false negative mechanisms’ that identify the specific component inside the detector architecture that failed to detect the object. Using these mechanisms, we explore how different computer vision datasets and their inherent characteristics can influence object detector failures. Specifically, we investigate the false negative mechanisms of Faster R-CNN and RetinaNet across five computer vision datasets, namely Microsoft COCO, Pascal VOC, ExDark, ObjectNet, and COD10K. Our results show that object size and class influence the false negative mechanisms of object detectors. We also show that comparing the false negative mechanisms of a single object class across different datasets can highlight potentially unknown biases in datasets.
Can the Mathematical Correctness of Object Configurations Affect the Accuracy of Their Perception?

We investigate a new type of dataset bias based on the mathematical correctness of object configurations in visual scenes, and how this bias can affect the accuracy of computer vision models. Our experiments demonstrate how CNNs trained to detect and recognize individual objects are capable of implicitly learning simple mathematical relationships between them directly from pixel data; moreover, models that are trained with a dataset bias (e.g., all examples are mathematically correct) can suffer in performance when evaluated on test data without this bias. We found evidence for this effect in two settings: (1) object detection of math symbols in images of arithmetic expressions, and (2) object detection of moving particles from images produced by a physics simulator. Importantly, the semantic bias that we study is based not just on simple co-occurrence patterns in each image, but rather on higher-order semantic rules that generalize to unique combinations of objects not seen during training. While the magnitude of the effect was small, the accuracy difference was statistically reliable.
Poisons that are learned faster are more effective

Imperceptible poisoning attacks on entire datasets have recently been touted as methods for protecting data privacy. However, among a number of defenses preventing the practical use of these techniques, early-stopping stands out as a simple, yet effective defense. To gauge poisons’ vulnerability to early-stopping, we benchmark error-minimizing, error-maximizing, and synthetic poisons in terms of peak test accuracy over 100 epochs and make a number of surprising observations. First, we find that poisons that reach a low training loss faster have lower peak test accuracy. Second, we find that a current state-of-the-art error-maximizing poison is 7× less effective when poison training is stopped at epoch 8. Third, we find that stronger, more transferable adversarial attacks do not make stronger poisons. We advocate for evaluating poisons in terms of peak test accuracy.