OpenRaiser/Intern-Atlas huggingface dataset:
Uncertainty in Contrastive Learning: On the Predictability of Downstream Performance
Venue: arXiv cs.LG 2022
Year: 2022

The superior performance of some of today's state-of-the-art deep learning models is to some extent owed to extensive (self-)supervised contrastive pretraining on large-scale datasets. In contrastive learning, the network is presented with pairs of positive (similar) and negative (dissimilar) datapoints and is trained to find an embedding vector for each datapoint, i.e., a representation, which can be further fine-tuned for various downstream tasks. In order to safely deploy these models in critical decision-making systems, it is crucial to equip them with a measure of their uncertainty or reliability. However, due to the pairwise nature of training a contrastive model, and the lack of absolute labels on the output (an abstract embedding vector), adapting conventional uncertainty estimation techniques to such models is non-trivial. In this work, we study whether the uncertainty of such a representation can be quantified for a single datapoint in a meaningful way. In other words, we explore if the downstream performance on a given datapoint is predictable, directly from its pre-trained embedding. We show that this goal can be achieved by directly estimating the distribution of the training data in the embedding space and accounting for the local consistency of the representations. Our experiments show that this notion of uncertainty for an embedding vector often strongly correlates with its downstream accuracy.
A coherence parameter characterizing generative compressed sensing with Fourier measurements
Venue: arXiv cs.IT 2022
Year: 2022

In Bora et al. (2017), a mathematical framework was developed for compressed sensing guarantees in the setting where the measurement matrix is Gaussian and the signal structure is the range of a generative neural network (GNN). The problem of compressed sensing with GNNs has since been extensively analyzed when the measurement matrix and/or network weights follow a subgaussian distribution. We move beyond the subgaussian assumption, to measurement matrices that are derived by sampling uniformly at random rows of a unitary matrix (including subsampled Fourier measurements as a special case). Specifically, we prove the first known restricted isometry guarantee for generative compressed sensing with subsampled isometries and provide recovery bounds, addressing an open problem of Scarlett et al. (2022, p. 10). Recovery efficacy is characterized by the coherence, a new parameter, which measures the interplay between the range of the network and the measurement matrix. Our approach relies on subspace counting arguments and ideas central to high-dimensional probability. Furthermore, we propose a regularization strategy for training GNNs to have favourable coherence with the measurement operator. We provide compelling numerical simulations that support this regularized training strategy: our strategy yields low coherence networks that require fewer measurements for signal recovery. This, together with our theoretical results, supports coherence as a natural quantity for characterizing generative compressed sensing with subsampled isometries.
Cycle Encoding of a StyleGAN Encoder for Improved Reconstruction and Editability
Venue: arXiv cs.CV 2022
Year: 2022

GAN inversion aims to invert an input image into the latent space of a pre-trained GAN. Despite the recent advances in GAN inversion, there remain challenges to mitigate the tradeoff between distortion and editability, i.e. reconstructing the input image accurately and editing the inverted image with a small visual quality drop. The recently proposed pivotal tuning model makes significant progress towards reconstruction and editability, by using a two-step approach that first inverts the input image into a latent code, called pivot code, and then alters the generator so that the input image can be accurately mapped into the pivot code. Here, we show that both reconstruction and editability can be improved by a proper design of the pivot code. We present a simple yet effective method, named cycle encoding, for a high-quality pivot code. The key idea of our method is to progressively train an encoder in varying spaces according to a cycle scheme: W->W+->W. This training methodology preserves the properties of both W and W+ spaces, i.e. high editability of W and low distortion of W+. To further decrease the distortion, we also propose to refine the pivot code with an optimization-based method, where a regularization term is introduced to reduce the degradation in editability. Qualitative and quantitative comparisons to several state-of-the-art methods demonstrate the superiority of our approach.
Polynomial Threshold Functions for Decision Lists
Venue: arXiv cs.CC 2022
Year: 2022

For $S \subseteq \{0,1\}^n$ a Boolean function $f \colon S \to \{-1,1\}$ is a polynomial threshold function (PTF) of degree $d$ and weight $W$ if there is a polynomial $p$ with integer coefficients of degree $d$ and with sum of absolute coefficients $W$ such that $f(x) = \text{sign}(p(x))$ for all $x \in S$. We study a representation of decision lists as PTFs over Boolean cubes $\{0,1\}^n$ and over Hamming balls $\{0,1\}^{n}_{\leq k}$. As our first result, we show that for all $d = O\left( \left( \frac{n}{\log n}\right)^{1/3}\right)$ any decision list over $\{0,1\}^n$ can be represented by a PTF of degree $d$ and weight $2^{O(n/d^2)}$. This improves the result by Klivans and Servedio [Klivans, Servedio, 2006] by a $\log^2 d$ factor in the exponent of the weight. Our bound is tight for all $d = O\left( \left( \frac{n}{\log n}\right)^{1/3}\right)$ due to the matching lower bound by Beigel [Beigel, 1994]. For decision lists over a Hamming ball $\{0,1\}^n_{\leq k}$ we show that the upper bound on weight above can be drastically improved to $n^{O(\sqrt{k})}$ for $d = Θ(\sqrt{k})$. We also show that similar improvement is not possible for smaller degrees by proving the lower bound $W = 2^{Ω(n/d^2)}$ for all $d = O(\sqrt{k})$. \end{abstract}
Neural Greedy Pursuit for Feature Selection
Venue: arXiv cs.LG 2022
Year: 2022

We propose a greedy algorithm to select $N$ important features among $P$ input features for a non-linear prediction problem. The features are selected one by one sequentially, in an iterative loss minimization procedure. We use neural networks as predictors in the algorithm to compute the loss and hence, we refer to our method as neural greedy pursuit (NGP). NGP is efficient in selecting $N$ features when $N \ll P$, and it provides a notion of feature importance in a descending order following the sequential selection procedure. We experimentally show that NGP provides better performance than several feature selection methods such as DeepLIFT and Drop-one-out loss. In addition, we experimentally show a phase transition behavior in which perfect selection of all $N$ features without false positives is possible when the training data size exceeds a threshold.
Bounding generalization error with input compression: An empirical study with infinite-width networks
Venue: arXiv cs.LG 2022
Year: 2022

Estimating the Generalization Error (GE) of Deep Neural Networks (DNNs) is an important task that often relies on availability of held-out data. The ability to better predict GE based on a single training set may yield overarching DNN design principles to reduce a reliance on trial-and-error, along with other performance assessment advantages. In search of a quantity relevant to GE, we investigate the Mutual Information (MI) between the input and final layer representations, using the infinite-width DNN limit to bound MI. An existing input compression-based GE bound is used to link MI and GE. To the best of our knowledge, this represents the first empirical study of this bound. In our attempt to empirically falsify the theoretical bound, we find that it is often tight for best-performing models. Furthermore, it detects randomization of training labels in many cases, reflects test-time perturbation robustness, and works well given only few training samples. These results are promising given that input compression is broadly applicable where MI can be estimated with confidence.
Deep Analysis of Visual Product Reviews
Venue: arXiv cs.CV 2022
Year: 2022

With the proliferation of the e-commerce industry, analyzing customer feedback is becoming indispensable to a service provider. In recent days, it can be noticed that customers upload the purchased product images with their review scores. In this paper, we undertake the task of analyzing such visual reviews, which is very new of its kind. In the past, the researchers worked on analyzing language feedback, but here we do not take any assistance from linguistic reviews that may be absent, since a recent trend can be observed where customers prefer to quickly upload the visual feedback instead of typing language feedback. We propose a hierarchical architecture, where the higher-level model engages in product categorization, and the lower-level model pays attention to predicting the review score from a customer-provided product image. We generated a database by procuring real visual product reviews, which was quite challenging. Our architecture obtained some promising results by performing extensive experiments on the employed database. The proposed hierarchical architecture attained a 57.48% performance improvement over the single-level best comparable architecture.
SeasoNet: A Seasonal Scene Classification, segmentation and Retrieval dataset for satellite Imagery over Germany
Venue: arXiv cs.CV 2022
Year: 2022

This work presents SeasoNet, a new large-scale multi-label land cover and land use scene understanding dataset. It includes $1\,759\,830$ images from Sentinel-2 tiles, with 12 spectral bands and patch sizes of up to $ 120 \ \mathrm{px} \times 120 \ \mathrm{px}$. Each image is annotated with large scale pixel level labels from the German land cover model LBM-DE2018 with land cover classes based on the CORINE Land Cover database (CLC) 2018 and a five times smaller minimum mapping unit (MMU) than the original CLC maps. We provide pixel synchronous examples from all four seasons, plus an additional snowy set. These properties make SeasoNet the currently most versatile and biggest remote sensing scene understanding dataset with possible applications ranging from scene classification over land cover mapping to content-based cross season image retrieval and self-supervised feature learning. We provide baseline results by evaluating state-of-the-art deep networks on the new dataset in scene classification and semantic segmentation scenarios.
HSE-NN Team at the 4th ABAW Competition: Multi-task Emotion Recognition and Learning from Synthetic Images
Venue: arXiv cs.CV 2022
Year: 2022

In this paper, we present the results of the HSE-NN team in the 4th competition on Affective Behavior Analysis in-the-wild (ABAW). The novel multi-task EfficientNet model is trained for simultaneous recognition of facial expressions and prediction of valence and arousal on static photos. The resulting MT-EmotiEffNet extracts visual features that are fed into simple feed-forward neural networks in the multi-task learning challenge. We obtain performance measure 1.3 on the validation set, which is significantly greater when compared to either performance of baseline (0.3) or existing models that are trained only on the s-Aff-Wild2 database. In the learning from synthetic data challenge, the quality of the original synthetic training set is increased by using the super-resolution techniques, such as Real-ESRGAN. Next, the MT-EmotiEffNet is fine-tuned on the new training set. The final prediction is a simple blending ensemble of pre-trained and fine-tuned MT-EmotiEffNets. Our average validation F1 score is 18% greater than the baseline convolutional neural network.
Industry Led Use-Case Development for Human-Swarm Operations
Venue: arXiv cs.RO 2022
Year: 2022

In the domain of unmanned vehicles, autonomous robotic swarms promise to deliver increased efficiency and collective autonomy. How these swarms will operate in the future, and what communication requirements and operational boundaries will arise are yet to be sufficiently defined. A workshop was conducted with 11 professional unmanned-vehicle operators and designers with the objective of identifying use-cases for developing and testing robotic swarms. Three scenarios were defined by experts and were then compiled to produce a single use case outlining the scenario, objectives, agents, communication requirements and stages of operation when collaborating with highly autonomous swarms. Our compiled use case is intended for researchers, designers, and manufacturers alike to test and tailor their design pipeline to accommodate for some of the key issues in human-swarm ininteraction. Examples of application include informing simulation development, forming the basis of further design workshops, and identifying trust issues that may arise between human operators and the swarm.
Inclusive Privacy Design for Older Adults Living in Ambient Assisted Living
Venue: arXiv cs.CY 2022
Year: 2022

Ambient assisted living (AAL) environments support independence and quality of life of older adults However, in an AAL environment, privacy-related issues (e.g., unawareness, information disclosure, and lack of support) directly impact older adults and bystanders (e.g., caregivers, service providers, etc.). We explore the privacy challenges that both older adults and bystanders face in AAL. We call for inclusive privacy design and recommend following areas of improvement: consent, notification, and consideration for cultural differences.
Design of Coded Caching Schemes with Linear Subpacketizations Based on Injective Arc Coloring of Regular Digraphs
Venue: arXiv cs.IT 2022
Year: 2022

Coded caching is an effective technique to decongest the amount of traffic in the backhaul link. In such a scheme, each file hosted in the server is divided into a number of packets to pursue a low transmission rate based on the delicate design of contents cached into users and broadcast messages. However, the implementation complexity of this scheme increases with the number of packets. It is desirable to design a scheme with a small subpacketization level and a relatively low transmission rate. Recently, placement delivery array (PDA) was proposed to address the subpacketization bottleneck of coded caching. This paper investigates the design PDA from a new perspective, i.e., the injective arc coloring of regular digraphs. It is shown that the injective arc coloring of a regular digraph can yield a PDA with the same number of rows and columns. Based on this, a new class of regular digraphs are defined and the upper bounds on the injective chromatic index of such digraphs are derived. Consequently, some new coded caching schemes with a linear subpacketization level and a small transmission rate are proposed, one of which generalizes the existing scheme for the scenario with a more flexible number of users.
RV4JaCa -- Runtime Verification for Multi-Agent Systems
Venue: arXiv cs.MA 2022
Year: 2022

This paper presents a Runtime Verification (RV) approach for Multi-Agent Systems (MAS) using the JaCaMo framework. Our objective is to bring a layer of security to the MAS. This layer is capable of controlling events during the execution of the system without needing a specific implementation in the behaviour of each agent to recognise the events. MAS have been used in the context of hybrid intelligence. This use requires communication between software agents and human beings. In some cases, communication takes place via natural language dialogues. However, this kind of communication brings us to a concern related to controlling the flow of dialogue so that agents can prevent any change in the topic of discussion that could impair their reasoning. We demonstrate the implementation of a monitor that aims to control this dialogue flow in a MAS that communicates with the user through natural language to aid decision-making in hospital bed allocation.
Multi-Task Learning for Emotion Descriptors Estimation at the fourth ABAW Challenge
Venue: arXiv cs.CV 2022
Year: 2022

Facial valence/arousal, expression and action unit are related tasks in facial affective analysis. However, the tasks only have limited performance in the wild due to the various collected conditions. The 4th competition on affective behavior analysis in the wild (ABAW) provided images with valence/arousal, expression and action unit labels. In this paper, we introduce multi-task learning framework to enhance the performance of three related tasks in the wild. Feature sharing and label fusion are used to utilize their relations. We conduct experiments on the provided training and validating data.
OTPose: Occlusion-Aware Transformer for Pose Estimation in Sparsely-Labeled Videos
Venue: arXiv cs.CV 2022
Year: 2022

Although many approaches for multi-human pose estimation in videos have shown profound results, they require densely annotated data which entails excessive man labor. Furthermore, there exists occlusion and motion blur that inevitably lead to poor estimation performance. To address these problems, we propose a method that leverages an attention mask for occluded joints and encodes temporal dependency between frames using transformers. First, our framework composes different combinations of sparsely annotated frames that denote the track of the overall joint movement. We propose an occlusion attention mask from these combinations that enable encoding occlusion-aware heatmaps as a semi-supervised task. Second, the proposed temporal encoder employs transformer architecture to effectively aggregate the temporal relationship and keypoint-wise attention from each time step and accurately refines the target frame's final pose estimation. We achieve state-of-the-art pose estimation results for PoseTrack2017 and PoseTrack2018 datasets and demonstrate the robustness of our approach to occlusion and motion blur in sparsely annotated video data.
Task-adaptive Spatial-Temporal Video Sampler for Few-shot Action Recognition
Venue: arXiv cs.CV 2022
Year: 2022

A primary challenge faced in few-shot action recognition is inadequate video data for training. To address this issue, current methods in this field mainly focus on devising algorithms at the feature level while little attention is paid to processing input video data. Moreover, existing frame sampling strategies may omit critical action information in temporal and spatial dimensions, which further impacts video utilization efficiency. In this paper, we propose a novel video frame sampler for few-shot action recognition to address this issue, where task-specific spatial-temporal frame sampling is achieved via a temporal selector (TS) and a spatial amplifier (SA). Specifically, our sampler first scans the whole video at a small computational cost to obtain a global perception of video frames. The TS plays its role in selecting top-T frames that contribute most significantly and subsequently. The SA emphasizes the discriminative information of each frame by amplifying critical regions with the guidance of saliency maps. We further adopt task-adaptive learning to dynamically adjust the sampling strategy according to the episode task at hand. Both the implementations of TS and SA are differentiable for end-to-end optimization, facilitating seamless integration of our proposed sampler with most few-shot action recognition methods. Extensive experiments show a significant boost in the performances on various benchmarks including long-term videos.The code is available at https://github.com/R00Kie-Liu/Sampler
Joint Precoding and Artificial Noise Design for MU-MIMO Wiretap Channels
Venue: arXiv cs.IT 2022
Year: 2022

Secure precoding superimposed with artificial noise (AN) is a promising transmission technique to improve security by harnessing the superposition nature of the wireless medium. However, finding a jointly optimal precoding and AN structure is very challenging in downlink multi-user multiple-input multiple-output (MU-MIMO) wiretap channels with multiple eavesdroppers. The major challenge in maximizing the secrecy rate arises from the non-convexity and non-smoothness of the rate function. Traditionally, an alternating optimization framework that identifies beamforming vectors and AN covariance matrix has been adopted; yet this alternating approach has limitations in maximizing the secrecy rate. In this paper, we put forth a novel secure precoding algorithm that jointly and simultaneously optimizes the beams and AN covariance matrix for maximizing the secrecy rate when a transmitter has either perfect or partial channel knowledge of eavesdroppers. To this end, we first establish an approximate secrecy rate in a smooth function. Then, we derive the first-order optimality condition in the form of the nonlinear eigenvalue problem (NEP). We present a computationally efficient algorithm to identify the principal eigenvector of the NEP as a suboptimal solution for secure precoding. Simulations demonstrate that the proposed methods improve secrecy rate significantly compared to the existing secure precoding methods.
PFMC: a parallel symbolic model checker for security protocol verification
Venue: arXiv cs.LO 2022
Year: 2022

We present an investigation into the design and implementation of a parallel model checker for security protocol verification that is based on a symbolic model of the adversary, where instantiations of concrete terms and messages are avoided until needed to resolve a particular assertion. We propose to build on this naturally lazy approach to parallelise this symbolic state exploration and evaluation. We utilise the concept of strategies in Haskell, which abstracts away from the low-level details of thread management and modularly adds parallel evaluation strategies (encapsulated as a monad in Haskell). We build on an existing symbolic model checker, OFMC, which is already implemented in Haskell. We show that there is a very significant speed up of around 3-5 times improvement when moving from the original single-threaded implementation of OFMC to our multi-threaded version, for both the Dolev-Yao attacker model and more general algebraic attacker models. We identify several issues in parallelising the model checker: among others, controlling growth of memory consumption, balancing lazy vs strict evaluation, and achieving an optimal granularity of parallelism.
ViGAT: Bottom-up event recognition and explanation in video using factorized graph attention network
Venue: arXiv cs.CV 2022
Year: 2022

In this paper a pure-attention bottom-up approach, called ViGAT, that utilizes an object detector together with a Vision Transformer (ViT) backbone network to derive object and frame features, and a head network to process these features for the task of event recognition and explanation in video, is proposed. The ViGAT head consists of graph attention network (GAT) blocks factorized along the spatial and temporal dimensions in order to capture effectively both local and long-term dependencies between objects or frames. Moreover, using the weighted in-degrees (WiDs) derived from the adjacency matrices at the various GAT blocks, we show that the proposed architecture can identify the most salient objects and frames that explain the decision of the network. A comprehensive evaluation study is performed, demonstrating that the proposed approach provides state-of-the-art results on three large, publicly available video datasets (FCVID, Mini-Kinetics, ActivityNet).