OpenRaiser/Intern-Atlas huggingface dataset:
Improved Image Generation via Sparse Modeling
Venue: arXiv cs.CV 2022
Year: 2022

The interest of the deep learning community in image synthesis has grown massively in recent years. Nowadays, deep generative methods, and especially Generative Adversarial Networks (GANs), are leading to state-of-the-art performance, capable of synthesizing images that appear realistic. While the efforts for improving the quality of the generated images are extensive, most attempts still consider the generator part as an uncorroborated "black-box". In this paper, we aim to provide a better understanding and design of the image generation process. We interpret existing generators as implicitly relying on sparsity-inspired models. More specifically, we show that generators can be viewed as manifestations of the Convolutional Sparse Coding (CSC) and its Multi-Layered version (ML-CSC) synthesis processes. We leverage this observation by explicitly enforcing a sparsifying regularization on appropriately chosen activation layers in the generator, and demonstrate that this leads to improved image synthesis. Furthermore, we show that the same rationale and benefits apply to generators serving inverse problems, demonstrated on the Deep Image Prior (DIP) method.
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Venue: arXiv cs.CV 2022
Year: 2022

Our objective in this work is video-text retrieval - in particular a joint embedding that enables efficient text-to-video retrieval. The challenges in this area include the design of the visual architecture and the nature of the training data, in that the available large scale video-text training datasets, such as HowTo100M, are noisy and hence competitive performance is achieved only at scale through large amounts of compute. We address both these challenges in this paper. We propose an end-to-end trainable model that is designed to take advantage of both large-scale image and video captioning datasets. Our model is an adaptation and extension of the recent ViT and Timesformer architectures, and consists of attention in both space and time. The model is flexible and can be trained on both image and video text datasets, either independently or in conjunction. It is trained with a curriculum learning schedule that begins by treating images as 'frozen' snapshots of video, and then gradually learns to attend to increasing temporal context when trained on video datasets. We also provide a new video-text pretraining dataset WebVid-2M, comprised of over two million videos with weak captions scraped from the internet. Despite training on datasets that are an order of magnitude smaller, we show that this approach yields state-of-the-art results on standard downstream video-retrieval benchmarks including MSR-VTT, MSVD, DiDeMo and LSMDC.
Graph Attention Networks for Channel Estimation in RIS-assisted Satellite IoT Communications
Venue: arXiv cs.NI 2022
Year: 2022

Direct-to-satellite (DtS) communication has gained importance recently to support globally connected Internet of things (IoT) networks. However, relatively long distances of densely deployed satellite networks around the Earth cause a high path loss. In addition, since high complexity operations such as beamforming, tracking and equalization have to be performed in IoT devices partially, both the hardware complexity and the need for high-capacity batteries of IoT devices increase. The reconfigurable intelligent surfaces (RISs) have the potential to increase the energy-efficiency and to perform complex signal processing over the transmission environment instead of IoT devices. But, RISs need the information of the cascaded channel in order to change the phase of the incident signal. This study evaluates the pilot signal as a graph and incorporates this information into the graph attention networks (GATs) to track the phase relation through pilot signaling. The proposed GAT-based channel estimation method examines the performance of the DtS IoT networks for different RIS configurations to solve the challenging channel estimation problem. It is shown that the proposed GAT both demonstrates a higher performance with increased robustness under changing conditions and has lower computational complexity compared to conventional deep learning methods. Moreover, bit error rate performance is investigated for RIS designs with discrete and non-uniform phase shifts under channel estimation based on the proposed method. One of the findings in this study is that the channel models of the operating environment and the performance of the channel estimation method must be considered during RIS design to exploit performance improvement as far as possible.
Flexible Instruction-Set Semantics via Type Classes
Venue: arXiv cs.LO 2022
Year: 2022

Instruction sets, from families like x86 and ARM, are at the center of many ambitious formal-methods projects. Many verification, synthesis, programming, and debugging tools rely on formal semantics of instruction sets, but different tools can use semantics in rather different ways. As a result, a central challenge for that community is how semantics should be written and what techniques should be used to connect them to new use cases. The best-known work applying single semantics across quite-different tools relies on domain-specific languages like Sail, where the language and its translation tools are specialized to the realm of instruction sets. We decided to explore a different approach, with semantics written in a carefully chosen subset of Haskell. This style does not depend on any new language translators, relying instead on parameterization of semantics over type-class instances. As a result, a semantics can be a first-class object within a logic, and application of a semantics for a new kind of tool can be a first-class operation in the logic, allowing sharing of theorems across applications. Our case study is for the open RISC-V instruction-set family, and we have used a single core semantics to support testing, interactive proof, and model checking of both software and hardware. We especially highlight an application of a first-class semantics within Coq that can be instantiated in different ways within one proof: simulation between variants where multiplication is implemented in hardware or in the machine code of a particular software trap handler.
The Effects of Spectral Dimensionality Reduction on Hyperspectral Pixel Classification: A Case Study
Venue: arXiv cs.CV 2022
Year: 2022

This paper presents a systematic study of the effects of hyperspectral pixel dimensionality reduction on the pixel classification task. We use five dimensionality reduction methods -- PCA, KPCA, ICA, AE, and DAE -- to compress 301-dimensional hyperspectral pixels. Compressed pixels are subsequently used to perform pixel classifications. Pixel classification accuracies together with compression method, compression rates, and reconstruction errors provide a new lens to study the suitability of a compression method for the task of pixel classification. We use three high-resolution hyperspectral image datasets, representing three common landscape types (i.e. urban, transitional suburban, and forests) collected by the Remote Sensing and Spatial Ecosystem Modeling laboratory of the University of Toronto. We found that PCA, KPCA, and ICA post greater signal reconstruction capability; however, when compression rates are more than 90\% these methods show lower classification scores. AE and DAE methods post better classification accuracy at 95\% compression rate, however their performance drops as compression rate approaches 97\%. Our results suggest that both the compression method and the compression rate are important considerations when designing a hyperspectral pixel classification pipeline.
Symmetric Subrank of Tensors and Applications
Venue: arXiv cs.CC 2022
Year: 2022

Strassen (Strassen, J. Reine Angew. Math., 375/376, 1987) introduced the subrank of a tensor as a natural extension of matrix rank to tensors. Subrank measures the largest diagonal tensor that can be obtained by applying linear operations to the different indices (legs) of the tensor (just like the matrix rank measures the largest diagonal matrix that can be obtained using row and column operations). Motivated by problems in combinatorics and complexity theory we introduce the new notion of symmetric subrank of tensors by restricting these linear operations to be the same for each index. We prove precise relations and separations between subrank and symmetric subrank. We prove that for symmetric tensors the subrank and the symmetric subrank are asymptotically equal. This proves the asymptotic subrank analogon of a conjecture known as Comon's conjecture in the theory of tensors. This result allows us to prove a strong connection between the general and symmetric version of an asymptotic duality theorem of Strassen. We introduce a representation-theoretic method to asymptotically bound the symmetric subrank called the symmetric quantum functional in analogy with the quantum functionals (Christandl, Vrana, Zuiddam, J. Amer. Math. Soc., 2021), and we study the relations between these functionals.
Evolution of Symbiosis in the Game of Life: Three Characteristics of Successful Symbiotes
Venue: arXiv cs.NE 2022
Year: 2022

In past work, we developed a computational model of the evolution of symbiotic entities (Model-S), based on Conway's Game of Life. In this article, we examine three trends that biologists have observed in the evolution of symbiotes. (1) Management: If one partner is able to control the symbiotic relation, this control can reduce conflict; thus, evolutionary selection favours symbiotes that have a manager. (2) Mutualism: Although partners in a symbiote often have conflicting needs, evolutionary selection favours symbiotes in which partners are better off together inside the symbiote than they would be as individuals outside of the symbiote. (3) Interaction: Repeated interaction among partners in symbiosis tends to promote increasing fitness due to evolutionary selection. We have added new components to Model-S that allow us to observe these three trends in runs of Model-S. The new components are analogous to the practice of staining cells in biology research, to reveal patterns that are not usually visible. When we measure the fitness of a symbiote by making it compete with other symbiotes, we find that fitter symbiotes have significantly more management, mutualism, and interaction than less fit symbiotes. These results confirm the trends observed in nature by biologists. Model-S allows biologists to study these evolutionary trends and other characteristics of symbiosis in ways that are not tractable with living organisms.
Covariance estimation under one-bit quantization
Venue: arXiv cs.IT 2022
Year: 2022

We consider the classical problem of estimating the covariance matrix of a subgaussian distribution from i.i.d. samples in the novel context of coarse quantization, i.e., instead of having full knowledge of the samples, they are quantized to one or two bits per entry. This problem occurs naturally in signal processing applications. We introduce new estimators in two different quantization scenarios and derive non-asymptotic estimation error bounds in terms of the operator norm. In the first scenario we consider a simple, scale-invariant one-bit quantizer and derive an estimation result for the correlation matrix of a centered Gaussian distribution. In the second scenario, we add random dithering to the quantizer. In this case we can accurately estimate the full covariance matrix of a general subgaussian distribution by collecting two bits per entry of each sample. In both scenarios, our bounds apply to masked covariance estimation. We demonstrate the near-optimality of our error bounds by deriving corresponding (minimax) lower bounds and using numerical simulations.
TATL: Task Agnostic Transfer Learning for Skin Attributes Detection
Venue: arXiv cs.CV 2022
Year: 2022

Existing skin attributes detection methods usually initialize with a pre-trained Imagenet network and then fine-tune on a medical target task. However, we argue that such approaches are suboptimal because medical datasets are largely different from ImageNet and often contain limited training samples. In this work, we propose \emph{Task Agnostic Transfer Learning (TATL)}, a novel framework motivated by dermatologists' behaviors in the skincare context. TATL learns an attribute-agnostic segmenter that detects lesion skin regions and then transfers this knowledge to a set of attribute-specific classifiers to detect each particular attribute. Since TATL's attribute-agnostic segmenter only detects skin attribute regions, it enjoys ample data from all attributes, allows transferring knowledge among features, and compensates for the lack of training data from rare attributes. We conduct extensive experiments to evaluate the proposed TATL transfer learning mechanism with various neural network architectures on two popular skin attributes detection benchmarks. The empirical results show that TATL not only works well with multiple architectures but also can achieve state-of-the-art performances while enjoying minimal model and computational complexities. We also provide theoretical insights and explanations for why our transfer learning framework performs well in practice.
Some considerations about reviewing and open-access in scientific publishing
Venue: arXiv cs.DL 2022
Year: 2022

Scientific research changed profoundly over the last 30 years, in all its aspects. Scientific publishing has changed as well, mainly because of the strong increased number of submitted papers and because of the appearance of Open Access journals and publishers. We propose some reflections on these issues.
Generalization of GANs and overparameterized models under Lipschitz continuity
Venue: arXiv cs.LG 2022
Year: 2022

Generative adversarial networks (GANs) are so complex that the existing learning theories do not provide a satisfactory explanation for why GANs have great success in practice. The same situation also remains largely open for deep neural networks. To fill this gap, we introduce a Lipschitz theory to analyze generalization. We demonstrate its simplicity by analyzing generalization and consistency of overparameterized neural networks. We then use this theory to derive Lipschitz-based generalization bounds for GANs. Our bounds show that penalizing the Lipschitz constant of the GAN loss can improve generalization. This result answers the long mystery of why the popular use of Lipschitz constraint for GANs often leads to great success, empirically without a solid theory. Finally but surprisingly, we show that, when using Dropout or spectral normalization, both \emph{truly deep} neural networks and GANs can generalize well without the curse of dimensionality.
Subjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR)
Venue: arXiv cs.MM 2022
Year: 2022

Recent studies have shown that it is possible to characterize subject bias and variance in subjective assessment tests. Apparent differences among subjects can, for the most part, be explained by random factors. Building on that theory, we propose a subjective test design where three to four team members each rate the stimuli multiple times. The results are comparable to a high performing objective metric. This provides a quick and simple way to analyze new technologies and perform pre-tests for subjective assessment.
HIH: Towards More Accurate Face Alignment via Heatmap in Heatmap
Venue: arXiv cs.CV 2022
Year: 2022

Heatmap-based regression overcomes the lack of spatial and contextual information of direct coordinate regression, and has revolutionized the task of face alignment. Yet it suffers from quantization errors caused by neglecting subpixel coordinates in image resizing and network downsampling. In this paper, we first quantitatively analyze the quantization error on benchmarks, which accounts for more than 1/3 of the whole prediction errors for state-of-the-art methods. To tackle this problem, we propose a novel Heatmap In Heatmap(HIH) representation and a coordinate soft-classification (CSC) method, which are seamlessly integrated into the classic hourglass network. The HIH representation utilizes nested heatmaps to jointly represent the coordinate label: one heatmap called integer heatmap stands for the integer coordinate, and the other heatmap named decimal heatmap represents the subpixel coordinate. The range of a decimal heatmap makes up one pixel in the corresponding integer heatmap. Besides, we transfer the offset regression problem to an interval classification task, and CSC regards the confidence of the pixel as the probability of the interval. Meanwhile, CSC applying the distribution loss leverage the soft labels generated from the Gaussian distribution function to guide the offset heatmap training, which makes it easier to learn the distribution of coordinate offsets. Extensive experiments on challenging benchmark datasets demonstrate that our HIH can achieve state-of-the-art results. In particular, our HIH reaches 4.08 NME (Normalized Mean Error) on WFLW, and 3.21 on COFW, which exceeds previous methods by a significant margin.
Rethinking the Backdoor Attacks' Triggers: A Frequency Perspective
Venue: arXiv cs.LG 2022
Year: 2022

Backdoor attacks have been considered a severe security threat to deep learning. Such attacks can make models perform abnormally on inputs with predefined triggers and still retain state-of-the-art performance on clean data. While backdoor attacks have been thoroughly investigated in the image domain from both attackers' and defenders' sides, an analysis in the frequency domain has been missing thus far. This paper first revisits existing backdoor triggers from a frequency perspective and performs a comprehensive analysis. Our results show that many current backdoor attacks exhibit severe high-frequency artifacts, which persist across different datasets and resolutions. We further demonstrate these high-frequency artifacts enable a simple way to detect existing backdoor triggers at a detection rate of 98.50% without prior knowledge of the attack details and the target model. Acknowledging previous attacks' weaknesses, we propose a practical way to create smooth backdoor triggers without high-frequency artifacts and study their detectability. We show that existing defense works can benefit by incorporating these smooth triggers into their design consideration. Moreover, we show that the detector tuned over stronger smooth triggers can generalize well to unseen weak smooth triggers. In short, our work emphasizes the importance of considering frequency analysis when designing both backdoor attacks and defenses in deep learning.
Computation and Bribery of Voting Power in Delegative Simple Games
Venue: arXiv cs.GT 2022
Year: 2022

Following Zhang and Grossi~(AAAI 2021), we study in more depth a variant of weighted voting games in which agents' weights are induced by a transitive support structure. This class of simple games is notably well suited to study the relative importance of agents in the liquid democracy framework. We first propose a pseudo-polynomial time algorithm to compute the Banzhaf and Shapley-Shubik indices for this class of game. Then, we study a bribery problem, in which one tries to maximize/minimize the voting power/weight of a given agent by changing the support structure under a budget constraint. We show that these problems are computationally hard and provide several parameterized complexity results.
The human side of Software Engineering Teams: an investigation of contemporary challenges
Venue: arXiv cs.SE 2022
Year: 2022

There have been recent calls for research on the human side of software engineering and its impact on various factors such as productivity, developer happiness and project success. An analysis of which challenges in software engineering teams are most frequent is still missing. We aim to provide a starting point for a theory about relevant human challenges and their causes in software engineering. We establish a reusable set of challenges and start out by investigating the effect of team virtualization. Virtual teams often use digital communication and consist of members with different nationalities. We designed a survey instrument and asked respondents to assess the frequency and criticality of a set of challenges, separated in context "within teams" as well as "between teams and clients", compiled from previous empiric work, blog posts and pilot survey feedback. For the team challenges, we asked if mitigation measures were already in place. Respondents were also asked to provide information about their team setup. The survey also measured Schwartz human values. Finally, respondents were asked if there were additional challenges at their workplace. We report on the results obtained from 192 respondents. We present a set of challenges that takes the survey feedback into account and introduce two categories of challenges; "interpersonal" and "intrapersonal". We found no evidence for links between human values and challenges. We found some significant links between the number of distinct nationalities in a team and certain challenges, with less frequent and critical challenges occurring if 2-3 different nationalities were present compared to a team having members of just one nationality or more than three. A higher degree of virtualization seems to increase the frequency of some human challenges.
Modeling Object Dissimilarity for Deep Saliency Prediction
Venue: arXiv cs.CV 2022
Year: 2022

Saliency prediction has made great strides over the past two decades, with current techniques modeling low-level information, such as color, intensity and size contrasts, and high-level ones, such as attention and gaze direction for entire objects. Despite this, these methods fail to account for the dissimilarity between objects, which affects human visual attention. In this paper, we introduce a detection-guided saliency prediction network that explicitly models the differences between multiple objects, such as their appearance and size dissimilarities. Our approach allows us to fuse our object dissimilarities with features extracted by any deep saliency prediction network. As evidenced by our experiments, this consistently boosts the accuracy of the baseline networks, enabling us to outperform the state-of-the-art models on three saliency benchmarks, namely SALICON, MIT300 and CAT2000. Our project page is at https://github.com/IVRL/DisSal.
ACERAC: Efficient reinforcement learning in fine time discretization
Venue: arXiv cs.LG 2022
Year: 2022

One of the main goals of reinforcement learning (RL) is to provide a~way for physical machines to learn optimal behavior instead of being programmed. However, effective control of the machines usually requires fine time discretization. The most common RL methods apply independent random elements to each action, which is not suitable in that setting. It is not feasible because it causes the controlled system to jerk, and does not ensure sufficient exploration since a~single action is not long enough to create a~significant experience that could be translated into policy improvement. In our view these are the main obstacles that prevent application of RL in contemporary control systems. To address these pitfalls, in this paper we introduce an RL framework and adequate analytical tools for actions that may be stochastically dependent in subsequent time instances. We also introduce an RL algorithm that approximately optimizes a~policy that produces such actions. It applies experience replay to adjust likelihood of sequences of previous actions to optimize expected $n$-step returns the policy yields. The efficiency of this algorithm is verified against four other RL methods (CDAU, PPO, SAC, ACER) in four simulated learning control problems (Ant, HalfCheetah, Hopper, and Walker2D) in diverse time discretization. The algorithm introduced here outperforms the competitors in most cases considered.