OpenRaiser/Intern-Atlas huggingface dataset:
Decidability and Periodicity of Low Complexity Tilings
Venue: arXiv cs.DM 2022
Year: 2022

In this paper we study colorings (or tilings) of the two-dimensional grid $\mathbb{Z}^2$. A coloring is said to be valid with respect to a set $P$ of $n\times m$ rectangular patterns if all $n\times m$ sub-patterns of the coloring are in $P$. A coloring $c$ is said to be of low complexity with respect to a rectangle if there exist $m,n\in\mathbb{N}$ and a set $P$ of $n\times m$ rectangular patterns such that $c$ is valid with respect to $P$ and $|P|\leq nm$. Open since it was stated in 1997, Nivat's conjecture states that such a coloring is necessarily periodic. If Nivat's conjecture is true, all valid colorings with respect to $P$ such that $|P|\leq mn$ must be periodic. We prove that there exists at least one periodic coloring among the valid ones. We use this result to investigate the tiling problem, also known as the domino problem, which is well known to be undecidable in its full generality. However, we show that it is decidable in the low-complexity setting. Then, we use our result to show that Nivat's conjecture holds for uniformly recurrent configurations. These results also extend to other convex shapes in place of the rectangle.\\ After that, we prove that the $nm$ bound is multiplicatively optimal for the decidability of the domino problem, as for all $\varepsilon>0$ it is undecidable to determine if there exists a valid coloring for a given $m,n\in \mathbb{N}$ and set of rectangular patterns $P$ of size $n\times m$ such that $|P|\leq (1+\varepsilon)nm$. We prove a slightly better bound in the case where $m=n$, as well as constructing aperiodic SFTs of pretty low complexity.\\ This paper is an extended version of a paper published in STACS 2020.
Are Query-Based Ontology Debuggers Really Helping Knowledge Engineers?
Venue: arXiv cs.AI 2022
Year: 2022

Real-world semantic or knowledge-based systems, e.g., in the biomedical domain, can become large and complex. Tool support for the localization and repair of faults within knowledge bases of such systems can therefore be essential for their practical success. Correspondingly, a number of knowledge base debugging approaches, in particular for ontology-based systems, were proposed throughout recent years. Query-based debugging is a comparably recent interactive approach that localizes the true cause of an observed problem by asking knowledge engineers a series of questions. Concrete implementations of this approach exist, such as the OntoDebug plug-in for the ontology editor Protégé. To validate that a newly proposed method is favorable over an existing one, researchers often rely on simulation-based comparisons. Such an evaluation approach however has certain limitations and often cannot fully inform us about a method's true usefulness. We therefore conducted different user studies to assess the practical value of query-based ontology debugging. One main insight from the studies is that the considered interactive approach is indeed more efficient than an alternative algorithmic debugging based on test cases. We also observed that users frequently made errors in the process, which highlights the importance of a careful design of the queries that users need to answer.
Taming the Knight's Tour: Minimizing Turns and Crossings
Venue: arXiv cs.DS 2022
Year: 2022

We introduce two new metrics of "simplicity" for knight's tours: the number of turns and the number of crossings. We give a novel algorithm that produces tours with $9.25n+O(1)$ turns and $12n+O(1)$ crossings on an $n\times n$ board, and we show lower bounds of $(6-ε)n$ and $4n-O(1)$ on the respective problems of minimizing these metrics. Hence, our algorithm achieves approximation ratios of $9.25/6+o(1)$ and $3+o(1)$. Our algorithm takes linear time and is fully parallelizable, i.e., the tour can be computed in $O(n^2/p)$ time using $p$ processors in the CREW PRAM model. We generalize our techniques to rectangular boards, high-dimensional boards, symmetric tours, odd boards with a missing corner, and tours for $(1,4)$-leapers. In doing so, we show that these extensions also admit a constant approximation ratio on the minimum number of turns, and on the number of crossings in most cases.
swTVM: Towards Optimized Tensor Code Generation for Deep Learning on Sunway Many-Core Processor
Venue: arXiv cs.LG 2022
Year: 2022

The flourish of deep learning frameworks and hardware platforms has been demanding an efficient compiler that can shield the diversity in both software and hardware in order to provide application portability. Among the existing deep learning compilers, TVM is well known for its efficiency in code generation and optimization across diverse hardware devices. In the meanwhile, the Sunway many-core processor renders itself as a competitive candidate for its attractive computational power in both scientific computing and deep learning workloads. This paper combines the trends in these two directions. Specifically, we propose swTVM that extends the original TVM to support ahead-of-time compilation for architecture requiring cross-compilation such as Sunway. In addition, we leverage the architecture features during the compilation such as core group for massive parallelism, DMA for high bandwidth memory transfer and local device memory for data locality, in order to generate efficient codes for deep learning workloads on Sunway. The experiment results show that the codes generated by swTVM achieves 1.79x on average compared to the state-of-the-art deep learning framework on Sunway, across six representative benchmarks. This work is the first attempt from the compiler perspective to bridge the gap of deep learning and Sunway processor particularly with productivity and efficiency in mind. We believe this work will encourage more people to embrace the power of deep learning and Sunway many-core processor.
Deep Localization of Mixed Image Tampering Techniques
Venue: arXiv cs.CV 2022
Year: 2022

With technological advances leading to an increase in mechanisms for image tampering, fraud detection methods must continue to be upgraded to match their sophistication. One problem with current methods is that they require prior knowledge of the method of forgery in order to determine which features to extract from the image to localize the region of interest. When a machine learning algorithm is used to learn different types of tampering from a large set of various image types, with a large enough database we can easily classify which images are tampered. However, we still are left with the question of which features to train on, and how to localize the manipulation. In this work, deep learning for object detection is adapted to tampering detection to solve these two problems, while fusing features from multiple classic techniques for improved accuracy. A Multi-stream version of the Faster RCNN network will be employed with the second stream having an input of the element-wise sum of the ELA and BAG error maps to provide even higher accuracy than a single stream alone.
Constructing minimal telescopers for rational functions in three discrete variables
Venue: arXiv cs.SC 2022
Year: 2022

We present a new algorithm for constructing minimal telescopers for rational functions in three discrete variables. This is the first discrete reduction-based algorithm that goes beyond the bivariate case. The termination of the algorithm is guaranteed by a known existence criterion of telescopers. Our approach has the important feature that it avoids the potentially costly computation of certificates. Computational experiments are also provided so as to illustrate the efficiency of our approach.
Finite-Memory Strategies in POMDPs with Long-Run Average Objectives
Venue: arXiv cs.GT 2022
Year: 2022

Partially observable Markov decision processes (POMDPs) are standard models for dynamic systems with probabilistic and nondeterministic behaviour in uncertain environments. We prove that in POMDPs with long-run average objective, the decision maker has approximately optimal strategies with finite memory. This implies notably that approximating the long-run value is recursively enumerable, as well as a weak continuity property of the value with respect to the transition function.
Randomization of Approximate Bilinear Computation for Matrix Multiplication
Venue: arXiv cs.DS 2022
Year: 2022

We present a method for randomizing formulas for bilinear computation of matrix products. We consider the implications of such randomization when there are two sources of error: One due to the formula itself only being approximately correct, and one due to using floating point arithmetic. Our theoretical results and numerical experiments indicate that our method can improve performance when each of these error sources are present individually, as well as when they are present at the same time.
Accelerating Distributed Optimization via Fixed-time Convergent Flows: Extensions to Non-convex Functions and Consistent Discretization
Venue: arXiv cs.SY 2022
Year: 2022

Distributed optimization has gained significant attention in recent years, primarily fueled by the availability of a large amount of data and privacy-preserving requirements. This paper presents a fixed-time convergent optimization algorithm for solving a potentially non-convex optimization problem using a first-order multi-agent system. Each agent in the network can access only its private objective function, while local information exchange is permitted between the neighbors. The proposed optimization algorithm combines a fixed-time convergent distributed parameter estimation scheme with a fixed-time distributed consensus scheme as its solution methodology. The results are presented under the assumption that the team objective function is strongly convex, as opposed to the common assumptions in the literature requiring each of the local objective functions to be strongly convex. The results extend to the class of possibly non-convex team objective functions satisfying only the Polyak-Łojasiewicz (PL) inequality. It is also shown that the proposed continuous-time scheme, when discretized using Euler's method, leads to consistent discretization, i.e., the fixed-time convergence behavior is preserved under discretization. Numerical examples comprising large-scale distributed linear regression and training of neural networks corroborate our theoretical analysis.
A view of Estimation of Distribution Algorithms through the lens of Expectation-Maximization
Venue: arXiv cs.LG 2022
Year: 2022

We show that a large class of Estimation of Distribution Algorithms, including, but not limited to, Covariance Matrix Adaption, can be written as a Monte Carlo Expectation-Maximization algorithm, and as exact EM in the limit of infinite samples. Because EM sits on a rigorous statistical foundation and has been thoroughly analyzed, this connection provides a new coherent framework with which to reason about EDAs.
When can unlabeled data improve the learning rate?
Venue: arXiv cs.LG 2022
Year: 2022

In semi-supervised classification, one is given access both to labeled and unlabeled data. As unlabeled data is typically cheaper to acquire than labeled data, this setup becomes advantageous as soon as one can exploit the unlabeled data in order to produce a better classifier than with labeled data alone. However, the conditions under which such an improvement is possible are not fully understood yet. Our analysis focuses on improvements in the minimax learning rate in terms of the number of labeled examples (with the number of unlabeled examples being allowed to depend on the number of labeled ones). We argue that for such improvements to be realistic and indisputable, certain specific conditions should be satisfied and previous analyses have failed to meet those conditions. We then demonstrate examples where these conditions can be met, in particular showing rate changes from $1/\sqrt{\ell}$ to $e^{-c\ell}$ and from $1/\sqrt{\ell}$ to $1/\ell$. These results improve our understanding of what is and isn't possible in semi-supervised learning.
Efficient Kirszbraun Extension with Applications to Regression
Venue: arXiv cs.LG 2022
Year: 2022

We introduce a framework for performing regression between two Hilbert spaces. This is done based on Kirszbraun's extension theorem, to the best of our knowledge, the first application of this technique to supervised learning. We analyze the statistical and computational aspects of this method. We decompose this task into two stages: training (which corresponds operationally to smoothing/regularization) and prediction (which is achieved via Kirszbraun extension). Both are solved algorithmically via a novel multiplicative weight updates (MWU) scheme, which, for our problem formulation, achieves a quadratic runtime improvement over the state of the art. Our empirical results indicate a dramatic improvement over standard off-the-shelf solvers in our setting.
Sublinear Update Time Randomized Algorithms for Dynamic Graph Regression
Venue: arXiv cs.LG 2022
Year: 2022

A well-known problem in data science and machine learning is {\em linear regression}, which is recently extended to dynamic graphs. Existing exact algorithms for updating the solution of dynamic graph regression require at least a linear time (in terms of $n$: the size of the graph). However, this time complexity might be intractable in practice. In the current paper, we utilize {\em subsampled randomized Hadamard transform} and \textsf{CountSketch} to propose the first sublinear update time randomized algorithms for regression of general dynamic graphs. Suppose that we are given a $n\times d$ matrix embedding $\mathbf M$ of the graph, where $d \ll n$ and $\mathbf M$ has certain properties. Let $r$ be the number of samples required by subsampled randomized Hadamard transform for a $1\pm ε$ approximation, which is a sublinear of $n$. Our first algorithm supports edge insertion and edge deletion and updates the approximate solution in $O(rd)$ time. Our second algorithm is based on \textsf{CountSketch} and supports edge insertion, edge deletion, node insertion and node deletion. It updates the approximate solution in $O(qd)$ time, where $q=O\left(\frac{d^2}{ε^2} \log^6(d/ε) \right)$.
Continual learning with hypernetworks
Venue: arXiv cs.LG 2022
Year: 2022

Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks. To overcome this problem, we present a novel approach based on task-conditioned hypernetworks, i.e., networks that generate the weights of a target model based on task identity. Continual learning (CL) is less difficult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-specific weight realizations, which can be maintained in memory using a simple regularizer. Besides achieving state-of-the-art performance on standard CL benchmarks, additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories. Notably, such long memory lifetimes are achieved in a compressive regime, when the number of trainable hypernetwork weights is comparable or smaller than target network size. We provide insight into the structure of low-dimensional task embedding spaces (the input space of the hypernetwork) and show that task-conditioned hypernetworks demonstrate transfer learning. Finally, forward information transfer is further supported by empirical results on a challenging CL benchmark based on the CIFAR-10/100 image datasets.
SoK: Differential Privacies
Venue: arXiv cs.CR 2022
Year: 2022

Shortly after it was first introduced in 2006, differential privacy became the flagship data privacy definition. Since then, numerous variants and extensions were proposed to adapt it to different scenarios and attacker models. In this work, we propose a systematic taxonomy of these variants and extensions. We list all data privacy definitions based on differential privacy, and partition them into seven categories, depending on which aspect of the original definition is modified. These categories act like dimensions: variants from the same category cannot be combined, but variants from different categories can be combined to form new definitions. We also establish a partial ordering of relative strength between these notions by summarizing existing results. Furthermore, we list which of these definitions satisfy some desirable properties, like composition, post-processing, and convexity by either providing a novel proof or collecting existing ones.
One-Shot Neural Architecture Search via Compressive Sensing
Venue: arXiv cs.LG 2022
Year: 2022

Neural Architecture Search remains a very challenging meta-learning problem. Several recent techniques based on parameter-sharing idea have focused on reducing the NAS running time by leveraging proxy models, leading to architectures with competitive performance compared to those with hand-crafted designs. In this paper, we propose an iterative technique for NAS, inspired by algorithms for learning low-degree sparse Boolean functions. We validate our approach on the DARTs search space (Liu et al., 2018b) and NAS-Bench-201 (Yang et al., 2020). In addition, we provide theoretical analysis via upper bounds on the number of validation error measurements needed for reliable learning, and include ablation studies to further in-depth understanding of our technique.
Input Selection for Bandwidth-Limited Neural Network Inference
Venue: arXiv cs.LG 2022
Year: 2022

Data are often accommodated on centralized storage servers. This is the case, for instance, in remote sensing and astronomy, where projects produce several petabytes of data every year. While machine learning models are often trained on relatively small subsets of the data, the inference phase typically requires transferring significant amounts of data between the servers and the clients. In many cases, the bandwidth available per user is limited, which then renders the data transfer to be one of the major bottlenecks. In this work, we propose a framework that automatically selects the relevant parts of the input data for a given neural network. The model as well as the associated selection masks are trained simultaneously such that a good model performance is achieved while only a minimal amount of data is selected. During the inference phase, only those parts of the data have to be transferred between the server and the client. We propose both instance-independent and instance-dependent selection masks. The former ones are the same for all instances to be transferred, whereas the latter ones allow for variable transfer sizes per instance. Our experiments show that it is often possible to significantly reduce the amount of data needed to be transferred without affecting the model quality much.
Value Functions for Depth-Limited Solving in Zero-Sum Imperfect-Information Games
Venue: arXiv cs.AI 2022
Year: 2022

We provide a formal definition of depth-limited games together with an accessible and rigorous explanation of the underlying concepts, both of which were previously missing in imperfect-information games. The definition works for an arbitrary extensive-form game and is not tied to any specific game-solving algorithm. Moreover, this framework unifies and significantly extends three approaches to depth-limited solving that previously existed in extensive-form games and multiagent reinforcement learning but were not known to be compatible. A key ingredient of these depth-limited games are value functions. Focusing on two-player zero-sum imperfect-information games, we show how to obtain optimal value functions and prove that public information provides both necessary and sufficient context for computing them. We provide a domain-independent encoding of the domains that allows for approximating value functions even by simple feed-forward neural networks, which are then able to generalize to unseen parts of the game. We use the resulting value network to implement a depth-limited version of counterfactual regret minimization. In three distinct domains, we show that the algorithm's exploitability is roughly linearly dependent on the value network's quality and that it is not difficult to train a value network with which depth-limited CFR's performance is as good as that of CFR with access to the full game.
A Generalized Minimax Q-learning Algorithm for Two-Player Zero-Sum Stochastic Games
Venue: arXiv cs.LG 2022
Year: 2022

We consider the problem of two-player zero-sum games. This problem is formulated as a min-max Markov game in the literature. The solution of this game, which is the min-max payoff, starting from a given state is called the min-max value of the state. In this work, we compute the solution of the two-player zero-sum game utilizing the technique of successive relaxation that has been successfully applied in the literature to compute a faster value iteration algorithm in the context of Markov Decision Processes. We extend the concept of successive relaxation to the setting of two-player zero-sum games. We show that, under a special structure on the game, this technique facilitates faster computation of the min-max value of the states. We then derive a generalized minimax Q-learning algorithm that computes the optimal policy when the model information is not known. Finally, we prove the convergence of the proposed generalized minimax Q-learning algorithm utilizing stochastic approximation techniques, under an assumption on the boundedness of iterates. Through experiments, we demonstrate the effectiveness of our proposed algorithm.
Software Engineering Practices for Machine Learning
Venue: arXiv cs.SE 2022
Year: 2022

In the last couple of years we have witnessed an enormous increase of machine learning (ML) applications. More and more program functions are no longer written in code, but learnt from a huge amount of data samples using an ML algorithm. However, what is often overlooked is the complexity of managing the resulting ML models as well as bringing these into a real production system. In software engineering, we have spent decades on developing tools and methodologies to create, manage and assemble complex software modules. We present an overview of current techniques to manage complex software, and how this applies to ML models.
CaDIS: Cataract Dataset for Image Segmentation
Venue: arXiv cs.CV 2022
Year: 2022

Video feedback provides a wealth of information about surgical procedures and is the main sensory cue for surgeons. Scene understanding is crucial to computer assisted interventions (CAI) and to post-operative analysis of the surgical procedure. A fundamental building block of such capabilities is the identification and localization of surgical instruments and anatomical structures through semantic segmentation. Deep learning has advanced semantic segmentation techniques in the recent years but is inherently reliant on the availability of labelled datasets for model training. This paper introduces a dataset for semantic segmentation of cataract surgery videos complementing the publicly available CATARACTS challenge dataset. In addition, we benchmark the performance of several state-of-the-art deep learning models for semantic segmentation on the presented dataset. The dataset is publicly available at https://cataracts-semantic-segmentation2020.grand-challenge.org/.