OpenRaiser/Intern-Atlas huggingface dataset:
Dataset Distillation with Infinitely Wide Convolutional Networks
Venue: arXiv cs.LG 2022
Year: 2022

The effectiveness of machine learning algorithms arises from being able to extract useful features from large amounts of data. As model and dataset sizes increase, dataset distillation methods that compress large datasets into significantly smaller yet highly performant ones will become valuable in terms of training efficiency and useful feature extraction. To that end, we apply a novel distributed kernel based meta-learning framework to achieve state-of-the-art results for dataset distillation using infinitely wide convolutional neural networks. For instance, using only 10 datapoints (0.02% of original dataset), we obtain over 65% test accuracy on CIFAR-10 image classification task, a dramatic improvement over the previous best test accuracy of 40%. Our state-of-the-art results extend across many other settings for MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and SVHN. Furthermore, we perform some preliminary analyses of our distilled datasets to shed light on how they differ from naturally occurring data.
Content-aware Directed Propagation Network with Pixel Adaptive Kernel Attention
Venue: arXiv cs.CV 2022
Year: 2022

Convolutional neural networks (CNNs) have been not only widespread but also achieved noticeable results on numerous applications including image classification, restoration, and generation. Although the weight-sharing property of convolutions makes them widely adopted in various tasks, its content-agnostic characteristic can also be considered a major drawback. To solve this problem, in this paper, we propose a novel operation, called pixel adaptive kernel attention (PAKA). PAKA provides directivity to the filter weights by multiplying spatially varying attention from learnable features. The proposed method infers pixel-adaptive attention maps along the channel and spatial directions separately to address the decomposed model with fewer parameters. Our method is trainable in an end-to-end manner and applicable to any CNN-based models. In addition, we propose an improved information aggregation module with PAKA, called the hierarchical PAKA module (HPM). We demonstrate the superiority of our HPM by presenting state-of-the-art performance on semantic segmentation compared to the conventional information aggregation modules. We validate the proposed method through additional ablation studies and visualizing the effect of PAKA providing directivity to the weights of convolutions. We also show the generalizability of the proposed method by applying it to multi-modal tasks especially color-guided depth map super-resolution.
Learned Optimizers for Analytic Continuation
Venue: arXiv cs.LG 2022
Year: 2022

Traditional maximum entropy and sparsity-based algorithms for analytic continuation often suffer from the ill-posed kernel matrix or demand tremendous computation time for parameter tuning. Here we propose a neural network method by convex optimization and replace the ill-posed inverse problem by a sequence of well-conditioned surrogate problems. After training, the learned optimizers are able to give a solution of high quality with low time cost and achieve higher parameter efficiency than heuristic fully-connected networks. The output can also be used as a neural default model to improve the maximum entropy for better performance. Our methods may be easily extended to other high-dimensional inverse problems via large-scale pretraining.
Pseudo-LiDAR Based Road Detection
Venue: arXiv cs.CV 2022
Year: 2022

Road detection is a critically important task for self-driving cars. By employing LiDAR data, recent works have significantly improved the accuracy of road detection. Relying on LiDAR sensors limits the wide application of those methods when only cameras are available. In this paper, we propose a novel road detection approach with RGB being the only input during inference. Specifically, we exploit pseudo-LiDAR using depth estimation, and propose a feature fusion network where RGB and learned depth information are fused for improved road detection. To further optimize the network structure and improve the efficiency of the network. we search for the network structure of the feature fusion module using NAS techniques. Finally, be aware of that generating pseudo-LiDAR from RGB via depth estimation introduces extra computational costs and relies on depth estimation networks, we design a modality distillation strategy and leverage it to further free our network from these extra computational cost and dependencies during inference. The proposed method achieves state-of-the-art performance on two challenging benchmarks, KITTI and R2D.
Predicting Patch Correctness Based on the Similarity of Failing Test Cases
Venue: arXiv cs.SE 2022
Year: 2022

Towards predicting patch correctness in APR, we propose a simple, but novel hypothesis on how the link between the patch behaviour and failing test specifications can be drawn: similar failing test cases should require similar patches. We then propose BATS, an unsupervised learning-based system to predict patch correctness by checking patch Behaviour Against failing Test Specification. BATS exploits deep representation learning models for code and patches: for a given failing test case, the yielded embedding is used to compute similarity metrics in the search for historical similar test cases in order to identify the associated applied patches, which are then used as a proxy for assessing generated patch correctness. Experimentally, we first validate our hypothesis by assessing whether ground-truth developer patches cluster together in the same way that their associated failing test cases are clustered. Then, after collecting a large dataset of 1278 plausible patches (written by developers or generated by some 32 APR tools), we use BATS to predict correctness: BATS achieves an AUC between 0.557 to 0.718 and a recall between 0.562 and 0.854 in identifying correct patches. Compared against previous work, we demonstrate that our approach outperforms state-of-the-art performance in patch correctness prediction, without the need for large labeled patch datasets in contrast with prior machine learning-based approaches. While BATS is constrained by the availability of similar test cases, we show that it can still be complementary to existing approaches: used in conjunction with a recent approach implementing supervised learning, BATS improves the overall recall in detecting correct patches. We finally show that BATS can be complementary to the state-of-the-art PATCH-SIM dynamic approach of identifying the correct patches for APR tools.
$(1+ε)$-Approximate Shortest Paths in Dynamic Streams
Venue: arXiv cs.DS 2022
Year: 2022

Computing approximate shortest paths in the dynamic streaming setting is a fundamental challenge that has been intensively studied during the last decade. Currently existing solutions for this problem either build a sparse multiplicative spanner of the input graph and compute shortest paths in the spanner offline, or compute an exact single source BFS tree. Solutions of the first type are doomed to incur a stretch-space tradeoff of $2κ-1$ versus $n^{1+1/κ}$, for an integer parameter $κ$. (In fact, existing solutions also incur an extra factor of $1+ε$ in the stretch for weighted graphs, and an additional factor of $\log^{O(1)}n$ in the space.) The only existing solution of the second type uses $n^{1/2 - O(1/κ)}$ passes over the stream (for space $O(n^{1+1/κ})$), and applies only to unweighted graphs. In this paper we show that $(1+ε)$-approximate single-source shortest paths can be computed in this setting with $\tilde{O}(n^{1+1/κ})$ space using just \emph{constantly} many passes in unweighted graphs, and polylogarithmically many passes in weighted graphs (assuming $ε$ and $κ$ are constant). Moreover, in fact, the same result applies for multi-source shortest paths, as long as the number of sources is $O(n^{1/κ})$. We achieve these results by devising efficient dynamic streaming constructions of $(1 + ε, β)$-spanners and hopsets. We believe that these constructions are of independent interest.
Chance constrained conic-segmentation support vector machine with uncertain data
Venue: arXiv cs.LG 2022
Year: 2022

Support vector machines (SVM) is one of the well known supervised classes of learning algorithms. Furthermore, the conic-segmentation SVM (CS-SVM) is a natural multiclass analogue of the standard binary SVM, as CS-SVM models are dealing with the situation where the exact values of the data points are known. This paper studies CS-SVM when the data points are uncertain or mislabelled. With some properties known for the distributions, a chance-constrained CS-SVM approach is used to ensure the small probability of misclassification for the uncertain data. The geometric interpretation is presented to show how CS-SVM works. Finally, we present experimental results to investigate the chance constrained CS-SVM's performance.
Study on Precoding Optimization Algorithms in Massive MIMO System with Multi-Antenna Users
Venue: arXiv cs.IT 2022
Year: 2022

The paper studies the multi-user precoding problem as a non-convex optimization problem for wireless multiple input and multiple output (MIMO) systems. In our work, we approximate the target Spectral Efficiency function with a novel computationally simpler function. Then, we reduce the precoding problem to an unconstrained optimization task using a special differential projection method and solve it by the Quasi-Newton L-BFGS iterative procedure to achieve gains in capacity. We are testing the proposed approach in several scenarios generated using Quadriga - open-source software for generating realistic radio channel impulse response. Our method shows monotonic improvement over heuristic methods with reasonable computation time. The proposed L-BFGS optimization scheme is novel in this area and shows a significant advantage over the standard approaches. The proposed method has a simple implementation and can be a good reference for other heuristic algorithms in this field.
Learning the shape of female breasts: an open-access 3D statistical shape model of the female breast built from 110 breast scans
Venue: arXiv cs.CV 2022
Year: 2022

We present the Regensburg Breast Shape Model (RBSM) -- a 3D statistical shape model of the female breast built from 110 breast scans acquired in a standing position, and the first publicly available. Together with the model, a fully automated, pairwise surface registration pipeline used to establish dense correspondence among 3D breast scans is introduced. Our method is computationally efficient and requires only four landmarks to guide the registration process. A major challenge when modeling female breasts from surface-only 3D breast scans is the non-separability of breast and thorax. In order to weaken the strong coupling between breast and surrounding areas, we propose to minimize the variance outside the breast region as much as possible. To achieve this goal, a novel concept called breast probability masks (BPMs) is introduced. A BPM assigns probabilities to each point of a 3D breast scan, telling how likely it is that a particular point belongs to the breast area. During registration, we use BPMs to align the template to the target as accurately as possible inside the breast region and only roughly outside. This simple yet effective strategy significantly reduces the unwanted variance outside the breast region, leading to better statistical shape models in which breast shapes are quite well decoupled from the thorax. The RBSM is thus able to produce a variety of different breast shapes as independently as possible from the shape of the thorax. Our systematic experimental evaluation reveals a generalization ability of 0.17 mm and a specificity of 2.8 mm. To underline the expressiveness of the proposed model, we finally demonstrate in two showcase applications how the RBSM can be used for surgical outcome simulation and the prediction of a missing breast from the remaining one. Our model is available at https://www.rbsm.re-mic.de/.
Social Processes: Self-Supervised Meta-Learning over Conversational Groups for Forecasting Nonverbal Social Cues
Venue: arXiv cs.LG 2022
Year: 2022

Free-standing social conversations constitute a yet underexplored setting for human behavior forecasting. While the task of predicting pedestrian trajectories has received much recent attention, an intrinsic difference between these settings is how groups form and disband. Evidence from social psychology suggests that group members in a conversation explicitly self-organize to sustain the interaction by adapting to one another's behaviors. Crucially, the same individual is unlikely to adapt similarly across different groups; contextual factors such as perceived relationships, attraction, rapport, etc., influence the entire spectrum of participants' behaviors. A question arises: how can we jointly forecast the mutually dependent futures of conversation partners by modeling the dynamics unique to every group? In this paper, we propose the Social Process (SP) models, taking a novel meta-learning and stochastic perspective of group dynamics. Training group-specific forecasting models hinders generalization to unseen groups and is challenging given limited conversation data. In contrast, our SP models treat interaction sequences from a single group as a meta-dataset: we condition forecasts for a sequence from a given group on other observed-future sequence pairs from the same group. In this way, an SP model learns to adapt its forecasts to the unique dynamics of the interacting partners, generalizing to unseen groups in a data-efficient manner. Additionally, we first rethink the task formulation itself, motivating task requirements from social science literature that prior formulations have overlooked. For our formulation of Social Cue Forecasting, we evaluate the empirical performance of our SP models against both non-meta-learning and meta-learning approaches with similar assumptions. The SP models yield improved performance on synthetic and real-world behavior datasets.
Text Mining Undergraduate Engineering Programs' Applications: the Role of Gender, Nationality, and Socio-economic Status
Venue: arXiv cs.CY 2022
Year: 2022

Women, visible minorities, and other socially disadvantaged groups continue to be underrepresented in STEM education. Understanding students' motivations for pursuing a STEM major, and the roles gender, nationality, parental education attainment, and socio-economic background play in shaping students' motivations can support the design of more effective recruitment efforts towards these groups. In this paper, we propose and develop a novel text mining approach incorporating the Latent Dirichlet Allocation and word embeddings to analyze applicants' motivational factors for choosing an engineering program. We apply the proposed method to a dataset of 43,645 applications to the engineering school of a large Canadian university. We then investigate the relationship between applicants' gender, nationality, and family income and educational attainment, and their stated motivations for applying to their engineering program of choice. We find that interest in technology and the desire to make social impact are the two most powerful motivators for applicants. Additionally, while we find significant motivational differences related to applicants' nationality and family socio-economic status, gender has the strongest and the most robust impact on students' motivations for studying engineering.
Safest Nearby Neighbor Queries in Road Networks (Full Version)
Venue: arXiv cs.DB 2022
Year: 2022

Traditional route planning and k nearest neighbors queries only consider distance or travel time and ignore road safety altogether. However, many travellers prefer to avoid risky or unpleasant road conditions such as roads with high crime rates (e.g., robberies, kidnapping, riots etc.) and bumpy roads. To facilitate safe travel, we introduce a novel query for road networks called the k safest nearby neighbors (kSNN) query. Given a query location $v_l$, a distance constraint $d_c$ and a point of interest $p_i$, we define the safest path from $v_l$ to $p_i$ as the path with the highest path safety score among all the paths from $v_l$ to $p_i$ with length less than $d_c$. The path safety score is computed considering the road safety of each road segment on the path. Given a query location $v_l$, a distance constraint $d_c$ and a set of POIs P, a kSNN query returns k POIs with the k highest path safety scores in P along with their respective safest paths from the query location. We develop two novel indexing structures called Ct-tree and a safety score based Voronoi diagram (SNVD). We propose two efficient query processing algorithms each exploiting one of the proposed indexes to effectively refine the search space using the properties of the index. Our extensive experimental study on real datasets demonstrates that our solution is on average an order of magnitude faster than the baselines.
A Framework for Adversarial Streaming via Differential Privacy and Difference Estimators
Venue: arXiv cs.DS 2022
Year: 2022

Classical streaming algorithms operate under the (not always reasonable) assumption that the input stream is fixed in advance. Recently, there is a growing interest in designing robust streaming algorithms that provide provable guarantees even when the input stream is chosen adaptively as the execution progresses. We propose a new framework for robust streaming that combines techniques from two recently suggested frameworks by Hassidim et al. [NeurIPS 2020] and by Woodruff and Zhou [FOCS 2021]. These recently suggested frameworks rely on very different ideas, each with its own strengths and weaknesses. We combine these two frameworks into a single hybrid framework that obtains the ``best of both worlds'', thereby solving a question left open by Woodruff and Zhou.
An Efficient DP-SGD Mechanism for Large Scale NLP Models
Venue: arXiv cs.CL 2022
Year: 2022

Recent advances in deep learning have drastically improved performance on many Natural Language Understanding (NLU) tasks. However, the data used to train NLU models may contain private information such as addresses or phone numbers, particularly when drawn from human subjects. It is desirable that underlying models do not expose private information contained in the training data. Differentially Private Stochastic Gradient Descent (DP-SGD) has been proposed as a mechanism to build privacy-preserving models. However, DP-SGD can be prohibitively slow to train. In this work, we propose a more efficient DP-SGD for training using a GPU infrastructure and apply it to fine-tuning models based on LSTM and transformer architectures. We report faster training times, alongside accuracy, theoretical privacy guarantees and success of Membership inference attacks for our models and observe that fine-tuning with proposed variant of DP-SGD can yield competitive models without significant degradation in training time and improvement in privacy protection. We also make observations such as looser theoretical $ε, δ$ can translate into significant practical privacy gains.
Long-Range Optical Wireless Information and Power Transfer
Venue: arXiv cs.ET 2022
Year: 2022

Simultaneous wireless information and power transfer (SWIPT) is a remarkable technology to support both the data and the energy transfer in the era of Internet of Things (IoT). In this paper, we proposed a long-range optical wireless information and power transfer system utilizing retro-reflectors, a gain medium, a telescope internal modulator to form the resonant beam, achieving high-power and high-rate SWIPT. We adopt the transfer matrix, which can depict the beam modulated, resonator stability, transmission loss, and beam distribution. Then, we provide a model for energy harvesting and data receiving, which can evaluate the SWIPT performance. Numerical results illustrate that the proposed system can simultaneously supply 0$\sim$9 W electrical power and 18 bit/s/Hz spectral efficiency over 20 m distance.
Sequential Blocked Matching
Venue: arXiv cs.GT 2022
Year: 2022

We consider a sequential blocked matching (SBM) model where strategic agents repeatedly report ordinal preferences over a set of services to a central planner. The planner's goal is to elicit agents' true preferences and design a policy that matches services to agents in order to maximize the expected social welfare with the added constraint that each matched service can be \emph{blocked} or unavailable for a number of time periods. Naturally, SBM models the repeated allocation of reusable services to a set of agents where each allocated service becomes unavailable for a fixed duration. We first consider the offline SBM setting, where the strategic agents are aware of their true preferences. We measure the performance of any policy by \emph{distortion}, the worst-case multiplicative approximation guaranteed by any policy. For the setting with $s$ services, we establish lower bounds of $Ω(s)$ and $Ω(\sqrt{s})$ on the distortions of any deterministic and randomised mechanisms, respectively. We complement these results by providing approximately truthful, measured by \emph{incentive ratio}, deterministic and randomised policies based on random serial dictatorship which match our lower bounds. Our results show that there is a significant improvement if one considers the class of randomised policies. Finally, we consider the online SBM setting with bandit feedback where each agent is initially unaware of her true preferences, and the planner must facilitate each agent in the learning of their preferences through the matching of services over time. We design an approximately truthful mechanism based on the Explore-then-Commit paradigm, which achieves logarithmic dynamic approximate regret.
Learning to Control DC Motor for Micromobility in Real Time with Reinforcement Learning
Venue: arXiv cs.LG 2022
Year: 2022

Autonomous micromobility has been attracting the attention of researchers and practitioners in recent years. A key component of many micro-transport vehicles is the DC motor, a complex dynamical system that is continuous and non-linear. Learning to quickly control the DC motor in the presence of disturbances and uncertainties is desired for various applications that require robustness and stability. Techniques to accomplish this task usually rely on a mathematical system model, which is often insufficient to anticipate the effects of time-varying and interrelated sources of non-linearities. While some model-free approaches have been successful at the task, they rely on massive interactions with the system and are trained in specialized hardware in order to fit a highly parameterized controller. In this work, we learn to steer a DC motor via sample-efficient reinforcement learning. Using data collected from hardware interactions in the real world, we additionally build a simulator to experiment with a wide range of parameters and learning strategies. With the best parameters found, we learn an effective control policy in one minute and 53 seconds on a simulation and in 10 minutes and 35 seconds on a physical system.