OpenRaiser/Intern-Atlas huggingface dataset:
Analysis and Design of Quadratic Neural Networks for Regression, Classification, and Lyapunov Control of Dynamical Systems
Venue: arXiv cs.LG 2022
Year: 2022

This paper addresses the analysis and design of quadratic neural networks, which have been recently introduced in the literature, and their applications to regression, classification, system identification and control of dynamical systems. These networks offer several advantages, the most important of which are the fact that the architecture is a by-product of the design and is not determined a-priori, their training can be done by solving a convex optimization problem so that the global optimum of the weights is achieved, and the input-output mapping can be expressed analytically by a quadratic form. It also appears from several examples that these networks work extremely well using only a small fraction of the training data. The results in the paper cast regression, classification, system identification, stability and control design as convex optimization problems, which can be solved efficiently with polynomial-time algorithms to a global optimum. Several examples will show the effectiveness of quadratic neural networks in applications.
Scheduling under Non-Uniform Job and Machine Delays
Venue: arXiv cs.DS 2022
Year: 2022

We study the problem of scheduling precedence-constrained jobs on heterogenous machines in the presence of non-uniform job and machine communication delays. We are given as input $n$ unit size precedence-ordered jobs and $m$ related machines such that machine $i$ can execute up to $m_i$ jobs at a time. Each machine $i$ has an in-delay $ρ^{\mathrm{in}}_i$ and out-delay $ρ^{\mathrm{out}}_i$. Likewise, each job $v$ has an in-delay $ρ^{\mathrm{in}}_v$ and out-delay $ρ^{\mathrm{out}}_v$. In a schedule, job $v$ may be executed on machine $i$ at time $t$ if each predecessor $u$ of $v$ is completed on $i$ before time $t$ or on any machine $j$ before time $t - (ρ^{\mathrm{in}}_i + ρ^{\mathrm{out}}_j + ρ^{\mathrm{out}}_u + ρ^{\mathrm{in}}_v)$. The goal is to construct a schedule that minimizes makespan. We consider schedules that allow duplication of jobs as well as schedules which do not. When duplication is allowed, we provide an asymptotic $\mathrm{polylog}(n)$-approximation algorithms both when duplication is allowed and when it is not. We also obtain a true $\mathrm{polylog}(n)$-approximation for symmetric machine and job delays. These are the first polylogarithmic approximation algorithms for scheduling with non-uniform communication delays. We also consider a more general model, where the delay can be an arbitrary function of the job and the machine executing it: job $v$ can be executed on machine $i$ at time $t$ if all of $v$'s predecessors are executed on $i$ by time $t-1$ or on any machine by time $t - ρ_{v,i}$. We present an approximation-preserving reduction from the Unique Machines Precedence-constrained Scheduling (UMPS) problem, first defined in [DKRSTZ22], to this job-machine delay model. The reduction entails logarithmic hardness for this delay setting, as well as polynomial hardness if the conjectured hardness of UMPS holds.
Retrieval-Augmented Transformer for Image Captioning
Venue: arXiv cs.CV 2022
Year: 2022

Image captioning models aim at connecting Vision and Language by providing natural language descriptions of input images. In the past few years, the task has been tackled by learning parametric models and proposing visual feature extraction advancements or by modeling better multi-modal connections. In this paper, we investigate the development of an image captioning approach with a kNN memory, with which knowledge can be retrieved from an external corpus to aid the generation process. Our architecture combines a knowledge retriever based on visual similarities, a differentiable encoder, and a kNN-augmented attention layer to predict tokens based on the past context and on text retrieved from the external memory. Experimental results, conducted on the COCO dataset, demonstrate that employing an explicit external memory can aid the generation process and increase caption quality. Our work opens up new avenues for improving image captioning models at larger scale.
Perception-Aware Attack: Creating Adversarial Music via Reverse-Engineering Human Perception
Venue: arXiv cs.SD 2022
Year: 2022

Recently, adversarial machine learning attacks have posed serious security threats against practical audio signal classification systems, including speech recognition, speaker recognition, and music copyright detection. Previous studies have mainly focused on ensuring the effectiveness of attacking an audio signal classifier via creating a small noise-like perturbation on the original signal. It is still unclear if an attacker is able to create audio signal perturbations that can be well perceived by human beings in addition to its attack effectiveness. This is particularly important for music signals as they are carefully crafted with human-enjoyable audio characteristics. In this work, we formulate the adversarial attack against music signals as a new perception-aware attack framework, which integrates human study into adversarial attack design. Specifically, we conduct a human study to quantify the human perception with respect to a change of a music signal. We invite human participants to rate their perceived deviation based on pairs of original and perturbed music signals, and reverse-engineer the human perception process by regression analysis to predict the human-perceived deviation given a perturbed signal. The perception-aware attack is then formulated as an optimization problem that finds an optimal perturbation signal to minimize the prediction of perceived deviation from the regressed human perception model. We use the perception-aware framework to design a realistic adversarial music attack against YouTube's copyright detector. Experiments show that the perception-aware attack produces adversarial music with significantly better perceptual quality than prior work.
Boosting Point-BERT by Multi-choice Tokens
Venue: arXiv cs.CV 2022
Year: 2022

Masked language modeling (MLM) has become one of the most successful self-supervised pre-training task. Inspired by its success, Point-BERT, as a pioneer work in point cloud, proposed masked point modeling (MPM) to pre-train point transformer on large scale unanotated dataset. Despite its great performance, we find the inherent difference between language and point cloud tends to cause ambiguous tokenization for point cloud. For point cloud, there doesn't exist a gold standard for point cloud tokenization. Point-BERT use a discrete Variational AutoEncoder (dVAE) as tokenizer, but it might generate different token ids for semantically-similar patches and generate the same token ids for semantically-dissimilar patches. To tackle above problem, we propose our McP-BERT, a pre-training framework with multi-choice tokens. Specifically, we ease the previous single-choice constraint on patch token ids in Point-BERT, and provide multi-choice token ids for each patch as supervision. Moreover, we utilitze the high-level semantics learned by transformer to further refine our supervision signals. Extensive experiments on point cloud classification, few-shot classification and part segmentation tasks demonstrate the superiority of our method, e.g., the pre-trained transformer achieves 94.1% accuracy on ModelNet40, 84.28% accuracy on the hardest setting of ScanObjectNN and new state-of-the-art performance on few-shot learning. We also demonstrate that our method not only improves the performance of Point-BERT on all downstream tasks, but also incurs almost no extra computational overhead. The code will be released in https://github.com/fukexue/McP-BERT.
Back in the US-SR: Unlimited Sampling and Sparse Super-Resolution with its Hardware Validation
Venue: arXiv cs.IT 2022
Year: 2022

The Unlimited Sensing Framework (USF) is a digital acquisition protocol that allows for sampling and reconstruction of high dynamic range signals. By acquiring modulo samples, the USF circumvents the clipping or saturation problem that is a fundamental bottleneck in conventional analog-to-digital converters (ADCs). In the context of the USF, several works have focused on bandlimited function classes and recently, a hardware validation of the modulo sampling approach has been presented. In a different direction, in this paper we focus on non-bandlimited function classes and consider the well-known super-resolution problem; we study the recovery of sparse signals (Dirac impulses) from low-pass filtered, modulo samples. Taking an end-to-end approach to USF based super-resolution, we present a novel recovery algorithm (US-SR) that leverages a doubly sparse structure of the modulo samples. We derive a sampling criterion for the US-SR method. A hardware experiment with the modulo ADC demonstrates the empirical robustness of our method in a realistic, noisy setting, thus validating its practical utility.
BrainActivity1: A Framework of EEG Data Collection and Machine Learning Analysis for College Students
Venue: arXiv cs.DB 2022
Year: 2022

Using Machine Learning and Deep Learning to predict cognitive tasks from electroencephalography (EEG) signals has been a fast-developing area in Brain-Computer Interfaces (BCI). However, during the COVID-19 pandemic, data collection and analysis could be more challenging than before. This paper explored machine learning algorithms that can run efficiently on personal computers for BCI classification tasks. Also, we investigated a way to conduct such BCI experiments remotely via Zoom. The results showed that Random Forest and RBF SVM performed well for EEG classification tasks. The remote experiment during the pandemic yielded several challenges, and we discussed the possible solutions; nevertheless, we developed a protocol that grants non-experts who are interested a guideline for such data collection.
Cubic Goldreich-Levin
Venue: arXiv cs.DS 2022
Year: 2022

In this paper, we give a cubic Goldreich-Levin algorithm which makes polynomially-many queries to a function $f \colon \mathbb F_p^n \to \mathbb C$ and produces a decomposition of $f$ as a sum of cubic phases and a small error term. This is a natural higher-order generalization of the classical Goldreich-Levin algorithm. The classical (linear) Goldreich-Levin algorithm has wide-ranging applications in learning theory, coding theory and the construction of pseudorandom generators in cryptography, as well as being closely related to Fourier analysis. Higher-order Goldreich-Levin algorithms on the other hand involve central problems in higher-order Fourier analysis, namely the inverse theory of the Gowers $U^k$ norms, which are well-studied in additive combinatorics. The only known result in this direction prior to this work is the quadratic Goldreich-Levin theorem, proved by Tulsiani and Wolf in 2011. The main step of their result involves an algorithmic version of the $U^3$ inverse theorem. More complications appear in the inverse theory of the $U^4$ and higher norms. Our cubic Goldreich-Levin algorithm is based on algorithmizing recent work by Gowers and Milićević who proved new quantitative bounds for the $U^4$ inverse theorem. Our cubic Goldreich-Levin algorithm is constructed from two main tools: an algorithmic $U^4$ inverse theorem and an arithmetic decomposition result in the style of the Frieze-Kannan graph regularity lemma. As one application of our main theorem we solve the problem of self-correction for cubic Reed-Muller codes beyond the list decoding radius. Additionally we give a purely combinatorial result: an improvement of the quantitative bounds on the $U^4$ inverse theorem.
Federated Selective Aggregation for Knowledge Amalgamation
Venue: arXiv cs.CV 2022
Year: 2022

In this paper, we explore a new knowledge-amalgamation problem, termed Federated Selective Aggregation (FedSA). The goal of FedSA is to train a student model for a new task with the help of several decentralized teachers, whose pre-training tasks and data are different and agnostic. Our motivation for investigating such a problem setup stems from a recent dilemma of model sharing. Many researchers or institutes have spent enormous resources on training large and competent networks. Due to the privacy, security, or intellectual property issues, they are, however, not able to share their own pre-trained models, even if they wish to contribute to the community. The proposed FedSA offers a solution to this dilemma and makes it one step further since, again, the learned student may specialize in a new task different from all of the teachers. To this end, we proposed a dedicated strategy for handling FedSA. Specifically, our student-training process is driven by a novel saliency-based approach that adaptively selects teachers as the participants and integrates their representative capabilities into the student. To evaluate the effectiveness of FedSA, we conduct experiments on both single-task and multi-task settings. Experimental results demonstrate that FedSA effectively amalgamates knowledge from decentralized models and achieves competitive performance to centralized baselines.
JDRec: Practical Actor-Critic Framework for Online Combinatorial Recommender System
Venue: arXiv cs.IR 2022
Year: 2022

A combinatorial recommender (CR) system feeds a list of items to a user at a time in the result page, in which the user behavior is affected by both contextual information and items. The CR is formulated as a combinatorial optimization problem with the objective of maximizing the recommendation reward of the whole list. Despite its importance, it is still a challenge to build a practical CR system, due to the efficiency, dynamics, personalization requirement in online environment. In particular, we tear the problem into two sub-problems, list generation and list evaluation. Novel and practical model architectures are designed for these sub-problems aiming at jointly optimizing effectiveness and efficiency. In order to adapt to online case, a bootstrap algorithm forming an actor-critic reinforcement framework is given to explore better recommendation mode in long-term user interaction. Offline and online experiment results demonstrate the efficacy of proposed JDRec framework. JDRec has been applied in online JD recommendation, improving click through rate by 2.6% and synthetical value for the platform by 5.03%. We will publish the large-scale dataset used in this study to contribute to the research community.
Portrait Interpretation and a Benchmark
Venue: arXiv cs.CV 2022
Year: 2022

We propose a task we name Portrait Interpretation and construct a dataset named Portrait250K for it. Current researches on portraits such as human attribute recognition and person re-identification have achieved many successes, but generally, they: 1) may lack mining the interrelationship between various tasks and the possible benefits it may bring; 2) design deep models specifically for each task, which is inefficient; 3) may be unable to cope with the needs of a unified model and comprehensive perception in actual scenes. In this paper, the proposed portrait interpretation recognizes the perception of humans from a new systematic perspective. We divide the perception of portraits into three aspects, namely Appearance, Posture, and Emotion, and design corresponding sub-tasks for each aspect. Based on the framework of multi-task learning, portrait interpretation requires a comprehensive description of static attributes and dynamic states of portraits. To invigorate research on this new task, we construct a new dataset that contains 250,000 images labeled with identity, gender, age, physique, height, expression, and posture of the whole body and arms. Our dataset is collected from 51 movies, hence covering extensive diversity. Furthermore, we focus on representation learning for portrait interpretation and propose a baseline that reflects our systematic perspective. We also propose an appropriate metric for this task. Our experimental results demonstrate that combining the tasks related to portrait interpretation can yield benefits. Code and dataset will be made public.
Two-Stream UNET Networks for Semantic Segmentation in Medical Images
Venue: arXiv cs.CV 2022
Year: 2022

Recent advances of semantic image segmentation greatly benefit from deeper and larger Convolutional Neural Network (CNN) models. Compared to image segmentation in the wild, properties of both medical images themselves and of existing medical datasets hinder training deeper and larger models because of overfitting. To this end, we propose a novel two-stream UNET architecture for automatic end-to-end medical image segmentation, in which intensity value and gradient vector flow (GVF) are two inputs for each stream, respectively. We demonstrate that two-stream CNNs with more low-level features greatly benefit semantic segmentation for imperfect medical image datasets. Our proposed two-stream networks are trained and evaluated on the popular medical image segmentation benchmarks, and the results are competitive with the state of the art. The code will be released soon.
Traffic Sign Detection With Event Cameras and DCNN
Venue: arXiv cs.CV 2022
Year: 2022

In recent years, event cameras (DVS - Dynamic Vision Sensors) have been used in vision systems as an alternative or supplement to traditional cameras. They are characterised by high dynamic range, high temporal resolution, low latency, and reliable performance in limited lighting conditions -- parameters that are particularly important in the context of advanced driver assistance systems (ADAS) and self-driving cars. In this work, we test whether these rather novel sensors can be applied to the popular task of traffic sign detection. To this end, we analyse different representations of the event data: event frame, event frequency, and the exponentially decaying time surface, and apply video frame reconstruction using a deep neural network called FireNet. We use the deep convolutional neural network YOLOv4 as a detector. For particular representations, we obtain a detection accuracy in the range of 86.9-88.9% mAP@0.5. The use of a fusion of the considered representations allows us to obtain a detector with higher accuracy of 89.9% mAP@0.5. In comparison, the detector for the frames reconstructed with FireNet is characterised by an accuracy of 72.67% mAP@0.5. The results obtained illustrate the potential of event cameras in automotive applications, either as standalone sensors or in close cooperation with typical frame-based cameras.
Time Series Forecasting Models Copy the Past: How to Mitigate
Venue: arXiv cs.LG 2022
Year: 2022

Time series forecasting is at the core of important application domains posing significant challenges to machine learning algorithms. Recently neural network architectures have been widely applied to the problem of time series forecasting. Most of these models are trained by minimizing a loss function that measures predictions' deviation from the real values. Typical loss functions include mean squared error (MSE) and mean absolute error (MAE). In the presence of noise and uncertainty, neural network models tend to replicate the last observed value of the time series, thus limiting their applicability to real-world data. In this paper, we provide a formal definition of the above problem and we also give some examples of forecasts where the problem is observed. We also propose a regularization term penalizing the replication of previously seen values. We evaluate the proposed regularization term both on synthetic and real-world datasets. Our results indicate that the regularization term mitigates to some extent the aforementioned problem and gives rise to more robust models.
Safe and Robust Experience Sharing for Deterministic Policy Gradient Algorithms
Venue: arXiv cs.LG 2022
Year: 2022

Learning in high dimensional continuous tasks is challenging, mainly when the experience replay memory is very limited. We introduce a simple yet effective experience sharing mechanism for deterministic policies in continuous action domains for the future off-policy deep reinforcement learning applications in which the allocated memory for the experience replay buffer is limited. To overcome the extrapolation error induced by learning from other agents' experiences, we facilitate our algorithm with a novel off-policy correction technique without any action probability estimates. We test the effectiveness of our method in challenging OpenAI Gym continuous control tasks and conclude that it can achieve a safe experience sharing across multiple agents and exhibits a robust performance when the replay memory is strictly limited.
Towards the Probabilistic Fusion of Learned Priors into Standard Pipelines for 3D Reconstruction
Venue: arXiv cs.CV 2022
Year: 2022

The best way to combine the results of deep learning with standard 3D reconstruction pipelines remains an open problem. While systems that pass the output of traditional multi-view stereo approaches to a network for regularisation or refinement currently seem to get the best results, it may be preferable to treat deep neural networks as separate components whose results can be probabilistically fused into geometry-based systems. Unfortunately, the error models required to do this type of fusion are not well understood, with many different approaches being put forward. Recently, a few systems have achieved good results by having their networks predict probability distributions rather than single values. We propose using this approach to fuse a learned single-view depth prior into a standard 3D reconstruction system. Our system is capable of incrementally producing dense depth maps for a set of keyframes. We train a deep neural network to predict discrete, nonparametric probability distributions for the depth of each pixel from a single image. We then fuse this "probability volume" with another probability volume based on the photometric consistency between subsequent frames and the keyframe image. We argue that combining the probability volumes from these two sources will result in a volume that is better conditioned. To extract depth maps from the volume, we minimise a cost function that includes a regularisation term based on network predicted surface normals and occlusion boundaries. Through a series of experiments, we demonstrate that each of these components improves the overall performance of the system.
PASTA-GAN++: A Versatile Framework for High-Resolution Unpaired Virtual Try-on
Venue: arXiv cs.CV 2022
Year: 2022

Image-based virtual try-on is one of the most promising applications of human-centric image generation due to its tremendous real-world potential. In this work, we take a step forwards to explore versatile virtual try-on solutions, which we argue should possess three main properties, namely, they should support unsupervised training, arbitrary garment categories, and controllable garment editing. To this end, we propose a characteristic-preserving end-to-end network, the PAtch-routed SpaTially-Adaptive GAN++ (PASTA-GAN++), to achieve a versatile system for high-resolution unpaired virtual try-on. Specifically, our PASTA-GAN++ consists of an innovative patch-routed disentanglement module to decouple the intact garment into normalized patches, which is capable of retaining garment style information while eliminating the garment spatial information, thus alleviating the overfitting issue during unsupervised training. Furthermore, PASTA-GAN++ introduces a patch-based garment representation and a patch-guided parsing synthesis block, allowing it to handle arbitrary garment categories and support local garment editing. Finally, to obtain try-on results with realistic texture details, PASTA-GAN++ incorporates a novel spatially-adaptive residual module to inject the coarse warped garment feature into the generator. Extensive experiments on our newly collected UnPaired virtual Try-on (UPT) dataset demonstrate the superiority of PASTA-GAN++ over existing SOTAs and its ability for controllable garment editing.