Miracle-dz/UniFAR-FTUs huggingface dataset:
Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture

Membership inference attacks are a key measure to evaluate privacy leakage in machine learning (ML) models. These attacks aim to distinguish training members from non-members by exploiting differential behavior of the models on member and non-member inputs. The goal of this work is to train ML models that have high membership privacy while largely preserving their utility; we therefore aim for an empirical membership privacy guarantee as opposed to the provable privacy guarantees provided by techniques like differential privacy, as such techniques are shown to deteriorate model utility. Specifically, we propose a new framework to train privacy-preserving models that induces similar behavior on member and non-member inputs to mitigate membership inference attacks. Our framework, called SELENA, has two major components. The first component and the core of our defense is a novel ensemble architecture for training. This architecture, which we call Split-AI, splits the training data into random subsets, and trains a model on each subset of the data. We use an adaptive inference strategy at test time: our ensemble architecture aggregates the outputs of only those models that did not contain the input sample in their training data. We prove that our Split-AI architecture defends against a large family of membership inference attacks, however, it is susceptible to new adaptive attacks. Therefore, we use a second component in our framework called Self-Distillation to protect against such stronger attacks. The Self-Distillation component (self-)distills the training dataset through our Split-AI ensemble, without using any external public datasets. Through extensive experiments on major benchmark datasets we show that SELENA presents a superior trade-off between membership privacy and utility compared to the state of the art.
Sudden appearance of a thick Dirichlet wall in a cavity

We investigate the vacuum properties of a massless scalar field theory in constrained spatial geometry, namely, instantaneous appearance of a thick Dirichlet boundary inside a one dimensional (1D) Dirichlet cavity and divides it into two parts. Our work presents the calculations for the energy density and particle number density created. The expression of the energy density is found to be dependent on the size nature of the appearing wall, more precisely, it depends on whether the length of the wall is rational or irrational.
Natural Attribute-based Shift Detection

Despite the impressive performance of deep networks in vision, language, and healthcare, unpredictable behaviors on samples from the distribution different than the training distribution cause severe problems in deployment. For better reliability of neural-network-based classifiers, we define a new task, natural attribute-based shift (NAS) detection, to detect the samples shifted from the training distribution by some natural attribute such as age of subjects or brightness of images. Using the natural attributes present in existing datasets, we introduce benchmark datasets in vision, language, and medical for NAS detection. Further, we conduct an extensive evaluation of prior representative out-of-distribution (OOD) detection methods on NAS datasets and observe an inconsistency in their performance. To understand this, we provide an analysis on the relationship between the location of NAS samples in the feature space and the performance of distance- and confidence-based OOD detection methods. Based on the analysis, we split NAS samples into three categories and further suggest a simple modification to the training objective to obtain an improved OOD detection method that is capable of detecting samples from all NAS categories.
Non-Euclidean Contractivity of Recurrent Neural Networks

Critical questions in dynamical neuroscience and machine learning are related to the study of recurrent neural networks and their stability, robustness, and computational efficiency. These properties can be simultaneously established via a contraction analysis.This paper develops a comprehensive contraction theory for recurrent neural networks. First, for non-Euclidean ℓ1/ℓ∞ logarithmic norms, we establish quasiconvexity with respect to positive diagonal weights and closed-form worst-case expressions over certain matrix polytopes. Second, for locally Lipschitz maps (e.g., arising as activation functions), we show that their one-sided Lipschitz constant equals the essential supremum of the logarithmic norm of their Jacobian. Third and final, we apply these general results to classes of recurrent neural circuits, including Hopfield, firing rate, Persidskii, Lur’e and other models. For each model, we compute the optimal contraction rate and corresponding weighted non-Euclidean norm via a linear program or, in some special cases, via a Hurwitz condition on the Metzler majorant of the synaptic matrix. Our non-Euclidean analysis establishes also absolute, connective, and total contraction properties.
The Fitting height of finite groups with a fixed-point-free automorphism satisfying an identity

Motivated by classic theorems of Thompson and Berger on the Fitting height of finite groups with a fixed-point-free automorphism of coprime order, we conjecture that, for every non-zero polynomial f(x) = a0 + a1x+ · · ·+ adx d ∈ Z[x], there is an integer k > 0 with the following property. Let G be a finite (solvable) group with a fixed-point-free automorphism α satisfying gcd(|G|, k) = 1 and {g0 · α(g)1 · α(g)2 · · ·α(g)d |g ∈ G} = {1}. Then the Fitting height of G is at most the number of irreducible factors of f(x). We confirm the conjecture for a large family of polynomials with explicit constants k.
Invariant Language Modeling

Modern pretrained language models are critical components of NLP pipelines. Yet, they suffer from spurious correlations, poor out-of-domain generalization, and biases.Inspired by recent progress in causal machine learning, in particular the invariant risk minimization (IRM) paradigm, we propose invariant language modeling, a framework for learning invariant representations that generalize better across multiple environments. In particular, we adapt a game-theoretic implementation of IRM (IRM-games) to language models, where the invariance emerges from a specific training schedule in which all the environments compete to optimize their own environment-specific loss by updating subsets of the model in a round-robin fashion.We focused on controlled experiments to precisely demonstrate the ability of our method to (i) remove structured noise, (ii) ignore specific spurious correlations without affecting global performance, and (iii) achieve better out-of-domain generalization.These benefits come with a negligible computational overhead compared to standard training, do not require changing the local loss, and can be applied to any language model. We believe this framework is promising to help mitigate spurious correlations and biases in language models.
Improving Robustness using Generated Data

Recent work argues that robust training requires substantially larger datasets than those required for standard classification. On CIFAR-10 and CIFAR-100, this translates into a sizable robust-accuracy gap between models trained solely on data from the original training set and those trained with additional data extracted from the"80 Million Tiny Images"dataset (TI-80M). In this paper, we explore how generative models trained solely on the original training set can be leveraged to artificially increase the size of the original training set and improve adversarial robustness to $\ell_p$ norm-bounded perturbations. We identify the sufficient conditions under which incorporating additional generated data can improve robustness, and demonstrate that it is possible to significantly reduce the robust-accuracy gap to models trained with additional real data. Surprisingly, we even show that even the addition of non-realistic random data (generated by Gaussian sampling) can improve robustness. We evaluate our approach on CIFAR-10, CIFAR-100, SVHN and TinyImageNet against $\ell_\infty$ and $\ell_2$ norm-bounded perturbations of size $\epsilon = 8/255$ and $\epsilon = 128/255$, respectively. We show large absolute improvements in robust accuracy compared to previous state-of-the-art methods. Against $\ell_\infty$ norm-bounded perturbations of size $\epsilon = 8/255$, our models achieve 66.10% and 33.49% robust accuracy on CIFAR-10 and CIFAR-100, respectively (improving upon the state-of-the-art by +8.96% and +3.29%). Against $\ell_2$ norm-bounded perturbations of size $\epsilon = 128/255$, our model achieves 78.31% on CIFAR-10 (+3.81%). These results beat most prior works that use external data.
Towards Instance-Optimal Offline Reinforcement Learning with Pessimism

We study the offline reinforcement learning (offline RL) problem, where the goal is to learn a reward-maximizing policy in an unknown Markov Decision Process (MDP) using the data coming from a policy μ. In particular, we consider the sample complexity problems of offline RL for finite-horizon MDPs. Prior works study this problem based on different data-coverage assumptions, and their learning guarantees are expressed by the covering coefficients which lack the explicit characterization of system quantities. In this work, we analyze the Adaptive Pessimistic Value Iteration (APVI) algorithm and derive the suboptimality upper bound that nearly matches O  H ∑
A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models

Large pre-trained vision-language (VL) models can learn a new task with a handful of examples and generalize to a new task without fine-tuning.However, these VL models are hard to deploy for real-world applications due to their impractically huge sizes and slow inference speed.To solve this limitation, we study prompt-based low-resource learning of VL tasks with our proposed method, FewVLM, relatively smaller than recent few-shot learners.For FewVLM, we pre-train a sequence-to-sequence transformer model with prefix language modeling (PrefixLM) and masked language modeling (MaskedLM). Furthermore, we analyze the effect of diverse prompts for few-shot tasks.Experimental results on VQA show that FewVLM with prompt-based learning outperforms Frozen which is 31x larger than FewVLM by 18.2% point and achieves comparable results to a 246x larger model, PICa.In our analysis, we observe that (1) prompts significantly affect zero-shot performance but marginally affect few-shot performance, (2) models with noisy prompts learn as quickly as hand-crafted prompts given larger training data, and (3) MaskedLM helps VQA tasks while PrefixLM boosts captioning performance. Our code is publicly available at https://github.com/woojeongjin/FewVLM
Finding Everything within Random Binary Networks

A recent work by Ramanujan et al. (2020) pro-vides signiﬁcant empirical evidence that sufﬁ-ciently overparameterized, random neural networks contain untrained subnetworks that achieve state-of-the-art accuracy on several predictive tasks. A follow-up line of theoretical work pro-vides justiﬁcation of these ﬁndings by proving that slightly overparameterized neural networks, with commonly used continuous-valued random initializations can indeed be pruned to approximate any target network. In this work, we show that the amplitude of those random weights does not even matter. We prove that any target network can be approximated up to arbitrary accuracy by simply pruning a random network of binary {± 1 } weights that is only a polylogarithmic factor wider and deeper than the target network.
Domains of Magnetic Pressure Balance in Parker Solar Probe Observations of the Solar Wind

The Parker Solar Probe (PSP) spacecraft is performing the first in situ exploration of the solar wind within 0.2 au of the Sun. Initial observations confirmed the Alfvénic nature of aligned fluctuations of the magnetic field B and velocity V in solar wind plasma close to the Sun, in domains of nearly constant magnetic field magnitude ∣ B ∣, i.e., approximate magnetic pressure balance. Such domains are interrupted by particularly strong fluctuations, including but not limited to radial field (polarity) reversals, known as switchbacks. It has been proposed that nonlinear Kelvin–Helmholtz instabilities form near magnetic boundaries in the nascent solar wind leading to extensive shear-driven dynamics, strong turbulent fluctuations including switchbacks, and mixing layers that involve domains of approximate magnetic pressure balance. In this work we identify and analyze various aspects of such domains using data from the first five PSP solar encounters. The filling fraction of domains, a measure of Alfvénicity, varies from median values of 90% within 0.2 au to 38% outside 0.9 au, with strong fluctuations. We find an inverse association between the mean domain duration and plasma β. We examine whether the mean domain duration is also related to the crossing time of spatial structures frozen into the solar wind flow for extreme cases of the aspect ratio. Our results are inconsistent with long, thin domains aligned along the radial or Parker spiral direction, and compatible with isotropic domains, which is consistent with prior observations of isotropic density fluctuations or flocculae in the solar wind.
HRFormer: High-Resolution Transformer for Dense Prediction

We present a High-Resolution Transformer (HRFormer) that learns high-resolution representations for dense prediction tasks, in contrast to the original Vision Transformer that produces low-resolution representations and has high memory and computational cost. We take advantage of the multi-resolution parallel design introduced in high-resolution convolutional networks (HRNet), along with local-window self-attention that performs self-attention over small non-overlapping image windows, for improving the memory and computation efficiency. In addition, we introduce a convolution into the FFN to exchange information across the disconnected image windows. We demonstrate the effectiveness of the High-Resolution Transformer on both human pose estimation and semantic segmentation tasks, e.g., HRFormer outperforms Swin transformer by $1.3$ AP on COCO pose estimation with $50\%$ fewer parameters and $30\%$ fewer FLOPs. Code is available at: https://github.com/HRNet/HRFormer.
Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs

We study episodic reinforcement learning (RL) in non-stationary linear kernel Markov decision processes (MDPs). In this setting, both the reward function and the transition kernel are linear with respect to the given feature maps and are allowed to vary over time, as long as their respective parameter variations do not exceed certain variation budgets. We propose the $\underline{\text{p}}$eriodically $\underline{\text{r}}$estarted $\underline{\text{o}}$ptimistic $\underline{\text{p}}$olicy $\underline{\text{o}}$ptimization algorithm (PROPO), which is an optimistic policy optimization algorithm with linear function approximation. PROPO features two mechanisms: sliding-window-based policy evaluation and periodic-restart-based policy improvement, which are tailored for policy optimization in a non-stationary environment. In addition, only utilizing the technique of sliding window, we propose a value-iteration algorithm. We establish dynamic upper bounds for the proposed methods and a matching minimax lower bound which shows the (near-) optimality of the proposed methods. To our best knowledge, PROPO is the first provably efficient policy optimization algorithm that handles non-stationarity.
TIP: Task-Informed Motion Prediction for Intelligent Systems

—Motion prediction is important for intelligent driving systems, providing the future distributions of road agent behaviors and supporting various decision making tasks. Existing motion predictors are often optimized and evaluated via task-agnostic measures based on prediction accuracy. Such measures fail to account for the use of prediction in downstream tasks, and could result in sub-optimal task performance. We propose a task-informed motion prediction framework that jointly reasons about prediction accuracy and task utility, to better support downstream tasks through its predictions. The task utility function does not require the full task information, but rather a speciﬁcation of the utility of the task, resulting in predictors that serve a wide range of downstream tasks. We demonstrate our framework on two use cases of task utilities, in the context of autonomous driving and parallel autonomy, and show the advantage of task-informed predictors over task-agnostic ones on the Waymo Open Motion dataset.
Vit-GAN: Image-to-image Translation with Vision Transformes and Conditional GANS

In this paper, we have developed a general-purpose architecture, Vit-Gan, capable of performing most of the image-to-image translation tasks from semantic image segmentation to single image depth perception. This paper is a follow-up paper, an extension of generator based model [1] in which the obtained results were very promising. This opened the possibility of further improvements with adversarial architecture. We used a unique vision transformers-based generator architecture and Conditional GANs(cGANs) with a Markovian Discriminator (PatchGAN) (https://github.com/YigitGunduc/vit-gan). In the present work, we use images as conditioning arguments. It is observed that the obtained results are more realistic than the commonly used architectures.
Global well-posedness, stability and instability for the non-viscous Oldroyd-B model

In this paper we consider the 3-dimensional incompressible Oldroyd-B model. First, we establish two results of the global existence for different kinds of the coupling coefficient k. Then, we prove that the solutions (u, τ) are globally steady when k → k > 0, though (u, τ) corresponds to different decays for different kinds of k > 0 . Finally, we show that the energy of u(t, x) will have a jump when k → 0 in large time, which implies a nonsteady phenomenon. In a word, we find an interesting physical phenomenon of (1.2) such that smaller coupling coefficient k will have a better impact for the energy dissipation of (u, τ), but k can’t be too small to zero, or the dissipation will vanish instantly. While the damping term τ and Du always bring the well impact for the energy dissipation.
Provable Hierarchy-Based Meta-Reinforcement Learning

Hierarchical reinforcement learning (HRL) has seen widespread interest as an approach to tractable learning of complex modular behaviors. However, existing work either assume access to expert-constructed hierarchies, or use hierarchy-learning heuristics with no provable guarantees. To address this gap, we analyze HRL in the meta-RL setting, where a learner learns latent hierarchical structure during meta-training for use in a downstream task. We consider a tabular setting where natural hierarchical structure is embedded in the transition dynamics. Analogous to supervised meta-learning theory, we provide"diversity conditions"which, together with a tractable optimism-based algorithm, guarantee sample-efficient recovery of this natural hierarchy. Furthermore, we provide regret bounds on a learner using the recovered hierarchy to solve a meta-test task. Our bounds incorporate common notions in HRL literature such as temporal and state/action abstractions, suggesting that our setting and analysis capture important features of HRL in practice.
BERRY–ESSEEN BOUND AND LOCAL LIMIT THEOREM FOR THE COEFFICIENTS OF PRODUCTS OF RANDOM MATRICES

<jats:p>Let <jats:inline-formula> <jats:alternatives> <jats:inline-graphic xmlns:xlink="http://www.w3. org/1999/xlink" mime-subtype="png" xlink:href="S1474748022000561_inline1. png" /> <jats:tex-math> $\mu $ </jats:tex-math> </jats:alternatives> </jats:inline-formula> be a probability measure on <jats:inline-formula> <jats:alternatives> <jats:inline-graphic xmlns:xlink="http://www.w3. org/1999/xlink" mime-subtype="png" xlink:href="S1474748022000561_inline2. png" /> <jats:tex-math> $\mathrm {GL}_d(\mathbb {R})$ </jats:tex-math> </jats:alternatives> </jats:inline-formula>, and denote by <jats:inline-formula> <jats:alternatives> <jats:inline-graphic xmlns:xlink="http://www.w3. org/1999/xlink" mime-subtype="png" xlink:href="S1474748022000561_inline3. png" /> <jats:tex-math> $S_n:= g_n \cdots g_1$ </jats:tex-math> </jats:alternatives> </jats:inline-formula> the associated random matrix product, where <jats:inline-formula> <jats:alternatives> <jats:inline-graphic xmlns:xlink="http://www.w3. org/1999/xlink" mime-subtype="png" xlink:href="S1474748022000561_inline4. png" /> <jats:tex-math> $g_j$ </jats:tex-math> </jats:alternatives> </jats:inline-formula> are i.i. d. with law <jats:inline-formula> <jats:alternatives> <jats:inline-graphic xmlns:xlink="http://www.w3. org/1999/xlink" mime-subtype="png" xlink:href="S1474748022000561_inline5. png" /> <jats:tex-math> $\mu $ </jats:tex-math> </jats:alternatives> </jats:inline-formula>. Under the assumptions that <jats:inline-formula> <jats:alternatives> <jats:inline-graphic xmlns:xlink="http://www.w3. org/1999/xlink" mime-subtype="png" xlink:href="S1474748022000561_inline6. png" /> <jats:tex-math> $\mu $ </jats:tex-math> </jats:alternatives> </jats:inline-formula> has a finite exponential moment and generates a proximal and strongly irreducible semigroup, we prove a Berry–Esseen bound with the optimal rate <jats:inline-formula> <jats:alternatives> <jats:inline-graphic xmlns:xlink="http://www.w3. org/1999/xlink" mime-subtype="png" xlink:href="S1474748022000561_inline7. png" /> <jats:tex-math> $O(1/\sqrt n)$ </jats:tex-math> </jats:alternatives> </jats:inline-formula> for the coefficients of <jats:inline-formula> <jats:alternatives> <jats:inline-graphic xmlns:xlink="http://www.w3. org/1999/xlink" mime-subtype="png" xlink:href="S1474748022000561_inline8. png" /> <jats:tex-math> $S_n$ </jats:tex-math> </jats:alternatives> </jats:inline-formula>, settling a long-standing question considered since the fundamental work of Guivarc’h and Raugi. The local limit theorem for the coefficients is also obtained, complementing a recent partial result of Grama, Quint and Xiao. </jats:p>
Andreev Reflection in the Fractional Quantum Hall State

Önder Gül,1,* Yuval Ronen,1,* Si Young Lee,1,2,* Hassan Shapourian,1,3 Jonathan Zauberman,1 Young Hee Lee,2,4 Kenji Watanabe,5 Takashi Taniguchi,6 Ashvin Vishwanath,1 Amir Yacoby,1 and Philip Kim1,✉ 1Department of Physics, Harvard University, Cambridge, MA, USA 2Center for Integrated Nanostructure Physics (CINAP), Institute for Basic Science (IBS), Suwon, Republic of Korea 3Department of Physics, Massachusetts Institute of Technology, Cambridge, MA, USA 4Department of Energy Science, Sungkyunkwan University, Suwon, Republic of Korea 5Research Center for Functional Materials, National Institute for Materials Science, Tsukuba, Japan 6International Center for Materials Nanoarchitectonics, National Institute for Materials Science, Tsukuba, Japan