Miracle-dz/UniFAR-FTUs huggingface dataset:
Differentially Private Bias-Term only Fine-tuning of Foundation Models

We study the problem of differentially private (DP) fine-tuning of large pre-trained models -- a recent privacy-preserving approach suitable for solving downstream tasks with sensitive data. Existing work has demonstrated that high accuracy is possible under strong privacy constraint, yet requires significant computational overhead or modifications to the network architecture. We propose differentially private bias-term fine-tuning (DP-BiTFiT), which matches the state-of-the-art accuracy for DP algorithms and the efficiency of the standard BiTFiT. DP-BiTFiT is model agnostic (not modifying the network architecture), parameter efficient (only training about $0.1\%$ of the parameters), and computation efficient (almost removing the overhead caused by DP, in both the time and space complexity). On a wide range of tasks, DP-BiTFiT is $2\sim 30\times$ faster and uses $2\sim 8\times$ less memory than DP full fine-tuning, even faster than the standard full fine-tuning. This amazing efficiency enables us to conduct DP fine-tuning on language and vision tasks with long-sequence texts and high-resolution images, which were computationally difficult using existing methods.
Arithmetic progressions among powerful numbers

In this paper, we study k -term arithmetic progressions N, N + d, . , N +( k − 1) d of powerful numbers. Under the abc -conjecture, we obtain d ≫ ǫ N 1 / 2 − ǫ . On the other hand, there exist inﬁnitely many 3-term arithmetic progressions of powerful numbers with d ≪ N 1 / 2 unconditionally. We also prove some partial results when k ≥ 4 and pose some open questions.
PrivTrace: Differentially Private Trajectory Synthesis by Adaptive Markov Model

Publishing trajectory data (individual's movement information) is very useful, but it also raises privacy concerns. To handle the privacy concern, in this paper, we apply differential privacy, the standard technique for data privacy, together with Markov chain model, to generate synthetic trajectories. We notice that existing studies all use Markov chain model and thus propose a framework to analyze the usage of the Markov chain model in this problem. Based on the analysis, we come up with an effective algorithm PrivTrace that uses the first-order and second-order Markov model adaptively. We evaluate PrivTrace and existing methods on synthetic and real-world datasets to demonstrate the superiority of our method.
A deep learning approach for focal-plane wavefront sensing using vortex phase diversity

The performance of high-contrast imaging instruments is limited by wavefront errors, in particular by non-common path aberrations (NCPAs). Focal-plane wavefront sensing (FPWFS) is appropriate to handle NCPAs because it measures the aberration where it matters the most, that is to say at the science focal plane. Phase retrieval from focal-plane images results, nonetheless, in a sign ambiguity for even modes of the pupil-plane phase. The phase diversity methods currently used to solve the sign ambiguity tend to reduce the science duty cycle, that is, the fraction of observing time dedicated to science. In this work, we explore how we can combine the phase diversity provided by a vortex coronagraph with modern deep learning techniques to perform efficient FPWFS without losing observing time. We applied the state-of-the-art convolutional neural network EfficientNet-B4 to infer phase aberrations from simulated focal-plane images. The two cases of scalar and vector vortex coronagraphs (SVC and VVC) were considered using a single post-coronagraphic point spread function (PSF) or two PSFs obtained by splitting the circular polarization states, respectively. The sign ambiguity has been properly lifted in both cases even at low signal-to-noise ratios (S/Ns). Using either the SVC or the VVC, we have reached a very similar performance compared to using phase diversity with a defocused PSF, except for high levels of aberrations where the SVC slightly underperforms compared to the other approaches. The models finally show great robustness when trained on data with a wide range of wavefront errors and noise levels. The proposed FPWFS technique provides a 100% science duty cycle for instruments using a vortex coronagraph and does not require any additional hardware in the case of the SVC.
Taming Fat-Tailed ("Heavier-Tailed" with Potentially Infinite Variance) Noise in Federated Learning

A key assumption in most existing works on FL algorithms' convergence analysis is that the noise in stochastic first-order information has a finite variance. Although this assumption covers all light-tailed (i.e., sub-exponential) and some heavy-tailed noise distributions (e.g., log-normal, Weibull, and some Pareto distributions), it fails for many fat-tailed noise distributions (i.e., ``heavier-tailed'' with potentially infinite variance) that have been empirically observed in the FL literature. To date, it remains unclear whether one can design convergent algorithms for FL systems that experience fat-tailed noise. This motivates us to fill this gap in this paper by proposing an algorithmic framework called FAT-Clipping (\ul{f}ederated \ul{a}veraging with \ul{t}wo-sided learning rates and \ul{clipping}), which contains two variants: FAT-Clipping per-round (FAT-Clipping-PR) and FAT-Clipping per-iteration (FAT-Clipping-PI). Specifically, for the largest $\alpha \in (1,2]$ such that the fat-tailed noise in FL still has a bounded $\alpha$-moment, we show that both variants achieve $\mathcal{O}((mT)^{\frac{2-\alpha}{\alpha}})$ and $\mathcal{O}((mT)^{\frac{1-\alpha}{3\alpha-2}})$ convergence rates in the strongly-convex and general non-convex settings, respectively, where $m$ and $T$ are the numbers of clients and communication rounds. Moreover, at the expense of more clipping operations compared to FAT-Clipping-PR, FAT-Clipping-PI further enjoys a linear speedup effect with respect to the number of local updates at each client and being lower-bound-matching (i.e., order-optimal). Collectively, our results advance the understanding of designing efficient algorithms for FL systems that exhibit fat-tailed first-order oracle information.
Compositional Generalization in Unsupervised Compositional Representation Learning: A Study on Disentanglement and Emergent Language

Deep learning models struggle with compositional generalization, i.e. the ability to recognize or generate novel combinations of observed elementary concepts. In hopes of enabling compositional generalization, various unsupervised learning algorithms have been proposed with inductive biases that aim to induce compositional structure in learned representations (e.g. disentangled representation and emergent language learning). In this work, we evaluate these unsupervised learning algorithms in terms of how well they enable compositional generalization. Specifically, our evaluation protocol focuses on whether or not it is easy to train a simple model on top of the learned representation that generalizes to new combinations of compositional factors. We systematically study three unsupervised representation learning algorithms - $\beta$-VAE, $\beta$-TCVAE, and emergent language (EL) autoencoders - on two datasets that allow directly testing compositional generalization. We find that directly using the bottleneck representation with simple models and few labels may lead to worse generalization than using representations from layers before or after the learned representation itself. In addition, we find that the previously proposed metrics for evaluating the levels of compositionality are not correlated with actual compositional generalization in our framework. Surprisingly, we find that increasing pressure to produce a disentangled representation produces representations with worse generalization, while representations from EL models show strong compositional generalization. Taken together, our results shed new light on the compositional generalization behavior of different unsupervised learning algorithms with a new setting to rigorously test this behavior, and suggest the potential benefits of delevoping EL learning algorithms for more generalizable representations.
On some possible combinations of mixing rates for strictly stationary, reversible Markov chains

A class of examples is constructed to show that for strictly stationary Markov chains that are reversible, the simultaneous mixing rates for the ρ -mixing and strong mixing ( α -mixing) conditions can be fairly arbitrary, within certain unavoidable tight restrictions. The examples constructed here have the added property that the mixing rate for the absolute regularity ( β -mixing) condition is within a constant factor of that for strong mixing.
Pinwheels as Lagrangian barriers

The complex projective plane CP^2 contains certain Lagrangian CW-complexes called pinwheels, which have interesting rigidity properties related to solutions of the Markov equation. We compute the Gromov width of the complement of pinwheels and show that it is strictly smaller than the Gromov width of CP^2, meaning that pinwheels are Lagrangian barriers in the sense of Biran. The accumulation points of the set of these Gromov widths are in a simple bijection with the Lagrange spectrum below 3.
PROPERTIES OF UMBRAL DOTS AS MEASURED FROM THE NEW SOLAR TELESCOPE DATA AND MHD SIMULATIONS

We studied bright umbral dots (UDs) detected in a moderate size sunspot and compared their statistical properties to recent MHD models. The study is based on high-resolution data recorded by the New Solar Telescope at the Big Bear Solar Observatory and three-dimensional (3D) MHD simulations of sunspots. Observed UDs, living longer than 150 s, were detected and tracked in a 46 minute long data set, using an automatic detection code. A total of 1553 (620) UDs were detected in the photospheric (low chromospheric) data. Our main findings are (1) none of the analyzed UDs is precisely circular, (2) the diameter–intensity relationship only holds in bright umbral areas, and (3) UD velocities are inversely related to their lifetime. While nearly all photospheric UDs can be identified in the low chromospheric images, some small closely spaced UDs appear in the low chromosphere as a single cluster. Slow-moving and long-living UDs seem to exist in both the low chromosphere and photosphere, while fast-moving and short-living UDs are mainly detected in the photospheric images. Comparison to the 3D MHD simulations showed that both types of UDs display, on average, very similar statistical characteristics. However, (1) the average number of observed UDs per unit area is smaller than that of the model UDs, and (2) on average, the diameter of model UDs is slightly larger than that of observed ones.
LPT: Long-tailed Prompt Tuning for Image Classification

For long-tailed classification, most works often pretrain a big model on a large-scale dataset, and then fine-tune the whole model for adapting to long-tailed data. Though promising, fine-tuning the whole pretrained model tends to suffer from high cost in computation and deployment of different models for different tasks, as well as weakened generalization ability for overfitting to certain features of long-tailed data. To alleviate these issues, we propose an effective Long-tailed Prompt Tuning method for long-tailed classification. LPT introduces several trainable prompts into a frozen pretrained model to adapt it to long-tailed data. For better effectiveness, we divide prompts into two groups: 1) a shared prompt for the whole long-tailed dataset to learn general features and to adapt a pretrained model into target domain; and 2) group-specific prompts to gather group-specific features for the samples which have similar features and also to empower the pretrained model with discrimination ability. Then we design a two-phase training paradigm to learn these prompts. In phase 1, we train the shared prompt via supervised prompt tuning to adapt a pretrained model to the desired long-tailed domain. In phase 2, we use the learnt shared prompt as query to select a small best matched set for a group of similar samples from the group-specific prompt set to dig the common features of these similar samples, then optimize these prompts with dual sampling strategy and asymmetric GCL loss. By only fine-tuning a few prompts while fixing the pretrained model, LPT can reduce training and deployment cost by storing a few prompts, and enjoys a strong generalization ability of the pretrained model. Experiments show that on various long-tailed benchmarks, with only ~1.1% extra parameters, LPT achieves comparable performance than previous whole model fine-tuning methods, and is more robust to domain-shift.
Analysis of the primal-dual central path for nonlinear semidefinite optimization without the nondegeneracy condition

We study properties of the central path underlying a nonlinear semidefinite optimization problem, called NSDP for short. The latest radical work on this topic was contributed by Yamashita and Yabe (2012): they proved that the Jacobian of a certain equation-system derived from the Karush-Kuhn-Tucker (KKT) conditions of the NSDP is nonsingular at a KKT point under the second-order sufficient condition (SOSC), the strict complementarity condition (SC), and the nondegeneracy condition (NC). This yields uniqueness and existence of the central path through the implicit function theorem. In this paper, we consider the following three assumptions on a KKT point: the strong SOSC, the SC, and the Mangasarian-Fromovitz constraint qualification. Under the absence of the NC, the Lagrange multiplier set is not necessarily a singleton and the nonsingularity of the above-mentioned Jacobian is no longer valid. Nonetheless, we establish that the central path exists uniquely, and moreover prove that the dual component of the path converges to the so-called analytic center of the Lagrange multiplier set. As another notable result, we clarify a region around the central path where Newton's equations relevant to primal-dual interior point methods are uniquely solvable.
Improving Kerr QND measurement sensitivity via squeezed light

In ref [Phys. Rev. A 106, 013720], the scheme of quantum non-demolition measurement of optical quanta that uses a resonantly enhanced Kerr nonlinearity in optical microresonators was analyzed theoretically. It was shown that using the modern high-Q microresonators, it is possible to achieve the sensitivity several times better than the standard quantum limit. Here we propose and analyze in detail a significantly improved version of that scheme. We show, that by using a squeezed quantum state of the probe beam and the anti-squeezing (parametric amplification) of this beam at the output of the microresonator, it is possible to reduce the measurement imprecision by about one order of magnitude. The resulting sensitivity allows to generate and verify multi-photon non-Gaussian quantum states of light, making the scheme considered here interesting for the quantum information processing tasks.
Defining a Credible Interval Is Not Always Possible with “Point-Null” Priors: A Lesser-Known Correlate of the Jeffreys-Lindley Paradox

In many common situations, a Bayesian credible interval will be, given the same data, very similar to a frequentist confidence interval, and researchers will interpret these intervals in a similar fashion. However, no predictable similarity exists when credible intervals are based on model-averaged posteriors whenever one of the two nested models under consideration is a so called ''point-null''. Not only can this model-averaged credible interval be quite different than the frequentist confidence interval, in some cases it may be undefined. This is a lesser-known correlate of the Jeffreys-Lindley paradox and is of particular interest given the popularity of the Bayes factor for testing point-null hypotheses.
Wide Attention Is The Way Forward For Transformers

The Transformer is an extremely powerful and prominent deep learning architecture. In this work, we challenge the commonly held belief in deep learning that going deeper is better, and show an alternative design approach that is building wider attention Transformers. We demonstrate that wide single layer Transformer models can compete with or outperform deeper ones in a variety of Natural Language Processing (NLP) tasks when both are trained from scratch. The impact of changing the model aspect ratio on Transformers is then studied systematically. This ratio balances the number of layers and the number of attention heads per layer while keeping the total number of attention heads and all other hyperparameters constant. On average, across 4 NLP tasks and 10 attention types, single layer wide models perform 0.3% better than their deep counterparts. We show an in-depth evaluation and demonstrate how wide models require a far smaller memory footprint and can run faster on commodity hardware, in addition, these wider models are also more interpretable. For example, a single layer Transformer on the IMDb byte level text classification has 3.1x faster inference latency on a CPU than its equally accurate deeper counterpart, and is half the size. We therefore put forward wider and shallower models as a viable and desirable alternative for small models on NLP tasks, and as an important area of research for domains beyond this.
CERBERUS: Simple and Effective All-In-One Automotive Perception Model with Multi Task Learning

—Perceiving the surrounding environment is essen- tial for enabling autonomous or assisted driving functionalities. Common tasks in this domain include detecting road users, as well as determining lane boundaries and classifying driving conditions. Over the last few years, a large variety of pow- erful Deep Learning models have been proposed to address individual tasks of camera-based automotive perception with astonishing performances. However, the limited capabilities of in-vehicle embedded computing platforms cannot cope with the computational effort required to run a heavy model for each individual task. In this work, we present CERBERUS (CEnteR Based End-to-end peRception Using a Single model), a lightweight model that leverages a multitask-learning approach to enable the execution of multiple perception tasks at the cost of a single inference. The code will be made publicly available at https://github.com/cscribano/CERBERUS .
Autocorrelations of stellar light and mass at z∼ 0 and ∼1: from SDSS to DEEP2

We present measurements of projected autocorrelation functions wp(rp) for the stellar mass of galaxies and for their light in the U, B and V bands, using data from the third data release of the DEEP2 Galaxy Redshift Survey and the final data release of the Sloan Digital Sky Survey (SDSS). We investigate the clustering bias of stellar mass and light by comparing these to projected autocorrelations of dark matter estimated from the Millennium Simulations (MS) at z = 1 and 0.07, the median redshifts of our galaxy samples. All of the autocorrelation and bias functions show systematic trends with spatial scale and waveband which are impressively similar at the two redshifts. This shows that the well-established environmental dependence of stellar populations in the local Universe is already in place at z = 1. The recent MS-based galaxy formation simulation of Guo et al. (2011) reproduces the scale-dependent clustering of luminosity to an accuracy better than 30% in all bands and at both redshifts, but substantially overpredicts mass autocorrelations at separations below about 2 Mpc. Further comparison of the shapes of our stellar mass bias functions with those predicted by the model suggests that both the SDSS and DEEP2 data prefer a fluctuation amplitude of σ8 � 0.8 rather than the σ8 = 0.9 assumed by the MS.
MALM: Mixing Augmented Language Modeling for Zero-Shot Machine Translation

Large pre-trained language models have brought remarkable progress in NLP. Pre-training and Fine-tuning have given state-of-art performance across tasks in text processing. Data Augmentation techniques have also helped build state-of-art models on low or zero resource tasks. Many works in the past have attempted at learning a single massively multilingual machine translation model for zero-shot translation. Although those translation models are producing correct translations, the main challenge is those models are producing the wrong languages for zero-shot translation. This work and its results indicate that prompt conditioned large models do not suffer from off-target language errors i.e. errors arising due to translation to wrong languages. We empirically demonstrate the effectiveness of self-supervised pre-training and data augmentation for zero-shot multi-lingual machine translation.
WANDERING STARS: AN ORIGIN OF ESCAPED POPULATIONS

We demonstrate that stars beyond the virial radii of galaxies may be generated by the gravitational impulse received by a satellite as it passes through the pericenter of its orbit around its parent. These stars may become energetically unbound (escaped stars), or may travel to further than a few virial radii for longer than a few Gyr, but still remain energetically bound to the system (wandering stars). Larger satellites (10%–100% the mass of the parent), and satellites on more radial orbits are responsible for the majority of this ejected population. Wandering stars could be observable on Mpc scales via classical novae, and on 100 Mpc scales via Type Ia supernova. The existence of such stars would imply a corresponding population of barely bound, old, high-velocity stars orbiting the Milky Way, generated by the same physical mechanism during the Galaxy's formation epoch. Sizes and properties of these combined populations should place some constraints on the orbits and masses of the progenitor objects from which they came, providing insight into the merging histories of galaxies in general and the Milky Way in particular.
Language-Aware Soft Prompting for Vision & Language Foundation Models

This paper is on soft prompt learning for Vision & Language (V&L) models. Similarly to their NLP counterparts, V&L models can be adapted to a downstream task by learning soft continuous prompts using a few training examples. Current methods learn the soft prompts by minimizing a cross-entropy loss using as class weights the features obtained by passing the prompts plus the class names through the text encoder. Such methods, however, signiﬁcantly overﬁt the training data suffering from large accuracy degradation when tested on unseen classes from the same domain. Our main contribution, in this paper, is a surprisingly simple approach to alleviate this problem: we use a second cross entropy loss to minimize the distance between the learned soft prompts and a set of hand-engineered manual prompts (obtained by prompt engineering). The proposed loss can be interpreted in multiple ways including as a regularizer, as a means for language-based augmentation, and as a way of learning more discriminative class centroids. Importantly, our formulation is inherently amenable to including, during training, virtual classes, i.e. class names for which no visual samples are available, further increasing the robustness of the learned prompts. Through extensive evaluations on 11 datasets, we show that our approach (a) signiﬁcantly outperforms all prior works on soft prompting, and (b) matches and surpasses, for the ﬁrst time, the accuracy on novel classes obtained by hand-crafted prompts and CLIP for the majority of the test datasets. Code will be made available.