Miracle-dz/UniFAR-FTUs huggingface dataset:
CLIP-KD: An Empirical Study of Distilling CLIP Models

CLIP has become a promising language-supervised visual pre-training framework and achieves excellent performance over a wide range of tasks. This paper aims to distill small CLIP models supervised by a large teacher CLIP model. We propose several distillation strategies, including relation, feature, gradient and contrastive paradigm, to examine the impact on CLIP distillation. We show that the simplest feature mimicry with MSE loss performs best. Moreover, interactive contrastive learning and relation-based distillation are also critical in performance improvement. We apply the unified method to distill several student networks trained on 15 million (image, text) pairs. Distillation improves the student CLIP models consistently over zero-shot ImageNet classification and cross-modal retrieval benchmarks. We hope our empirical study will become an important baseline for future CLIP distillation research. The code is available at \url{https://github.com/winycg/CLIP-KD}.
ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting

Diffusion-based image super-resolution (SR) methods are mainly limited by the low inference speed due to the requirements of hundreds or even thousands of sampling steps. Existing acceleration sampling techniques inevitably sacrifice performance to some extent, leading to over-blurry SR results. To address this issue, we propose a novel and efficient diffusion model for SR that significantly reduces the number of diffusion steps, thereby eliminating the need for post-acceleration during inference and its associated performance deterioration. Our method constructs a Markov chain that transfers between the high-resolution image and the low-resolution image by shifting the residual between them, substantially improving the transition efficiency. Additionally, an elaborate noise schedule is developed to flexibly control the shifting speed and the noise strength during the diffusion process. Extensive experiments demonstrate that the proposed method obtains superior or at least comparable performance to current state-of-the-art methods on both synthetic and real-world datasets, even only with 15 sampling steps. Our code and model are available at https://github.com/zsyOAOA/ResShift.
Diurnal modulation of electron recoils from DM-nucleon scattering through the Migdal effect

Halo dark matter (DM) particles could lose energy due to the scattering off nuclei within the Earth before reaching the underground detectors of DM direct detection experiments. This Earth shielding effect can result in diurnal modulation of the DM-induced recoil event rates observed underground due to the self-rotation of the Earth. For electron recoil signals from DM-electron scatterings, the current experimental constraints are very stringent such that the diurnal modulation cannot be observed for halo DM. We propose a novel type of diurnal modulation effect: diurnal modulation in electron recoil signals induced by DM-nucleon scattering via the Migdal effect. We set so far the most stringent constraints on DM-nucleon scattering cross section via the Migdal effect for sub-GeV DM using the S2-only data of PandaX-II and PandaX-4T with improved simulations of the Earth shielding effect. Based on the updated constraints, we show that the Migdal effect induced diurnal modulation of electron events can still be significant in the low energy region, and can be probed by experiments such as PandaX-4T in the near future.
Causal Fair Machine Learning via Rank-Preserving Interventional Distributions

A decision can be defined as fair if equal individuals are treated equally and unequals unequally. Adopting this definition, the task of designing machine learning models that mitigate unfairness in automated decision-making systems must include causal thinking when introducing protected attributes. Following a recent proposal, we define individuals as being normatively equal if they are equal in a fictitious, normatively desired (FiND) world, where the protected attribute has no (direct or indirect) causal effect on the target. We propose rank-preserving interventional distributions to define an estimand of this FiND world and a warping method for estimation. Evaluation criteria for both the method and resulting model are presented and validated through simulations and empirical data. With this, we show that our warping approach effectively identifies the most discriminated individuals and mitigates unfairness.
A First Look at On-device Models in iOS Apps

Powered by the rising popularity of deep learning techniques on smartphones, on-device deep learning models are being used in vital fields like finance, social media, and driving assistance. Because of the transparency of the Android platform and the on-device models inside, on-device models on Android smartphones have been proven to be extremely vulnerable. However, due to the challenge in accessing and analysing iOS app files, despite iOS being a mobile platform as popular as Android, there are no relevant works on on-device models in iOS apps. Since the functionalities of the same app on Android and iOS platforms are similar, the same vulnerabilities may exist on both platforms. In this paper, we present the first empirical study about on-device models in iOS apps, including their adoption of deep learning frameworks, structure, functionality, and potential security issues. We study why current developers use different on-device models for one app between iOS and Android. We propose a more general attack against white-box models that does not rely on pre-trained models and a new adversarial attack approach based on our findings to target iOS’s gray-box on-device models. Our results show the effectiveness of our approaches. Finally, we successfully exploit the vulnerabilities of on-device models to attack real-world iOS apps.
UniFormaly: Towards Task-Agnostic Unified Framework for Visual Anomaly Detection

Visual anomaly detection aims to learn normality from normal images, but existing approaches are fragmented across various tasks: defect detection, semantic anomaly detection, multi-class anomaly detection, and anomaly clustering. This one-task-one-model approach is resource-intensive and incurs high maintenance costs as the number of tasks increases. We present UniFormaly, a universal and powerful anomaly detection framework. We emphasize the necessity of our off-the-shelf approach by pointing out a suboptimal issue in online encoder-based methods. We introduce Back Patch Masking (BPM) and top k-ratio feature matching to achieve unified anomaly detection. BPM eliminates irrelevant background regions using a self-attention map from self-supervised ViTs. This operates in a task-agnostic manner and alleviates memory storage consumption, scaling to tasks with large-scale datasets. Top k-ratio feature matching unifies anomaly levels and tasks by casting anomaly scoring into multiple instance learning. Finally, UniFormaly achieves outstanding results on various tasks and datasets. Codes are available at https://github.com/YoojLee/Uniformaly.
Size-Ramsey numbers of structurally sparse graphs

Size-Ramsey numbers are a central notion in combinatorics and have been widely studied since their introduction by Erd\H{o}s, Faudree, Rousseau and Schelp in 1978. Research has mainly focused on the size-Ramsey numbers of $n$-vertex graphs with constant maximum degree $\Delta$. For example, graphs which also have constant treewidth are known to have linear size-Ramsey numbers. On the other extreme, the canonical examples of graphs of unbounded treewidth are the grid graphs, for which the best known bound has only very recently been improved from $O(n^{3/2})$ to $O(n^{5/4})$ by Conlon, Nenadov and Truji\'c. In this paper, we prove a common generalization of these results by establishing new bounds on the size-Ramsey numbers in terms of treewidth (which may grow as a function of $n$). As a special case, this yields a bound of $\tilde{O}(n^{3/2 - 1/2\Delta})$ for proper minor-closed classes of graphs. In particular, this bound applies to planar graphs, addressing a question of Kamcev, Liebenau, Wood and Yepremyan. Our proof combines methods from structural graph theory and classic Ramsey-theoretic embedding techniques, taking advantage of the product structure exhibited by graphs with bounded treewidth.
Classified Mixed Model Prediction

ABSTRACT Many practical problems are related to prediction, where the main interest is at subject (e.g., personalized medicine) or (small) sub-population (e.g., small community) level. In such cases, it is possible to make substantial gains in prediction accuracy by identifying a class that a new subject belongs to. This way, the new subject is potentially associated with a random effect corresponding to the same class in the training data, so that method of mixed model prediction can be used to make the best prediction. We propose a new method, called classified mixed model prediction (CMMP), to achieve this goal. We develop CMMP for both prediction of mixed effects and prediction of future observations, and consider different scenarios where there may or may not be a “match” of the new subject among the training-data subjects. Theoretical and empirical studies are carried out to study the properties of CMMP, including prediction intervals based on CMMP, and its comparison with existing methods. In particular, we show that, even if the actual match does not exist between the class of the new observations and those of the training data, CMMP still helps in improving prediction accuracy. Two real-data examples are considered. Supplementary materials for this article are available online.
An Estimator for the Sensitivity to Perturbations of Deep Neural Networks

For Deep Neural Networks (DNNs) to become useful in safety-critical applications, such as self-driving cars and disease diagnosis, they must be stable to perturbations in input and model parameters. Characterizing the sensitivity of a DNN to perturbations is necessary to determine minimal bit-width precision that may be used to safely represent the network. However, no general result exists that is capable of predicting the sensitivity of a given DNN to round-off error, noise, or other perturbations in input. This paper derives an estimator that can predict such quantities. The estimator is derived via inequalities and matrix norms, and the resulting quantity is roughly analogous to a condition number for the entire neural network. An approximation of the estimator is tested on two Convolutional Neural Networks, AlexNet and VGG-19, using the ImageNet dataset. For each of these networks, the tightness of the estimator is explored via random perturbations and adversarial attacks.
A preconditioned MINRES method for optimal control of wave equations and its asymptotic spectral distribution theory

In this work, we propose a novel preconditioned Krylov subspace method for solving an optimal control problem of wave equations, after explicitly identifying the asymptotic spectral distribution of the involved sequence of linear coefficient matrices from the optimal control problem. Namely, we first show that the all-at-once system stemming from the wave control problem is associated to a structured coefficient matrix-sequence possessing an eigenvalue distribution. Then, based on such a spectral distribution of which the symbol is explicitly identified, we develop an ideal preconditioner and two parallel-in-time preconditioners for the saddle point system composed of two block Toeplitz matrices. For the ideal preconditioner, we show that the eigenvalues of the preconditioned matrix-sequence all belong to the set $\left(-\frac{3}{2},-\frac{1}{2}\right)\bigcup \left(\frac{1}{2},\frac{3}{2}\right)$ well separated from zero, leading to mesh-independent convergence when the minimal residual method is employed. The proposed {parallel-in-time} preconditioners can be implemented efficiently using fast Fourier transforms or discrete sine transforms, and their effectiveness is theoretically shown in the sense that the eigenvalues of the preconditioned matrix-sequences are clustered around $\pm 1$, which leads to rapid convergence. When these parallel-in-time preconditioners are not fast diagonalizable, we further propose modified versions which can be efficiently inverted. Several numerical examples are reported to verify our derived localization and spectral distribution result and to support the effectiveness of our proposed preconditioners and the related advantages with respect to the relevant literature.
Efficiently Learning One-Hidden-Layer ReLU Networks via Schur Polynomials

We study the problem of PAC learning a linear combination of $k$ ReLU activations under the standard Gaussian distribution on $\mathbb{R}^d$ with respect to the square loss. Our main result is an efficient algorithm for this learning task with sample and computational complexity $(dk/\epsilon)^{O(k)}$, where $\epsilon>0$ is the target accuracy. Prior work had given an algorithm for this problem with complexity $(dk/\epsilon)^{h(k)}$, where the function $h(k)$ scales super-polynomially in $k$. Interestingly, the complexity of our algorithm is near-optimal within the class of Correlational Statistical Query algorithms. At a high-level, our algorithm uses tensor decomposition to identify a subspace such that all the $O(k)$-order moments are small in the orthogonal directions. Its analysis makes essential use of the theory of Schur polynomials to show that the higher-moment error tensors are small given that the lower-order ones are.
The correlation between the 500 pc scale molecular gas masses and AGN powers for massive elliptical galaxies

Massive molecular clouds have been discovered in massive elliptical galaxies at the center of galaxy clusters. Some of this cold gas is expected to flow in the central supermassive black holes and activate galactic nucleus (AGN) feedback. In this study, we analyze archival Atacama Large Millimeter Array (ALMA) data of nine massive elliptical galaxies, focusing on CO line emissions, to explore the circumnuclear gas. We show that the mass of the molecular gas within a fixed radius (500 pc) from the AGNs (Mmol ∼ 107–108 M⊙) is correlated with the jet power estimated from X-ray cavities (Pcav ∼ 1042–1045 erg s−1). The mass accretion rate of the circumnuclear gas $\dot{M}$ also has a correlation with Pcav. On the other hand, the continuum luminosities at ∼1.4 GHz and ∼100–300 GHz have no correlation with Mmol. These results indicate that the circumnuclear gas is sustaining the long-term AGN activities (∼107 yr) rather than the current ones. The circumnuclear gas mass is a better indicator of the jet power than the continuum luminosity, which probably changes on a shorter time scale. We also study the origin of the continuum emission from the AGNs at ∼100–300 GHz and find that it is mostly synchrotron radiation. For low-luminosity AGNs, however, dust emission appears to contaminate the continuum.
Gaussian decomposition of magic states for matchgate computations

Magic states were originally introduced as a resource that enables universal quantum computation using classically simulable Clifford gates. This concept has been extended to matchgate circuits (MGCs) which are made of two-qubit nearest-neighbour quantum gates defined by a set of algebraic constraints. In our work, we study the Gaussian rank of a quantum state -- defined as the minimum number of terms in any decomposition of that state into Gaussian states -- and associated quantities: the Gaussian Fidelity and the Gaussian Extent. We investigate the algebraic structure of Gaussian states and find and describe the independent sets of constraints upper-bounding the dimension of the manifold of Gaussian states. Furthermore, we describe the form of linearly dependent triples of Gaussian states and find the dimension of the manifold of solutions. By constructing the corresponding $\epsilon$-net for the Gaussian states, we are able to obtain upper bounds on the Gaussian fidelity. We identify a family of extreme points of the feasible set for the Dual Gaussian extent problem and show that Gaussian extent is multiplicative on systems of 4 qubits; and further that it is multiplicative on primal points whose optimal dual witness is in the above family. These extreme points turn out to be closely related to Extended Hamming Codes. We show that optimal dual witnesses are unique almost-surely, when the primal point lies in the interior of the normal cone of an extreme point. Furthermore, we show that the Gaussian rank of two copies of our canonical magic state is 4 for symmetry-restricted decompositions. Numerical investigation suggests that no low-rank decompositions exist of either 2 or 3 copies of the magic state. Finally, we consider approximate Gaussian rank and present approximate decompositions for selected magic states.
Learning Navigational Visual Representations with Semantic Map Supervision

Being able to perceive the semantics and the spatial structure of the environment is essential for visual navigation of a household robot. However, most existing works only employ visual backbones pre-trained either with independent images for classification or with self-supervised learning methods to adapt to the indoor navigation domain, neglecting the spatial relationships that are essential to the learning of navigation. Inspired by the behavior that humans naturally build semantically and spatially meaningful cognitive maps in their brains during navigation, in this paper, we propose a novel navigational-specific visual representation learning method by contrasting the agent’s egocentric views and semantic maps (Ego2-Map). We apply the visual transformer as the backbone encoder and train the model with data collected from the large-scale HabitatMatterport3D environments. Ego2-Map learning transfers the compact and rich information from a map, such as objects, structure and transition, to the agent’s egocentric representations for navigation. Experiments show that agents using our learned representations on object-goal navigation outperform recent visual pre-training methods. Moreover, our representations significantly improve vision-and-language navigation in continuous environments for both high-level and low-level action spaces, achieving new state-of-the-art results of 47% SR and 41% SPL on the test server.
CryptoMask : Privacy-preserving Face Recognition

Face recognition is a widely-used technique for identification or verification, where a verifier checks whether a face image matches anyone stored in a database. However, in scenarios where the database is held by a third party, such as a cloud server, both parties are concerned about data privacy. To address this concern, we propose CryptoMask, a privacy-preserving face recognition system that employs homomorphic encryption (HE) and secure multi-party computation (MPC). We design a new encoding strategy that leverages HE properties to reduce communication costs and enable efficient similarity checks between face images, without expensive homomorphic rotation. Additionally, CryptoMask leaks less information than existing state-of-the-art approaches. CryptoMask only reveals whether there is an image matching the query or not, whereas existing approaches additionally leak sensitive intermediate distance information. We conduct extensive experiments that demonstrate CryptoMask's superior performance in terms of computation and communication. For a database with 100 million 512-dimensional face vectors, CryptoMask offers ${\thicksim}5 \times$ and ${\thicksim}144 \times$ speed-ups in terms of computation and communication, respectively.
An adaptive model for the optical properties of excited gold

We study the temperature-dependent optical properties of gold over a broad energy spectrum covering photon energies below and above the interband threshold. We apply a semi-analytical Drude-Lorentz model with temperature-dependent oscillator parameters. Our approximations are based on the distribution of electrons over the active bands with a density of states provided by density functional theory. This model can be easily adapted to other materials with similar band structures and can also be applied to the case of occupational nonequilibrium. Our calculations show a strong enhancement of the intraband response with increasing electron temperature while the interband component decreases. Moreover, our model compares well with density functional theory-based calculations for the reflectivity of highly excited gold and reproduces many of its key features. Applying our methods to thin films shows a sensitive nonlinear dependence of the reflection and absorption on the electron temperature. These features are more prominent at small photon energies and can be highlighted with polarized light. Our findings offer valuable insights for modeling ultrafast processes, in particular, the pathways of energy deposition in laser-excited samples.
FDCT: Fast Depth Completion for Transparent Objects

Depth completion is crucial for many robotic tasks such as autonomous driving, 3-D reconstruction, and manipulation. Despite the significant progress, existing methods remain computationally intensive and often fail to meet the real-time requirements of low-power robotic platforms. Additionally, most methods are designed for opaque objects and struggle with transparent objects due to the special properties of reflection and refraction. To address these challenges, we propose a Fast Depth Completion framework for Transparent objects (FDCT), which also benefits downstream tasks like object pose estimation. To leverage local information and avoid overfitting issues when integrating it with global information, we design a new fusion branch and shortcuts to exploit low-level features and a loss function to suppress overfitting. This results in an accurate and user-friendly depth rectification framework which can recover dense depth estimation from RGB-D images alone. Extensive experiments demonstrate that FDCT can run about 70 FPS with a higher accuracy than the state-of-the-art methods. We also demonstrate that FDCT can improve pose estimation in object grasping tasks.
Client-Level Differential Privacy via Adaptive Intermediary in Federated Medical Imaging

Despite recent progress in enhancing the privacy of federated learning (FL) via differential privacy (DP), the trade-off of DP between privacy protection and performance is still underexplored for real-world medical scenario. In this paper, we propose to optimize the trade-off under the context of client-level DP, which focuses on privacy during communications. However, FL for medical imaging involves typically much fewer participants (hospitals) than other domains (e.g., mobile devices), thus ensuring clients be differentially private is much more challenging. To tackle this problem, we propose an adaptive intermediary strategy to improve performance without harming privacy. Specifically, we theoretically find splitting clients into sub-clients, which serve as intermediaries between hospitals and the server, can mitigate the noises introduced by DP without harming privacy. Our proposed approach is empirically evaluated on both classification and segmentation tasks using two public datasets, and its effectiveness is demonstrated with significant performance improvements and comprehensive analytical studies. Code is available at: https://github.com/med-air/Client-DP-FL.