Miracle-dz/UniFAR-FTUs huggingface dataset:
Cryo-Near-Field Photovoltage Microscopy of Heavy-Fermion Twisted Symmetric Trilayer Graphene

Ever since the initial experimental observation of correlated insulators and superconductivity in the flat Dirac bands of magic angle twisted bilayer graphene, a search for the microscopic description that explains its strong electronic interactions has begun. While the seemingly disagreeing electronic transport and scanning tunneling microscopy experiments suggest a dichotomy between local and extended electronic orbitals, definitive experimental evidence merging the two patterns together has been much sought after. Here, we report on the local photothermoelectric measurements in the flat electronic bands of twisted symmetric trilayer graphene (TSTG). We use a cryogenic scanning near-field optical microscope with an oscillating atomic force microscopy (AFM) tip irradiated by the infrared photons to create a nanoscopic hot spot in the planar samples, which generates a photocurrent that we probe globally. We observe a breakdown of the non-interacting Mott formalism at low temperatures (10K), signaling the importance of the electronic interactions. Our measurements reveal an overall negative offset of the Seebeck coefficient and significant peaks of the local photovoltage values at all positive integer fillings of the TSTG's moir\'e superlattice, further indicating a substantial deviation from the classical two-band semiconductor Seebeck response. We explain these observations using the interacting topological heavy-fermion model. In addition, our data reveal a spatial variation of the relative interaction strength dependent on the measured local twist angle (1.2{\deg} - 1.6{\deg}). Our findings provide experimental evidence of heavy fermion behaviour in the topological flat bands of moir\'e graphene and epitomize an avenue to apply local thermoelectric measurements to other strongly correlated materials in the disorder-free limit.
Emergent Word Order Universals from Cognitively-Motivated Language Models

The world's languages exhibit certain so-called typological or implicational universals; for example, Subject-Object-Verb (SOV) word order typically employs postpositions. Explaining the source of such biases is a key goal in linguistics. We study the word-order universals through a computational simulation with language models (LMs). Our experiments show that typologically typical word orders tend to have lower perplexity estimated by LMs with cognitively plausible biases: syntactic biases, specific parsing strategies, and memory limitations. This suggests that the interplay of these cognitive biases and predictability (perplexity) can explain many aspects of word-order universals. This also showcases the advantage of cognitively-motivated LMs, which are typically employed in cognitive modeling, in the computational simulation of language universals.
Memory attacks in network nonlocality and self-testing

We study what can or cannot be certified in communication scenarios where the assumption of independence and identical distribution (iid) between experimental rounds fails. In this respect, we prove that membership tests for non-convex sets of correlations cannot be formulated in the non-iid regime. Similarly, it is impossible to self-test non-extreme quantum operations, such as mixed states, or noisy quantum measurements, unless one allows more than a single use thereof within the same experimental round. One consequence of our results is that non-classicality in causal networks without inputs cannot be experimentally demonstrated. By analyzing optimal non-iid strategies in the triangle scenario, we raise the need to take into account the prior communication required to set up a causal network.
Triple-Encoders: Representations That Fire Together, Wire Together

Search-based dialog models typically re-encode the dialog history at every turn, incurring high cost. Curved Contrastive Learning, a representation learning method that encodes relative distances between utterances into the embedding space via a bi-encoder, has recently shown promising results for dialog modeling at far superior efficiency. While high efficiency is achieved through independently encoding utterances, this ignores the importance of contextualization. To overcome this issue, this study introduces triple-encoders, which efficiently compute distributed utterance mixtures from these independently encoded utterances through a novel hebbian inspired co-occurrence learning objective without using any weights. Empirically, we find that triple-encoders lead to a substantial improvement over bi-encoders, and even to better zero-shot generalization than single-vector representation models without requiring re-encoding. Our code/model is publicly available.
Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvement

Replacing non-polynomial functions (e.g., non-linear activation functions such as ReLU) in a neural network with their polynomial approximations is a standard practice in privacy-preserving machine learning. The resulting neural network, called polynomial approximation of neural network (PANN) in this paper, is compatible with advanced cryptosystems to enable privacy-preserving model inference. Using ``highly precise'' approximation, state-of-the-art PANN offers similar inference accuracy as the underlying backbone model. However, little is known about the effect of approximation, and existing literature often determined the required approximation precision empirically. In this paper, we initiate the investigation of PANN as a standalone object. Specifically, our contribution is two-fold. Firstly, we provide an explanation on the effect of approximate error in PANN. In particular, we discovered that (1) PANN is susceptible to some type of perturbations; and (2) weight regularisation significantly reduces PANN's accuracy. We support our explanation with experiments. Secondly, based on the insights from our investigations, we propose solutions to increase inference accuracy for PANN. Experiments showed that combination of our solutions is very effective: at the same precision, our PANN is 10% to 50% more accurate than state-of-the-arts; and at the same accuracy, our PANN only requires a precision of $2^{-9}$ while state-of-the-art solution requires a precision of $2^{-12}$ using the ResNet-20 model on CIFAR-10 dataset.
Existence of an optimal shape for the first eigenvalue of polyharmonic operators

We prove the existence of an open set minimizing the first eigenvalue of the Dirichlet polylaplacian of order $m\geq1$ under volume constraint. Moreover, the corresponding eigenfunction is shown to enjoy $C^{m-1,\alpha}$ H\"older regularity. This is performed for dimension $2\leq d\leq 4m$. In particular, our analysis answers the question of the existence of an optimal shape for the clamped plate up to dimension $8$.
Stealing the Invisible: Unveiling Pre-Trained CNN Models through Adversarial Examples and Timing Side-Channels

Machine learning, with its myriad applications, has become an integral component of numerous technological systems. A common practice in this domain is the use of transfer learning, where a pre-trained model's architecture, readily available to the public, is fine-tuned to suit specific tasks. As Machine Learning as a Service (MLaaS) platforms increasingly use pre-trained models in their backends, it's crucial to safeguard these architectures and understand their vulnerabilities. In this work, we present an approach based on the observation that the classification patterns of adversarial images can be used as a means to steal the models. Furthermore, the adversarial image classifications in conjunction with timing side channels can lead to a model stealing method. Our approach, designed for typical user-level access in remote MLaaS environments exploits varying misclassifications of adversarial images across different models to fingerprint several renowned Convolutional Neural Network (CNN) and Vision Transformer (ViT) architectures. We utilize the profiling of remote model inference times to reduce the necessary adversarial images, subsequently decreasing the number of queries required. We have presented our results over 27 pre-trained models of different CNN and ViT architectures using CIFAR-10 dataset and demonstrate a high accuracy of 88.8% while keeping the query budget under 20.
Geometric and arithmetic theta correspondences

Geometric/arithmetic theta correspondences provide correspondences between automorphic forms and cohomology classes/algebraic cycles on Shimura varieties. We give an introduction focusing on the example of unitary groups and highlight recent advances in the arithmetic theory (also known as the Kudla program) and their applications. These are expanded lecture notes for the IHES 2022 Summer School on the Langlands Program.
Deep learning methods for Hamiltonian parameter estimation and magnetic domain image generation in twisted van der Waals magnets

The application of twist engineering in van der Waals magnets has opened new frontiers in the field of two-dimensional magnetism, yielding distinctive magnetic domain structures. Despite the introduction of numerous theoretical methods, limitations persist in terms of accuracy or efficiency due to the complex nature of the magnetic Hamiltonians pertinent to these systems. In this study, we introduce a deep-learning approach to tackle these challenges. Utilizing customized, fully connected networks, we develop two deep-neural-network kernels that facilitate efficient and reliable analysis of twisted van der Waals magnets. Our regression model is adept at estimating the magnetic Hamiltonian parameters of twisted bilayer CrI3 from its magnetic domain images generated through atomistic spin simulations. The generative model excels in producing precise magnetic domain images from the provided magnetic parameters. The trained networks for these models undergo thorough validation, including statistical error analysis and assessment of robustness against noisy injections. These advancements not only extend the applicability of deep-learning methods to twisted van der Waals magnets but also streamline future investigations into these captivating yet poorly understood systems.
Lax-Wendroff Flux Reconstruction on adaptive curvilinear meshes with error based time stepping for hyperbolic conservation laws

Lax-Wendroff Flux Reconstruction (LWFR) is a single-stage, high order, quadrature free method for solving hyperbolic conservation laws. This work extends the LWFR scheme to solve conservation laws on curvilinear meshes with adaptive mesh refinement (AMR). The scheme uses a subcell based blending limiter to perform shock capturing and exploits the same subcell structure to obtain admissibility preservation on curvilinear meshes. It is proven that the proposed extension of LWFR scheme to curvilinear grids preserves constant solution (free stream preservation) under the standard metric identities. For curvilinear meshes, linear Fourier stability analysis cannot be used to obtain an optimal CFL number. Thus, an embedded-error based time step computation method is proposed for LWFR method which reduces fine-tuning process required to select a stable CFL number using the wave speed based time step computation. The developments are tested on compressible Euler's equations, validating the blending limiter, admissibility preservation, AMR algorithm, curvilinear meshes and error based time stepping.
Modularized Networks for Few-shot Hateful Meme Detection

In this paper, we address the challenge of detecting hateful memes in the low-resource setting where only a few labeled examples are available. Our approach leverages the compositionality of Low-rank adaptation (LoRA), a widely used parameter-efficient tuning technique. We commence by fine-tuning large language models (LLMs) with LoRA on selected tasks pertinent to hateful meme detection, thereby generating a suite of LoRA modules. These modules are capable of essential reasoning skills for hateful meme detection. We then use the few available annotated samples to train a module composer, which assigns weights to the LoRA modules based on their relevance. The model's learnable parameters are directly proportional to the number of LoRA modules. This modularized network, underpinned by LLMs and augmented with LoRA modules, exhibits enhanced generalization in the context of hateful meme detection. Our evaluation spans three datasets designed for hateful meme detection in a few-shot learning context. The proposed method demonstrates superior performance to traditional in-context learning, which is also more computationally intensive during inference.We then use the few available annotated samples to train a module composer, which assigns weights to the LoRA modules based on their relevance. The model's learnable parameters are directly proportional to the number of LoRA modules. This modularized network, underpinned by LLMs and augmented with LoRA modules, exhibits enhanced generalization in the context of hateful meme detection. Our evaluation spans three datasets designed for hateful meme detection in a few-shot learning context. The proposed method demonstrates superior performance to traditional in-context learning, which is also more computationally intensive during inference.
Aligning Modalities in Vision Large Language Models via Preference Fine-tuning

Instruction-following Vision Large Language Models (VLLMs) have achieved significant progress recently on a variety of tasks. These approaches merge strong pre-trained vision models and large language models (LLMs). Since these components are trained separately, the learned representations need to be aligned with joint training on additional image-language pairs. This procedure is not perfect and can cause the model to hallucinate - provide answers that do not accurately reflect the image, even when the core LLM is highly factual and the vision backbone has sufficiently complete representations. In this work, we frame the hallucination problem as an alignment issue, tackle it with preference tuning. Specifically, we propose POVID to generate feedback data with AI models. We use ground-truth instructions as the preferred response and a two-stage approach to generate dispreferred data. First, we prompt GPT-4V to inject plausible hallucinations into the correct answer. Second, we distort the image to trigger the inherent hallucination behavior of the VLLM. This is an automated approach, which does not rely on human data generation or require a perfect expert, which makes it easily scalable. Finally, both of these generation strategies are integrated into an RLHF pipeline via Direct Preference Optimization. In experiments across broad benchmarks, we show that we can not only reduce hallucinations, but improve model performance across standard benchmarks, outperforming prior approaches. Our data and code are available at https://github.com/YiyangZhou/POVID.
AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods

The choice of batch sizes in stochastic gradient optimizers is critical for model training. However, the practice of varying batch sizes throughout the training process is less explored compared to other hyperparameters. We investigate adaptive batch size strategies derived from adaptive sampling methods, traditionally applied only in stochastic gradient descent. Given the significant interplay between learning rates and batch sizes, and considering the prevalence of adaptive gradient methods in deep learning, we emphasize the need for adaptive batch size strategies in these contexts. We introduce AdAdaGrad and its scalar variant AdAdaGradNorm, which incrementally increase batch sizes during training, while model updates are performed using AdaGrad and AdaGradNorm. We prove that AdaGradNorm converges with high probability at a rate of $\mathscr{O}(1/K)$ for finding a first-order stationary point of smooth nonconvex functions within $K$ iterations. AdaGrad also demonstrates similar convergence properties when integrated with a novel coordinate-wise variant of our adaptive batch size strategies. Our theoretical claims are supported by numerical experiments on various image classification tasks, highlighting the enhanced adaptability of progressive batching protocols in deep learning and the potential of such adaptive batch size strategies with adaptive gradient optimizers in large-scale model training.
On a q-analogue of the Zeta polynomial of posets

We introduce a q-analogue of the classical Zeta polynomial of finite partially ordered sets, as a polynomial in one variable x with coefficients depending on the indeterminate q. We prove some properties of this polynomial invariant, including its behaviour with respect to duality, product and disjoint union. The leading term is a q-analogue of the number of maximal chains, but not always with non-negative coefficients. The value at q=0 turns out to be essentially the characteristic polynomial.
Speculative Streaming: Fast LLM Inference without Auxiliary Models

Speculative decoding is a prominent technique to speed up the inference of a large target language model based on predictions of an auxiliary draft model. While effective, in application-specific settings, it often involves fine-tuning both draft and target models to achieve high acceptance rates. As the number of downstream tasks grows, these draft models add significant complexity to inference systems. We propose Speculative Streaming, a single-model speculative decoding method that fuses drafting into the target model by changing the fine-tuning objective from next token prediction to future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 - 3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and Meaning Representation, without sacrificing generation quality. Additionally, Speculative Streaming is parameter-efficient. It achieves on-par/higher speed-ups than Medusa-style architectures while using ~10000X fewer extra parameters, making it well-suited for resource-constrained devices.
WildFake: A Large-scale Challenging Dataset for AI-Generated Images Detection

The extraordinary ability of generative models enabled the generation of images with such high quality that human beings cannot distinguish Artificial Intelligence (AI) generated images from real-life photographs. The development of generation techniques opened up new opportunities but concurrently introduced potential risks to privacy, authenticity, and security. Therefore, the task of detecting AI-generated imagery is of paramount importance to prevent illegal activities. To assess the generalizability and robustness of AI-generated image detection, we present a large-scale dataset, referred to as WildFake, comprising state-of-the-art generators, diverse object categories, and real-world applications. WildFake dataset has the following advantages: 1) Rich Content with Wild collection: WildFake collects fake images from the open-source community, enriching its diversity with a broad range of image classes and image styles. 2) Hierarchical structure: WildFake contains fake images synthesized by different types of generators from GANs, diffusion models, to other generative models. These key strengths enhance the generalization and robustness of detectors trained on WildFake, thereby demonstrating WildFake's considerable relevance and effectiveness for AI-generated detectors in real-world scenarios. Moreover, our extensive evaluation experiments are tailored to yield profound insights into the capabilities of different levels of generative models, a distinctive advantage afforded by WildFake's unique hierarchical structure.
The Effects of Group Discussion and Role-playing Training on Self-efficacy, Support-seeking, and Reporting Phishing Emails: Evidence from a Mixed-design Experiment

Organizations rely on phishing interventions to enhance employees' vigilance and safe responses to phishing emails that bypass technical solutions. While various resources are available to counteract phishing, studies emphasize the need for interactive and practical training approaches. To investigate the effectiveness of such an approach, we developed and delivered two anti-phishing trainings, group discussion and role-playing, at a European university. We conducted a pre-registered experiment (N = 105), incorporating repeated measures at three time points, a control group, and three in-situ phishing tests. Both trainings enhanced employees' anti-phishing self-efficacy and support-seeking intention in within-group analyses. Only the role-playing training significantly improved support-seeking intention when compared to the control group. Participants in both trainings reported more phishing tests and demonstrated heightened vigilance to phishing attacks compared to the control group. We discuss practical implications for evaluating and improving phishing interventions and promoting safe responses to phishing threats within organizations.
Asymptotic Gaussian Fluctuations of Eigenvectors in Spectral Clustering

The performance of spectral clustering relies on the fluctuations of the entries of the eigenvectors of a similarity matrix, which has been left uncharacterized until now. In this letter, it is shown that the signal $+$ noise structure of a general spike random matrix model is transferred to the eigenvectors of the corresponding Gram kernel matrix and the fluctuations of their entries are Gaussian in the large-dimensional regime. This CLT-like result was the last missing piece to precisely predict the classification performance of spectral clustering. The proposed proof is very general and relies solely on the rotational invariance of the noise. Numerical experiments on synthetic and real data illustrate the universality of this phenomenon.
Scaling Limit of Asymptotically-free Self-interacting Random Walks to Brownian Motion Perturbed at Extrema

We show convergence of a family of one-dimensional self-interacting random walks to Brownian motion perturbed at extrema under the diffusive scaling. This completes the functional limit theorem in [KMP23] for the asymptotically free case when $0<p \leq \frac{1}{2}$. The approach is to approximate the total drift experienced by the walker via analyzing directed edge local times, described by the branching-like processes. The analysis depends on the diffusion approximation of the branching-like processes obtained in the Ray-Knight type framework.
The Competition Complexity of Prophet Inequalities

We study the classic single-choice prophet inequality problem through a resource augmentation lens. Our goal is to bound the $(1-\varepsilon)$-competition complexity of different types of online algorithms. This metric asks for the smallest $k$ such that the expected value of the online algorithm on $k$ copies of the original instance, is at least a $(1-\varepsilon)$-approximation to the expected offline optimum on a single copy. We show that block threshold algorithms, which set one threshold per copy, are optimal and give a tight bound of $k = \Theta(\log \log 1/\varepsilon)$. This shows that block threshold algorithms approach the offline optimum doubly-exponentially fast. For single threshold algorithms, we give a tight bound of $k = \Theta(\log 1/\varepsilon)$, establishing an exponential gap between block threshold algorithms and single threshold algorithms. Our model and results pave the way for exploring resource-augmented prophet inequalities in combinatorial settings. In line with this, we present preliminary findings for bipartite matching with one-sided vertex arrivals, as well as in XOS combinatorial auctions. Our results have a natural competition complexity interpretation in mechanism design and pricing applications.