Miracle-dz/UniFAR-FTUs huggingface dataset:
Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval

The ability to accurately model the fitness landscape of protein sequences is critical to a wide range of applications, from quantifying the effects of human variants on disease likelihood, to predicting immune-escape mutations in viruses and designing novel biotherapeutic proteins. Deep generative models of protein sequences trained on multiple sequence alignments have been the most successful approaches so far to address these tasks. The performance of these methods is however contingent on the availability of sufficiently deep and diverse alignments for reliable training. Their potential scope is thus limited by the fact many protein families are hard, if not impossible, to align. Large language models trained on massive quantities of non-aligned protein sequences from diverse families address these problems and show potential to eventually bridge the performance gap. We introduce Tranception, a novel transformer architecture leveraging autoregressive predictions and retrieval of homologous sequences at inference to achieve state-of-the-art fitness prediction performance. Given its markedly higher performance on multiple mutants, robustness to shallow alignments and ability to score indels, our approach offers significant gain of scope over existing approaches. To enable more rigorous model testing across a broader range of protein families, we develop ProteinGym -- an extensive set of multiplexed assays of variant effects, substantially increasing both the number and diversity of assays compared to existing benchmarks.
Image Harmonization with Region-wise Contrastive Learning

—Image harmonization task aims at harmonizing different composite foreground regions according to speciﬁc background image. Previous methods would rather focus on improving the reconstruction ability of the generator by some internal enhancements such as attention, adaptive normalization and light adjustment, etc. . However, they pay less attention to discriminating the foreground and background appearance features within a restricted generator, which becomes a new chal-lenge in image harmonization task. In this paper, we propose a novel image harmonization framework with external style fusion and region-wise contrastive learning scheme. For the external style fusion, we leverage the external background appearance from the encoder as the style reference to generate harmonized foreground in the decoder. This approach enhances the harmonization ability of the decoder by external background guidance. Moreover, for the contrastive learning scheme, we design a region-wise contrastive loss function for image harmonization task. Speciﬁcally, we ﬁrst introduce a straight-forward samples generation method that selects negative samples from the output harmonized foreground region and selects positive samples from the ground-truth background region. Our method attempts to bring together corresponding positive and negative samples by maximizing the mutual information between the foreground and background styles, which desirably makes our harmonization network more robust to discriminate the foreground and background style features when harmonizing composite images. Extensive experiments on the benchmark datasets show that our method can achieve a clear improvement in harmonization quality and demonstrate the good generalization capability in real-scenario applications.
Subverting machines, fluctuating identities: Re-learning human categorization

Most machine learning systems that interact with humans construct some notion of a person’s “identity,” yet the default paradigm in AI research envisions identity with essential attributes that are discrete and static. In stark contrast, strands of thought within critical theory present a conception of identity as malleable and constructed entirely through interaction; a doing rather than a being. In this work, we distill some of these ideas for machine learning practitioners and introduce a theory of identity as autopoiesis, circular processes of formation and function. We argue that the default paradigm of identity used by the field immobilizes existing identity categories and the power differentials that co-occur, due to the absence of iterative feedback to our models. This includes a critique of emergent AI fairness practices that continue to impose the default paradigm. Finally, we apply our theory to sketch approaches to autopoietic identity through multilevel optimization and relational learning. While these ideas raise many open questions, we imagine the possibilities of machines that are capable of expressing human identity as a relationship perpetually in flux.
Feudal Multi-Agent Reinforcement Learning with Adaptive Network Partition for Traffic Signal Control

Multi-agent reinforcement learning (MARL) has been ap- plied and shown great potential in multi-intersections trafﬁc signal control, where multiple agents, one for each intersec- tion, must cooperate together to optimize trafﬁc ﬂow. To encourage global cooperation, previous work partitions the traf- ﬁc network into several regions and learns policies for agents in a feudal structure. However, static network partition fails to adapt to dynamic trafﬁc ﬂow, which will changes frequently over time. To address this, we propose a novel feudal MARL approach with adaptive network partition. Speciﬁcally, we ﬁrst partition the network into several regions according to the trafﬁc ﬂow. To do this, we propose two approaches: one is directly to use graph neural network (GNN) to generate the network partition, and the other is to use Monte-Carlo tree search (MCTS) to ﬁnd the best partition with criteria com- puted by GNN. Then, we design a variant of Qmix using GNN to handle various dimensions of input, given by the dy- namic network partition. Finally, we use a feudal hierarchy to manage agents in each partition and promote global coopera- tion. By doing so, agents are able to adapt to the trafﬁc ﬂow as required in practice. We empirically evaluate our method both in a synthetic trafﬁc grid and real-world trafﬁc networks of three cities, widely used in the literature. Our experimen- tal results conﬁrm that our method can achieve better performance, in terms of average travel time and queue length, than several leading methods for trafﬁc signal control.
Dynamic cluster structure and predictive modelling of music creation style distributions

We investigate the dynamics of music creation style distributions to understand cultural evolution involving intelligence to create complex artefacts. Previous work suggested that a music creation style can be quantified as statistics describing a generative process of music data, and that the distribution of music creation styles in a society has cluster structure related to the presence of different musical genres. To find patterns in the dynamics of the cluster structure, we analysed statistics of melodies in Japanese popular music data and statistics of audio features in American popular music data. Using statistical modelling methods, we found that intra-cluster dynamics, such as the contraction and the shift of a cluster, as well as inter-cluster dynamics represented by clusters’ relative frequencies, often exhibit notable dynamical modes. Additionally, to compare the individual contributions of these different dynamical aspects for predicting future creation style distributions, we constructed a fitness-based evolutionary model and found that the predictions of cluster frequencies and cluster variances often have comparable contributions. Our results highlight the relevance of intra-cluster dynamics in music style evolution, which have often been overlooked in previous studies. The present methodology can be applied to cultural artefacts whose generative process can be characterized by a discrete probability distribution.
Diffusion Monte Carlo calculations of fully heavy compact hexaquarks

We used a diffusion Monte Carlo technique to describe the properties of fully-heavy compact arrangements (no dibaryon molecules) including six quarks and no antiquarks within the framewok of a constituent quark model. Only arrangements whose wavefunctions were eigenvectors of $L^2$ with eigenvalue $\ell$ = 0 were taken into account, what means that we only considered the subset of all the possible color-spin combinations that make the total wavefunctions antisymmetric with respect to the interchange of any two quarks of the same type. This means arrangements with spin $S=0$ for $cccccc$, $bbbbbb$, $cccccb$, $bbbbbc$ and $cccbbb$ and spin $S=0,1,2$ for the $ccccbb$ and $bbbbcc$ hexaquarks. In all cases, the masses of the six-quark arrangements are larger that the ones corresponding to the sum of any of the two baryons we can split them into, but smaller that the ones for a set of six isolated quarks, i.e., all them are bound systems. The analysis of their structure indicates that all the hexaquarks considered in this work are compact objects, except the $cccbbb$, that appears to be a loose association of two baryons.
Spartan: Differentiable Sparsity via Regularized Transportation

We present Spartan, a method for training sparse neural network models with a predetermined level of sparsity. Spartan is based on a combination of two techniques: (1) soft top-k masking of low-magnitude parameters via a regularized optimal transportation problem and (2) dual averaging-based parameter updates with hard sparsification in the forward pass. This scheme realizes an exploration-exploitation tradeoff: early in training, the learner is able to explore various sparsity patterns, and as the soft top-k approximation is gradually sharpened over the course of training, the balance shifts towards parameter optimization with respect to a fixed sparsity mask. Spartan is sufficiently flexible to accommodate a variety of sparsity allocation policies, including both unstructured and block structured sparsity, as well as general cost-sensitive sparsity allocation mediated by linear models of per-parameter costs. On ImageNet-1K classification, Spartan yields 95% sparse ResNet-50 models and 90% block sparse ViT-B/16 models while incurring absolute top-1 accuracy losses of less than 1% compared to fully dense training.
Writes hurt: lessons in cache design for optane NVRAM

Intel® Optane™ DC Persistent Memory resides on the memory bus and approaches DRAM in access latency. One avenue for its adoption is to employ it in place of persistent storage; another is to use it as a cheaper and denser extension of DRAM. In pursuit of the latter goal, we present the design of a volatile Optane NVRAM cache as a component in a storage engine underlying MongoDB. The primary innovation in our design is a new cache admission policy. We discover that on Optane NVRAM, known for its limited write throughput, the presence of writes disproportionately affects the throughput of reads, much more so than on DRAM. Therefore, an admission policy that indiscriminately admits new data (and thus generates writes), severely limits the rate of data retrieval and results in exceedingly poor performance for the cache overall. We design an admission policy that balances the rate of admission with the rate of lookups using dynamically observed characteristics of the workload. Our implementation outperforms OpenCAS (an off-the-shelf Optane-based block cache) in all cases, and Intel Memory Mode in cases where the database size exceeds the available NVRAM. Our cache is decoupled from the rest of the storage engine and uses generic metrics to guide its admission policy; this design can be easily adopted in other systems.
Infinite-randomness criticality in monitored quantum dynamics with static disorder

We consider a model of monitored quantum dynamics with quenched spatial randomness: specifically, random quantum circuits with spatially varying measurement rates. These circuits undergo a measurement-induced phase transition (MIPT) in their entanglement structure, but the nature of the critical point differs drastically from the case with constant measurement rate. In particular, at the critical measurement rate, we find that the entanglement of a subsystem of size $\ell$ scales as $S \sim \sqrt{\ell}$; moreover, the dynamical critical exponent $z = \infty$. The MIPT is flanked by Griffiths phases with continuously varying dynamical exponents. We argue for this infinite-randomness scenario on general grounds and present numerical evidence that it captures some features of the universal critical properties of MIPT using large-scale simulations of Clifford circuits. These findings demonstrate that the relevance and irrelevance of perturbations to the MIPT can naturally be interpreted using a powerful heuristic known as the Harris criterion.
Learning with Stochastic Orders

Learning high-dimensional distributions is often done with explicit likelihood modeling or implicit modeling via minimizing integral probability metrics (IPMs). In this paper, we expand this learning paradigm to stochastic orders, namely, the convex or Choquet order between probability measures. Towards this end, exploiting the relation between convex orders and optimal transport, we introduce the Choquet-Toland distance between probability measures, that can be used as a drop-in replacement for IPMs. We also introduce the Variational Dominance Criterion (VDC) to learn probability measures with dominance constraints, that encode the desired stochastic order between the learned measure and a known baseline. We analyze both quantities and show that they suffer from the curse of dimensionality and propose surrogates via input convex maxout networks (ICMNs), that enjoy parametric rates. We provide a min-max framework for learning with stochastic orders and validate it experimentally on synthetic and high-dimensional image generation, with promising results. Finally, our ICMNs class of convex functions and its derived Rademacher Complexity are of independent interest beyond their application in convex orders.
Decaying warm dark matter revisited

Decaying dark matter models provide a physically motivated way of channeling energy between the matter and radiation sectors. In principle, this could affect the predicted value of the Hubble constant in such a way as to accommodate the discrepancies between CMB inferences and local measurements of the same. Here, we revisit the model of warm dark matter decaying non-relativistically to invisible radiation. In particular, we rederive the background and perturbation equations starting from a decaying neutrino model and describe a new, computationally efficient method of computing the decay product perturbations up to large multipoles. We conduct MCMC analyses to constrain all three model parameters, for the first time including the mass of the decaying species, and assess the ability of the model to alleviate the Hubble and σ 8 tensions, the latter being the discrepancy between the CMB and weak gravitational lensing constraints on the amplitude of matter fluctuations on an 8 h -1 Mpc-1 scale. We find that the model reduces the H 0 tension from ∼ 4σ to ∼ 3σ and neither alleviates nor worsens the S 8 ≡ σ 8 (Ω m /0.3)0.5 tension, ultimately showing only mild improvements with respect to ΛCDM. However, the values of the model-specific parameters favoured by data is found to be well within the regime of relativistic decays where inverse processes are important, rendering a conclusive evaluation of the decaying warm dark matter model open to future work.
Oblique transition in hypersonic double-wedge flow

Abstract We utilize resolvent and weakly nonlinear analyses in combination with direct numerical simulations (DNS) to identify mechanisms for oblique transition in a Mach $5$ hypersonic flow over an adiabatic slender double wedge. Even though the laminar separated flow is globally stable, resolvent analysis demonstrates significant amplification of unsteady external disturbances to the linearized flow equations. These disturbances are introduced upstream of the separation zone and they lead to the appearance of oblique waves further downstream. We demonstrate that the large amplification of oblique waves arises from the growth of fluctuation shear stress due to streamline curvature of the laminar base flow in the separated shear layer. This is in contrast to the attached boundary layers, where no such mechanism exists. We also use a weakly nonlinear analysis to show that the resolvent operator associated with linearization around the laminar base flow governs the evolution of steady reattachment streaks that arise from quadratic interactions of unsteady oblique waves. These quadratic interactions generate vortical excitations in the reattaching shear layer which lead to the formation of streaks in the recirculation zone and their subsequent amplification, breakdown and transition to turbulence downstream. Our analysis of the energy budget shows that deceleration of the base flow near reattachment is primarily responsible for amplification of steady streaks. Finally, we employ DNS to examine latter stages of transition to turbulence and demonstrate the predictive power of a weakly nonlinear input–output framework in uncovering triggering mechanisms for oblique transition in separated high-speed boundary layer flows.
CIGMO: Categorical invariant representations in a deep generative framework

Data of general object images have two most common structures: (1) each object of a given shape can be rendered in multiple different views, and (2) shapes of objects can be categorized in such a way that the diversity of shapes is much larger across categories than within a category. Existing deep generative models can typically capture either structure, but not both. In this work, we introduce a novel deep generative model, called CIGMO, that can learn to represent category, shape, and view factors from image data. The model is comprised of multiple modules of shape representations that are each specialized to a particular category and disentangled from view representation, and can be learned using a group-based weakly supervised learning method. By empirical investigation, we show that our model can effectively discover categories of object shapes despite large view variation and quantitatively supersede various previous methods including the state-of-the-art invariant clustering algorithm. Further, we show that our approach using category-specialization can enhance the learned shape representation to better perform down-stream tasks such as one-shot object identification as well as shape-view disentanglement.
Learning to Control Linear Systems can be Hard

In this paper, we study the statistical difficulty of learning to control linear systems. We focus on two standard benchmarks, the sample complexity of stabilization, and the regret of the online learning of the Linear Quadratic Regulator (LQR). Prior results state that the statistical difficulty for both benchmarks scales polynomially with the system state dimension up to system-theoretic quantities. However, this does not reveal the whole picture. By utilizing minimax lower bounds for both benchmarks, we prove that there exist non-trivial classes of systems for which learning complexity scales dramatically, i.e. exponentially, with the system dimension. This situation arises in the case of underactuated systems, i.e. systems with fewer inputs than states. Such systems are structurally difficult to control and their system theoretic quantities can scale exponentially with the system dimension dominating learning complexity. Under some additional structural assumptions (bounding systems away from uncontrollability), we provide qualitatively matching upper bounds. We prove that learning complexity can be at most exponential with the controllability index of the system, that is the degree of underactuation.
Entanglement entropy scaling of noisy random quantum circuits in two dimensions

Whether noisy quantum devices without error correction can provide quantum advantage over classical computers is a critical issue of current quantum computation. In this work, the random quantum circuits, which are used as the paradigm model to demonstrate quantum advantage, are simulated with depolarizing noise on experiment relevant two-dimensional architecture. With comprehensive numerical simulation and theoretical analysis, we find that the maximum achievable operator entanglement entropy, which indicates maximal simulation cost, has area law scaling with the system size for constant noise rate. On the other hand, we also find that the maximum achievable operator entanglement entropy has power law scaling with the noise rate for fixed system size, and the volume law scaling can be obtained only if the noise rate decreases when system size increase.
Gradient damage models for heterogeneous materials

In this paper we study the asymptotic behaviour of phase-field functionals of Ambrosio and Tortorelli type allowing for small-scale oscillations both in the volume and in the diffuse surface term. The functionals under examination can be interpreted as an instance of a static gradient damage model for heterogeneous materials. Depending on the mutual vanishing rate of the approximation and of the oscillation parameters, the effective behaviour of the model is fully characterised by means of Γ-convergence.
Membership Inference Attack Using Self Influence Functions

Member inference (MI) attacks aim to determine if a specific data sample was used to train a machine learning model. Thus, MI is a major privacy threat to models trained on private sensitive data, such as medical records. In MI attacks one may consider the black-box settings, where the model's parameters and activations are hidden from the adversary, or the white-box case where they are available to the attacker. In this work, we focus on the latter and present a novel MI attack for it that employs influence functions, or more specifically the samples' self-influence scores, to perform the MI prediction. We evaluate our attack on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets, using versatile architectures such as AlexNet, ResNet, and DenseNet. Our attack method achieves new state-of-the-art results for both training with and without data augmentations. Code is available at https://github.com/giladcohen/sif_mi_attack.
Textural-Structural Joint Learning for No-Reference Super-Resolution Image Quality Assessment

—Image super-resolution (SR) has been widely investigated in recent years. However, it is challenging to fairly estimate the performances of various SR methods, as the lack of reliable and accurate criteria for perceptual quality. Existing SR image quality assessment (IQA) metrics usually concentrate on the speciﬁc kind of degradation without distinguishing the visual sensitive areas, which have no adaptive ability to describe the diverse SR degeneration situations. In this paper, we focus on the textural and structural degradation of image SR which acts as a critical role for visual perception, and design a dual stream network to jointly explore the textural and structural information for quality prediction, dubbed TSNet. By mimicking the human vision system (HVS) that pays more attention to the signiﬁcant areas of the image, we develop the spatial attention mechanism to make the visual-sensitive areas more distinguishable, which improves the prediction accuracy. Feature normalization (F-Norm) is also developed to investigate the inherent spatial correlation of SR features and boost the network representation capacity. Experimental results show the proposed TSNet predicts the visual quality more accurate than the state-of-the-art IQA methods, and demonstrates better consistency with the human’s perspective. The source code will be made available at http://github.com/yuqing-liu-dut/NRIQA SR.