Miracle-dz/UniFAR-FTUs huggingface dataset:
Smooth Bilevel Programming for Sparse Regularization

Iteratively reweighted least square (IRLS) is a popular approach to solve sparsity-enforcing regression problems in machine learning. State of the art approaches are more efficient but typically rely on specific coordinate pruning schemes. In this work, we show how a surprisingly simple reparametrization of IRLS, coupled with a bilevel resolution (instead of an alternating scheme) is able to achieve top performances on a wide range of sparsity (such as Lasso, group Lasso and trace norm regularizations), regularization strength (including hard constraints), and design matrices (ranging from correlated designs to differential operators). Similarly to IRLS, our method only involves linear systems resolutions, but in sharp contrast, corresponds to the minimization of a smooth function. Despite being non-convex, we show that there is no spurious minima and that saddle points are"ridable", so that there always exists a descent direction. We thus advocate for the use of a BFGS quasi-Newton solver, which makes our approach simple, robust and efficient. We perform a numerical benchmark of the convergence speed of our algorithm against state of the art solvers for Lasso, group Lasso, trace norm and linearly constrained problems. These results highlight the versatility of our approach, removing the need to use different solvers depending on the specificity of the ML problem under study.
Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA?

Recent work has investigated the interesting question using pre-trained language models (PLMs) as knowledge bases for answering open questions. However, existing work is limited in using small benchmarks with high test-train overlaps. We construct a new dataset of closed-book QA using SQuAD, and investigate the performance of BART. Experiments show that it is challenging for BART to remember training facts in high precision, and also challenging to answer closed-book questions even if relevant knowledge is retained. Some promising directions are found, including decoupling the knowledge memorizing process and the QA finetune process, forcing the model to recall relevant knowledge when question answering.
Supercurrent diode effect and finite-momentum superconductors

Significance Our work shows a fascinating application of finite-momentum superconductivity, the supercurrent diode effect, which is being reported in a growing number of experiments. We show that, under external magnetic field, Cooper pairs can acquire finite momentum so that critical currents in the direction parallel and antiparallel to the Cooper pair momentum become unequal.
Normalization for Multimodal Type Theory

We prove normalization for MTT, a general multimodal dependent type theory capable of expressing modal type theories for guarded recursion, internalized parametricity, and various other prototypical modal situations. We prove that deciding type checking and conversion in MTT can be reduced to deciding the equality of modalities in the underlying modal situation, immediately yielding a type checking algorithm for all instantiations of MTT in the literature. This proof follows from a generalization of synthetic Tait computability—an abstract approach to gluing proofs—to account for modalities. This extension is based on MTT itself, so that this proof also constitutes a significant case study of MTT.
Resource-Aware Cost-Sharing Mechanisms with Priors

In a decentralized system with m machines, we study the selfish scheduling problem where each user strategically chooses which machine to use. Each machine incurs a cost, which is a function of the total load assigned to it, and some cost-sharing mechanism distributes this cost among the machine's users. The users choose a machine aiming to minimize their own share of the cost, so the cost-sharing mechanism induces a game among them. We approach this problem from the perspective of a designer who can select which cost-sharing mechanism to use, aiming to minimize the price of anarchy (PoA) of the induced games. Recent work introduced the class of resource-aware cost-sharing mechanisms, whose decisions can depend on the set of machines in the system, but are oblivious to the total number of users. These mechanisms can guarantee low PoA bounds for instances where the cost functions of the machines are all convex or concave, but can suffer from very high PoA for cost functions that deviate from these families. In this paper we show that if we enhance the class of resource-aware mechanisms with some prior information regarding the users, then they can achieve low PoA for a much more general family of cost functions. We first show that, as long as the mechanism knows just two of the participating users, then it can assign special roles to them and ensure a constant PoA. We then extend this idea to settings where the mechanism has access to the probability with which each user is present in the system. For all these instances, we provide a mechanism that achieves an expected PoA that is logarithmic in the expected number of users.
Gradient Boosted Binary Histogram Ensemble for Large-scale Regression

In this paper, we propose a gradient boosting algorithm for large-scale regression problems called Gradient Boosted Binary Histogram Ensemble (GBBHE) based on binary histogram partition and ensemble learning. From the theoretical perspective, by assuming the Hölder continuity of the target function, we establish the statistical convergence rate of GBBHE in the space C and C, where a lower bound of the convergence rate for the base learner demonstrates the advantage of boosting. Moreover, in the space C, we prove that the number of iterations to achieve the fast convergence rate can be reduced by using ensemble regressor as the base learner, which improves the computational efficiency. In the experiments, compared with other state-of-the-art algorithms such as gradient boosted regression tree (GBRT), Breiman’s forest, and kernel-based methods, our GBBHE algorithm shows promising performance with less running time on large-scale datasets.
DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification

Attention is sparse in vision transformers. We observe the final prediction in vision transformers is only based on a subset of most informative tokens, which is sufficient for accurate image recognition. Based on this observation, we propose a dynamic token sparsification framework to prune redundant tokens progressively and dynamically based on the input. Specifically, we devise a lightweight prediction module to estimate the importance score of each token given the current features. The module is added to different layers to prune redundant tokens hierarchically. To optimize the prediction module in an end-to-end manner, we propose an attention masking strategy to differentiably prune a token by blocking its interactions with other tokens. Benefiting from the nature of self-attention, the unstructured sparse tokens are still hardware friendly, which makes our framework easy to achieve actual speed-up. By hierarchically pruning 66% of the input tokens, our method greatly reduces 31%~37% FLOPs and improves the throughput by over 40% while the drop of accuracy is within 0.5% for various vision transformers. Equipped with the dynamic token sparsification framework, DynamicViT models can achieve very competitive complexity/accuracy trade-offs compared to state-of-the-art CNNs and vision transformers on ImageNet. Code is available at https://github.com/raoyongming/DynamicViT
When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations

Vision Transformers (ViTs) and MLPs signal further efforts on replacing hand-wired features or inductive biases with general-purpose neural architectures. Existing works empower the models by massive data, such as large-scale pre-training and/or repeated strong data augmentations, and still report optimization-related problems (e.g., sensitivity to initialization and learning rates). Hence, this paper investigates ViTs and MLP-Mixers from the lens of loss geometry, intending to improve the models' data efficiency at training and generalization at inference. Visualization and Hessian reveal extremely sharp local minima of converged models. By promoting smoothness with a recently proposed sharpness-aware optimizer, we substantially improve the accuracy and robustness of ViTs and MLP-Mixers on various tasks spanning supervised, adversarial, contrastive, and transfer learning (e.g., +5.3\% and +11.0\% top-1 accuracy on ImageNet for ViT-B/16 and Mixer-B/16, respectively, with the simple Inception-style preprocessing). We show that the improved smoothness attributes to sparser active neurons in the first few layers. The resultant ViTs outperform ResNets of similar size and throughput when trained from scratch on ImageNet without large-scale pre-training or strong data augmentations. Model checkpoints are available at \url{https://github.com/google-research/vision_transformer}.
Accuracy Improvement Technique of DNN for Accelerating CFD Simulator

There is a Computational fluid dynamics (CFD) method of incorporating the DNN inference to reduce the computational cost. The reduction is realized by replacing some calculations by DNN inference. The cost reduction depends on the implementation method of the DNN and the accuracy of the DNN inference. Thus, we propose two techniques suitable to infer flow field on the CFD grid. The first technique is to infer the flow field of the steady state from the airfoil shape. We use the position on the coordinates of the grid point and the distance from the surface of the airfoil as input information for the DNN. The second method uses the customized mean square error as a loss function. The size of the associated area for each grid point was multiplied by the square error. This method compensates for the effect caused by the size of the associated area of nonuniform allocation of grid points. The evaluation results show that the CFD incorporated first technique achieves 1.7x speedup against the CFD without DNN, while maintaining equivalent result quality. By implementing the second technique, the CFD realized further 2.3x speed up against the CFD with first technique only.
Improving Event Causality Identification via Self-Supervised Representation Learning on External Causal Statement

Current models for event causality identification (ECI) mainly adopt a supervised framework, which heavily rely on labeled data for training. Unfortunately, the scale of current annotated datasets is relatively limited, which cannot provide sufficient support for models to capture useful indicators from causal statements, especially for handing those new, unseen cases. To alleviate this problem, we propose a novel approach, shortly named CauSeRL, which leverages external causal statements for event causality identification. First of all, we design a self-supervised framework to learn context-specific causal patterns from external causal statements. Then, we adopt a contrastive transfer strategy to incorporate the learned context-specific causal patterns into the target ECI model. Experimental results show that our method significantly outperforms previous methods on EventStoryLine and Causal-TimeBank (+2.0 and +3.4 points on F1 value respectively).
The energy cost and optimal design of networks for biological discrimination

Many biological processes discriminate between correct and incorrect substrates through the kinetic proofreading mechanism that enables lower error at the cost of higher energy dissipation. Elucidating physico-chemical constraints for global minimization of dissipation and error is important for understanding enzyme evolution. Here, we identify theoretically a fundamental error–cost bound that tightly constrains the performance of proofreading networks under any parameter variations preserving the rate discrimination between substrates. The bound is kinetically controlled, i.e. completely determined by the difference between the transition state energies on the underlying free energy landscape. The importance of the bound is analysed for three biological processes. DNA replication by T7 DNA polymerase is shown to be nearly optimized, i.e. its kinetic parameters place it in the immediate proximity of the error–cost bound. The isoleucyl-tRNA synthetase (IleRS) of E. coli also operates close to the bound, but further optimization is prevented by the need for reaction speed. In contrast, E. coli ribosome operates in a high-dissipation regime, potentially in order to speed up protein production. Together, these findings establish a fundamental error–dissipation relation in biological proofreading networks and provide a theoretical framework for studying error–dissipation trade-off in other systems with biological discrimination.
A Jacobian criterion for nonsingularity in mixed characteristic

We give a version of the usual Jacobian characterization of the defining ideal of the singular locus in the equal characteristic case: the new theorem is valid for essentially affine algebras over a complete local algebra over a mixed characteristic discrete valuation ring. The result makes use of the minors of a matrix that includes a row coming from the values of a p-derivation. To study the analogue of modules of differentials associated with the mixed Jacobian matrices that arise in our context, we introduce and investigate the notion of a perivation, which may be thought of, roughly, as a linearization of the notion of p-derivation. We also develop a mixed characteristic analogue of the positive characteristic Γ-construction, and apply this to give additional nonsingularity criteria.
Unsharp Mask Guided Filtering

The goal of this paper is guided image filtering, which emphasizes the importance of structure transfer during filtering by means of an additional guidance image. Where classical guided filters transfer structures using hand-designed functions, recent guided filters have been considerably advanced through parametric learning of deep networks. The state-of-the-art leverages deep networks to estimate the two core coefficients of the guided filter. In this work, we posit that simultaneously estimating both coefficients is suboptimal, resulting in halo artifacts and structure inconsistencies. Inspired by unsharp masking, a classical technique for edge enhancement that requires only a single coefficient, we propose a new and simplified formulation of the guided filter. Our formulation enjoys a filtering prior from a low-pass filter and enables explicit structure transfer by estimating a single coefficient. Based on our proposed formulation, we introduce a successive guided filtering network, which provides multiple filtering results from a single network, allowing for a trade-off between accuracy and efficiency. Extensive ablations, comparisons and analysis show the effectiveness and efficiency of our formulation and network, resulting in state-of-the-art results across filtering tasks like upsampling, denoising, and cross-modality filtering. Code is available at https://github.com/shizenglin/Unsharp-Mask-Guided-Filtering.
Morphological classification of galaxies with deep learning: comparing 3-way and 4-way CNNs

Classifying the morphologies of galaxies is an important step in understanding their physical properties and evolutionary histories. The advent of large-scale surveys has hastened the need to develop techniques for automated morphological classification. We train and test several convolutional neural network (CNN) architectures to classify the morphologies of galaxies in both a 3-class (elliptical, lenticular, and spiral) and a 4-class (+irregular/miscellaneous) schema with a data set of 14 034 visually classified SDSS images. We develop a new CNN architecture that outperforms existing models in both 3-way and 4-way classifications, with overall classification accuracies of 83 and 81 per cent, respectively. We also compare the accuracies of 2-way/binary classifications between all four classes, showing that ellipticals and spirals are most easily distinguished (>98 per cent accuracy), while spirals and irregulars are hardest to differentiate (78 per cent accuracy). Through an analysis of all classified samples, we find tentative evidence that misclassifications are physically meaningful, with lenticulars misclassified as ellipticals tending to be more massive, among other trends. We further combine our binary CNN classifiers to perform a hierarchical classification of samples, obtaining comparable accuracies (81 per cent) to the direct 3-class CNN, but considerably worse accuracies in the 4-way case (65 per cent). As an additional verification, we apply our networks to a small sample of Galaxy Zoo images, obtaining accuracies of 92, 82, and 77 per cent for the binary, 3-way, and 4-way classifications, respectively.
Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models

This paper studies how to automatically generate a natural language text that describes the facts in knowledge graph (KG). Considering the few-shot setting, we leverage the excellent capacities of pretrained language models (PLMs) in language understanding and generation. We make three major technical contributions, namely representation alignment for bridging the semantic gap between KG encodings and PLMs, relation-biased KG linearization for deriving better input representations, and multi-task learning for learning the correspondence between KG and text. Extensive experiments on three benchmark datasets have demonstrated the effectiveness of our model on KG-to-text generation task. In particular, our model outperforms all comparison methods on both fully-supervised and few-shot settings. Our code and datasets are available at https://github.com/RUCAIBox/Few-Shot-KG2Text.
A Dataset and Baselines for Multilingual Reply Suggestion

Reply suggestion models help users process emails and chats faster. Previous work only studies English reply suggestion. Instead, we present MRS, a multilingual reply suggestion dataset with ten languages. MRS can be used to compare two families of models: 1) retrieval models that select the reply from a fixed set and 2) generation models that produce the reply from scratch. Therefore, MRS complements existing cross-lingual generalization benchmarks that focus on classification and sequence labeling tasks. We build a generation model and a retrieval model as baselines for MRS. The two models have different strengths in the monolingual setting, and they require different strategies to generalize across languages. MRS is publicly available at https://github.com/zhangmozhi/mrs.
Global Convergence of Multi-Agent Policy Gradient in Markov Potential Games

Potential games are arguably one of the most important and widely studied classes of normal form games. They define the archetypal setting of multi-agent coordination as all agent utilities are perfectly aligned with each other via a common potential function. Can this intuitive framework be transplanted in the setting of Markov Games? What are the similarities and differences between multi-agent coordination with and without state dependence? We present a novel definition of Markov Potential Games (MPG) that generalizes prior attempts at capturing complex stateful multi-agent coordination. Counter-intuitively, insights from normal-form potential games do not carry over as MPGs can consist of settings where state-games can be zero-sum games. In the opposite direction, Markov games where every state-game is a potential game are not necessarily MPGs. Nevertheless, MPGs showcase standard desirable properties such as the existence of deterministic Nash policies. In our main technical result, we prove fast convergence of independent policy gradient to Nash policies by adapting recent gradient dominance property arguments developed for single agent MDPs to multi-agent learning settings.
Discovery of a Giant Molecular Loop in the Central Region of NGC 253

NGC 253 is a starburst galaxy of SAB(s)c type with increasing interest because of its high activity at unrivaled closeness. Its energetic event is manifested as the vertical gas features in its central molecular zone, for which stellar feedback was proposed as the driving engine. In order to pursue details of the activity, we have undertaken a kinematic analysis of the ALMA archive data of 12CO(J = 3 − 2) emission at the highest resolution ∼3 pc. We revealed that one of the non-rotating gas components in the central molecular zone shows a loop-like structure of ∼200 pc radius. The loop-like structure is associated with a star cluster, whereas the cluster is not inside the loop-like structure and is not likely as the driver of the loop-like structure formation. Further, we find that the bar potential of NGC 253 seems to be too weak to drive the gas motion by the eccentric orbit. As an alternative, we frame a scenario that magnetic acceleration by the Parker instability is responsible for the creation of the loop-like structure. We show that the observed loop-like structure properties are similar to those in the Milky Way, and argue that recent magneto-hydrodynamics simulations lend support for the picture having the magnetic field strength of ≳100 μG. We suggest that cluster formation was triggered by the falling gas to the footpoint of the loop, which is consistent with a typical dynamical timescale of the loop ∼1 Myr.
Physics from Photons at the LHC

LHC collisions can act as a source of photons in the initial state. This mechanism plays an important role in the production of particles with electroweak couplings, and a precise account of photon-initiated (PI) production at the LHC is a key ingredient in the LHC precision physics programme. I will discuss the possibility of modelling PI processes directly via the structure function approach. This can provide percent level precision in the production cross sections, and is therefore well positioned to account for LHC precision requirements. This formalism in addition allows one to make use of another useful feature of photons, namely that they are colour-singlet and can often be emitted elastically (or quasi-elastically) from the proton. I will discuss recent work on applications of the structure function approach to precision calculations of PI production in the inclusive mode, and to `exclusive' processes with rapidity gaps, which can provide a unique probe of the Standard Model and physics beyond it.
Axial Anomaly in Galaxies and the Dark Universe

Motivated by the SU(2)CMB modification of the cosmological model ΛCDM, we consider isolated fuzzy-dark-matter lumps, made of ultralight axion particles whose masses arise due to distinct SU(2) Yang–Mills scales and the Planck mass MP. In contrast to SU(2)CMB, these Yang–Mills theories are in confining phases (zero temperature) throughout most of the Universe’s history and associate with the three lepton flavours of the Standard Model of particle physics. As the Universe expands, axionic fuzzy dark matter comprises a three-component fluid which undergoes certain depercolation transitions when dark energy (a global axion condensate) is converted into dark matter. We extract the lightest axion mass ma,e=0.675×10−23eV from well motivated model fits to observed rotation curves in low-surface-brightness galaxies (SPARC catalogue). Since the virial mass of an isolated lump solely depends on MP and the associated Yang–Mills scale the properties of an e-lump predict those of μ- and τ-lumps. As a result, a typical e-lump virial mass ∼6.3×1010M⊙ suggests that massive compact objects in galactic centers such as Sagittarius A* in the Milky Way are (merged) μ- and τ-lumps. In addition, τ-lumps may constitute globular clusters. SU(2)CMB is always thermalised, and its axion condensate never has depercolated. If the axial anomaly indeed would link leptons with dark matter and the CMB with dark energy then this would demystify the dark Universe through a firmly established feature of particle physics.