Miracle-dz/UniFAR-FTUs huggingface dataset:
Table and Image Generation for Investigating Knowledge of Entities in Pre-trained Vision and Language Models

In this paper, we propose a table and image generation task to verify how the knowledge about entities acquired from natural language is retained in Vision & Language (V & L) models. This task consists of two parts: the first is to generate a table containing knowledge about an entity and its related image, and the second is to generate an image from an entity with a caption and a table containing related knowledge of the entity. In both tasks, the model must know the entities used to perform the generation properly. We created the Wikipedia Table and Image Generation (WikiTIG) dataset from about 200,000 infoboxes in English Wikipedia articles to perform the proposed tasks. We evaluated the performance on the tasks with respect to the above research question using the V & L model OFA, which has achieved state-of-the-art results in multiple tasks. Experimental results show that OFA forgets part of its entity knowledge by pre-training as a complement to improve the performance of image related tasks.
On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research

This evidence-based position paper critiques current research practices within the language model pre-training literature. Despite rapid recent progress afforded by increasingly better pre-trained language models (PLMs), current PLM research practices often conflate different possible sources of model improvement, without conducting proper ablation studies and principled comparisons between different models under comparable conditions. These practices (i) leave us ill-equipped to understand which pre-training approaches should be used under what circumstances; (ii) impede reproducibility and credit assignment; and (iii) render it difficult to understand: “How exactly does each factor contribute to the progress that we have today? ” We provide a case in point by revisiting the success of BERT over its baselines, ELMo and GPT-1, and demonstrate how — under comparable conditions where the baselines are tuned to a similar extent — these baselines (and even-simpler variants thereof) can, in fact, achieve competitive or better performance than BERT. These findings demonstrate how disentangling different factors of model improvements can lead to valuable new insights. We conclude with recommendations for how to encourage and incentivize this line of work, and accelerate progress towards a better and more systematic understanding of what factors drive the progress of our foundation models today.
MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Poses

We propose a generalizable neural radiance fields - MonoNeRF, that can be trained on large-scale monocular videos of moving in static scenes without any ground-truth annotations of depth and camera poses. MonoNeRF follows an Autoencoder-based architecture, where the encoder estimates the monocular depth and the camera pose, and the decoder constructs a Multiplane NeRF representation based on the depth encoder feature, and renders the input frames with the estimated camera. The learning is supervised by the reconstruction error. Once the model is learned, it can be applied to multiple applications including depth estimation, camera pose estimation, and single-image novel view synthesis. More qualitative results are available at: https://oasisyang.github. io/mononerf .
Training Data Attribution for Diffusion Models

Diffusion models have become increasingly popular for synthesizing high-quality samples based on training datasets. However, given the oftentimes enormous sizes of the training datasets, it is difficult to assess how training data impact the samples produced by a trained diffusion model. The difficulty of relating diffusion model inputs and outputs poses significant challenges to model explainability and training data attribution. Here we propose a novel solution that reveals how training data influence the output of diffusion models through the use of ensembles. In our approach individual models in an encoded ensemble are trained on carefully engineered splits of the overall training data to permit the identification of influential training examples. The resulting model ensembles enable efficient ablation of training data influence, allowing us to assess the impact of training data on model outputs. We demonstrate the viability of these ensembles as generative models and the validity of our approach to assessing influence.
Emergence of near-TAP free energy functional in the SK model at high temperature

We study the SK model at inverse temperature $\beta>0$ and strictly positive field $h>0$ in the region of $(\beta,h)$ where the replica-symmetric formula is valid. An integral representation of the partition function derived from the Hubbard-Stratonovitch transformation combined with a duality formula is used to prove that the infinite volume free energy of the SK model can be expressed as a variational formula on the space of magnetisations, $m$. The resulting free energy functional differs from that of Thouless, Anderson and Palmer (TAP) by the term $-\frac{\beta^2}{4}\left(q-q_{\text{EA}}(m)\right)^2$ where $q_{\text{EA}}(m)$ is the Edwards-Anderson parameter and $q$ is the minimiser of the replica-symmetric formula. Thus, both functionals have the same critical points and take the same value on the subspace of magnetisations satisfying $q_{\text{EA}}(m)=q$. This result is based on an in-depth study of the global maximum of this near-TAP free energy functional using Bolthausen's solutions of the TAP equations, Bandeira&van Handel's bounds on the spectral norm of non-homogeneous Wigner-type random matrices, and Gaussian comparison techniques. It holds for $(\beta,h)$ in a large subregion of the de Almeida and Thouless high-temperature stability region.
The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation

Denoising diffusion probabilistic models have transformed image generation with their impressive fidelity and diversity. We show that they also excel in estimating optical flow and monocular depth, surprisingly, without task-specific architectures and loss functions that are predominant for these tasks. Compared to the point estimates of conventional regression-based methods, diffusion models also enable Monte Carlo inference, e.g., capturing uncertainty and ambiguity in flow and depth. With self-supervised pre-training, the combined use of synthetic and real data for supervised training, and technical innovations (infilling and step-unrolled denoising diffusion training) to handle noisy-incomplete training data, and a simple form of coarse-to-fine refinement, one can train state-of-the-art diffusion models for depth and optical flow estimation. Extensive experiments focus on quantitative performance against benchmarks, ablations, and the model's ability to capture uncertainty and multimodality, and impute missing values. Our model, DDVM (Denoising Diffusion Vision Model), obtains a state-of-the-art relative depth error of 0.074 on the indoor NYU benchmark and an Fl-all outlier rate of 3.26\% on the KITTI optical flow benchmark, about 25\% better than the best published method. For an overview see https://diffusion-vision.github. io.
Improved Stability and Generalization Guarantees of the Decentralized SGD Algorithm

This paper presents a new generalization error analysis for Decentralized Stochastic Gradient Descent (D-SGD) based on algorithmic stability. The obtained results overhaul a series of recent works that suggested an increased instability due to decentralization and a detrimental impact of poorly-connected communication graphs on generalization. On the contrary, we show, for convex, strongly convex and non-convex functions, that D-SGD can always recover generalization bounds analogous to those of classical SGD, suggesting that the choice of graph does not matter. We then argue that this result is coming from a worst-case analysis, and we provide a refined data-dependent generalization bound for general convex functions. This new bound reveals that the choice of graph can in fact improve the worst-case bound in certain regimes, and that surprisingly, a poorly-connected graph can even be beneficial.
Spectral and Jet Properties of the Quasi-thermal-dominated GRB 210121A, GRB 210610B, and GRB 221022B

Some quasi-thermal (QT)-dominated gamma-ray bursts (GRBs) could be well described by a multicolor blackbody (mBB) function or a combined model of a BB plus a nonthermal (NT) component. In this analysis, two QT radiation-dominated bursts with known emission properties (GRB 210610B, likely from a hybrid jet, and GRB 210121A, with a spectrum consistent with nondissipative photospheric emission from a pure hot fireball) are used to make a comparison between these two models. To diagnose the magnetization properties of the central engine, the “top-down” approach proposed by Gao and Zhang is adopted. It is found that diagnoses based on these two models provide similar conclusions qualitatively; however, the model with mBB (or mBB+NT) may give more reasonable physical explanations. This implies that impacts from the GRB jet structure and geometrical broadening on the observed spectrum should be considered. However, conservatively, these methods may be not sensitive enough to distinguish between a pure hot fireball and a mildly magnetized hybrid jet. Some other information is necessary to provide more evidence when determining the jet properties for similar GRBs. Based on these considerations, we suggest that the photospheric emission of GRB 221022B is from a hot jet, where dissipation is caused by an internal shock mechanism due to the increasing Lorentz factor with time, which makes its prompt emission display typical evolution from thermal to NT.
EnfoMax: Domain Entropy and Mutual Information Maximization for Domain Generalized Face Anti-spoofing

The face anti-spoofing (FAS) method performs well under intra-domain setups. However, its cross-domain performance is unsatisfactory. As a result, the domain generalization (DG) method has gained more attention in FAS. Existing methods treat FAS as a simple binary classification task and propose a heuristic training objective to learn domain-invariant features. However, there is no theoretical explanation of what a domain-invariant feature is. Additionally, the lack of theoretical support makes domain generalization techniques such as adversarial training lack training stability. To address these issues, this paper proposes the EnfoMax framework, which uses information theory to analyze cross-domain FAS tasks. This framework provides theoretical guarantees and optimization objectives for domain-generalized FAS tasks. EnfoMax maximizes the domain entropy and mutual information of live samples in source domains without using adversarial learning. Experimental results demonstrate that our approach performs well on extensive public datasets and outperforms state-of-the-art methods.
Diverse origins for non-repeating fast radio bursts: Rotational radio transient sources and cosmological compact binary merger remnants

A large number of fast radio bursts (FRBs) detected with the CHIME telescope have enabled investigations of their energy distributions in different redshift intervals, incorporating the consideration of the selection effects of CHIME. As a result, we obtained a non-evolving energy function (EF) for the high-energy FRBs (HEFRBs) of energies E ≳ 2 × 1038 erg, which takes the form of a power law with a low-energy exponential cutoff. On the contrary, the energy distribution of the low-energy FRBs (LEFRBs) obviously cannot be described by the same EF. Including the lowest dispersion measure (DM) samples, the LEFRBs are concentrated towards the Galactic plane and their latitude distribution is similar to that of Galactic rotational radio transients (RRATs). These indications hint that LEFRBs might compose a special type of RRATs, with relatively higher DMs and energies (i.e., ∼1028 − 31 erg for a reference distance of ∼10 kpc if they belong to the Milky Way). Finally, we revisit the redshift-dependent event rate of HEFRBs and confirm that they could be produced by the remnants of cosmological compact binary mergers.
Fair and Optimal Classification via Post-Processing

To mitigate the bias exhibited by machine learning models, fairness criteria can be integrated into the training process to ensure fair treatment across all demographics, but it often comes at the expense of model performance. Understanding such tradeoffs, therefore, underlies the design of fair algorithms. To this end, this paper provides a complete characterization of the inherent tradeoff of demographic parity on classification problems, under the most general multi-group, multi-class, and noisy setting. Specifically, we show that the minimum error rate achievable by randomized and attribute-aware fair classifiers is given by the optimal value of a Wasserstein-barycenter problem. On the practical side, our findings lead to a simple post-processing algorithm that derives fair classifiers from score functions, which yields the optimal fair classifier when the score is Bayes optimal. We provide suboptimality analysis and sample complexity for our algorithm, and demonstrate its effectiveness on benchmark datasets.
Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicity

Modern predictive models are often deployed to environments in which computational budgets are dynamic. Anytime algorithms are well-suited to such environments as, at any point during computation, they can output a prediction whose quality is a function of computation time. Early-exit neural networks have garnered attention in the context of anytime computation due to their capability to provide intermediate predictions at various stages throughout the network. However, we demonstrate that current early-exit networks are not directly applicable to anytime settings, as the quality of predictions for individual data points is not guaranteed to improve with longer computation. To address this shortcoming, we propose an elegant post-hoc modification, based on the Product-of-Experts, that encourages an early-exit network to become gradually confident. This gives our deep models the property of conditional monotonicity in the prediction quality -- an essential stepping stone towards truly anytime predictive modeling using early-exit architectures. Our empirical results on standard image-classification tasks demonstrate that such behaviors can be achieved while preserving competitive accuracy on average.
ATEM: A Topic Evolution Model for the Detection of Emerging Topics in Scientific Archives

This paper presents ATEM, a novel framework for studying topic evolution in scientific archives. ATEM is based on dynamic topic modeling and dynamic graph embedding techniques that explore the dynamics of content and citations of documents within a scientific corpus. ATEM explores a new notion of contextual emergence for the discovery of emerging interdisciplinary research topics based on the dynamics of citation links in topic clusters. Our experiments show that ATEM can efficiently detect emerging cross-disciplinary topics within the DBLP archive of over five million computer science articles.
DiffPack: A Torsional Diffusion Model for Autoregressive Protein Side-Chain Packing

Proteins play a critical role in carrying out biological functions, and their 3D structures are essential in determining their functions. Accurately predicting the conformation of protein side-chains given their backbones is important for applications in protein structure prediction, design and protein-protein interactions. Traditional methods are computationally intensive and have limited accuracy, while existing machine learning methods treat the problem as a regression task and overlook the restrictions imposed by the constant covalent bond lengths and angles. In this work, we present DiffPack, a torsional diffusion model that learns the joint distribution of side-chain torsional angles, the only degrees of freedom in side-chain packing, by diffusing and denoising on the torsional space. To avoid issues arising from simultaneous perturbation of all four torsional angles, we propose autoregressively generating the four torsional angles from $\chi_1$ to $\chi_4$ and training diffusion models for each torsional angle. We evaluate the method on several benchmarks for protein side-chain packing and show that our method achieves improvements of $11.9\%$ and $13.5\%$ in angle accuracy on CASP13 and CASP14, respectively, with a significantly smaller model size ($60\times$ fewer parameters). Additionally, we show the effectiveness of our method in enhancing side-chain predictions in the AlphaFold2 model. Code is available at https://github.com/DeepGraphLearning/DiffPack.
The alpha and helion particle charge radius difference from spectroscopy of quantum-degenerate helium

Accurate spectroscopic measurements of calculable systems provide a powerful method for testing the Standard Model and extracting fundamental constants. Recently, spectroscopic measurements of finite nuclear size effects in normal and muonic hydrogen resulted in unexpectedly large adjustments of the proton charge radius and the Rydberg constant. We measured the $2^3\mathrm{S}\rightarrow2^1\mathrm{S}$ transition frequency in a Fermi gas of $^3$He with an order of magnitude higher accuracy than before. Together with a previous measurement in a $^4$He Bose-Einstein condensate, a squared charge radius difference $r^2_h - r^2_{\alpha} = 1.0757(15)\ \mathrm{fm^2}$ is determined between the helion and alpha particle. This measurement provides a benchmark with unprecedented accuracy for nuclear structure calculations. A deviation of 3.6$\sigma$ is found with a determination (arXiv:2305.11679) based on spectroscopy of muonic helium ions.
A Data-driven Region Generation Framework for Spatiotemporal Transportation Service Management

MAUP (modifiable areal unit problem) is a fundamental problem for spatial data management and analysis. As an instantiation of MAUP in online transportation platforms, region generation (i.e., specifying the areal unit for service operations) is the first and vital step for supporting spatiotemporal transportation services such as ride-sharing and freight transport. Most existing region generation methods are manually specified (e.g., fixed-size grids), suffering from poor spatial semantic meaning and inflexibility to meet service operation requirements. In this paper, we propose RegionGen, a data-driven region generation framework that can specify regions with key characteristics (e.g., good spatial semantic meaning and predictability) by modeling region generation as a multi-objective optimization problem. First, to obtain good spatial semantic meaning, RegionGen segments the whole city into atomic spatial elements based on road networks and obstacles (e.g., rivers). Then, it clusters the atomic spatial elements into regions by maximizing various operation characteristics, which is formulated as a multi-objective optimization problem. For this optimization problem, we propose a multi-objective co-optimization algorithm. Extensive experiments verify that RegionGen can generate more suitable regions than traditional methods for spatiotemporal service management.
Benchmarking Large Language Models on CMExam - A Comprehensive Chinese Medical Exam Dataset

Recent advancements in large language models (LLMs) have transformed the field of question answering (QA). However, evaluating LLMs in the medical field is challenging due to the lack of standardized and comprehensive datasets. To address this gap, we introduce CMExam, sourced from the Chinese National Medical Licensing Examination. CMExam consists of 60K+ multiple-choice questions for standardized and objective evaluations, as well as solution explanations for model reasoning evaluation in an open-ended manner. For in-depth analyses of LLMs, we invited medical professionals to label five additional question-wise annotations, including disease groups, clinical departments, medical disciplines, areas of competency, and question difficulty levels. Alongside the dataset, we further conducted thorough experiments with representative LLMs and QA algorithms on CMExam. The results show that GPT-4 had the best accuracy of 61.6% and a weighted F1 score of 0.617. These results highlight a great disparity when compared to human accuracy, which stood at 71.6%. For explanation tasks, while LLMs could generate relevant reasoning and demonstrate improved performance after finetuning, they fall short of a desired standard, indicating ample room for improvement. To the best of our knowledge, CMExam is the first Chinese medical exam dataset to provide comprehensive medical annotations. The experiments and findings of LLM evaluation also provide valuable insights into the challenges and potential solutions in developing Chinese medical QA systems and LLM evaluation pipelines. The dataset and relevant code are available at https://github.com/williamliujl/CMExam.
EffCRN: An Efficient Convolutional Recurrent Network for High-Performance Speech Enhancement

Fully convolutional recurrent neural networks (FCRNs) have shown state-of-the-art performance in single-channel speech enhancement. However, the number of parameters and the FLOPs/second of the original FCRN are restrictively high. A further important class of efficient networks is the CRUSE topology, serving as reference in our work. By applying a number of topological changes at once, we propose both an efficient FCRN (FCRN15), and a new family of efficient convolutional recurrent neural networks (EffCRN23, EffCRN23lite). We show that our FCRN15 (875K parameters) and EffCRN23lite (396K) outperform the already efficient CRUSE5 (85M) and CRUSE4 (7.2M) networks, respectively, w.r. t. PESQ, DNSMOS and DeltaSNR, while requiring about 94% less parameters and about 20% less #FLOPs/frame. Thereby, according to these metrics, the FCRN/EffCRN class of networks provides new best-in-class network topologies for speech enhancement.
DVFO: Dynamic Voltage, Frequency Scaling and Workload Offloading for DNN Edge Inference

Due to edge device resource constraints and different characteristics of deep neural network (DNN) models, it is a big challenge to optimize DNN inference performance in terms of energy consumption and inference latency on edge devices. In addition to the dynamic voltage frequency scaling (DVFS) technique, the edge-cloud architecture provides a collaborative approach to efficient DNN inference. However, current edge-cloud collaborative inference methods have not optimized various compute resources on edge devices. Thus, we propose DVFO, a novel DVFS-enabled edge-cloud collaborative inference framework, which jointly optimize DVFS and offloading parameters via deep reinforcement learning (DRL). Specifically, DVFO automatically co-optimizes 1) CPU, GPU and memory frequencies of edge devices, and 2) feature maps to be offloaded to cloud servers. In addition, it leverages a thinking-while-moving concurrent mechanism to accelerate the DRL learning process, and a spatialchannel attention mechanism to extract DNN feature maps of secondary importance for workload offloading. This approach improves energy efficiency and inference latency for different DNN models under various edge-cloud network conditions. Experimental results on different datasets show that DVFO reduces the average energy consumption by 33% compared to state-ofthe-art schemes. Moreover, DVFO achieves up to 54% end-to-end inference latency reduction.
Counting Pollicott--Ruelle resonances for Axiom A flows

In this paper, we count the number of Pollicott--Ruelle resonances for open hyperbolic systems and Axiom A flows. In particular, we prove polynomial upper bounds and sublinear lower bounds on the number of resonances with modulus less than $r$ in strips for open hyperbolic systems and Axiom A flows with a transversality condition.