Miracle-dz/UniFAR-FTUs huggingface dataset:
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks

We introduce Syntax-Aware Fill-In-the-Middle (SAFIM), a new benchmark for evaluating Large Language Models (LLMs) on the code Fill-in-the-Middle (FIM) task. This benchmark focuses on syntax-aware completions of program structures such as code blocks and conditional expressions, and includes 17,720 examples from multiple programming languages, sourced from recent code submissions after April 2022 to minimize data contamination. SAFIM provides a robust framework with various prompt designs and novel syntax-aware post-processing techniques, facilitating accurate and fair comparisons across LLMs. Our comprehensive evaluation of 15 LLMs shows that FIM pretraining not only enhances FIM proficiency but also improves Left-to-Right (L2R) inference using LLMs. Our findings challenge conventional beliefs and suggest that pretraining methods and data quality have more impact than model size. SAFIM thus serves as a foundational platform for future research in effective pretraining strategies for code LLMs. The evaluation toolkit and dataset are available at https://github.com/gonglinyuan/safim, and the leaderboard is available at https://safimbenchmark.com.
Recovery Guarantees of Unsupervised Neural Networks for Inverse Problems trained with Gradient Descent

Advanced machine learning methods, and more prominently neural networks, have become standard to solve inverse problems over the last years. However, the theoretical recovery guarantees of such methods are still scarce and difficult to achieve. Only recently did unsupervised methods such as Deep Image Prior (DIP) get equipped with convergence and recovery guarantees for generic loss functions when trained through gradient flow with an appropriate initialization. In this paper, we extend these results by proving that these guarantees hold true when using gradient descent with an appropriately chosen step-size/learning rate. We also show that the discretization only affects the overparametrization bound for a two-layer DIP network by a constant and thus that the different guarantees found for the gradient flow will hold for gradient descent.
Non-additivity in many-body interactions between membrane-deforming spheres increases disorder

Membrane-induced interactions have been predicted to be important for the organization of membrane proteins. Measurements of the interactions between two and three membrane deforming objects have revealed their non-additive nature. They are thought to lead to complex many-body effects, however, experimental evidence is lacking to date. We here present an experimental method to measure many-body effects in membrane-mediated interactions using colloidal spheres placed between a deflated giant unilamellar vesicles and a planar substrate. The thus confined colloidal particles cause a large deformation of the membrane while not being physochemically attached to it and interact through it. Two particles are found to attract with a maximum force of 0.2~pN. For three particles, we observe a preference for forming compact equilateral triangles over a linear arrangement. We use numerical energy minimization to establish that the attraction stems from a reduction in the membrane-deformation energy caused by the particles. Confining up to 36 particles, we find a preference for hexagonally close packed clusters. However, with increasing number of particles the order of the confined particles decreases, while at the same time, diffusivity of the particles increases. Our experiments for the first time show that the non-additive nature of membrane-mediated interactions affects the interactions and arrangements and ultimately leads to spherical aggregates with liquid-like order of potential importance for cellular processes.
An Improved Algorithm for Learning Drifting Discrete Distributions

We present a new adaptive algorithm for learning discrete distributions under distribution drift. In this setting, we observe a sequence of independent samples from a discrete distribution that is changing over time, and the goal is to estimate the current distribution. Since we have access to only a single sample for each time step, a good estimation requires a careful choice of the number of past samples to use. To use more samples, we must resort to samples further in the past, and we incur a drift error due to the bias introduced by the change in distribution. On the other hand, if we use a small number of past samples, we incur a large statistical error as the estimation has a high variance. We present a novel adaptive algorithm that can solve this trade-off without any prior knowledge of the drift. Unlike previous adaptive results, our algorithm characterizes the statistical error using data-dependent bounds. This technicality enables us to overcome the limitations of the previous work that require a fixed finite support whose size is known in advance and that cannot change over time. Additionally, we can obtain tighter bounds depending on the complexity of the drifting distribution, and also consider distributions with infinite support.
Experimental set-up for thermal measurements at the nanoscale using an SThM probe with niobium nitride thermometer

Scanning Thermal Microscopy (SThM) has become an important measurement tool for characterizing the thermal properties of materials at the nanometer scale. This technique requires a SThM probe that combines an Atomic Force Microscopy (AFM) probe and a very sensitive resistive thermometry; the thermometer being located at the apex of the probe tip allows the mapping of temperature or thermal properties of nanostructured materials with very high spatial resolution. The high interest of the SThM technique in the field of thermal nanoscience currently suffers from a low temperature sensitivity despite its high spatial resolution. To address this challenge, we developed a high vacuum-based AFM system hosting a highly sensitive niobium nitride (NbN) SThM probe to demonstrate its unique performance. As a proof of concept, we utilized this custom-built system to carry out thermal measurements using the 3$\omega$ method. By measuring the $V_{3\omega}$ voltage on the NbN resistive thermometer in vacuum conditions we were able to determine the SThM probe's thermal conductance and thermal time constant. The performance of the probe is demonstrated by doing thermal measurements in-contact with a sapphire sample.
A Monte Carlo algorithm for spin foam intertwiners

Monte Carlo algorithms are barely considered in spin foam quantum gravity. Due to the quantum nature of spin foam amplitudes one cannot readily apply them, and the present sign problem is a threat to convergence and thus efficiency. Yet, ultimately the severity of the sign problem in spin foams is not known. In this article we propose a new probability distribution for coherent (boundary) intertwiners, which we use to define a Markov Chain Monte Carlo algorithm. We apply this algorithm to the SU(2) coherent vertex amplitude for various Regge-type boundary data and find convergent, accurate results at far lower costs than the explicit calculation. The resources are instead used to increase the size of boundary spins, bridging the gap to the asymptotic formulae. While the sign problem is not solved, it is under control in the vast majority of cases. We close by discussing how this algorithm can be extended to larger triangulations with boundary and 4d Lorentzian spin foam models. More speculatively, we discuss how this algorithm can be used to sample (Regge-geometric) bulk intertwiners.
TopicDiff: A Topic-enriched Diffusion Approach for Multimodal Conversational Emotion Detection

Multimodal Conversational Emotion (MCE) detection, generally spanning across the acoustic, vision and language modalities, has attracted increasing interest in the multimedia community. Previous studies predominantly focus on learning contextual information in conversations with only a few considering the topic information in single language modality, while always neglecting the acoustic and vision topic information. On this basis, we propose a model-agnostic Topic-enriched Diffusion (TopicDiff) approach for capturing multimodal topic information in MCE tasks. Particularly, we integrate the diffusion model into neural topic model to alleviate the diversity deficiency problem of neural topic model in capturing topic information. Detailed evaluations demonstrate the significant improvements of TopicDiff over the state-of-the-art MCE baselines, justifying the importance of multimodal topic information to MCE and the effectiveness of TopicDiff in capturing such information. Furthermore, we observe an interesting finding that the topic information in acoustic and vision is more discriminative and robust compared to the language.
Constraints on Metastable Dark Energy Decaying into Dark Matter

We revisit the proposal that an energy transfer from dark energy into dark matter can be described in field theory by a first order phase transition. We analyze the model proposed in Ref. Abdalla et al. (2013), using updated constraints on the decay time of a metastable dark energy from the work of Ref. Shafieloo et al. (2018). The results of our analysis show no prospects for potentially observable signals that could distinguish this scenario. We also show that such model would not drive a complete transition to a dark matter dominated phase even in a distant future. Nevertheless, the model is not excluded by the latest data and we confirm that the mass of the dark matter particle that would result from such a process corresponds to an axion-like particle, which is currently one of the best motivated dark matter candidates. We argue that extensions to this model, possibly with additional couplings, still deserve further attention as it could provide an interesting and viable description for an interacting dark sector scenario based in a single scalar field.
Bayesian Hierarchical Probabilistic Forecasting of Intraday Electricity Prices

We present a first study of Bayesian forecasting of electricity prices traded on the German continuous intraday market which fully incorporates parameter uncertainty. Our target variable is the IDFull price index, forecasts are given in terms of posterior predictive distributions. For validation we use the exceedingly volatile electricity prices of 2022, which have hardly been the subject of forecasting studies before. As a benchmark model, we use all available intraday transactions at the time of forecast creation to compute a current value for the IDFull. According to the weak-form efficiency hypothesis, it would not be possible to significantly improve this benchmark built from last price information. We do, however, observe statistically significant improvement in terms of both point measures and probability scores. Finally, we challenge the declared gold standard of using LASSO for feature selection in electricity price forecasting by presenting strong statistical evidence that Orthogonal Matching Pursuit (OMP) leads to better forecasting performance.
Landraces and Crop Genetic Improvement

Landraces are repository of gene pool that enrich biodiversity and maintain and stabilize ecosystem in a sustainable way to make it functional. Cultivation of traditional crops in different regions of the world, aside maintaining biodiversity in agriculture, also avails humanity of regulatory services such as nutrient cycling, carbon sequestration, control of soil erosion, reduction of greenhouse gas emission and control of hydrological processes. However, man through over-exploitation of some plant species with utter neglect to some other either deliberately or otherwise through modern agricultural systems that promote cultivation of a few high-input and high-yielding crop species caused disaffection to biodiversity with consequences of reduction in its regulatory services. In this chapter, different landraces of crops are examined, their usefulness in the maintenance of genetic diversity is explored, and implications of their depletion are discussed.
A preliminary study on new biodegradable films to cover silages.

Two different silage cover films, one made of standard polyethylene (PE) and the other made of Mater-Bi® biodegradable plastic (MB), with two different thicknesses (60 and 120 )m) have been compared. A whole maize crop was chopped, treated with a mixture of Lactobacillus plantarum, L. buchneri, and Enterococcus faecium, and ensiled in plastic bags with four replications for each treatment. After 55 and 110 d of conservation, all the silages were well fermented with no differences in fermentative and nutritional quality between the treatments, and few differences in mould count and aerobic stability after 110 d of conservation. These results showed the possibility of successfully developing a biodegradable cover for silage for 4 months after ensiling.
Not all tickets are equal and we know it: Guiding pruning with domain-specific knowledge

Neural structure learning is of paramount importance for scientific discovery and interpretability. Yet, contemporary pruning algorithms that focus on computational resource efficiency face algorithmic barriers to select a meaningful model that aligns with domain expertise. To mitigate this challenge, we propose DASH, which guides pruning by available domain-specific structural information. In the context of learning dynamic gene regulatory network models, we show that DASH combined with existing general knowledge on interaction partners provides data-specific insights aligned with biology. For this task, we show on synthetic data with ground truth information and two real world applications the effectiveness of DASH, which outperforms competing methods by a large margin and provides more meaningful biological insights. Our work shows that domain specific structural information bears the potential to improve model-derived scientific insights.
Chlorine and zinc co-doping effects on the electronic structure and optical properties of {\gamma}-CuI

The effects of chlorine (Cl) and zinc (Zn) co-doping on the electronic structure and optical properties of the zinc blende ({\gamma}) phase of copper iodide ({\gamma}-CuI) scintillator material are investigated by using first-principles density functional theory calculations. The band structure, density of states, dielectric function, absorption coefficients, and reflectivity were analyzed before and after doping. Results show co-doping significantly modifies the band structure, reduces the band gap, and generates impurity energy levels. Cl doping enhances absorption in the high energy region while reducing visible light absorption. Zn doping induces a redshift in absorption and n-type conductivity at high concentrations. With suitable co-doping ratios, the absorption coefficient and reflectivity of {\gamma}-CuI can be optimized in the visible range to improve scintillation light yield. The calculations provide guidance for co-doping {\gamma}-CuI scintillators to achieve superior detection performance. The n-type conductivity also makes doped {\gamma}-CuI promising for optoelectronic applications.
Will GPT-4 Run DOOM?

We show that GPT-4's reasoning and planning capabilities extend to the 1993 first-person shooter Doom. This large language model (LLM) is able to run and play the game with only a few instructions, plus a textual description--generated by the model itself from screenshots--about the state of the game being observed. We find that GPT-4 can play the game to a passable degree: it is able to manipulate doors, combat enemies, and perform pathing. More complex prompting strategies involving multiple model calls provide better results. While further work is required to enable the LLM to play the game as well as its classical, reinforcement learning-based counterparts, we note that GPT-4 required no training, leaning instead on its own reasoning and observational capabilities. We hope our work pushes the boundaries on intelligent, LLM-based agents in video games. We conclude by discussing the ethical implications of our work.
CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusion

Recent advancements in text-to-image generative systems have been largely driven by diffusion models. However, single-stage text-to-image diffusion models still face challenges, in terms of computational efficiency and the refinement of image details. To tackle the issue, we propose CogView3, an innovative cascaded framework that enhances the performance of text-to-image diffusion. CogView3 is the first model implementing relay diffusion in the realm of text-to-image generation, executing the task by first creating low-resolution images and subsequently applying relay-based super-resolution. This methodology not only results in competitive text-to-image outputs but also greatly reduces both training and inference costs. Our experimental results demonstrate that CogView3 outperforms SDXL, the current state-of-the-art open-source text-to-image diffusion model, by 77.0\% in human evaluations, all while requiring only about 1/2 of the inference time. The distilled variant of CogView3 achieves comparable performance while only utilizing 1/10 of the inference time by SDXL.
ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models

Large language models (LLMs) have achieved unprecedented performance in various applications, yet their evaluation remains a critical issue. Existing hallucination benchmarks are either static or lack adjustable complexity for thorough analysis. We contend that utilizing existing relational databases is a promising approach for constructing benchmarks due to their accurate knowledge description via functional dependencies. We propose ERBench to automatically convert any relational database into a benchmark based on the entity-relationship (ER) model. Our key idea is to construct questions using the database schema, records, and functional dependencies such that they can be automatically verified. In addition, we use foreign key constraints to join relations and construct multihop questions, which can be arbitrarily complex and used to debug the intermediate answers of LLMs. Finally, ERBench supports continuous evaluation, multimodal questions, and various prompt engineering techniques. In our experiments, we construct an LLM benchmark using databases of multiple domains and make an extensive comparison of contemporary LLMs. We observe that better LLMs like GPT-4 can handle a larger variety of question types, but are by no means perfect. Also, correct answers do not necessarily imply correct rationales, which is an important evaluation that ERBench does better than other benchmarks for various question types. Code is available at https: //github.com/DILAB-KAIST/ERBench.
A note on the singular value decomposition of idempotent and involutory matrices

It is known that singular values of idempotent matrices are either zero or larger or equal to one \cite{HouC63}. We state exactly how many singular values greater than one, equal to one, and equal to zero there are. Moreover, we derive a singular value decomposition of idempotent matrices which reveals a tight relationship between its left and right singular vectors. The same idea is used to augment a discovery regarding the singular values of involutory matrices as presented in \cite{FasH20}.
CommitBench: A Benchmark for Commit Message Generation

Writing commit messages is a tedious daily task for many software developers, and often remains neglected. Automating this task has the potential to save time while ensuring that messages are informative. A high-quality dataset and an objective benchmark are vital preconditions for solid research and evaluation towards this goal. We show that existing datasets exhibit various problems, such as the quality of the commit selection, small sample sizes, duplicates, privacy issues, and missing licenses for redistribution. This can lead to unusable models and skewed evaluations, where inferior models achieve higher evaluation scores due to biases in the data. We compile a new large-scale dataset, CommitBench, adopting best practices for dataset creation. We sample commits from diverse projects with licenses that permit redistribution and apply our filtering and dataset enhancements to improve the quality of generated commit messages. We use CommitBench to compare existing models and show that other approaches are outperformed by a Transformer model pretrained on source code. We hope to accelerate future research by publishing the source code( https://github.com/Maxscha/commitbench ).
Physics-based distinction of nonequilibrium effects in near-wall modeling of turbulent separation bubble with and without sweep

Pressure-gradient-induced separation of swept and unswept turbulent boundary layers, based on the DNS studies of Coleman et al. (J. Fluid Mech. 2018&2019), have been analyzed for various nonequilibrium effects. The goal is to isolate physical processes critical to near-wall flow modeling. The decomposition of skin friction into contributing physical terms, proposed by Renard and Deck (J. Fluid Mech. 2016) (short: RD decomposition), affords several key insights into the near-wall physics of these flows. In the unswept case, spatial growth term (encapsulating nonequilibrium effects) and TKE production appear to be the dominant contributing terms in the RD decomposition in the separated and pressure-gradient zones, but a closer inspection reveals that only the spatial growth term dominates in the inner layer close to the separation bubble, implying a strong need for incorporating nonequilibrium terms in the wall modeling of this case. The comparison of streamwise RD decomposition of swept and unswept cases shows that a larger accumulated Clauser-pressure-gradient parameter history in the latter energizes the outer dynamics in the APG, leading to diminished separation bubble size in the unswept case. The spanwise RD decomposition in the swept case indicates that the downstream spanwise flow largely retains the upstream ZPG characteristics. This seems to ease the near-wall modeling challenge in the separated region, especially for basic models with an inherent log-law assumption. Wall-modeled LES of the swept and unswept cases are then performed using three wall models, validating many of the modeling implications from the DNS. In particular, the extension of RD decomposition to wall models underpins the criticality of spatial growth term close to the separation bubble, and the corresponding superior predictions by the PDE wall model due to its accurate capturing of this term.
Sora as an AGI World Model? A Complete Survey on Text-to-Video Generation

Text-to-video generation marks a significant frontier in the rapidly evolving domain of generative AI, integrating advancements in text-to-image synthesis, video captioning, and text-guided editing. This survey critically examines the progression of text-to-video technologies, focusing on the shift from traditional generative models to the cutting-edge Sora model, highlighting developments in scalability and generalizability. Distinguishing our analysis from prior works, we offer an in-depth exploration of the technological frameworks and evolutionary pathways of these models. Additionally, we delve into practical applications and address ethical and technological challenges such as the inability to perform multiple entity handling, comprehend causal-effect learning, understand physical interaction, perceive object scaling and proportioning, and combat object hallucination which is also a long-standing problem in generative models. Our comprehensive discussion covers the topic of enablement of text-to-video generation models as human-assistive tools and world models, as well as eliciting model's shortcomings and summarizing future improvement direction that mainly centers around training datasets and evaluation metrics (both automatic and human-centered). Aimed at both newcomers and seasoned researchers, this survey seeks to catalyze further innovation and discussion in the growing field of text-to-video generation, paving the way for more reliable and practical generative artificial intelligence technologies.