Miracle-dz/UniFAR-FTUs huggingface dataset:
On the Parameterization of Second-Order Optimization Effective Towards the Infinite Width

Second-order optimization has been developed to accelerate the training of deep neural networks and it is being applied to increasingly larger-scale models. In this study, towards training on further larger scales, we identify a specific parameterization for second-order optimization that promotes feature learning in a stable manner even if the network width increases significantly. Inspired by a maximal update parameterization, we consider a one-step update of the gradient and reveal the appropriate scales of hyperparameters including random initialization, learning rates, and damping terms. Our approach covers two major second-order optimization algorithms, K-FAC and Shampoo, and we demonstrate that our parameterization achieves higher generalization performance in feature learning. In particular, it enables us to transfer the hyperparameters across models with different widths.
Predicting Line-Level Defects by Capturing Code Contexts with Hierarchical Transformers

Software defects consume 40% of the total budget in software development and cost the global economy billions of dollars every year. Unfortunately, despite the use of many software quality assurance (SQA) practices in software development (e.g., code review, continuous integration), defects may still exist in the official release of a software product. Therefore, prioritizing SQA efforts for the vulnerable areas of the codebase is essential to ensure the high quality of a software release. Predicting software defects at the line level could help prioritize the SQA effort but is a highly challenging task given that only ~3% of lines of a codebase could be defective. Existing works on line-level defect prediction often fall short and cannot fully leverage the line-level defect information. In this paper, we propose Bugsplorer, a novel deep-learning technique for line-level defect prediction. It leverages a hierarchical structure of transformer models to represent two types of code elements: code tokens and code lines. Unlike the existing techniques that are optimized for file-level defect prediction, Bugsplorer is optimized for a line-level defect prediction objective. Our evaluation with five performance metrics shows that Bugsplorer has a promising capability of predicting defective lines with 26-72% better accuracy than that of the state-of-the-art technique. It can rank the first 20% defective lines within the top 1-3% suspicious lines. Thus, Bugsplorer has the potential to significantly reduce SQA costs by ranking defective lines higher.
Synergistic Anchored Contrastive Pre-training for Few-Shot Relation Extraction

Few-shot Relation Extraction (FSRE) aims to extract relational facts from a sparse set of labeled corpora. Recent studies have shown promising results in FSRE by employing Pre-trained Language Models (PLMs) within the framework of supervised contrastive learning, which considers both instances and label facts. However, how to effectively harness massive instance-label pairs to encompass the learned representation with semantic richness in this learning paradigm is not fully explored. To address this gap, we introduce a novel synergistic anchored contrastive pre-training framework. This framework is motivated by the insight that the diverse viewpoints conveyed through instance-label pairs capture incomplete yet complementary intrinsic textual semantics. Specifically, our framework involves a symmetrical contrastive objective that encompasses both sentence-anchored and label-anchored contrastive losses. By combining these two losses, the model establishes a robust and uniform representation space. This space effectively captures the reciprocal alignment of feature distributions among instances and relational facts, simultaneously enhancing the maximization of mutual information across diverse perspectives within the same relation. Experimental results demonstrate that our framework achieves significant performance enhancements compared to baseline models in downstream FSRE tasks. Furthermore, our approach exhibits superior adaptability to handle the challenges of domain shift and zero-shot relation extraction. Our code is available online at https://github.com/AONE-NLP/FSRE-SaCon.
Two-Channel Extended Kalman Filtering with Intermittent Measurements

We consider two nonlinear state estimation problems in a setting where an extended Kalman filter receives measurements from two sets of sensors via two channels (2C). In the stochastic-2C problem, the channels drop measurements stochastically, whereas in 2C scheduling, the estimator chooses when to read each channel. In the first problem, we generalize linear-case 2C analysis to obtain -- for a given pair of channel arrival rates -- boundedness conditions for the trace of the error covariance, as well as a worst-case upper bound. For scheduling, an optimization problem is solved to find arrival rates that balance low channel usage with low trace bounds, and channels are read deterministically with the expected periods corresponding to these arrival rates. We validate both solutions in simulations for linear and nonlinear dynamics; as well as in a real experiment with an underwater robot whose position is being intermittently found in a UAV camera image.
Dynamically selected steady states and criticality in non-reciprocal networks

Diverse equilibrium systems with heterogeneous interactions lie at the edge of stability. Such marginally stable states are dynamically selected as the most abundant ones or as those with the largest basins of attraction. On the other hand, systems with non-reciprocal (or asymmetric) interactions are inherently out of equilibrium, and exhibit a rich variety of steady states, including fixed points, limit cycles and chaotic trajectories. How are steady states dynamically selected away from equilibrium? We address this question in a simple neural network model, with a tunable level of non-reciprocity. Our study reveals different types of ordered phases and it shows how non-equilibrium steady states are selected in each phase. In the spin-glass region, the system exhibits marginally stable behaviour for reciprocal (or symmetric) interactions and it smoothly transitions to chaotic dynamics, as the non-reciprocity (or asymmetry) in the couplings increases. Such region, on the other hand, shrinks and eventually disappears when couplings become anti-symmetric. Our results are relevant to advance the knowledge of disordered systems beyond the paradigm of reciprocal couplings, and to develop an interface between statistical physics of equilibrium spin-glasses and dynamical systems theory.
Labrador: Exploring the Limits of Masked Language Modeling for Laboratory Data

In this work we introduce Labrador, a pre-trained Transformer model for laboratory data. Labrador and BERT were pre-trained on a corpus of 100 million lab test results from electronic health records (EHRs) and evaluated on various downstream outcome prediction tasks. Both models demonstrate mastery of the pre-training task but neither consistently outperform XGBoost on downstream supervised tasks. Our ablation studies reveal that transfer learning shows limited effectiveness for BERT and achieves marginal success with Labrador. We explore the reasons for the failure of transfer learning and suggest that the data generating process underlying each patient cannot be characterized sufficiently using labs alone, among other factors. We encourage future work to focus on joint modeling of multiple EHR data categories and to include tree-based baselines in their evaluations.
Tracking Any Object Amodally

Amodal perception, the ability to comprehend complete object structures from partial visibility, is a fundamental skill, even for infants. Its significance extends to applications like autonomous driving, where a clear understanding of heavily occluded objects is essential. However, modern detection and tracking algorithms often overlook this critical capability, perhaps due to the prevalence of modal annotations in most datasets. To address the scarcity of amodal data, we introduce the TAO-Amodal benchmark, featuring 880 diverse categories in thousands of video sequences. Our dataset includes amodal and modal bounding boxes for visible and occluded objects, including objects that are partially out-of-frame. To enhance amodal tracking with object permanence, we leverage a lightweight plug-in module, the amodal expander, to transform standard, modal trackers into amodal ones through fine-tuning on a few hundred video sequences with data augmentation. We achieve a 3.3\% and 1.6\% improvement on the detection and tracking of occluded objects on TAO-Amodal. When evaluated on people, our method produces dramatic improvements of 2x compared to state-of-the-art modal baselines.
Equilibrium Computation in Multi-Stage Auctions and Contests

We compute equilibrium strategies in multi-stage games with continuous signal and action spaces as they are widely used in the management sciences and economics. Examples include sequential sales via auctions, multi-stage elimination contests, and Stackelberg competitions. In sequential auctions, analysts are required to derive not just single bids but bid functions for all possible signals or values that a bidder might have in multiple stages. Due to the continuity of the signal and action spaces, these bid functions come from an infinite dimensional space. While such models are fundamental to game theory and its applications, equilibrium strategies are rarely known. The resulting system of non-linear differential equations is considered intractable for all but elementary models. This has been limiting progress in game theory and is a barrier to its adoption in the field. We show that Deep Reinforcement Learning and self-play can learn equilibrium bidding strategies for various multi-stage games without making parametric assumptions on the bid function. We find equilibrium in models that have not yet been explored analytically and new asymmetric equilibrium bid functions for established models of sequential auctions. The verification of equilibrium is challenging in such games due to the continuous signal and action spaces. We introduce a verification algorithm and prove that the error of this verifier decreases when considering Lipschitz continuous strategies with increasing levels of discretization and sample sizes.
CAManim: Animating end-to-end network activation maps

Deep neural networks have been widely adopted in numerous domains due to their high performance and accessibility to developers and application-specific end-users. Fundamental to image-based applications is the development of Convolutional Neural Networks (CNNs), which possess the ability to automatically extract features from data. However, comprehending these complex models and their learned representations, which typically comprise millions of parameters and numerous layers, remains a challenge for both developers and end-users. This challenge arises due to the absence of interpretable and transparent tools to make sense of black-box models. There exists a growing body of Explainable Artificial Intelligence (XAI) literature, including a collection of methods denoted Class Activation Maps (CAMs), that seek to demystify what representations the model learns from the data, how it informs a given prediction, and why it, at times, performs poorly in certain tasks. We propose a novel XAI visualization method denoted CAManim that seeks to simultaneously broaden and focus end-user understanding of CNN predictions by animating the CAM-based network activation maps through all layers, effectively depicting from end-to-end how a model progressively arrives at the final layer activation. Herein, we demonstrate that CAManim works with any CAM-based method and various CNN architectures. Beyond qualitative model assessments, we additionally propose a novel quantitative assessment that expands upon the Remove and Debias (ROAD) metric, pairing the qualitative end-to-end network visual explanations assessment with our novel quantitative"yellow brick ROAD"assessment (ybROAD). This builds upon prior research to address the increasing demand for interpretable, robust, and transparent model assessment methodology, ultimately improving an end-user's trust in a given model's predictions.
Adaptive Smooth Activation for Improved Disease Diagnosis and Organ Segmentation from Radiology Scans

In this study, we propose a new activation function, called Adaptive Smooth Activation Unit (ASAU), tailored for optimized gradient propagation, thereby enhancing the proficiency of convolutional networks in medical image analysis. We apply this new activation function to two important and commonly used general tasks in medical image analysis: automatic disease diagnosis and organ segmentation in CT and MRI. Our rigorous evaluation on the RadImageNet abdominal/pelvis (CT and MRI) dataset and Liver Tumor Segmentation Benchmark (LiTS) 2017 demonstrates that our ASAU-integrated frameworks not only achieve a substantial (4.80\%) improvement over ReLU in classification accuracy (disease detection) on abdominal CT and MRI but also achieves 1\%-3\% improvement in dice coefficient compared to widely used activations for `healthy liver tissue' segmentation. These improvements offer new baselines for developing a diagnostic tool, particularly for complex, challenging pathologies. The superior performance and adaptability of ASAU highlight its potential for integration into a wide range of image classification and segmentation tasks.
Intrinsic Image Diffusion for Indoor Single-view Material Estimation

We present Intrinsic Image Diffusion, a generative model for appearance decomposition of indoor scenes. Given a single input view, we sample multiple possible material explanations represented as albedo, roughness, and metallic maps. Appearance decomposition poses a considerable challenge in computer vision due to the inherent ambiguity between lighting and material properties and the lack of real datasets. To address this issue, we advocate for a probabilistic formulation, where instead of attempting to directly predict the true material properties, we employ a conditional generative model to sample from the solution space. Furthermore, we show that utilizing the strong learned prior of recent diffusion models trained on large-scale real-world images can be adapted to material estimation and highly improves the generalization to real images. Our method produces significantly sharper, more consistent, and more detailed materials, outperforming state-of-the-art methods by $1.5dB$ on PSNR and by $45\%$ better FID score on albedo prediction. We demonstrate the effectiveness of our approach through experiments on both synthetic and real-world datasets.
CUDC: A Curiosity-Driven Unsupervised Data Collection Method with Adaptive Temporal Distances for Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to learn an effective policy from a pre-collected dataset. Most existing works are to develop sophisticated learning algorithms, with less emphasis on improving the data collection process. Moreover, it is even challenging to extend the single-task setting and collect a task-agnostic dataset that allows an agent to perform multiple downstream tasks. In this paper, we propose a Curiosity-driven Unsupervised Data Collection (CUDC) method to expand feature space using adaptive temporal distances for task-agnostic data collection and ultimately improve learning efficiency and capabilities for multi-task offline RL. To achieve this, CUDC estimates the probability of the k-step future states being reachable from the current states, and adapts how many steps into the future that the dynamics model should predict. With this adaptive reachability mechanism in place, the feature representation can be diversified, and the agent can navigate itself to collect higher-quality data with curiosity. Empirically, CUDC surpasses existing unsupervised methods in efficiency and learning performance in various downstream offline RL tasks of the DeepMind control suite.
An Alternate View on Optimal Filtering in an RKHS

Kernel Adaptive Filtering (KAF) are mathematically principled methods which search for a function in a Reproducing Kernel Hilbert Space. While they work well for tasks such as time series prediction and system identification they are plagued by a linear relationship between number of training samples and model size, hampering their use on the very large data sets common in today's data saturated world. Previous methods try to solve this issue by sparsification. We describe a novel view of optimal filtering which may provide a route towards solutions in a RKHS which do not necessarily have this linear growth in model size. We do this by defining a RKHS in which the time structure of a stochastic process is still present. Using correntropy [11], an extension of the idea of a covariance function, we create a time based functional which describes some potentially nonlinear desired mapping function. This form of a solution may provide a fruitful line of research for creating more efficient representations of functionals in a RKHS, while theoretically providing computational complexity in the test set similar to Wiener solution.
Computational insights into phase equilibria between wide-gap semiconductors and contact materials

Novel wide-band-gap semiconductors are needed for next-generation power electronic but there is a gap between a promising material and a functional device. Finding stable contacts is one of the major challenges, which is currently dealt with mainly via trial and error. Herein, we computationally investigate the thermochemistry and phase co-existence at the junction between three wide gap semiconductors, $\beta$-Ga$_{2}$O$_{3}$, GeO$_2$, and GaN, and possible contact materials. The pool of possible contacts includes 47 elemental metals and 4 common $n$-type transparent conducting oxides (ZnO, TiO$_2$, SnO$_2$, and In$_2$O$_3$). We use first-principles thermodynamics to model the Gibbs free energies of chemical reactions as a function of the gas pressure (p$_{\mathrm{O}_2}$/p$_{\mathrm{N}_2}$) and equilibrium temperature. We deduce whether a semiconductor/contact interface will be stable at relevant conditions, possibly influencing the long-term reliability and performance of devices. We generally find that most elemental metals tend to oxidize or nitridize and form various interface oxide/nitride layers. Exceptions include select late- and post-transition metals, and in case of GaN also the alkali metals, which are predicted to exhibit stable coexistence, although in many cases at relatively low gas partial pressures. Similar is true for the transparent conducting oxides, for which in most cases we predict a preference toward forming ternary oxides when in contact with $\beta$-Ga$_{2}$O$_{3}$ and GeO$_{2}$. The only exception is SnO$_2$, which can form stable contacts with both oxides. Finally, we show how the same approach can be used to predict gas partial pressure vs. temperature phase diagrams to help direct synthesis of ternary compounds. We believe these results provide a valuable guidance in selecting contact materials to wide-gap semiconductors and suitable growth conditions.
Kilonova-Targeting Lightcurve Classification for Wide Field Survey Telescope

With the enhancement of the sensitivity of gravitational wave (GW) detectors and capabilities of large survey facilities, such as the Vera Rubin Observatory Legacy Survey of Space and Time (LSST) and the 2.5 m Wide Field Survey Telescope (WFST), we now have the potential to detect an increasing number of distant kilonova (KN). However, distinguishing KN from the plethora of detected transients in ongoing and future follow-up surveys presents a significant challenge. In this study, our objective is to establish an efficient classification mechanism tailored for the follow-up survey conducted by WFST, with a specific focus on identifying KN associated with GW. We employ a novel temporal convolutional neural network architecture, trained using simulated multi-band photometry lasting for 3 days by WFST, accompanied by contextual information, i.e., luminosity distance information by GW. By comparison of the choices of contextual information, we can reach 95% precision and 94% recall for our best model. It also performs good validation of photometry data on AT2017gfo and AT2019npv. Furthermore, we investigate the ability of the model to distinguish KN in a GW follow-up survey. We conclude that there is over 80% probability that we can capture true KN in 20 selected candidates among ∼250 detected astrophysical transients that have passed the real–bogus filter and cross-matching.
Neural Network Approximation for Pessimistic Offline Reinforcement Learning

Deep reinforcement learning (RL) has shown remarkable success in specific offline decision-making scenarios, yet its theoretical guarantees are still under development. Existing works on offline RL theory primarily emphasize a few trivial settings, such as linear MDP or general function approximation with strong assumptions and independent data, which lack guidance for practical use. The coupling of deep learning and Bellman residuals makes this problem challenging, in addition to the difficulty of data dependence. In this paper, we establish a non-asymptotic estimation error of pessimistic offline RL using general neural network approximation with C-mixing data regarding the structure of networks, the dimension of datasets, and the concentrability of data coverage, under mild assumptions. Our result shows that the estimation error consists of two parts: the first converges to zero at a desired rate on the sample size with partially controllable concentrability, and the second becomes negligible if the residual constraint is tight. This result demonstrates the explicit efficiency of deep adversarial offline RL frameworks. We utilize the empirical process tool for C-mixing sequences and the neural network approximation theory for the Holder class to achieve this. We also develop methods to bound the Bellman estimation error caused by function approximation with empirical Bellman constraint perturbations. Additionally, we present a result that lessens the curse of dimensionality using data with low intrinsic dimensionality and function classes with low complexity. Our estimation provides valuable insights into the development of deep offline RL and guidance for algorithm model design.
Insights into the LiMn2O4 Cathode Stability in Aqueous Electrolyte

LiMn2O4 (LMO), cathodes present large stability when cycled in aqueous electrolytes, contrasting its behavior in conventional organic electrolytes in Lithium-ion batteries (LIBs). To elucidate the mechanisms underlying this distinctive behavior, we employ unconventional characterization techniques, including Variable Energy Positron Annihilation Lifetime Spectroscopy (VEPALS), Tip-Enhanced Raman Spectroscopy (TERS) and macro-Raman Spectroscopy (with mm-size laser spot). These still rather unexplored techniques in the battery field provide complementary information across different length scales, revealing previously hidden features. VEPALS offers atomic-scale insights, uncovering cationic defects and sub-nanometer pores that tend to collapse with cycling. TERS, operating at the nanometric range at the surface, captured the presence of Mn3O4 and its dissolution with cycling, elucidating dynamic changes during operation. Additionally, TERS highlights SO42- accumulation at grain boundaries. Macro-Raman Spectroscopy focuses on the micrometer scale, depicting small changes in the cathode's long-range order, suggesting a slow but progressive loss of crystalline quality under operation. Integrating these techniques provides a comprehensive assessment of LMO cathode stability in aqueous electrolytes, offering multifaceted insights into phase and defect evolution that can help to rationalize the origin of such stability when compared to conventional organic electrolytes. Our findings advance the understanding of LMO behavior in aqueous environments and provide guidelines for its development for next-generation LIBs.