Miracle-dz/UniFAR-FTUs huggingface dataset:
Imitation Learning from Observation with Automatic Discount Scheduling

Humans often acquire new skills through observation and imitation. For robotic agents, learning from the plethora of unlabeled video demonstration data available on the Internet necessitates imitating the expert without access to its action, presenting a challenge known as Imitation Learning from Observations (ILfO). A common approach to tackle ILfO problems is to convert them into inverse reinforcement learning problems, utilizing a proxy reward computed from the agent's and the expert's observations. Nonetheless, we identify that tasks characterized by a progress dependency property pose significant challenges for such approaches; in these tasks, the agent needs to initially learn the expert's preceding behaviors before mastering the subsequent ones. Our investigation reveals that the main cause is that the reward signals assigned to later steps hinder the learning of initial behaviors. To address this challenge, we present a novel ILfO framework that enables the agent to master earlier behaviors before advancing to later ones. We introduce an Automatic Discount Scheduling (ADS) mechanism that adaptively alters the discount factor in reinforcement learning during the training phase, prioritizing earlier rewards initially and gradually engaging later rewards only when the earlier behaviors have been mastered. Our experiments, conducted on nine Meta-World tasks, demonstrate that our method significantly outperforms state-of-the-art methods across all tasks, including those that are unsolvable by them.
Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity

This survey addresses the crucial issue of factuality in Large Language Models (LLMs). As LLMs find applications across diverse domains, the reliability and accuracy of their outputs become vital. We define the Factuality Issue as the probability of LLMs to produce content inconsistent with established facts. We first delve into the implications of these inaccuracies, highlighting the potential consequences and challenges posed by factual errors in LLM outputs. Subsequently, we analyze the mechanisms through which LLMs store and process facts, seeking the primary causes of factual errors. Our discussion then transitions to methodologies for evaluating LLM factuality, emphasizing key metrics, benchmarks, and studies. We further explore strategies for enhancing LLM factuality, including approaches tailored for specific domains. We focus two primary LLM configurations standalone LLMs and Retrieval-Augmented LLMs that utilizes external data, we detail their unique challenges and potential enhancements. Our survey offers a structured guide for researchers aiming to fortify the factual reliability of LLMs.
Markov Process Jump Times and Their Cox Construction

In this short paper, we connect the procedure of constructing a totally inaccessible stopping time for a given process using the well-known Cox construction, dependent on an independent exponential random variable; with naturally occurring jump times of Feller processes. Ultimately, we show that these two phenomena are not only related, but are in fact two examples of the same object. We link this fact to the behaviour of predictable stopping times, by proving that they can always be written as the hitting time of zero of a continuous process.
HaarNet: Large-scale Linear-Morphological Hybrid Network for RGB-D Semantic Segmentation

Signals from different modalities each have their own combination algebra which affects their sampling processing. RGB is mostly linear; depth is a geometric signal following the operations of mathematical morphology. If a network obtaining RGB-D input has both kinds of operators available in its layers, it should be able to give effective output with fewer parameters. In this paper, morphological elements in conjunction with more familiar linear modules are used to construct a mixed linear-morphological network called HaarNet. This is the first large-scale linear-morphological hybrid, evaluated on a set of sizeable real-world datasets. In the network, morphological Haar sampling is applied to both feature channels in several layers, which splits extreme values and high-frequency information such that both can be processed to improve both modalities. Moreover, morphologically parameterised ReLU is used, and morphologically-sound up-sampling is applied to obtain a full-resolution output. Experiments show that HaarNet is competitive with a state-of-the-art CNN, implying that morphological networks are a promising research direction for geometry-based learning tasks.
VeCLIP: Improving CLIP Training via Visual-enriched Captions

Large-scale web-crawled datasets are fundamental for the success of pre-training vision-language models, such as CLIP. However, the inherent noise and potential irrelevance of web-crawled AltTexts pose challenges in achieving precise image-text alignment. Existing methods utilizing large language models (LLMs) for caption rewriting have shown promise on small, curated datasets like CC3M and CC12M. This study introduces a scalable pipeline for noisy caption rewriting. Unlike recent LLM rewriting techniques, we emphasize the incorporation of visual concepts into captions, termed as Visual-enriched Captions (VeCap). To ensure data diversity, we propose a novel mixed training scheme that optimizes the utilization of AltTexts alongside newly generated VeCap. We showcase the adaptation of this method for training CLIP on large-scale web-crawled datasets, termed VeCLIP. Employing this cost-effective pipeline, we effortlessly scale our dataset up to 300 million samples named VeCap dataset. Our results show significant advantages in image-text alignment and overall model performance. For example, VeCLIP achieves up to +25.2% gain in COCO and Flickr30k retrieval tasks under the 12M setting. For data efficiency, VeCLIP achieves +3% gain while only using 14% of the data employed in the vanilla CLIP and 11% in ALIGN. We also note the VeCap data is complementary with other well curated datasets good for zero-shot classification tasks. When combining VeCap and DFN, our model can achieve strong performance on both of image-text retrieval and zero-shot classification tasks, e.g. 83.1% accuracy@1 on ImageNet zero-shot for a H/14 model. We release the pre-trained models at https://github.com/apple/ml-veclip.
Hypergraph Neural Networks through the Lens of Message Passing: A Common Perspective to Homophily and Architecture Design

Most of the current hypergraph learning methodologies and benchmarking datasets in the hypergraph realm are obtained by lifting procedures from their graph analogs, leading to overshadowing specific characteristics of hypergraphs. This paper attempts to confront some pending questions in that regard: Q1 Can the concept of homophily play a crucial role in Hypergraph Neural Networks (HNNs)? Q2 Is there room for improving current HNN architectures by carefully addressing specific characteristics of higher-order networks? Q3 Do existing datasets provide a meaningful benchmark for HNNs? To address them, we first introduce a novel conceptualization of homophily in higher-order networks based on a Message Passing (MP) scheme, unifying both the analytical examination and the modeling of higher-order networks. Further, we investigate some natural, yet mostly unexplored, strategies for processing higher-order structures within HNNs such as keeping hyperedge-dependent node representations, or performing node/hyperedge stochastic samplings, leading us to the most general MP formulation up to date -MultiSet-, as well as to an original architecture design, MultiSetMixer. Finally, we conduct an extensive set of experiments that contextualize our proposals and successfully provide insights about our inquiries.
Accelerating Vision Transformers Based on Heterogeneous Attention Patterns

Recently, Vision Transformers (ViTs) have attracted a lot of attention in the field of computer vision. Generally, the powerful representative capacity of ViTs mainly benefits from the self-attention mechanism, which has a high computation complexity. To accelerate ViTs, we propose an integrated compression pipeline based on observed heterogeneous attention patterns across layers. On one hand, different images share more similar attention patterns in early layers than later layers, indicating that the dynamic query-by-key self-attention matrix may be replaced with a static self-attention matrix in early layers. Then, we propose a dynamic-guided static self-attention (DGSSA) method where the matrix inherits self-attention information from the replaced dynamic self-attention to effectively improve the feature representation ability of ViTs. On the other hand, the attention maps have more low-rank patterns, which reflect token redundancy, in later layers than early layers. In a view of linear dimension reduction, we further propose a method of global aggregation pyramid (GLAD) to reduce the number of tokens in later layers of ViTs, such as Deit. Experimentally, the integrated compression pipeline of DGSSA and GLAD can accelerate up to 121% run-time throughput compared with DeiT, which surpasses all SOTA approaches.
IBoxCLA: Towards Robust Box-supervised Segmentation of Polyp via Improved Box-dice and Contrastive Latent-anchors

Box-supervised polyp segmentation attracts increasing attention for its cost-effective potential. Existing solutions often rely on learning-free methods or pretrained models to laboriously generate pseudo masks, triggering Dice constraint subsequently. In this paper, we found that a model guided by the simplest box-filled masks can accurately predict polyp locations/sizes, but suffers from shape collapsing. In response, we propose two innovative learning fashions, Improved Box-dice (IBox) and Contrastive Latent-Anchors (CLA), and combine them to train a robust box-supervised model IBoxCLA. The core idea behind IBoxCLA is to decouple the learning of location/size and shape, allowing for focused constraints on each of them. Specifically, IBox transforms the segmentation map into a proxy map using shape decoupling and confusion-region swapping sequentially. Within the proxy map, shapes are disentangled, while locations/sizes are encoded as box-like responses. By constraining the proxy map instead of the raw prediction, the box-filled mask can well supervise IBoxCLA without misleading its shape learning. Furthermore, CLA contributes to shape learning by generating two types of latent anchors, which are learned and updated using momentum and segmented polyps to steadily represent polyp and background features. The latent anchors facilitate IBoxCLA to capture discriminative features within and outside boxes in a contrastive manner, yielding clearer boundaries. We benchmark IBoxCLA on five public polyp datasets. The experimental results demonstrate the competitive performance of IBoxCLA compared to recent fully-supervised polyp segmentation methods, and its superiority over other box-supervised state-of-the-arts with a relative increase of overall mDice and mIoU by at least 6.5% and 7.5%, respectively.
S4C: Self-Supervised Semantic Scene Completion with Neural Fields

3D semantic scene understanding is a fundamental challenge in computer vision. It enables mobile agents to autonomously plan and navigate arbitrary environments. SSC formalizes this challenge as jointly estimating dense geometry and semantic information from sparse observations of a scene. Current methods for SSC are generally trained on 3D ground truth based on aggregated LiDAR scans. This process relies on special sensors and annotation by hand which are costly and do not scale well. To overcome this issue, our work presents the first self-supervised approach to SSC called S4C that does not rely on 3D ground truth data. Our proposed method can reconstruct a scene from a single image and only relies on videos and pseudo segmentation ground truth generated from off-the-shelf image segmentation network during training. Unlike existing methods, which use discrete voxel grids, we represent scenes as implicit semantic fields. This formulation allows querying any point within the camera frustum for occupancy and semantic class. Our architecture is trained through rendering-based self-supervised losses. Nonetheless, our method achieves performance close to fully supervised state-of-the-art methods. Additionally, our method demonstrates strong generalization capabilities and can synthesize accurate segmentation maps for far away viewpoints.
An efficient saddle search method for ordered phase transitions involving translational invariance

The bottleneck of studying phase transitions is the barrier-crossing process composed of escaping from the basin of the local minimum and finding the saddle point. Breaking the bottleneck requires designing efficient algorithms relevant to the properties of concrete phase transition. In this work, we propose an efficient nullspace-preserving saddle search (NPSS) method for a class of phase transitions involving translational invariance. These critical states in these phase transitions are usually degenerate. The NPSS overcomes the difficulty of degeneration by ensuring the ascent direction orthogonal to the kernel space of the initial minimum, then efficiently escapes from the basin and finds the saddle point. We apply the NPSS method to the phase transitions between crystals, and between crystal and quasicrystal, based on the Landau-Brazvoskii and Lifshitz-Petrich free energy functionals. Numerical results show a good performance of the proposed method. Finally, we investigate an important property of the inflection point, where symmetry-breaking begins to occur and nullspace is no longer maintained.
Low-lying zeros of a large orthogonal family of automorphic $L$-functions

We study a new orthogonal family of $L$-functions associated with holomorphic Hecke newforms of level $q$, averaged over $q \asymp Q$. To illustrate our methods, we prove a one level density result for this family with the support of the Fourier transform of the test function being extended to be inside $(-4, 4)$. The main techniques developed in this paper will be useful in developing further results for this family, including estimates for high moments, information on the vertical distribution of zeros, as well as critical line theorems.
SparseCoder: Advancing Source Code Analysis with Sparse Attention and Learned Token Pruning

As software projects rapidly evolve, software artifacts become more complex and defects behind get harder to identify. The emerging Transformer-based approaches, though achieving remarkable performance, struggle with long code sequences due to their self-attention mechanism, which scales quadratically with the sequence length. This paper introduces SparseCoder, an innovative approach incorporating sparse attention and learned token pruning (LTP) method (adapted from natural language processing) to address this limitation. Extensive experiments carried out on a large-scale dataset for vulnerability detection demonstrate the effectiveness and efficiency of SparseCoder, scaling from quadratically to linearly on long code sequence analysis in comparison to CodeBERT and RoBERTa. We further achieve 50% FLOPs reduction with a negligible performance drop of less than 1% comparing to Transformer leveraging sparse attention. Moverover, SparseCoder goes beyond making"black-box"decisions by elucidating the rationale behind those decisions. Code segments that contribute to the final decision can be highlighted with importance scores, offering an interpretable, transparent analysis tool for the software engineering landscape.
Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models

Considerable effort has been dedicated to mitigating toxicity, but existing methods often require drastic modifications to model parameters or the use of computationally intensive auxiliary models. Furthermore, previous approaches have often neglected the crucial factor of language's evolving nature over time. In this work, we present a comprehensive perspective on toxicity mitigation that takes into account its changing nature. We introduce Goodtriever, a flexible methodology that matches the current state-of-the-art toxicity mitigation while achieving 43% relative latency reduction during inference and being more computationally efficient. By incorporating a retrieval-based approach at decoding time, Goodtriever enables toxicity-controlled text generation. Our research advocates for an increased focus on adaptable mitigation techniques, which better reflect the data drift models face when deployed in the wild. Code and data are available at https://github.com/for-ai/goodtriever.
The sulfur species in hot rocky exoplanet atmospheres

The first JWST observations of hot Jupiters showed an unexpected detection of SO in their hydrogen‐rich atmospheres. We investigate how much sulfur can be expected in the atmospheres of rocky exoplanets and which sulfur molecules can be expected to be most abundant and detectable by transmission spectroscopy. We run thermochemical equilibrium models at the crust–atmosphere interface, considering surface temperatures 500–5000 K, surface pressures 1–100 bar, and various sets of element abundances based on common rock compositions. Between 1000 and 2000 K, we find gaseous sulfur concentrations of up to 25% above the rock in our models. SO, SO, HS, and S are by far the most abundant sulfur molecules. SO shows potentially detectable features in transmission spectra at about 4 m, between 7 and 8 m, and beyond 15 m. In contrast, the sometimes abundant HS molecule is difficult to detect in these spectra, which are mostly dominated by HO and CO. Although the molecule PS only occurs with concentrations ppm, it can cause a strong absorption feature between 0.3 and 0.65 m in some of our models for high surface pressures. The detection of sulfur molecules would enable a better characterization of the planetary surface.
A three-dimensional study of the tropospheric sulfur cycle

The global tropospheric distributions of seven important sulfur species were simulated with a global three-dimensional chemistry-transport model (IMAGES). Surface emission and deposition velocity maps were established for use as lower boundary conditions in the model. While anthropogenic SO2 emissions are by far the largest sulfur source in the northern midlatitudes, reduced sulfur compounds, notably dimethyl sulfide (DMS) predominate over most remote areas. Simulations were performed for the present-day (∼ 1985) atmosphere. The calculated distributions are compared with available observations. The model results are found to be generally within a factor of (at most) 2–3 of long-term observations. Comparison with campaign measurements is more difficult, mostly due to the strong dependence of sulfur species concentrations on local meteorological conditions. The results, however, indicate the need for future model refinements, especially with respect to biogenic emission estimates and parameterization of cloud processes. A sensitivity study is presented to discuss the uncertainties of the results related to several parameters (the decoupling of wet scavenging and convective transport for soluble species, volcanoes emission and deposition velocities). Results are also discussed in terms of global budgets and related variables and processes. Around 125 Tg S/yr of non-sea-salt (nss) sulfur compounds (DMS, CS2, H2S, COS, and SO2) are injected into the atmosphere. The balance is mainly maintained by nss-sulfates wet and dry deposition, and by SO2 dry deposition (94% of total sulfur deposition). It is found that DMS oxidation represents the main contribution to SO2 chemical production (80% of the chemical sources), and that the major sink of SO2 is provided by in-cloud oxidation (90% of the chemical sinks), under the assumption that all SO2 incorporated into clouds is oxidized. The calculated annual wet deposition of sulfates reaches 3 g S m−2 yr−1 over Europe and North America, while it is usually lower than 0.5 g S m−2 yr−1 in remote parts of the world. Estimations for the global lifetimes are 0.9 day for DMS, 4 days for CS2, 2.2 days for H2S, 0.6 day for SO2, 0.18 day for DMSO, 6.1 days for MSA, and 4.7 days for nss-sulfates.
A method for determining the average height of a large topographic ice sheet from observations of the echo received by a satellite altimeter

Abstract Ice-sheet surfaces have scales of fluctuation that arc similar to the diameter of the area illuminated by a satellite radar altimeter. The present theory of altimeter, developed to describe scattering from the ocean surface, does not deal properly with the geometry of ice-sheet surfaces. In this paper, the theory of altimeter is extended to cover this geometry. A general relation for the altimeter echo from a surface of unknown geometry is developed, including the effects of the penetration of the surface by the radar waves. This expression is linearized, using the characteristic operating geometry of satellite altimeters and the gentle nature of ice-sheet gradients. From this expression, an integral equation is derived, from whose solution the spatial average of the height of the surface relative to a spherical datum can be determined. The integral equation is of a Volterra type, which permits the uniqueness of the solution for the average height to be investigated simply. The method is extended to provide a solution for the spatial average of the height of a local region of the ice sheet, provided the region remains large in comparison with the area illuminated by the altimeter, and to deal with variations in the antenna bore-sight alignment. The results have a number of implications for the collection and reduction of echoes in an experiment to determine the average height of an ice sheet. The unique determination of the average height requires the echo to be known over a time interval that depends on the extrema of the surface, which therefore must be known a priori. The average height itself can be determined by the operation on the echo of a linear operator whose kernel is derived from the solution of the Volterra-type equation. This marks a change from the procedures currently used in practice to reduce echoes from ice sheets.
Airborne Laser Topographic Mapping: Applications to Hurricane Storm Surge Hazards

In the United States, the population and urbanization of the coastal zone is rapidly increasing. Currently, it is estimated that the population in the U.S. coastal zone increases on average by over 3600 people each day [Cullinton, 1998]. Cities along the Southeast and Gulf coast of the United States are particularly vulnerable to the hazards of hurricanes. The dramatic increases in the cost of hurricane dam-
Quantum Shadow Gradient Descent for Quantum Learning

This paper proposes a new procedure called quantum shadow gradient descent (QSGD) that addresses these key challenges. Our method has the benefits of a one-shot approach, in not requiring any sample duplication while having a convergence rate comparable to the ideal update rule using exact gradient computation. We propose a new technique for generating quantum shadow samples (QSS), which generates quantum shadows as opposed to classical shadows used in existing works. With classical shadows, the computations are typically performed on classical computers and, hence, are prohibitive since the dimension grows exponentially. Our approach resolves this issue by measurements of quantum shadows. As the second main contribution, we study more general non-product ansatz of the form $\exp\{i\sum_j \theta_j A_j\}$ that model variational Hamiltonians. We prove that the gradient can be written in terms of the gradient of single-parameter ansatzes that can be easily measured. Our proof is based on the Suzuki-Trotter approximation; however, our expressions are exact, unlike prior efforts that approximate non-product operators. As a result, existing gradient measurement techniques can be applied to more general VQAs followed by correction terms without any approximation penalty. We provide theoretical proofs, convergence analysis and verify our results through numerical experiments.