Miracle-dz/UniFAR-FTUs huggingface dataset:
DreamShard: Generalizable Embedding Table Placement for Recommender Systems

We study embedding table placement for distributed recommender systems, which aims to partition and place the tables on multiple hardware devices (e.g., GPUs) to balance the computation and communication costs. Although prior work has explored learning-based approaches for the device placement of computational graphs, embedding table placement remains to be a challenging problem because of 1) the operation fusion of embedding tables, and 2) the generalizability requirement on unseen placement tasks with different numbers of tables and/or devices. To this end, we present DreamShard, a reinforcement learning (RL) approach for embedding table placement. DreamShard achieves the reasoning of operation fusion and generalizability with 1) a cost network to directly predict the costs of the fused operation, and 2) a policy network that is efficiently trained on an estimated Markov decision process (MDP) without real GPU execution, where the states and the rewards are estimated with the cost network. Equipped with sum and max representation reductions, the two networks can directly generalize to any unseen tasks with different numbers of tables and/or devices without fine-tuning. Extensive experiments show that DreamShard substantially outperforms the existing human expert and RNN-based strategies with up to 19% speedup over the strongest baseline on large-scale synthetic tables and our production tables. The code is available at https://github.com/daochenzha/dreamshard
ProtoBandit: Efficient Prototype Selection via Multi-Armed Bandits

In this work, we propose a multi-armed bandit-based framework for identifying a compact set of informative data instances (i.e., the prototypes) from a source dataset $S$ that best represents a given target set $T$. Prototypical examples of a given dataset offer interpretable insights into the underlying data distribution and assist in example-based reasoning, thereby influencing every sphere of human decision-making. Current state-of-the-art prototype selection approaches require $O(|S||T|)$ similarity comparisons between source and target data points, which becomes prohibitively expensive for large-scale settings. We propose to mitigate this limitation by employing stochastic greedy search in the space of prototypical examples and multi-armed bandits for reducing the number of similarity comparisons. Our randomized algorithm, ProtoBandit, identifies a set of $k$ prototypes incurring $O(k^3|S|)$ similarity comparisons, which is independent of the size of the target set. An interesting outcome of our analysis is for the $k$-medoids clustering problem $T = S$ setting) in which we show that our algorithm ProtoBandit approximates the BUILD step solution of the partitioning around medoids (PAM) method in $O(k^3|S|)$ complexity. Empirically, we observe that ProtoBandit reduces the number of similarity computation calls by several orders of magnitudes ($100-1000$ times) while obtaining solutions similar in quality to those from state-of-the-art approaches.
Robustness of planar Fourier capture arrays to colour changes and lost pixels

Planar Fourier capture arrays (PFCAs) are optical sensors built entirely in standard microchip manufacturing flows. PFCAs are composed of ensembles of angle sensitive pixels (ASPs) that each report a single coefficient of the Fourier transform of the far-away scene. Here we characterize the performance of PFCAs under the following three non-optimal conditions. First, we show that PFCAs can operate while sensing light of a wavelength other than the design point. Second, if only a randomly-selected subset of 10% of the ASPs are functional, we can nonetheless reconstruct the entire far-away scene using compressed sensing. Third, if the wavelength of the imaged light is unknown, it can be inferred by demanding self-consistency of the outputs.
Deep optical observations of the γ-ray pulsar PSR J0007+7303 in the CTA 1 supernova remnant

ABSTRACT The Fermi Large Area Telescope (LAT) discovered the time signature of a radio-silent pul-sar coincident with RX J0007.0+7302, a plerion-like X-ray source at the centre of the CTA1 supernova remnant. The inferred timing parameters of the γ-ray pulsar PSR J0007+7303(P=315.8 ms; P˙ ∼3.6×10 −13 s s −1 ) point to a Vela-like neutron star, with an age compara-ble to that of CTA 1. The PSR J0007+7303 low distance (∼1.4kpc), interstellar absorption(A V ∼1.6), and relatively high energy loss rate (E˙ ∼4.5×10 35 erg s −1 ), make it a suit-able candidate for an optical follow-up. Here, we present deep optical observations of PSRJ0007+7303. The pulsar is not detected in the Gran Telescopio Canarias (GTC) images downto a limit of r ′ ∼ 27.6(3σ), the deepest ever obtained for this pulsar, while William Her-schel Telescope (WHT) images yield a limit of V ∼ 26.9. Our r ′ -band limit correspondsto an optical emission efﬁciency η opt ≡ L opt /E˙ . 9.4×10 −8 . This limit is more con-straining than those derived for other Vela-like pulsars, but is still above the measured opticalefﬁciency of the Vela pulsar. We compared the optical upper l imits with the extrapolation ofthe XMM-NewtonX-ray spectrum and found that the optical emission is compatible with theextrapolation of the X-ray power-law component, at variance with what is observed, e.g. inthe Vela pulsar.Key words: stars: neutron – pulsars: individual: PSR J0007+7303
Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees

Inverse reinforcement learning (IRL) aims to recover the reward function and the associated optimal policy that best fits observed sequences of states and actions implemented by an expert. Many algorithms for IRL have an inherently nested structure: the inner loop finds the optimal policy given parametrized rewards while the outer loop updates the estimates towards optimizing a measure of fit. For high dimensional environments such nested-loop structure entails a significant computational burden. To reduce the computational burden of a nested loop, novel methods such as SQIL [1] and IQ-Learn [2] emphasize policy estimation at the expense of reward estimation accuracy. However, without accurate estimated rewards, it is not possible to do counterfactual analysis such as predicting the optimal policy under different environment dynamics and/or learning new tasks. In this paper we develop a novel single-loop algorithm for IRL that does not compromise reward estimation accuracy. In the proposed algorithm, each policy improvement step is followed by a stochastic gradient step for likelihood maximization. We show that the proposed algorithm provably converges to a stationary solution with a finite-time guarantee. If the reward is parameterized linearly, we show the identified solution corresponds to the solution of the maximum entropy IRL problem. Finally, by using robotics control problems in MuJoCo and their transfer settings, we show that the proposed algorithm achieves superior performance compared with other IRL and imitation learning benchmarks.
Adaptive Leading Cruise Control in Mixed Traffic Considering Human Behavioral Diversity

This paper presents an adaptive leading cruise control strategy for the connected and automated vehicle (CAV) and first considers its impact on the following human-driven vehicle (HDV) with diverse driving characteristics in the unified optimization framework for improved holistic energy efficiency. The car-following behaviors of HDV are statistically calibrated using the Next Generation Simulation dataset. In a typical single-lane car-following scenario where CAVs and HDVs share the road, the longitudinal speed control of CAVs can substantially reduce the energy consumption of the following HDV by avoiding unnecessary acceleration and braking. Moreover, apart from the objectives including car-following safety and traffic efficiency, the energy efficiencies of both CAV and HDV are incorporated into the reward function of reinforcement learning. The specific driving pattern of the following HDV is learned in real-time from historical speed information to predict its acceleration and power consumption in the optimization horizon. A comprehensive simulation is conducted to statistically verify the positive impacts of CAV on the holistic energy efficiency of the mixed traffic flow with uncertain and diverse human driving behaviors. Simulation results indicate that the holistic energy efficiency is improved by 4.38% on average.
Phenaki: Variable Length Video Generation From Open Domain Textual Description

We present Phenaki, a model capable of realistic video synthesis, given a sequence of textual prompts. Generating videos from text is particularly challenging due to the computational cost, limited quantities of high quality text-video data and variable length of videos. To address these issues, we introduce a new model for learning video representation which compresses the video to a small representation of discrete tokens. This tokenizer uses causal attention in time, which allows it to work with variable-length videos. To generate video tokens from text we are using a bidirectional masked transformer conditioned on pre-computed text tokens. The generated video tokens are subsequently de-tokenized to create the actual video. To address data issues, we demonstrate how joint training on a large corpus of image-text pairs as well as a smaller number of video-text examples can result in generalization beyond what is available in the video datasets. Compared to the previous video generation methods, Phenaki can generate arbitrary long videos conditioned on a sequence of prompts (i.e. time variable text or a story) in open domain. To the best of our knowledge, this is the first time a paper studies generating videos from time variable prompts. In addition, compared to the per-frame baselines, the proposed video encoder-decoder computes fewer tokens per video but results in better spatio-temporal consistency.
BAYESIAN ANALYSIS TO IDENTIFY NEW STAR CANDIDATES IN NEARBY YOUNG STELLAR KINEMATIC GROUPS

We present a new method based on a Bayesian analysis to identify new members of nearby young kinematic groups. The analysis minimally takes into account the position, proper motion, magnitude, and color of a star, but other observables can be readily added (e.g., radial velocity, distance). We use this method to find new young low-mass stars in the β Pictoris and AB Doradus moving groups and in the TW Hydrae, Tucana-Horologium, Columba, Carina, and Argus associations. Starting from a sample of 758 mid-K to mid-M (K5V–M5V) stars showing youth indicators such as Hα and X-ray emission, our analysis yields 214 new highly probable low-mass members of the kinematic groups analyzed. One is in TW Hydrae, 37 in β Pictoris, 17 in Tucana-Horologium, 20 in Columba, 6 in Carina, 50 in Argus, 32 in AB Doradus, and the remaining 51 candidates are likely young but have an ambiguous membership to more than one association. The false alarm rate for new candidates is estimated to be 5% for β Pictoris and TW Hydrae, 10% for Tucana-Horologium, Columba, Carina, and Argus, and 14% for AB Doradus. Our analysis confirms the membership of 58 stars proposed in the literature. Firm membership confirmation of our new candidates will require measurement of their radial velocity (predicted by our analysis), parallax, and lithium 6708 Å equivalent width. We have initiated these follow-up observations for a number of candidates, and we have identified two stars (2MASSJ01112542+1526214, 2MASSJ05241914-1601153) as very strong candidate members of the β Pictoris moving group and one strong candidate member (2MASSJ05332558-5117131) of the Tucana-Horologium association; these three stars have radial velocity measurements confirming their membership and lithium detections consistent with young age.
Recycling Scraps: Improving Private Learning by Leveraging Intermediate Checkpoints

All state-of-the-art (SOTA) differentially private machine learning (DP ML) methods are iterative in nature, and their privacy analyses allow publicly releasing the intermediate training checkpoints. However, DP ML benchmarks, and even practical deployments, typically use only the final training checkpoint to make predictions. In this work, for the first time, we comprehensively explore various methods that aggregate intermediate checkpoints to improve the utility of DP training. Empirically, we demonstrate that checkpoint aggregations provide significant gains in the prediction accuracy over the existing SOTA for CIFAR10 and StackOverflow datasets, and that these gains get magnified in settings with periodically varying training data distributions. For instance, we improve SOTA StackOverflow accuracies to 22.7% (+0.43% absolute) for $\epsilon=8.2$, and 23.84% (+0.43%) for $\epsilon=18.9$. Theoretically, we show that uniform tail averaging of checkpoints improves the empirical risk minimization bound compared to the last checkpoint of DP-SGD. Lastly, we initiate an exploration into estimating the uncertainty that DP noise adds in the predictions of DP ML models. We prove that, under standard assumptions on the loss function, the sample variance from last few checkpoints provides a good approximation of the variance of the final model of a DP run. Empirically, we show that the last few checkpoints can provide a reasonable lower bound for the variance of a converged DP model.
The breakdown of current gyrochronology as evidenced by old coeval stars

Gyrochronology can yield useful ages for field main-sequence stars, a regime where other techniques are problematic. Typically, gyrochronology relations are calibrated using young (≲2 Gyr) clusters, but the constraints at older ages are scarce, making them potentially inaccurate and imprecise. In order to test the performance of existing relations, we construct samples of stellar pairs with coeval components, for a range of ages and with available rotation periods. These include randomly paired stars in clusters, and wide binaries in the Kepler field. We design indicators that, based on the measured rotation periods and expectations from gyrochronology, quantify the (dis)agreement between the coeval pairs and the gyrochronology calibrations under scrutiny. Our results show that wide binaries and cluster members are in better concordance with gyrochronology than samples of randomly paired field stars, confirming that the relations have predicting power. However, the agreement with the examined relations decreases for older stars, revealing a degradation of the examined relations with age, in agreement with recent works. This highlights the need for novel empirical constraints at older ages that may allow revised calibrations. Notably, using coeval stars to test gyrochronology poses the advantage of circumventing the need for age determinations while simultaneously exploiting larger samples at older ages. Our test is independent of any specific age–rotation relation, and it can be used to evaluate future spin-down models. In addition, taking gyrochronology at face value, we note that our results provide new empirical evidence that the components of field wide binaries are indeed coeval.
Game-theoretic statistics and safe anytime-valid inference

Safe anytime-valid inference (SAVI) provides measures of statistical evidence and certainty -- e-processes for testing and confidence sequences for estimation -- that remain valid at all stopping times, accommodating continuous monitoring and analysis of accumulating data and optional stopping or continuation for any reason. These measures crucially rely on test martingales, which are nonnegative martingales starting at one. Since a test martingale is the wealth process of a player in a betting game, SAVI centrally employs game-theoretic intuition, language and mathematics. We summarize the SAVI goals and philosophy, and report recent advances in testing composite hypotheses and estimating functionals in nonparametric settings.
Modeling the Extragalactic Background Light and the Cosmic Star Formation History

We present an updated model for the extragalactic background light (EBL) from stars and dust, over wavelengths ≈0.1–1000 μm. This model uses accurate theoretical stellar spectra, and tracks the evolution of star formation, stellar mass density, metallicity, and interstellar dust extinction and emission in the universe with redshift. Dust emission components are treated self-consistently, with stellar light absorbed by dust reradiated in the infrared as three blackbody components. We fit our model, with free parameters associated with star formation rate and dust extinction and emission, to a wide variety of data: luminosity density, stellar mass density, and dust extinction data from galaxy surveys; and γ-ray absorption optical depth data from γ-ray telescopes. Our results strongly constraint the star formation rate density and dust photon escape fraction of the universe out to redshift z = 10, about 90% of the history of the universe. We find our model result is, in some cases, below lower limits on the z = 0 EBL intensity, and below some low-z γ-ray absorption measurements.
GAPX: Generalized Autoregressive Paraphrase-Identification X

Paraphrase Identification is a fundamental task in Natural Language Processing. While much progress has been made in the field, the performance of many state-of-the-art models often suffer from distribution shift during inference time. We verify that a major source of this performance drop comes from biases introduced by negative examples. To overcome these biases, we propose in this paper to train two separate models, one that only utilizes the positive pairs and the other the negative pairs. This enables us the option of deciding how much to utilize the negative model, for which we introduce a perplexity based out-of-distribution metric that we show can effectively and automatically determine how much weight it should be given during inference. We support our findings with strong empirical results.
Simulations of galaxy formation with radiative transfer: hydrogen reionization and radiative feedback

We carry out hydrodynamical simulations of galaxy formation that simultaneously follow radiative transfer of hydrogen-ionizing photons, based on the optically thin variable Eddington tensor approximation as implemented in the GADGET code. We consider only star-forming galaxies as sources and examine to what extent they can yield a reasonable reionization history and thermal state of the intergalactic medium at redshifts around z ∼ 3. This serves as an important benchmark for our self-consistent methodology to simulate galaxy formation and reionization, and for future improvements through accounting of other sources and other wavelength ranges. We find that star formation alone is sufficient for reionizing the Universe by redshift z ∼ 6. For a suitable choice of the escape fraction and the heating efficiency, our models are approximately able to account at the same time for the one-point function and the power spectrum of the Lyman α forest. The radiation field has an important impact on the star formation rate density in our simulations and significantly lowers the gaseous and stellar fractions in low-mass dark matter haloes. Our results thus directly demonstrate the importance of radiative feedback for galaxy formation. The spatial and temporal importance of this effect can be studied accurately with the modelling technique explored here, allowing more faithful simulations of galaxy formation.
Multi-objective optimization via equivariant deep hypervolume approximation

Optimizing multiple competing objectives is a common problem across science and industry. The inherent inextricable trade-off between those objectives leads one to the task of exploring their Pareto front. A meaningful quantity for the purpose of the latter is the hypervolume indicator, which is used in Bayesian Optimization (BO) and Evolutionary Algorithms (EAs). However, the computational complexity for the calculation of the hypervolume scales unfavorably with increasing number of objectives and data points, which restricts its use in those common multi-objective optimization frameworks. To overcome these restrictions we propose to approximate the hypervolume function with a deep neural network, which we call DeepHV. For better sample efficiency and generalization, we exploit the fact that the hypervolume is scale-equivariant in each of the objectives as well as permutation invariant w.r. t. both the objectives and the samples, by using a deep neural network that is equivariant w.r. t. the combined group of scalings and permutations. We evaluate our method against exact, and approximate hypervolume methods in terms of accuracy, computation time, and generalization. We also apply and compare our methods to state-of-the-art multi-objective BO methods and EAs on a range of synthetic benchmark test cases. The results show that our methods are promising for such multi-objective optimization tasks.
Are All Losses Created Equal: A Neural Collapse Perspective

While cross entropy (CE) is the most commonly used loss to train deep neural networks for classification tasks, many alternative losses have been developed to obtain better empirical performance. Among them, which one is the best to use is still a mystery, because there seem to be multiple factors affecting the answer, such as properties of the dataset, the choice of network architecture, and so on. This paper studies the choice of loss function by examining the last-layer features of deep networks, drawing inspiration from a recent line work showing that the global optimal solution of CE and mean-square-error (MSE) losses exhibits a Neural Collapse phenomenon. That is, for sufficiently large networks trained until convergence, (i) all features of the same class collapse to the corresponding class mean and (ii) the means associated with different classes are in a configuration where their pairwise distances are all equal and maximized. We extend such results and show through global solution and landscape analyses that a broad family of loss functions including commonly used label smoothing (LS) and focal loss (FL) exhibits Neural Collapse. Hence, all relevant losses(i.e., CE, LS, FL, MSE) produce equivalent features on training data. Based on the unconstrained feature model assumption, we provide either the global landscape analysis for LS loss or the local landscape analysis for FL loss and show that the (only! ) global minimizers are neural collapse solutions, while all other critical points are strict saddles whose Hessian exhibit negative curvature directions either in the global scope for LS loss or in the local scope for FL loss near the optimal solution. The experiments further show that Neural Collapse features obtained from all relevant losses lead to largely identical performance on test data as well, provided that the network is sufficiently large and trained until convergence.