Miracle-dz/UniFAR-FTUs huggingface dataset:
Trends in Energy Estimates for Computing in AI/Machine Learning Accelerators, Supercomputers, and Compute-Intensive Applications

We examine the computational energy requirements of different systems driven by the geometrical scaling law (known as Moore's law or Dennard Scaling for geometry) and increasing use of Artificial Intelligence/ Machine Learning (AI/ML) over the last decade. With more scientific and technology applications based on data-driven discovery, machine learning methods, especially deep neural networks, have become widely used. In order to enable such applications, both hardware accelerators and advanced AI/ML methods have led to the introduction of new architectures, system designs, algorithms, and software. Our analysis of energy trends indicates three important observations: 1) Energy efficiency due to geometrical scaling is slowing down; 2) The energy efficiency at the bit-level does not translate into efficiency at the instruction-level, or at the system-level for a variety of systems, especially for large-scale AI/ML accelerators or supercomputers; 3) At the application level, general-purpose AI/ML methods can be computationally energy intensive, off-setting the gains in energy from geometrical scaling and special purpose accelerators. Further, our analysis provides specific pointers for integrating energy efficiency with performance analysis for enabling high-performance and sustainable computing in the future.
Anosov groups that are indiscrete in rank one

We exhibit Anosov subgroups of $\mathsf{SL}_{d}(\mathbb{R})$ that do not embed discretely in any rank-$1$ simple Lie group of noncompact type, or indeed, in any finite product of such Lie groups. These subgroups are isomorphic to free products $\Gamma * \Delta $, where $\Gamma $ is a uniform lattice in $\textsf{F}_{4}^{(-20)}$ and $\Delta $ is a uniform lattice in $\textsf{Sp}(m,1)$, $m \geq 51$.
BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model

This paper presents BERT-CTC, a novel formulation of end-to-end speech recognition that adapts BERT for connectionist temporal classification (CTC). Our formulation relaxes the conditional independence assumptions used in conventional CTC and incorporates linguistic knowledge through the explicit output dependency obtained by BERT contextual embedding. BERT-CTC attends to the full contexts of the input and hypothesized output sequences via the self-attention mechanism. This mechanism encourages a model to learn inner/inter-dependencies between the audio and token representations while maintaining CTC's training efficiency. During inference, BERT-CTC combines a mask-predict algorithm with CTC decoding, which iteratively refines an output sequence. The experimental results reveal that BERT-CTC improves over conventional approaches across variations in speaking styles and languages. Finally, we show that the semantic representations in BERT-CTC are beneficial towards downstream spoken language understanding tasks.
RIS-Based Steerable Beamforming Antenna with Near-Field Eigenmode Feeder

We present a novel and hardware-efficient space-fed antenna system that exploits the eigenmodes of the over-the-air propagation matrix from a small active multi-antenna feeder (AMAF) to a large reflective intelligent surface (RIS), both configured as standard linear arrays and placed in the near-field of each other. We demonstrate the flexibility of the proposed architecture by showing that it is capable of generating beam shapes for multiple applications, such as very narrow beams with low side lobes for space division multiple access communications, flat-top wide-angle patterns for short-range automotive sensing and sectorial beaconing, and monopulse-like patterns for radar angular tracking. A key parameter in our designs is the AMAF-RIS distance which must be optimized and it is generally much less than the Rayleigh distance. For a given AMAF array size, the optimal AMAF-RIS distance increases as a function of RIS size. The AMAF-RIS loss is almost exactly compensated by the larger aperture gain of the RIS, resulting in an almost constant RIS gain for increasing RIS sizes. This allows different beam angle selectivities to be selected for the same center beam gain. Active RF amplification is done at the AMAF only, thus resulting in a much lower hardware complexity than conventional active phased arrays for the same beamforming performance.
RLET: A Reinforcement Learning Based Approach for Explainable QA with Entailment Trees

Interpreting the reasoning process from questions to answers poses a challenge in approaching explainable QA. A recently proposed structured reasoning format, entailment tree, manages to offer explicit logical deductions with entailment steps in a tree structure. To generate entailment trees, prior single pass sequence-to-sequence models lack visible internal decision probability, while stepwise approaches are supervised with extracted single step data and cannot model the tree as a whole. In this work, we propose RLET, a Reinforcement Learning based Entailment Tree generation framework, which is trained utilising the cumulative signals across the whole tree. RLET iteratively performs single step reasoning with sentence selection and deduction generation modules, from which the training signal is accumulated across the tree with elaborately designed aligned reward function that is consistent with the evaluation. To the best of our knowledge, we are the first to introduce RL into the entailment tree generation task. Experiments on three settings of the EntailmentBank dataset demonstrate the strength of using RL framework.
GRaM-X: a new GPU-accelerated dynamical spacetime GRMHD code for Exascale computing with the Einstein Toolkit

We present GRaM-X (General Relativistic accelerated Magnetohydrodynamics on AMReX), a new GPU-accelerated dynamical-spacetime general relativistic magnetohydrodynamics (GRMHD) code which extends the GRMHD capability of Einstein Toolkit to GPU-based exascale systems. GRaM-X supports 3D adaptive mesh refinement (AMR) on GPUs via a new AMR driver for the Einstein Toolkit called CarpetX which in turn leverages AMReX, an AMR library developed for use by the United States DOE’s Exascale Computing Project. We use the Z4c formalism to evolve the Einstein equations and the Valencia formulation to evolve the equations of GRMHD. GRaM-X supports both analytic as well as tabulated equations of state. We implement TVD and WENO reconstruction methods as well as the HLLE Riemann solver. We test the accuracy of the code using a range of tests on static spacetime, e.g. 1D magnetohydrodynamics shocktubes, the 2D magnetic rotor and a cylindrical explosion, as well as on dynamical spacetimes, i.e. the oscillations of a 3D Tolman-Oppenheimer-Volkhof star. We find excellent agreement with analytic results and results of other codes reported in literature. We also perform scaling tests and find that GRaM-X shows a weak scaling efficiency of ∼40%–50% on 2304 nodes (13824 NVIDIA V100 GPUs) with respect to single-node performance on OLCF’s supercomputer Summit.
Continuous Attribution of Episodical Outcomes for More Efficient and Targeted Online Measurement

Online experimentation platforms collect user feedback at low cost and large scale. Some systems even support real-time or near real-time data processing, and can update metrics and statistics continuously. Many commonly used metrics, such as clicks and page views, can be observed without much delay. However, many important signals can only be observed after several hours or days, with noise adding up over the duration of the episode. When episodical outcomes follow a complex sequence of user-product interactions, it is difficult to understand which interactions lead to the final outcome. There is no obvious attribution logic for us to associate a positive or negative outcome back to the actions and choices we made at different times. This attribution logic is critical to unlocking more targeted and efficient measurement at a finer granularity that could eventually lead to the full capability of reinforcement learning. In this paper, we borrow the idea of Causal Surrogacy to model a long-term outcome using leading indicators that are incrementally observed and apply it as the value function to track the progress towards the final outcome and attribute incrementally to various user-product interaction steps. Applying this approach to the guest booking metric at Airbnb resulted in significant variance reductions of 50% to 85%, while aligning well with the booking metric itself. Continuous attribution allows us to assign a utility score to each product page-view, and this score can be flexibly further aggregated to a variety of units of interest, such as searches and listings. We provide multiple real-world applications of attribution to illustrate its versatility.
Mining Word Boundaries in Speech as Naturally Annotated Word Segmentation Data

Inspired by early research on exploring naturally annotated data for Chinese word segmentation (CWS), and also by recent research on integration of speech and text processing, this work for the first time proposes to mine word boundaries from parallel speech/text data. First we collect parallel speech/text data from two Internet sources that are related with CWS data used in our experiments. Then, we obtain character-level alignments and design simple heuristic rules for determining word boundaries according to pause duration between adjacent characters. Finally, we present an effective complete-then-train strategy that can better utilize extra naturally annotated data for model training. Experiments demonstrate our approach can significantly boost CWS performance in both cross-domain and low-resource scenarios.
Efficient Document Retrieval by End-to-End Refining and Quantizing BERT Embedding with Contrastive Product Quantization

Efficient document retrieval heavily relies on the technique of semantic hashing, which learns a binary code for every document and employs Hamming distance to evaluate document distances. However, existing semantic hashing methods are mostly established on outdated TFIDF features, which obviously do not contain lots of important semantic information about documents. Furthermore, the Hamming distance can only be equal to one of several integer values, significantly limiting its representational ability for document distances. To address these issues, in this paper, we propose to leverage BERT embeddings to perform efficient retrieval based on the product quantization technique, which will assign for every document a real-valued codeword from the codebook, instead of a binary code as in semantic hashing. Specifically, we first transform the original BERT embeddings via a learnable mapping and feed the transformed embedding into a probabilistic product quantization module to output the assigned codeword. The refining and quantizing modules can be optimized in an end-to-end manner by minimizing the probabilistic contrastive loss. A mutual information maximization based method is further proposed to improve the representativeness of codewords, so that documents can be quantized more accurately. Extensive experiments conducted on three benchmarks demonstrate that our proposed method significantly outperforms current state-of-the-art baselines.
A picture of the space of typical learnable tasks

We develop information geometric techniques to understand the representations learned by deep networks when they are trained on different tasks using supervised, meta-, semi-supervised and contrastive learning. We shed light on the following phenomena that relate to the structure of the space of tasks: (1) the manifold of probabilistic models trained on different tasks using different representation learning methods is effectively low-dimensional; (2) supervised learning on one task results in a surprising amount of progress even on seemingly dissimilar tasks; progress on other tasks is larger if the training task has diverse classes; (3) the structure of the space of tasks indicated by our analysis is consistent with parts of the Wordnet phylogenetic tree; (4) episodic meta-learning algorithms and supervised learning traverse different trajectories during training but they fit similar models eventually; (5) contrastive and semi-supervised learning methods traverse trajectories similar to those of supervised learning. We use classification tasks constructed from the CIFAR-10 and Imagenet datasets to study these phenomena.
CausalBench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Data

Causal inference is a vital aspect of multiple scientific disciplines and is routinely applied to high-impact applications such as medicine. However, evaluating the performance of causal inference methods in real-world environments is challenging due to the need for observations under both interventional and control conditions. Traditional evaluations conducted on synthetic datasets do not reflect the performance in real-world systems. To address this, we introduce CausalBench, a benchmark suite for evaluating network inference methods on real-world interventional data from large-scale single-cell perturbation experiments. CausalBench incorporates biologically-motivated performance metrics, including new distribution-based interventional metrics. A systematic evaluation of state-of-the-art causal inference methods using our CausalBench suite highlights how poor scalability of current methods limits performance. Moreover, methods that use interventional information do not outperform those that only use observational data, contrary to what is observed on synthetic benchmarks. Thus, CausalBench opens new avenues in causal network inference research and provides a principled and reliable way to track progress in leveraging real-world interventional data.
A Faster Sampler for Discrete Determinantal Point Processes

Discrete Determinantal Point Processes (DPPs) have a wide array of potential applications for subsampling datasets. They are however held back in some cases by the high cost of sampling. In the worst-case scenario, the sampling cost scales as O(n^3) where n is the number of elements of the ground set. A popular workaround to this prohibitive cost is to sample DPPs defined by low-rank kernels. In such cases, the cost of standard sampling algorithms scales as O(np^2 + nm^2) where m is the (average) number of samples of the DPP (usually m<<n) and p the rank of the kernel used to define the DPP (m \leq p \leq n). The first term, O(np^2), comes from a SVD-like step. We focus here on the second term of this cost, O(nm^2), and show that it can be brought down to O(nm + m^3 log m) without loss on the sampling's exactness. In practice, we observe very substantial speedups compared to the classical algorithm as soon as n>1000. The algorithm described here is a close variant of the standard algorithm for sampling continuous DPPs, and uses rejection sampling. In the specific case of projection DPPs, we also show that any additional sample can be drawn in time O(m^3 log m). Finally, an interesting by-product of the analysis is that a realisation from a DPP is typically contained in a subset of size O(m log m) formed using leverage score i.i. d. sampling.
The Near Infrared Imager and Slitless Spectrograph for JWST. V. Kernel Phase Imaging and Data Analysis

Kernel phase imaging (KPI) enables the direct detection of substellar companions and circumstellar dust close to and below the classical (Rayleigh) diffraction limit. The high-Strehl full pupil images provided by the James Webb Space Telescope (JWST) are ideal for application of the KPI technique. We present a kernel phase analysis of JWST NIRISS full pupil images taken during the instrument commissioning and compare the performance to closely related NIRISS aperture masking interferometry (AMI) observations. For this purpose, we develop and make publicly available the custom Kpi3Pipeline data reduction pipeline enabling the extraction of kernel phase observables from JWST images. The extracted observables are saved into a new and versatile kernel phase FITS file data exchange format. Furthermore, we present our new and publicly available fouriever toolkit which can be used to search for companions and derive detection limits from KPI, AMI, and long-baseline interferometry observations while accounting for correlated uncertainties in the model fitting process. Among the four KPI targets that were observed during NIRISS instrument commissioning, we discover a low-contrast (∼1:5) close-in (∼1 λ/D) companion candidate around CPD-66 562 and a new high-contrast (∼1:170) detection separated by ∼1.5 λ/D from 2MASS J062802.01-663738.0. The 5σ companion detection limits around the other two targets reach ∼6.5 mag at ∼200 mas and ∼7 mag at ∼400 mas. Comparing these limits to those obtained from the NIRISS AMI commissioning observations, we find that KPI and AMI perform similar in the same amount of observing time. Due to its 5.6 times higher throughput if compared to AMI, KPI is beneficial for observing faint targets and superior to AMI at separations ≳325 mas. At very small separations (≲100 mas) and between ∼250 and 325 mas, AMI slightly outperforms KPI which suffers from increased photon noise from the core and the first Airy ring of the point-spread function.
Optimal Antenna Placement for Two-Antenna Near-Field Wireless Power Transfer

Current trends in communication system design precipitate a change in the operating regime from the traditional far-field to the radiating near-field (Fresnel) region. We investigate the optimal transmit antenna placement for a multiple-input single-output (MISO) wireless power transfer (WPT) system designed for a three-dimensional cuboid room under line-of-sight (LoS) conditions in the Fresnel region. We formulate an optimisation problem for maximising the received power at the worst possible receiver location by considering the spherical nature of the electromagnetic (EM) wavefronts in the Fresnel region while assuming perfect knowledge of the channel at the transmitter. For the case of two transmit antennas, we derive a closed-form expression for the optimal positioning of the antennas which is purely determined by the geometry of the environment. If the room contains locations where the far-field approximation holds, the proposed positioning is shown to reduce to the far-field solution. The analytical solution is validated through simulation. Furthermore, the maximum received power at the locations yielding the worst performance is quantified and the power gain over the optimal far-field solution is presented. For the considered cuboid environment, we show that a distributed antenna system is optimal in the Fresnel region, whereas a co-located antenna architecture is ideal for the far-field.
Dissipation enhancement of cellular flows in general advection diffusion equations

The main contribution of this paper is twofold: (1) Recently, Iyer, Xu, and Zlato\v{s} studied the dissipation enhancement by cellular flows based on standard advection-diffusion equations via a stochastic method. We generalize their results to advective hyper-diffusion equations and advective nonlinear diffusion equations. (2) We prove there exist smooth incompressible flows that are relaxation enhancing to hyper-diffusion but not to standard diffusion.
HectoMAP: The Complete Redshift Survey (Data Release 2)

HectoMAP is a dense redshift survey of 95,403 galaxies based primarily on MMT spectroscopy with a median redshift z = 0.345. The survey covers 54.64 deg2 in a 1. °5 wide strip across the northern sky centered at a decl. of 43. °25. We report the redshift, the spectral indicator D n 4000, and the stellar mass. The red-selected survey is 81% complete for 55,962 galaxies with (g − r) > 1 and r < 20.5; it is 72% complete for 32,908 galaxies with (g − r) > 1, (r − i) > 0.5 and 20.5 < r < 21.3. Comparison of the survey basis SDSS photometry with the HSC-SSP photometry demonstrates that HectoMAP provides complete magnitude-limited surveys based on either photometric system. We update the comparison between the HSC-SSP photometric redshifts with HectoMAP spectroscopic redshifts; the comparison demonstrates that the HSC-SSP photometric redshifts have improved between the second and third data releases. HectoMAP is a foundation for examining the quiescent galaxy population (63% of the survey), clusters of galaxies, and the cosmic web. HectoMAP is completely covered by the HSC-SSP survey, thus enabling a variety of strong- and weak-lensing investigations.
Central limit theorem for intrinsic Frechet means in smooth compact Riemannian manifolds

We prove a central limit theorem (CLT) for the Fr´echet mean of independent and identically distributed observations in a compact Riemannian manifold assuming that the population Fr´echet mean is unique. Previous general CLT results in this setting have assumed that the cut locus of the Fr´echet mean lies outside the support of the population distribution. So far as we are aware, the CLT in the present paper is the ﬁrst which allows the cut locus to have co-dimension one or two when it is included in the support of the distribution. A key part of the proof is establishing an asymptotic approximation for the parallel transport of a certain vector ﬁeld. Whether or not a non-standard term arises in the CLT depends on whether the co-dimension of the cut locus is one or greater than one: in the former case a non-standard term appears but not in the latter case. This is the ﬁrst paper to give a general and explicit expression for the non-standard term which arises when the co-dimension of the cut locus is one.
Differentiable Data Augmentation for Contrastive Sentence Representation Learning

Fine-tuning a pre-trained language model via the contrastive learning framework with a large amount of unlabeled sentences or labeled sentence pairs is a common way to obtain high-quality sentence representations. Although the contrastive learning framework has shown its superiority on sentence representation learning over previous methods, the potential of such a framework is under-explored so far due to the simple method it used to construct positive pairs. Motivated by this, we propose a method that makes hard positives from the original training examples. A pivotal ingredient of our approach is the use of prefix that attached to a pre-trained language model, which allows for differentiable data augmentation during contrastive learning. Our method can be summarized in two steps: supervised prefix-tuning followed by joint contrastive fine-tuning with unlabeled or labeled examples. Our experiments confirm the effectiveness of our data augmentation approach. The proposed method yields significant improvements over existing methods under both semi-supervised and supervised settings. Our experiments under a low labeled data setting also show that our method is more label-efficient than the state-of-the-art contrastive learning methods.