Miracle-dz/UniFAR-FTUs huggingface dataset:
Distributional (Single) Index Models

Abstract A Distributional (Single) Index Model (DIM) is a semiparametric model for distributional regression, that is, estimation of conditional distributions given covariates. The method is a combination of classical single-index models for the estimation of the conditional mean of a response given covariates, and isotonic distributional regression. The model for the index is parametric, whereas the conditional distributions are estimated nonparametrically under a stochastic ordering constraint. We show consistency of our estimators and apply them to a highly challenging dataset on the length of stay (LoS) of patients in intensive care units. We use the model to provide skillful and calibrated probabilistic predictions for the LoS of individual patients, which outperform the available methods in the literature.
Curvature is Key: Sub-Sampled Loss Surfaces and the Implications for Large Batch Training

We study the effect of mini-batching on the loss landscape of deep neural networks using spiked, field-dependent random matrix theory. We show that the magnitude of the extremal values of the batch Hessian are larger than those of the empirical Hessian. Our framework yields an analytical expression for the maximal SGD learning rate as a function of batch size, informing practical optimisation schemes. We use this framework to demonstrate that accepted and empirically-proven schemes for adapting the learning rate emerge as special cases of our more general framework. For stochastic second order methods and adaptive methods, we derive that the minimal damping coefficient is proportional to the ratio of the learning rate to batch size. For adaptive methods, we show that for the typical setup of small learning rate and small damping, square root learning rate scalings with increasing batch-size should be employed. We validate our claims on the VGG/WideResNet architectures on the CIFAR-$100$ and ImageNet datasets.
Accelerating Online Reinforcement Learning with Offline Datasets

Reinforcement learning provides an appealing formalism for learning control policies from experience. However, the classic active formulation of reinforcement learning necessitates a lengthy active exploration process for each behavior, making it difficult to apply in real-world settings. If we can instead allow reinforcement learning to effectively use previously collected data to aid the online learning process, where the data could be expert demonstrations or more generally any prior experience, we could make reinforcement learning a substantially more practical tool. While a number of recent methods have sought to learn offline from previously collected data, it remains exceptionally difficult to train a policy with offline data and improve it further with online reinforcement learning. In this paper we systematically analyze why this problem is so challenging, and propose a novel algorithm that combines sample-efficient dynamic programming with maximum likelihood policy updates, providing a simple and effective framework that is able to leverage large amounts of offline data and then quickly perform online fine-tuning of reinforcement learning policies. We show that our method enables rapid learning of skills with a combination of prior demonstration data and online experience across a suite of difficult dexterous manipulation and benchmark tasks.
Byzantine-Robust Learning on Heterogeneous Datasets via Resampling

In Byzantine robust distributed optimization, a central server wants to train a machine learning model over data distributed across multiple workers. However, a fraction of these workers may deviate from the prescribed algorithm and send arbitrary messages to the server. While this problem has received significant attention recently, most current defenses assume that the workers have identical data. For realistic cases when the data across workers is heterogeneous (non-iid), we design new attacks which circumvent these defenses leading to significant loss of performance. We then propose a simple resampling scheme that adapts existing robust algorithms to heterogeneous datasets at a negligible computational cost. We theoretically and experimentally validate our approach, showing that combining resampling with existing robust algorithms is effective against challenging attacks.
Unsupervised Pansharpening Based on Self-Attention Mechanism

Pansharpening is to fuse a multispectral image (MSI) of low-spatial-resolution (LR) but rich spectral characteristics with a panchromatic image (PAN) of high spatial resolution (HR) but poor spectral characteristics. Traditional methods usually inject the extracted high-frequency details from PAN into the upsampled MSI. Recent deep learning endeavors are mostly supervised assuming that the HR MSI is available, which is unrealistic especially for satellite images. Nonetheless, these methods could not fully exploit the rich spectral characteristics in the MSI. Due to the wide existence of mixed pixels in satellite images where each pixel tends to cover more than one constituent material, pansharpening at the subpixel level becomes essential. In this article, we propose an unsupervised pansharpening (UP) method in a deep-learning framework to address the abovementioned challenges based on the self-attention mechanism (SAM), referred to as UP-SAM. The contribution of this article is threefold. First, the SAM is proposed where the spatial varying detail extraction and injection functions are estimated according to the attention representations indicating spectral characteristics of the MSI with subpixel accuracy. Second, such attention representations are derived from mixed pixels with the proposed stacked attention network powered with a stick-breaking structure to meet the physical constraints of mixed pixel formulations. Third, the detail extraction and injection functions are spatial varying based on the attention representations, which largely improves the reconstruction accuracy. Extensive experimental results demonstrate that the proposed approach is able to reconstruct sharper MSI of different types, with more details and less spectral distortion compared with the state-of-the-art.
Local Search is State of the Art for Neural Architecture Search Benchmarks

Local search is one of the simplest families of algorithms in combinatorial optimization, yet it yields strong approximation guarantees for canonical NP-Complete problems such as the traveling salesman problem and vertex cover. While it is a ubiquitous algorithm in theoretical computer science, local search is often neglected in hyperparameter optimization and neural architecture search. We show that the simplest local search instantiations achieve state-of-the-art results on multiple NAS benchmarks (NASBench-101 and NASBench-201), outperforming the most popular recent NAS algorithms. However, local search fails to perform well on the much larger DARTS search space. Motivated by these observations, we present a theoretical study which characterizes the performance of local search on graph optimization problems, backed by simulation results. This may be of independent interest beyond NAS. All code and materials needed to reproduce our results are publicly available at this https URL.
A non-conforming dual approach for adaptive Trust-Region Reduced Basis approximation of PDE-constrained optimization

In this contribution we propose and rigorously analyze new variants of adaptive Trust-Region methods for parameter optimization with PDE constraints and bilateral parameter constraints. The approach employs successively enriched Reduced Basis surrogate models that are constructed during the outer optimization loop and used as model function for the Trust-Region method. Each Trust-Region sub-problem is solved with the projected BFGS method. Moreover, we propose a non-conforming dual (NCD) approach to improve the standard RB approximation of the optimality system. Rigorous improved a posteriori error bounds are derived and used to prove convergence of the resulting NCD-corrected adaptive Trust-Region Reduced Basis algorithm. Numerical experiments demonstrate that this approach enables to reduce the computational demand for large scale or multi-scale PDE constrained optimization problems significantly.
The intrinsic SFRF and sSFRF of galaxies: comparing SDSS observation with IllustrisTNG simulation

The star formation rate function (SFRF) and specific star formation rate function (sSFRF) from observations are impacted by the Eddington bias, due to uncertainties in the estimated star formation rate (SFR). We develop a novel method to correct the Eddington bias and obtain the intrinsic SFRF and sSFRF from the Sloan Digital Sky Survey (SDSS) Data Release 7. The intrinsic SFRF is in good agreement with measurements from previous data in the literature that relied on UV SFRs but its high star-forming end is slightly lower than the corresponding IR and radio tracers. We demonstrate that the intrinsic sSFRF from SDSS has a bimodal form with one peak found at sSFR ∼ 10−9.7 yr−1 representing the star-forming objects while the other peak is found at sSFR ∼ 10−12 yr−1 representing the quenched population. Furthermore, we compare our observations with the predictions from the IllustrisTNG and Illustris simulations and affirm that the “TNG” model performs much better than its predecessor. However, we show that the simulated SFRF and CSFRD of TNG simulations are highly dependent on resolution, reflecting the limitations of the model and today’s state-of-the-art simulations. We demonstrate that the bimodal, two peaked sSFRF implied by the SDSS observations does not appear in TNG regardless of the adopted box-size or resolution. This tension reflects the need for inclusion of an additional efficient quenching mechanism in the TNG model.
PlumeNet: Large-Scale Air Quality Forecasting Using A Convolutional LSTM Network

This paper presents an engine able to forecast jointly the concentrations of the main pollutants harming people's health: nitrogen dioxyde (NO2), ozone (O3) and particulate matter (PM2.5 and PM10, which are respectively the particles whose diameters are below 2.5 um and 10 um respectively). The forecasts are performed on a regular grid (the results presented in the paper are produced with a 0.5° resolution grid over Europe and the United States) with a neural network whose architecture includes convolutional LSTM blocks. The engine is fed with the most recent air quality monitoring stations measures available, weather forecasts as well as air quality physical and chemical model (AQPCM) outputs. The engine can be used to produce air quality forecasts with long time horizons, and the experiments presented in this paper show that the 4 days forecasts beat very significantly simple benchmarks. A valuable advantage of the engine is that it does not need much computing power: the forecasts can be built in a few minutes on a standard GPU. Thus, they can be updated very frequently, as soon as new air quality measures are available (generally every hour), which is not the case of AQPCMs traditionally used for air quality forecasting. The engine described in this paper relies on the same principles as a prediction engine deployed and used by Plume Labs in several products aiming at providing air quality data to individuals and businesses.
Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation

Machine translation systems with inadequate document understanding can make errors when translating dropped or neutral pronouns into languages with gendered pronouns (e.g., English). Predicting the underlying gender of these pronouns is difficult since it is not marked textually and must instead be inferred from coreferent mentions in the context. We propose a novel cross-lingual pivoting technique for automatically producing high-quality gender labels, and show that this data can be used to fine-tune a BERT classifier with 92% F1 for Spanish dropped feminine pronouns, compared with 30-51% for neural machine translation models and 54-71% for a non-fine-tuned BERT model. We augment a neural machine translation model with labels from our classifier to improve pronoun translation, while still having parallelizable translation models that translate a sentence at a time.
Efficient Path Algorithms for Clustered Lasso and OSCAR

In high dimensional regression, feature clustering by their effects on outcomes is often as important as feature selection. For that purpose, clustered Lasso and octagonal shrinkage and clustering algorithm for regression (OSCAR) are used to make feature groups automatically by pairwise $L_1$ norm and pairwise $L_\infty$ norm, respectively. This paper proposes efficient path algorithms for clustered Lasso and OSCAR to construct solution paths with respect to their regularization parameters. Despite too many terms in exhaustive pairwise regularization, their computational costs are reduced by using symmetry of those terms. Simple equivalent conditions to check subgradient equations in each feature group are derived by some graph theories. The proposed algorithms are shown to be more efficient than existing algorithms in numerical experiments.
Black hole collapse and bounce in effective loop quantum gravity

We derive effective equations with loop quantum gravity corrections for the Lemaître–Tolman–Bondi family of space-times, and use these to study quantum gravity effects in the Oppenheimer–Snyder collapse model. For this model, after the formation of a black hole with an apparent horizon, quantum gravity effects become important in the space-time region where the energy density and space-time curvature scalars become comparable to the Planck scale. These quantum gravity effects first stop the collapse of the dust matter field when its energy density reaches the Planck scale, and then cause the dust field to begin slowly expanding. Due to this continued expansion, the matter field will eventually extend beyond the apparent horizon, at which point the horizon disappears and there is no longer a black hole. There are no singularities anywhere in this space-time. In addition, in the limit that edge effects are neglected, we show that the dynamics for the interior of the star of uniform energy density follow the loop quantum cosmology effective Friedman equation for the spatially flat Friedman–Lemaître–Robertson–Walker space-time. Finally, we estimate the lifetime of the black hole, as measured by a distant observer, to be ∼(GM)2/ℓ Pl.
RL-CycleGAN: Reinforcement Learning Aware Simulation-to-Real

Deep neural network based reinforcement learning (RL) can learn appropriate visual representations for complex tasks like vision-based robotic grasping without the need for manually engineering or prior learning a perception system. However, data for RL is collected via running an agent in the desired environment, and for applications like robotics, running a robot in the real world may be extremely costly and time consuming. Simulated training offers an appealing alternative, but ensuring that policies trained in simulation can transfer effectively into the real world requires additional machinery. Simulations may not match reality, and typically bridging the simulation-to-reality gap requires domain knowledge and task-specific engineering. We can automate this process by employing generative models to translate simulated images into realistic ones. However, this sort of translation is typically task-agnostic, in that the translated images may not preserve all features that are relevant to the task. In this paper, we introduce the RL-scene consistency loss for image translation, which ensures that the translation operation is invariant with respect to the Q-values associated with the image. This allows us to learn a task-aware translation. Incorporating this loss into unsupervised domain translation, we obtain the RL-CycleGAN, a new approach for simulation-to-real-world transfer for reinforcement learning. In evaluations of RL-CycleGAN on two vision-based robotics grasping tasks, we show that RL-CycleGAN offers a substantial improvement over a number of prior methods for sim-to-real transfer, attaining excellent real-world performance with only a modest number of real-world observations.
PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models

Abstract Pivot-based neural representation models have led to significant progress in domain adaptation for NLP. However, previous research following this approach utilize only labeled data from the source domain and unlabeled data from the source and target domains, but neglect to incorporate massive unlabeled corpora that are not necessarily drawn from these domains. To alleviate this, we propose PERL: A representation learning model that extends contextualized word embedding models such as BERT (Devlin et al., 2019) with pivot-based fine-tuning. PERL outperforms strong baselines across 22 sentiment classification domain adaptation setups, improves in-domain model performance, yields effective reduced-size models, and increases model stability.1
Using the Baryonic Tully–Fisher Relation to Measure Ho

We explore the use of the baryonic Tully–Fisher relation (bTFR) as a new distance indicator. Advances in near-IR imaging and stellar population models, plus precise rotation curves, have reduced the scatter in the bTFR such that distance is the dominant source of uncertainty. Using 50 galaxies with accurate distances from Cepheids or the tip magnitude of the red giant branch, we calibrate the bTFR on a scale independent of Ho. We then apply this calibrated bTFR to 95 independent galaxies from the SPARC sample, using CosmicFlows-3 velocities, to deduce the local value of Ho. We find Ho = 75.1 ± 2.3 (stat) ±1.5 (sys) km s−1 Mpc−1.
Risk bounds when learning infinitely many response functions by ordinary linear regression

Consider the problem of learning a large number of response functions simultaneously based on the same input variables. The training data consist of a single independent random sample of the input variables drawn from a common distribution together with the associated responses. The input variables are mapped into a high-dimensional linear space, called the feature space, and the response functions are modelled as linear functionals of the mapped features, with coefficients calibrated via ordinary least squares. We provide convergence guarantees on the worst-case excess prediction risk by controlling the convergence rate of the excess risk uniformly in the response function. The dimension of the feature map is allowed to tend to infinity with the sample size. The collection of response functions, although potentiallyinfinite, is supposed to have a finite Vapnik-Chervonenkis dimension. The bound derived can be applied when building multiple surrogate models in a reasonable computing time.
Deshufflegan: A Self-Supervised Gan to Improve Structure Learning

Generative Adversarial Networks (GANs) triggered an increased interest in problem of image generation due to their improved output image quality and versatility for expansion towards new methods. Numerous GAN-based works attempt to improve generation by architectural and loss-based extensions. We argue that one of the crucial points to improve the GAN performance in terms of realism and similarity to the original data distribution is to be able to provide the model with a capability to learn the spatial structure in data. To that end, we propose the DeshuffleGAN to enhance the learning of the discriminator and the generator, via a self-supervision approach. Specifically, we introduce a deshuffling task that solves a puzzle of randomly shuffled image tiles, which in turn helps the DeshuffleGAN learn to increase its expressive capacity for spatial structure and realistic appearance. We provide experimental evidence for the performance improvement in generated images, compared to the baseline methods, which is consistently observed over two different datasets.
The Landscape of Nonconvex-Nonconcave Minimax Optimization

Minimax optimization has become a central tool for modern machine learning with applications in robust optimization, game theory and training GANs. These applications are often nonconvex-nonconcave, but the existing theory is unable to identify and deal with the fundamental difficulties posed by nonconvex-nonconcave structures. We break this historical barrier by identifying three regions of nonconvex-nonconcave bilinear minimax problems and characterizing their different solution paths. For problems where the interaction between the agents is sufficiently strong, we derive global linear convergence guarantees. Conversely when the interaction between the agents is fairly weak, we derive local linear convergence guarantees. Between these two settings, we show that limiting cycles may occur, preventing the convergence of the solution path.
An Extended Integral Unit Commitment Formulation and an Iterative Algorithm for Convex Hull Pricing

To increase market transparency, independent system operators (ISOs) have been working on minimizing uplift payments based on convex hull pricing theorems. Along this direction, in this paper, based on the analysis of specific generator features in the Midcontinent ISO (MISO) system, besides reviewing integral formulations for several special cases, we develop two integral formulations of a single generator that can capture these features. We then build a compact convex hull pricing formulation based on these integral formulations, which guarantees to obtain an exact convex hull price by solving a large-scale linear program. Meanwhile, to address the computational challenges caused by the large-scale MISO system, we propose innovative iterative algorithms with convergence properties, plus a complementary algorithm, to obtain an approximated convex hull price. The case studies on MISO instances with and without transmission constraints indicate that our algorithms lead to an exact convex hull price for all numerical studies and, the solutions can be obtained within 20 minutes.
Feature Space Saturation during Training

We propose layer saturation - a simple, online-computable method for analyzing the information processing in neural networks. First, we show that a layer's output can be restricted to the eigenspace of its variance matrix without performance loss. We propose a computationally lightweight method for approximating the variance matrix during training. From the dimension of its lossless eigenspace we derive layer saturation - the ratio between the eigenspace dimension and layer width. We show that saturation seems to indicate which layers contribute to network performance. We demonstrate how to alter layer saturation in a neural network by changing network depth, filter sizes and input resolution. Furthermore, we show that well-chosen input resolution increases network performance by distributing the inference process more evenly across the network.