Miracle-dz/UniFAR-FTUs huggingface dataset:
A machine learning approach to optimal Tikhonov regularization I: Affine manifolds

Despite a variety of available techniques, such as discrepancy principle, generalized cross validation, and balancing principle, the issue of the proper regularization parameter choice for inverse problems still remains one of the relevant challenges in the field. The main difficulty lies in constructing an efficient rule, allowing to compute the parameter from given noisy data without relying either on any a priori knowledge of the solution, noise level or on the manual input. In this paper, we propose a novel method based on a statistical learning theory framework to approximate the high-dimensional function, which maps noisy data to the optimal Tikhonov regularization parameter. After an offline phase where we observe samples of the noisy data-to-optimal parameter mapping, an estimate of the optimal regularization parameter is computed directly from noisy data. Our assumptions are that ground truth solutions of the inverse problem are statistically distributed in a concentrated manner on (lower-dimensional) linear subspaces and the noise is sub-gaussian. We show that for our method to be efficient, the number of previously observed samples of the noisy data-to-optimal parameter mapping needs to scale at most linearly with the dimension of the solution subspace. We provide explicit error bounds on the approximation accuracy from noisy data of unobserved optimal regularization parameters and ground truth solutions. Even though the results are more of theoretical nature, we present a recipe for the practical implementation of the approach. We conclude with presenting numerical experiments verifying our theoretical results and illustrating the superiority of our method with respect to several state-of-the-art approaches in terms of accuracy or computational time for solving inverse problems of various types.
Modeling and Forecasting the Evolution of Preferences over Time: A Hidden Markov Model of Travel Behavior

Literature suggests that preferences, as denoted by taste parameters and consideration sets, may evolve over time in response to changes in demographic and situational variables, psychological, sociological and biological constructs, and available alternatives and their attributes. However, existing representations typically overlook the influence of past experiences on present preferences. This study develops, applies and tests a hidden Markov model with a discrete choice kernel to model and forecast the evolution of individual preferences and behaviors over long-range forecasting horizons. The hidden states denote different preferences i.e. modes considered in the choice set, and sensitivity to level-of-service attributes. The evolutionary path of those hidden states is hypothesized to be a first-order Markov process. The framework is applied to study the evolution of travel mode preferences, or modality styles, over time, in response to a major change in the public transportation system. We use longitudinal travel diary from Santiago, Chile. The dataset consists of four one-week pseudo travel diaries collected before and after the introduction of Transantiago, a complete redesign of the public transportation system in the city. Our model identifies four modality styles in the population: drivers, bus users, bus-metro users, and auto-metro users. The modality styles differ in terms of the travel modes that they consider and their sensitivity to level-of-service attributes. At the population level, there are significant shifts in the distribution of individuals across modality styles before and after the change in the system, but the distribution is relatively stable in the periods after the change. Finally, a comparison between the proposed dynamic framework and comparable static frameworks reveals differences in aggregate forecasts for different policy scenarios.
REPPlab: An R package for detecting clusters and outliers using exploratory projection pursuit

Abstract The R-package REPPlab is designed to explore multivariate data sets using one-dimensional unsupervised projection pursuit. It is useful as a preprocessing step to find clusters or as an outlier detection tool for multivariate data. Except from the packages tourr and rggobi, there is no implementation of exploratory projection pursuit tools available in R. REPPlab is an R interface for the Java program EPP-lab that implements four projection indices and three biologically inspired optimization algorithms. It also proposes new tools for plotting and combining the results and specific tools for outlier detection. The functionality of the package is illustrated through some simulations and using some real data.
Permuting incomplete paired data: a novel exact and asymptotic correct randomization test

ABSTRACT Various statistical tests have been developed for testing the equality of means in matched pairs with missing values. However, most existing methods are commonly based on certain distributional assumptions such as normality, 0-symmetry or homoscedasticity of the data. The aim of this paper is to develop a statistical test that is robust against deviations from such assumptions and also leads to valid inference in case of heteroscedasticity or skewed distributions. This is achieved by applying a clever randomization approach to handle missing data. The resulting test procedure is not only shown to be asymptotically correct but is also finitely exact if the distribution of the data is invariant with respect to the considered randomization group. Its small sample performance is further studied in an extensive simulation study and compared to existing methods. Finally, an illustrative data example is analysed.
Distributional Equivalence and Structure Learning for Bow-free Acyclic Path Diagrams

We consider the problem of structure learning for bow-free acyclic path diagrams (BAPs). BAPs can be viewed as a generalization of linear Gaussian DAG models that allow for certain hidden variables. We present a first method for this problem using a greedy score-based search algorithm. We also prove some necessary and some sufficient conditions for distributional equivalence of BAPs which are used in an algorithmic ap- proach to compute (nearly) equivalent model structures. This allows us to infer lower bounds of causal effects. We also present applications to real and simulated datasets using our publicly available R-package.
Estimation of the Discontinuous Leverage Effect: Evidence from the NASDAQ Order Book

An extensive empirical literature documents a generally negative correlation, named the ? leverage effect,? between asset returns and changes of volatility. It is more challenging to establish such a return-volatility relationship for jumps in high-frequency data. We propose new nonparametric methods to assess and test for a discontinuous leverage effect ? i.e. a relation between contemporaneous jumps in prices and volatility ? in high-frequency data with market microstructure noise. We present local tests and estimators for price jumps and volatility jumps. Five years of transaction data from 320 NASDAQ firms display no negative relation between price and volatility cojumps. We show, however, that there is a strong relation between price-volatility cojumps if one conditions on the sign of price jumps and whether the price jumps are market-wide or idiosyncratic.
Variable projection for nonsmooth models

Variable projection solves structured optimization problems by completely minimizing over a subset of the variables while iterating over the remaining variables. Over the last 30 years, the technique has been widely used, with empirical and theoretical results demonstrating both greater efficacy and greater stability compared to competing approaches. Classic examples have exploited closed form projections and smoothness of the objective function. We extend the approach to broader settings, where the projection subproblems can be nonsmooth, and can only be solved inexactly by iterative methods. We present a few case studies on problems occurring frequently in machine-learning and high-dimensional inference.
Nonparametric MANOVA in Mann-Whitney effects

Multivariate analysis of variance (MANOVA) is a powerful and versatile method to infer and quantify main and interaction effects in metric multivariate multi-factor data. It is, however, neither robust against change in units nor a meaningful tool for ordinal data. Thus, we propose a novel nonparametric MANOVA. Contrary to existing rank-based procedures we infer hypotheses formulated in terms of meaningful Mann-Whitney-type effects in lieu of distribution functions. The tests are based on a quadratic form in multivariate rank effect estimators and critical values are obtained by the bootstrap. This newly developed procedure consequently provides asymptotically exact and consistent inference for general models such as the nonparametric Behrens-Fisher problem as well as multivariate one-, two-, and higher-way crossed layouts. Computer simulations in small samples confirm the reliability of the developed method for ordinal as well as metric data with covariance heterogeneity. Finally, an analysis of a real data example illustrates the applicability and correct interpretation of the results.
Detecting and quantifying causal associations in large nonlinear time series datasets

A novel causal discovery method for estimating nonlinear interdependency networks from large time series datasets. Identifying causal relationships and quantifying their strength from observational time series data are key problems in disciplines dealing with complex dynamical systems such as the Earth system or the human body. Data-driven causal inference in such systems is challenging since datasets are often high dimensional and nonlinear with limited sample sizes. Here, we introduce a novel method that flexibly combines linear or nonlinear conditional independence tests with a causal discovery algorithm to estimate causal networks from large-scale time series datasets. We validate the method on time series of well-understood physical mechanisms in the climate system and the human heart and using large-scale synthetic datasets mimicking the typical properties of real-world data. The experiments demonstrate that our method outperforms state-of-the-art techniques in detection power, which opens up entirely new possibilities to discover and quantify causal networks from time series across a range of research fields.
Effect fusion using model-based clustering

Abstract: In social and economic studies many of the collected variables are measured on a nominal scale, often with a large number of categories. The definition of categories can be ambiguous and different classification schemes using either a finer or a coarser grid are possible. Categorization has an impact when such a variable is included as covariate in a regression model: a too fine grid will result in imprecise estimates of the corresponding effects, whereas with a too coarse grid important effects will be missed, resulting in biased effect estimates and poor predictive performance. To achieve an automatic grouping of the levels of a categorical covariate with essentially the same effect, we adopt a Bayesian approach and specify the prior on the level effects as a location mixture of spiky Normal components. Model-based clustering of the effects during MCMC sampling allows to simultaneously detect categories which have essentially the same effect size and identify variables with no effect at all. Fusion of level effects is induced by a prior on the mixture weights which encourages empty components. The properties of this approach are investigated in simulation studies. Finally, the method is applied to analyse effects of high-dimensional categorical predictors on income in Austria.
Optimal compromise between incompatible conditional probability distributions, with application to Objective Bayesian Kriging

Models are often defined through conditional rather than joint distributions, but it can be difficult to check whether the conditional distributions are compatible, i.e. whether there exists a joint probability distribution which generates them. When they are compatible, a Gibbs sampler can be used to sample from this joint distribution. When they are not, the Gibbs sampling algorithm may still be applied, resulting in a “pseudo-Gibbs sampler”. We show its stationary probability distribution to be the optimal compromise between the conditional distributions, in the sense that it minimizes a mean squared misfit between them and its own conditional distributions. This allows us to perform Objective Bayesian analysis of correlation parameters in Kriging models by using univariate conditional Jeffreys-rule posterior distributions instead of the widely used multivariate Jeffreys-rule posterior. This strategy makes the full-Bayesian procedure tractable. Numerical examples show it has near-optimal frequentist performance in terms of prediction interval coverage.
Surrogate Aided Unsupervised Recovery of Sparse Signals in Single Index Models for Binary Outcomes

We consider the recovery of regression coefficients, denoted by $\boldsymbol{\beta}_0$, for a single index model (SIM) relating a binary outcome $Y$ to a set of possibly high dimensional covariates $\boldsymbol{X}$, based on a large but 'unlabeled' dataset $\mathcal{U}$, with $Y$ never observed. On $\mathcal{U}$, we fully observe $\boldsymbol{X}$ and additionally, a surrogate $S$ which, while not being strongly predictive of $Y$ throughout the entirety of its support, can forecast it with high accuracy when it assumes extreme values. Such datasets arise naturally in modern studies involving large databases such as electronic medical records (EMR) where $Y$, unlike $(\boldsymbol{X}, S)$, is difficult and/or expensive to obtain. In EMR studies, an example of $Y$ and $S$ would be the true disease phenotype and the count of the associated diagnostic codes respectively. Assuming another SIM for $S$ given $\boldsymbol{X}$, we show that under sparsity assumptions, we can recover $\boldsymbol{\beta}_0$ proportionally by simply fitting a least squares LASSO estimator to the subset of the observed data on $(\boldsymbol{X}, S)$ restricted to the extreme sets of $S$, with $Y$ imputed using the surrogacy of $S$. We obtain sharp finite sample performance bounds for our estimator, including deterministic deviation bounds and probabilistic guarantees. We demonstrate the effectiveness of our approach through multiple simulation studies, as well as by application to real data from an EMR study conducted at the Partners HealthCare Systems.
Maximum regularized likelihood estimators: A general prediction theory and applications

Maximum regularized likelihood estimators (MRLEs) are arguably the most established class of estimators in high‐dimensional statistics. In this paper, we derive guarantees for MRLEs in the Kullback–Leibler divergence, a general measure of prediction accuracy. We assume only that the densities have a convex parametrization and that the regularization is definite and positive homogenous. The results thus apply to a very large variety of models and estimators, such as tensor regression and graphical models with convex and non‐convex regularized methods. A main conclusion is that MRLEs are broadly consistent in prediction—regardless of whether restricted eigenvalues or similar conditions hold. Copyright © 2018 John Wiley & Sons, Ltd.
Variable Selection in Causal Inference using a Simultaneous Penalization Method

Abstract In the causal adjustment setting, variable selection techniques based only on the outcome or only on the treatment allocation model can result in the omission of confounders and hence may lead to bias, or the inclusion of spurious variables and hence cause variance inflation, in estimation of the treatment effect. We propose a variable selection method using a penalized objective function that is based on both the outcome and treatment assignment models. The proposed method facilitates confounder selection in high-dimensional settings. We show that under some mild conditions our method attains the oracle property. The selected variables are used to form a doubly robust regression estimator of the treatment effect. Using the proposed method we analyze a set of data on economic growth and study the effect of life expectancy as a measure of population health on the average growth rate of gross domestic product per capita.
Extending hypothesis testing with persistent homology to three or more groups

We extend the work of Robinson and Turner to use hypothesis testing with persistence homology to test for measurable differences in shape between point clouds from three or more groups. Using samples of point clouds from three distinct groups, we conduct a large-scale simulation study to validate our proposed extension. We consider various combinations of groups, samples sizes and measurement errors in the simulation study, providing for each combination the percentage of $p$-values below an alpha-level of 0.05. Additionally, we apply our method to a Cardiotocography data set and find statistically significant evidence of measurable differences in shape between normal, suspect and pathologic health status groups.
Partial order similarity based on mutual information

Comparing the ranking of candidates by different voters is an important topic in social and information science with a high relevance from the point of view of practical applications. In general, ties and pairs of incomparable candidates may occur, thus, the alternative rankings are described by partial orders. Various distance measures between partial orders have already been introduced, where zero distance is corresponding to a perfect match between a pair of partial orders, and larger values signal greater differences. Here we take a different approach and propose a similarity measure based on adjusted mutual information. In general, the similarity value of unity is corresponding to exactly matching partial orders, while a low similarity is associated to a pair of independent partial orders. The time complexity of the computation of this similarity measure is $\mathcal{O}(\left|{\mathcal C}\right|^3)$ in the worst case, and $\mathcal{O}(\left|{\mathcal C}\right|^2\ln \left|{\mathcal C}\right|)$ in the typical case of partial orders corresponding to trees with constant branching number, where $\left|{\mathcal C}\right|$ denotes the number of candidates. An interesting feature of our approach is that the similarity measure is sensitive to the position of the disagreements in the ranking: Differences at the highly ranked candidates induce larger similarity drop compared to disagreements at the bottom candidates.
Likelihood Inference for Large Scale Stochastic Blockmodels With Covariates Based on a Divide-and-Conquer Parallelizable Algorithm With Communication

Abstract We consider a stochastic blockmodel equipped with node covariate information, that is, helpful in analyzing social network data. The key objective is to obtain maximum likelihood estimates of the model parameters. For this task, we devise a fast, scalable Monte Carlo EM type algorithm based on case-control approximation of the log-likelihood coupled with a subsampling approach. A key feature of the proposed algorithm is its parallelizability, by processing portions of the data on several cores, while leveraging communication of key statistics across the cores during each iteration of the algorithm. The performance of the algorithm is evaluated on synthetic datasets and compared with competing methods for blockmodel parameter estimation. We also illustrate the model on data from a Facebook derived social network enhanced with node covariate information. Supplemental materials for this article are available online.
Time Series Experiments and Causal Estimands: Exact Randomization Tests and Trading

Abstract We define causal estimands for experiments on single time series, extending the potential outcome framework to dealing with temporal data. Our approach allows the estimation of a broad class of these estimands and exact randomization-based p-values for testing causal effects, without imposing stringent assumptions. We further derive a general central limit theorem that can be used to conduct conservative tests and build confidence intervals for causal effects. Finally, we provide three methods for generalizing our approach to multiple units that are receiving the same class of treatment, over time. We test our methodology on simulated “potential autoregressions,” which have a causal interpretation. Our methodology is partially inspired by data from a large number of experiments carried out by a financial company who compared the impact of two different ways of trading equity futures contracts. We use our methodology to make causal statements about their trading methods. Supplementary materials for this article are available online.
Generalized Shock Model Based On The Frequency of Shocks: A Simple Approach

In a $\delta-$shock model, a system subject to randomly occurring shocks, the system fails when the time between two successive shocks lies below a threshold $\delta$. In this note, we study the generalization of this model where such $\delta-$shocks are accumulated and the system fails on the occurrence of $k^{th}$ such a $\delta-$shock. The probability distribution of the system failure time and the statistical characteristics are explicitly obtained. Normal approximation to the failure time distribution is proposed.
Honest data-adaptive inference for the average treatment effect under model misspecification using penalised bias-reduced double-robust estimation

The presence of confounding by high-dimensional variables complicates estimation of the average effect of a point treatment. On the one hand, it necessitates the use of variable selection strategies or more general data-adaptive high-dimensional statistical methods. On the other hand, the use of such techniques tends to result in biased estimators with a non-standard asymptotic behaviour. Double-robust estimators are vital for offering a resolution because they possess a so-called small bias property (Newey et al., 2004). This means that their bias vanishes faster than the bias in the nuisance parameter estimators when the relevant smoothing parameter goes to zero, making their performance less sensitive to smoothing (Chernozhukov et al., 2016). This property has been exploited to achieve valid (uniform) inference of the average causal effect when data-adaptive estimators of the propensity score and conditional outcome mean both converge to their respective truths at sufficiently fast rate (e.g., van der Laan, 2014; Farrell, 2015; Belloni et al., 2016). In this article, we extend this work in order to retain valid (uniform) inference when one of these estimators does not converge to the truth, regardless of which. This is done by generalising prior work for low-dimensional settings by Vermeulen and Vansteelandt (2015) to incorporate regularisation. The proposed penalised bias-reduced double-robust estimation strategy exhibits promising performance in extensive simulation studies and a data analysis, relative to competing proposals.