Miracle-dz/UniFAR-FTUs huggingface dataset:
Estimating local daytime population density from census and payroll data

ABSTRACT Daytime population density reflects where people commute and spend their waking hours. It carries significant weight as urban planners and engineers site transportation infrastructure and utilities, plan for disaster recovery, and assess urban vitality. Various methods with various drawbacks exist to estimate daytime population density across a metropolitan area, such as using census data, travel diaries, GPS traces, or publicly available payroll data. This study estimates the San Francisco Bay Area's tract-level daytime population density from US Census and LEHD LODES data. Estimated daytime densities are substantially more concentrated than corresponding night-time population densities, reflecting regional land use patterns. We conclude with a discussion of biases, limitations, and implications of this methodology.
Constrained maximum likelihood estimation of clusterwise linear regression models with unknown number of components

We consider an equivariant approach imposing data-driven bounds for the variances to avoid singular and spurious solutions in maximum likelihood (ML) estimation of clusterwise linear regression models. We investigate its use in the choice of the number of components and we propose a computational shortcut, which significantly reduces the computational time needed to tune the bounds on the data. In the simulation study and the two real-data applications, we show that the proposed methods guarantee a reliable assessment of the number of components compared to standard unconstrained methods, together with accurate model parameters estimation and cluster recovery.
Batch Data Processing and Gaussian Two-Armed Bandit

We consider the two-armed bandit problem as applied to data processing if there are two alternative processing methods available with different a priori unknown efficiencies. One should determine the most effective method and provide its predominant application. Gaussian two-armed bandit describes the batch, and possibly parallel, processing when the same methods are applied to sufficiently large packets of data and accumulated incomes are used for the control. If the number of packets is large enough then such control does not deteriorate the control performance, i.e. does not increase the minimax risk. For example, in case of 50 packets the minimax risk is about 2% larger than that one corresponding to one-by-one optimal processing. However, this is completely true only for methods with close efficiencies because otherwise there may be significant expected losses at the initial stage of control when both actions are applied turn-by-turn. To avoid significant losses at the initial stage of control one should take initial packets of data having smaller sizes.
Continuous-time GARCH process driven by semi-Lévy process

In this paper we study the simple semi-L\'evy driven continuous-time generalized autoregressive conditionally heteroscedastic (SS-COGARCH) process. The statistical properties of this process are characterized. This process has the potential to approximate any semi-L\'evy driven COGARCH processes. We show that the state representation of such SS-COGARCH process can be described by a random recurrence equation with periodic random coefficients. The almost sure absolute convergence of the state process is proved. The periodically stationary solution of the state process is shown which cause the volatility to be periodically stationary under some suitable conditions. Also it is shown that the increments with constant length of such SS-COGARCH process is itself a periodically correlated (PC) process. Finally, we apply some test to investigate the PC behavior of the increments (with constant length) of the simulated samples of proposed SS-COGARCH process.
On a Metropolis–Hastings importance sampling estimator

A classical approach for approximating expectations of functions w.r. t. partially known distributions is to compute the average of function values along a trajectory of a Metropolis-Hastings (MH) Markov chain. A key part in the MH algorithm is a suitable acceptance/rejection of a proposed state, which ensures the correct stationary distribution of the resulting Markov chain. However, the rejection of proposals causes highly correlated samples. In particular, when a state is rejected it is not taken any further into account. In contrast to that we consider a MH importance sampling estimator which explicitly incorporates all proposed states generated by the MH algorithm. The estimator satisfies a strong law of large numbers as well as a central limit theorem, and, in addition to that, we provide an explicit mean squared error bound. Remarkably, the asymptotic variance of the MH importance sampling estimator does not involve any correlation term in contrast to its classical counterpart. Moreover, although the analyzed estimator uses the same amount of information as the classical MH estimator, it can outperform the latter in scenarios of moderate dimensions as indicated by numerical experiments.
An accelerated variant of simulated annealing that converges under fast cooling

Given a target function $U$ to minimize on a finite state space $\mathcal{X}$, a proposal chain with generator $Q$ and a cooling schedule $T(t)$ that depends on time $t$, in this paper we study two types of simulated annealing (SA) algorithms with generators $M_{1,t}(Q,U,T(t))$ and $M_{2,t}(Q,U,T(t))$ respectively. While $M_{1,t}$ is the classical SA algorithm, we introduce a simple and greedy variant that we call $M_{2,t}$ which provably converges faster. Under any $T(t)$ that converges to $0$ and mild conditions on $Q$, the Markov chain generated by $M_{2,t}$ is weakly ergodic. When $T(t) > c_{M_2}/\log(t+1)$ follows the logarithmic cooling schedule, our proposed algorithm is strongly ergodic both in total variation and in relative entropy, and converges to the set of global minima, where $c_{M_2}$ is a constant that we explicitly identify. If $c_{M_1}$ is the optimal hill-climbing constant that appears in logarithmic cooling of $M_{1,t}$, we show that $c_{M_1} \geq c_{M_2}$ and give simple conditions under which $c_{M_1} > c_{M_2}$. Our proposed $M_{2,t}$ thus converges under a faster logarithmic cooling in this regime. The other situation that we investigate corresponds to $c_{M_1} > c_{M_2} = 0$, where we give a class of fast and non-logarithmic cooling schedule that works for $M_{2,t}$ (but not for $M_{1,t}$). To the best of our knowledge this is the first instance where strong ergodicity and convergence in relative entropy are proved for faster than logarithmic cooling. Finally, we give an algorithm to simulate $M_{2,t}$ by uniformization of Markov chains.
Particle-Based Adaptive-Lag Online Marginal Smoothing in General State-Space Models

We present a novel algorithm, an adaptive-lag smoother, approximating efficiently, in an online fashion, sequences of expectations under the marginal smoothing distributions in general state-space models. The algorithm evolves recursively a bank of estimators, one for each marginal, in resemblance with the so-called particle-based, rapid incremental smoother (PaRIS). Each estimator is propagated until a stopping criterion, measuring the fluctuations of the estimates, is met. The presented algorithm is furnished with theoretical results describing its asymptotic limit and memory usage.
Methodology for Designing Decision Support Systems for Visualising and Mitigating Supply Chain Cyber Risk from IoT Technologies

This paper proposes a methodology for designing decision support systems for visualising and mitigating the Internet of Things cyber risks. Digital technologies present new cyber risk in the supply chain which are often not visible to companies participating in the supply chains. This study investigates how the Internet of Things cyber risks can be visualised and mitigated in the process of designing business and supply chain strategies. The emerging DSS methodology present new findings on how digital technologies affect business and supply chain systems. Through epistemological analysis, the article derives with a decision support system for visualising supply chain cyber risk from Internet of Things digital technologies. Such methods do not exist at present and this represents the first attempt to devise a decision support system that would enable practitioners to develop a step by step process for visualising, assessing and mitigating the emerging cyber risk from IoT technologies on shared infrastructure in legacy supply chain systems.
Ultrahigh-Dimensional Robust and Efficient Sparse Regression Using Non-Concave Penalized Density Power Divergence

We propose a sparse regression method based on the non-concave penalized density power divergence loss function which is robust against infinitesimal contamination in very high dimensionality. Present methods of sparse and robust regression are based on $\ell _{1}$ -penalization, and their theoretical properties are not well-investigated. In contrast, we use a general class of folded concave penalties that ensure sparse recovery and consistent estimation of regression coefficients. We propose an alternating algorithm based on the Concave-Convex procedure to obtain our estimate, and demonstrate its robustness properties using influence function analysis. Under some conditions on the fixed design matrix and penalty function, we prove that this estimator possesses large-sample oracle properties in an ultrahigh-dimensional regime. The performance and effectiveness of our proposed method for parameter estimation and prediction compared to state-of-the-art are demonstrated through simulation studies.
An Information Criterion for Auxiliary Variable Selection in Incomplete Data Analysis

Statistical inference is considered for variables of interest, called primary variables, when auxiliary variables are observed along with the primary variables. We consider the setting of incomplete data analysis, where some primary variables are not observed. Utilizing a parametric model of joint distribution of primary and auxiliary variables, it is possible to improve the estimation of parametric model for the primary variables when the auxiliary variables are closely related to the primary variables. However, the estimation accuracy reduces when the auxiliary variables are irrelevant to the primary variables. For selecting useful auxiliary variables, we formulate the problem as model selection, and propose an information criterion for predicting primary variables by leveraging auxiliary variables. The proposed information criterion is an asymptotically unbiased estimator of the Kullback–Leibler divergence for complete data of primary variables under some reasonable conditions. We also clarify an asymptotic equivalence between the proposed information criterion and a variant of leave-one-out cross validation. Performance of our method is demonstrated via a simulation study and a real data example.
Metropolized Knockoff Sampling

Abstract Model-X knockoffs is a wrapper that transforms essentially any feature importance measure into a variable selection algorithm, which discovers true effects while rigorously controlling the expected fraction of false positives. A frequently discussed challenge to apply this method is to construct knockoff variables, which are synthetic variables obeying a crucial exchangeability property with the explanatory variables under study. This article introduces techniques for knockoff generation in great generality: we provide a sequential characterization of all possible knockoff distributions, which leads to a Metropolis–Hastings formulation of an exact knockoff sampler. We further show how to use conditional independence structure to speed up computations. Combining these two threads, we introduce an explicit set of sequential algorithms and empirically demonstrate their effectiveness. Our theoretical analysis proves that our algorithms achieve near-optimal computational complexity in certain cases. The techniques we develop are sufficiently rich to enable knockoff sampling in challenging models including cases where the covariates are continuous and heavy-tailed, and follow a graphical model such as the Ising model. Supplementary materials for this article are available online.
The True Cost of Stochastic Gradient Langevin Dynamics

The problem of posterior inference is central to Bayesian statistics and a wealth of Markov Chain Monte Carlo (MCMC) methods have been proposed to obtain asymptotically correct samples from the posterior. As datasets in applications grow larger and larger, scalability has emerged as a central problem for MCMC methods. Stochastic Gradient Langevin Dynamics (SGLD) and related stochastic gradient Markov Chain Monte Carlo methods offer scalability by using stochastic gradients in each step of the simulated dynamics. While these methods are asymptotically unbiased if the stepsizes are reduced in an appropriate fashion, in practice constant stepsizes are used. This introduces a bias that is often ignored. In this paper we study the mean squared error of Lipschitz functionals in strongly log- concave models with i.i. d. data of growing data set size and show that, given a batchsize, to control the bias of SGLD the stepsize has to be chosen so small that the computational cost of reaching a target accuracy is roughly the same for all batchsizes. Using a control variate approach, the cost can be reduced dramatically. The analysis is performed by considering the algorithms as noisy discretisations of the Langevin SDE which correspond to the Euler method if the full data set is used. An important observation is that the 1scale of the step size is determined by the stability criterion if the accuracy is required for consistent credible intervals. Experimental results confirm our theoretical findings.
Robust discrimination between long‐range dependence and a change in mean

In this article we introduce a robust to outliers Wilcoxon change‐point testing procedure, for distinguishing between short‐range dependent time series with a change in mean at unknown time and stationary long‐range dependent time series. We establish the asymptotic distribution of the test statistic under the null hypothesis for L1 near epoch dependent processes and show its consistency under the alternative. The Wilcoxon‐type testing procedure similarly as the CUSUM‐type testing procedure (of Berkes I., Horváth L., Kokoszka P. and Shao Q. 2006. Ann.Statist. 34:1140–1165), requires estimation of the location of a possible change‐point, and then using pre‐ and post‐break subsamples to discriminate between short and long‐range dependence. A simulation study examines the empirical size and power of the Wilcoxon‐type testing procedure in standard cases and with disturbances by outliers. It shows that in standard cases the Wilcoxon‐type testing procedure behaves equally well as the CUSUM‐type testing procedure but outperforms it in presence of outliers. We also apply both testing procedure to hydrologic data.
A statistical view on a surrogate model for estimating extreme events with an application to wind turbines

In the present paper we propose a surrogate model, which particularly aims at estimating extreme events from a vector of covariates and a suitable simulation environment. The first part introduces the model rigorously and discusses the flexibility of each of its components by drawing relations to literature within fields such as incomplete data, statistical matching, outlier detection and conditional probability estimation. In the second part of the paper we study the performance of the model in the estimation of extreme loads on an operating wind turbine from its operational statistics.
Causal inference from observational data: Estimating the effect of contributions on visitation frequency atLinkedIn

Randomized experiments (A/B testings) have become the standard way for web-facing companies to guide innovation, evaluate new products, and prioritize ideas. There are times, however, when running an experiment is too complicated (e.g., we have not built the infrastructure), costly (e.g., the intervention will have a substantial negative impact on revenue), and time-consuming (e.g., the effect may take months to materialize). Even if we can run an experiment, knowing the magnitude of the impact will significantly accelerate the product development life cycle by helping us prioritize tests and determine the appropriate traffic allocation for different treatment groups. In this setting, we should leverage observational data to quickly and cost-efficiently obtain a reliable estimate of the causal effect. Although causal inference from observational data has a long history, its adoption by data scientist in technology companies has been slow. In this paper, we rectify this by providing a brief introduction to the vast field of causal inference with a specific focus on the tools and techniques that data scientist can directly leverage. We illustrate how to apply some of these methodologies to measure the effect of contributions (e.g., post, comment, like or send private messages) on engagement metrics. Evaluating the impact of contributions on engagement through an A/B test requires encouragement design and the development of non-standard experimentation infrastructure, which can consume a tremendous amount of time and financial resources. We present multiple efficient strategies that exploit historical data to accurately estimate the contemporaneous (or instantaneous) causal effect of a user's contribution on her own and her neighbors' (i.e., the users she is connected to) subsequent visitation frequency. We apply these tools to LinkedIn data for several million members.
Geometric mean extension for data sets with zeros

There are numerous examples in different research fields where the use of the geometric mean is more appropriate than the arithmetic mean. Nevertheless, the geometric mean has a serious limitation in comparison with the arithmetic mean. Means are used to summarize the information in a large set of values in a single number; however, the geometric mean of a data set with at least one zero is always zero, as a result, the geometric mean does not capture any information about the non-zero values. The purpose of this short contribution is to review solutions proposed in the literature and their issues, followed by the formulation of a modified version of the geometric mean that can handle zeros while satisfying recovery + monotonicity properties.
Local Marchenko-Pastur Law for Random Bipartite Graphs

This paper is the first chapter of three of the author's undergraduate thesis. We study the random matrix ensemble of covariance matrices arising from random $(d_b, d_w)$-regular bipartite graphs on a set of $M$ black vertices and $N$ white vertices, for $d_b \gg \log^4 N$. We simultaneously prove that the Green's functions of these covariance matrices and the adjacency matrices of the underlying graphs agree with the corresponding limiting law (e.g. Marchenko-Pastur law for covariance matrices) down to the optimal scale. This is an improvement from the previously known mesoscopic results. We obtain eigenvector delocalization for the covariance matrix ensemble as consequence, as well as a weak rigidity estimate.
Phase Transition in the Recovery of Rank One Matrices Corrupted by Gaussian Noise

In datasets where the number of parameters is fixed and the number of samples is large, principal component analysis (PCA) is a powerful dimension reduction tool. However, in many contemporary datasets, when the number of parameters is comparable to the sample size, PCA can be misleading. A closely related problem is the following: is it possible to recover a rank-one matrix in the presence of a large amount of noise? In both situations, there is a phase transition in the eigen-structure of the matrix.
Downregulation of CHIP promotes ovarian cancer metastasis by inducing Snail‐mediated epithelial–mesenchymal transition

The epithelial–mesenchymal transition (EMT) plays a pivotal role in the conversion of early‐stage tumors into invasive malignancies. The transcription factor Snail, an extremely unstable protein whose subcellular levels are regulated by many E3 ubiquitin ligases, promotes EMT as well as associated pathological characteristics including migration, invasion, and metastasis. Through yeast two‐hybrid screening, we identified the carboxyl terminus of Hsc70‐interacting protein (CHIP) as a novel Snail ubiquitin ligase that interacts with Snail to induce ubiquitin‐mediated proteasomal degradation. Inhibition of CHIP expression increases Snail protein levels, induces EMT, and enhances in vitro migration and invasion as well as in vivo metastasis of ovarian cancer cells. In turn, Snail depletion abrogates all phenomena induced by CHIP depletion. Finally, Snail and CHIP expression is inversely correlated in ovarian tumor tissues. These findings establish the CHIP–Snail axis as a post‐translational mechanism of EMT and cancer metastasis regulation.
Sufficient Markov Decision Processes with Alternating Deep Neural Networks

Advances in mobile computing technologies have made it possible to monitor and apply data-driven interventions across complex systems in real time. Markov decision processes (MDPs) are the primary model for sequential decision problems with a large or indefinite time horizon. Choosing a representation of the underlying decision process that is both Markov and low-dimensional is non-trivial. We propose a method for constructing a low-dimensional representation of the original decision process for which: 1. the MDP model holds; 2. a decision strategy that maximizes mean utility when applied to the low-dimensional representation also maximizes mean utility when applied to the original process. We use a deep neural network to define a class of potential process representations and estimate the process of lowest dimension within this class. The method is illustrated using data from a mobile study on heavy drinking and smoking among college students.
Capturing between-tasks covariance and similarities using multivariate linear mixed models

We consider the problem of predicting several response variables using the same set of explanatory variables. This setting naturally induces a group structure over the coefficient matrix, in which every explanatory variable corresponds to a set of related coefficients. Most of the existing methods that utilize this group formation assume that the similarities between related coefficients arise solely through a joint sparsity structure. In this paper, we propose a procedure for constructing an estimator of a multivariate regression coefficient matrix that directly models and captures the within-group similarities, by employing a multivariate linear mixed model formulation, with a joint estimation of covariance matrices for coefficients and errors via penalized likelihood. Our approach, which we term Multivariate random Regression with Covariance Estimation (MrRCE) encourages structured similarity in parameters, in which coefficients for the same variable in related tasks sharing the same sign and similar magnitude. We illustrate the benefits of our approach in synthetic and real examples, and show that the proposed method outperforms natural competitors and alternative estimators under several model settings.