nils-herrmann/clean_scirepeval huggingface dataset:
Comment: Anova as a Tool for Structuring and Understanding Hierarchical Models

I agree with McCulloch that hierarchical models (which consider the persons in the experiment as a random sample from a hypothetical population) are a good idea for repeated measures data. As McCulloch points out, this assumption is typically well motivated by the goal of extrapolating the experimental findings to the population. He also explains why classical ANOVA is not the best tool for exploring such data. However, if we think about ANOVA more broadly—as a way of structuring statisical analyses rather than as one particular set of computations—I believe a unification is possible that will give us the benefits of hierarchical modeling (efficient estimation, even under imbalance, missing data, nonnormality, and other realistic data conditions, as discussed by McCulloch), while also preserving the benefits of ANOVA (the summary of a complicated model in terms of batches of coefficients and variance parameters). In my own areas of application, I have not found much use for F-tests and pvalues, but I have found concepts, such as decomposition of degrees of freedom, and estimation of the importance of different components of variation, to be helpful. In the embrace of maximum likelihood (or, more generally, Bayesian) estimation, I do not want to lose these helpful summaries. So, how can we get the most out of ANOVA in a likelihood/Bayesian modeling context? Each row of the ANOVA table corresponds to, and labels, a different batch of coefficients in the linear model. For example, McCulloch’s Table 1 has five rows, and his equation (1) has five subsetted coefficients. I would like to see, for each row of the ANOVA table, an estimate of the standard deviation of its batch of coefficients. This idea is discussed fully in Gelman (2005); for an example,  Thus, I agree with McCulloch’s point that hierarchical models and likelihood-based estimation can work better than classical ANOVA, especially in complex settings; but I would like to reserve a role for the concepts of ANOVA to help us understand fitted models. The goal here is not to test hypotheses of zero effects but rather to summarize the importance of each batch of coefficients. One of the most important advantages of model-summary tools is that they can facilitate the fitting of multiple models. With regard to McCulloch’s particular application discussed here, I would be interested in seeing interactions between person and alcohol and between person and pregnenolone. These batches of interactions would represent random samples from the interactions in the entire population and thus would have additional variance components, each estimated from n-1 degrees of freedom. An ANOVA display along the lines of Figure 1. could help us understand such a model and could lead to further investigation of treatment effects of interest. Hierarchical estimation tools of the sort discussed by McCulloch are crucial in allowing us to fit these models.
Voting as a rational choice : the effect of preferences regarding the well-being of others ∗

For voters with “social” preferences, the expected utility of voting is approximately independent of the size of the electorate, suggesting that rational voter turnouts can be substantial even in large elections. Less important elections are predicted to have lower turnout, but a feedback mechanism keeps turnout at a reasonable level under a wide range of conditions. The main contributions of this paper are: (1) to show how, for an individual with both selfish and social preferences, the social preferences will dominate in rational voting in large elections; (2) to show that rational socially-motivated voting has a feedback mechanism that stabilizes turnout at reasonable levels (e.g., 50% of the electorate); (3) to link the rational social-utility model of voter turnout with survey findings on socially-motivated vote choice.
COUNTY CSOUTH CGUIDE CUNEMP CBLPCT CCRIME CCONS IBLACK IMALE ITRIAL IACTCJS IREVOKE IDETAIN IPPRIS ICDRUG

In a predictive model, what is the expected change in the outcome associated with a unit change in one of the inputs? In a linear regression model without interactions, this average predictive effect is simply a regression coefficient (with associated uncertainty). In a model with nonlinearity or interactions, however, the average predictive effect in general depends on the values of the predictors. We consider various definitions based on averages over a population distribution of the predictors, and we compute standard errors based on uncertainty in model parameters. We illustrate with a study of criminal justice data for urban counties in the United States. The outcome of interest measures whether a convicted felon received a prison sentence rather than a jail or non-custodial sentence, with predictors available at both individual and county levels. We fit three models: a hierarchical logistic regression with varying coefficients for the within-county intercepts as well as for each individual predictor; a hierarchical model with varying intercepts only; and a non-hierarchical model that ignores the multilevel nature of the data. The regression coefficients have different interpretations for the different models; in contrast, the models can be compared directly using predictive effects. Furthermore, predictive effects clarify the interplay between the individual and county predictors for the hierarchical models, as well as illustrating the relative size of varying county effects.
Practical Issues in Implementing and Understanding Bayesian Ideal Point Estimation

Logistic regression models have been used in political science for estimating ideal points of legislators and Supreme Court justices. These models present estimation and identifiability challenges, such as improper variance estimates, scale and translation invariance, reflection invariance, and issues with outliers. We address these issues using Bayesian hierarchical modeling, linear transformations, informative regression predictors, and explicit modeling for outliers. In addition, we explore new ways to usefully display inferences and check model fit.
Reliability of a commercial kit to test groundwater for arsenic in Bangladesh.

A comparison of field and laboratory measurements of arsenic in groundwater of Araihazar, Bangladesh, indicates that the most widely used field kit correctly determined the status of 88% of 799 wells relative to the local standard of 50 microg/L As. Additional tests showthatthe inconsistencies, mainly underestimates in the 50-100 microg/L As range, can be avoided by increasing the reaction time from 20 to 40 min. Despite this limitation, the field data already compiled for millions of wells by the Bangladesh Arsenic Mitigation and Water Supply Project, in combination with information on well location and depth, should prove to be extremely useful to prioritize interventions in thousands of affected villages.
Two-Stage Regression and Multilevel Modeling: A Commentary

These articles demonstrate, in several different examples, the effectiveness of two-level regression: the procedure of fitting several separate regression models, and then fitting a second, higher-level, regression to the estimated coefficients (for example, fitting a separate regression model to survey data from each of several countries, then regressing the coefficient estimates on country-level predictors). For simplicity, we will refer to the first- and second-level units as “persons” and “countries,” respectively, but our points apply more generally.
Rich State, Poor State, Red State, Blue State: What's the Matter with Connecticut?

For decades, the Democrats have been viewed as the party of the poor, with the Republicans representing the rich. Recent presidential elections, however, have shown a reverse pattern, with Democrats performing well in the richer blue states in the northeast and coasts, and Republicans dominating in the red states in the middle of the country and the south. Through multilevel modeling of individuallevel survey data and county- and state-level demographic and electoral data, we reconcile these patterns.
Probabilistic feature analysis of facial perception of emotions

According to the hypothesis of configural encoding, the spatial relationships between the parts of the face function as an additional source of information in the facial perception of emotions. The paper analyses experimental data on the perception of emotion to investigate whether there is evidence for configural encoding in the processing of facial expressions. It is argued that analysis with a probabilistic feature model has several advantages that are not implied by, for example, a generalized linear modelling approach. First, the probabilistic feature model allows us to extract empirically the facial features that are relevant in processing the face, rather than focusing on the features that were manipulated in the experiment. Second, the probabilistic feature model allows a direct test of the hypothesis of configural encoding as it explicitly formalizes a mechanism for the way in which information about separate facial features is combined in processing the face. Third, the model allows us to account for a complex data structure while still yielding parameters that have a straightforward interpretation.
Comment: Fuzzy and Bayesian p-Values and u-Values

It is a pleasure to discuss this fascinating paper, which presents a new way to express uncertainty in p-values for composite null hypotheses. I particularly like the graphical display above equation (1.2). I have some brief comments about the confidence interval for binomial proportions and then some more to say about the relationship to Bayesian p-values. Geyer and Meeden’s fuzzy p-values are related to Bayesian p-values in that they recognize the uncertainty inherent in testing a composite null hypothesis. Fuzzy and Bayesian p-values also both allow for test statistics to depend on missing data, latent data and parameters, as well as on observed data. There is the potential to generalize these ideas to graphical test statistics and connect to exploratory data analysis.
Multiple Imputation for Model Checking: Completed‐Data Plots with Missing and Latent Data

In problems with missing or latent data, a standard approach is to first impute the unobserved data, then perform all statistical analyses on the completed dataset—corresponding to the observed data and imputed unobserved data—using standard procedures for complete‐data inference. Here, we extend this approach to model checking by demonstrating the advantages of the use of completed‐data model diagnostics on imputed completed datasets. The approach is set in the theoretical framework of Bayesian posterior predictive checks (but, as with missing‐data imputation, our methods of missing‐data model checking can also be interpreted as “predictive inference” in a non‐Bayesian context). We consider the graphical diagnostics within this framework. Advantages of the completed‐data approach include: (1) One can often check model fit in terms of quantities that are of key substantive interest in a natural way, which is not always possible using observed data alone. (2) In problems with missing data, checks may be devised that do not require to model the missingness or inclusion mechanism; the latter is useful for the analysis of ignorable but unknown data collection mechanisms, such as are often assumed in the analysis of sample surveys and observational studies. (3) In many problems with latent data, it is possible to check qualitative features of the model (for example, independence of two variables) that can be naturally formalized with the help of the latent data. We illustrate with several applied examples.
Survey Weighting and Regression

In the words of Andrew Gelman, “survey weighting is a mess.” F ull weighting (that is, creating weights based on cell counts in the population contingency table) on all covariat es is straightforward enough for estimating simple quantit ies like means. However, weights can be hard to calculate and eve n with known weights it gets tricky to estimate and calculate standard errors for more complicated quantities such as regression coefficients using weighting. Moreover, as more and more covariates are used to calculate weights, the w eights can become unstable and it may make more sense to use raking weights based on the covariates’ marginal dist ributions rather than the full contingency table. Finally, the population covariate contingency table may not be avail ble, so we may be forced to use raking weights based on the covariates’ marginal distributions.
Sampling for Bayesian Computation with Large Datasets

Multilevel models are extremely useful in handling large hierarchical datasets. However, computation can be a challenge, both in storage and CPU time per iteration of Gibbs sampler or other Markov chain Monte Carlo algorithms. We propose a computational strategy based on sampling the data, computing separate posterior distributions based on each sample, and then combining these to get a consensus posterior inference. With hierarchical data structures, we perform cluster sampling into subsets with the same structures as the original data. This reduces the number of parameters as well as sample size for each separate model fit. We illustrate with examples from climate modeling and newspaper marketing.
Analysis of variance: Why it is more important than ever?

Analysis of variance (ANOVA) is an extremely important method in exploratory and confirmatory data analysis. Unfortunately, in complex problems (e.g., split-plot designs), it is not always easy to set up an appropriate ANOVA. We propose a hierarchical analysis that automatically gives the correct ANOVA comparisons even in complex scenarios. The inferences for all means and variances are performed under a model with a separate batch of effects for each row of the ANOVA table. We connect to classical ANOVA by working with finite-sample variance components: fixed and random effects models are characterized by inferences about existing levels of a factor and new levels, respectively. We also introduce a new graphical display showing inferences about the standard deviations of each batch of effects. We illustrate with two examples from our applied data analysis, first illustrating the usefulness of our hierarchical computations and displays, and second showing how the ideas of ANOVA are helpful in understanding a previously fit hierarchical model.
R2WinBUGS: A Package for Running WinBUGS from R

The R2WinBUGS package provides convenient functions to call WinBUGS from R. It automatically writes the data and scripts in a format readable by WinBUGS for processing in batch mode, which is possible since version 1.4. After the WinBUGS process has finished, it is possible either to read the resulting data into R by the package itself--which gives a compact graphical summary of inference and convergence diagnostics--or to use the facilities of the coda package for further analyses of the output. Examples are given to demonstrate the usage of this package.
Bayesian Multilevel Estimation with Poststratification: State-Level Estimates from National Polls

We fit a multilevel logistic regression model for the mean of a binary response variable conditional on poststratification cells. This approach combines the modeling approach often used in small-area estimation with the population information used in poststratification (see Gelman and Little 1997, Survey Methodology 23:127–135). To validate the method, we apply it to U.S. preelection polls for 1988 and 1992, poststratified by state, region, and the usual demographic variables. We evaluate the model by comparing it to state-level election outcomes. The multilevel model outperforms more commonly used models in political science. We envision the most important usage of this method to be not forecasting elections but estimating public opinion on a variety of issues at the state level.
Rejoinder

My article presents four general kinds of model-based graphical diagnostics: 1. Displays of raw data, with a model used to simulate the reference distribution of the displays (as . Conversely, Figure 2 illustrates the difficulty, in general, of interpreting data displays without a comparison to a reference distribution. 2. Similar displays of tests of lower-dimensional data summaries (e.g., scalars in Figure 3, and vectors in Figures 4 and 5(a)) compared to simulations from fitted model— again, with discrepancies from the simulations illustrating aspects of the data not explained by the model. 3. Residual plots—put more generally, graphs of differences between data and fitted model which, if the model is true, should follow distributions with invariance properties such as independence and zero mean. Figures 5(b), 6, and 7 illustrate how violations of these invariance properties are visibly apparent without the need for explicit comparisons to simulated replications. 4. Displays of latent data/parameters (as in Figures 8 and 9) or of completed datasets (i.e., combinations of observed and missing or latent data, as in Figure 10). In methods 1 and 2 above, the displays and test summaries are functions of data alone, with the role of the model being to define a reference distribution for comparison. Thus, if a series of models is developed for a single dataset, one would hope to see the replications looking more and more like the actual data. An attractive example appears in Ripley (1988, chap. 6).
Using Image and Curve Registration for Measuring the Goodness of Fit of Spatial and Temporal Predictions

Conventional measures of model fit for indexed data (e.g., time series or spatial data) summarize errors in y, for instance by integrating (or summing) the squared difference between predicted and measured values over a range of x. We propose an approach which recognizes that errors can occur in the x‐direction as well. Instead of just measuring the difference between the predictions and observations at each site (or time), we first “deform” the predictions, stretching or compressing along the x‐direction or directions, so as to improve the agreement between the observations and the deformed predictions. Error is then summarized by (a) the amount of deformation in x, and (b) the remaining difference in y between the data and the deformed predictions (i.e., the residual error in y after the deformation). A parameter, λ, controls the tradeoff between (a) and (b), so that as λ→∞ no deformation is allowed, whereas for λ= 0 the deformation minimizes the errors in y. In some applications, the deformation itself is of interest because it characterizes the (temporal or spatial) structure of the errors. The optimal deformation can be computed by solving a system of nonlinear partial differential equations, or, for a unidimensional index, by using a dynamic programming algorithm. We illustrate the procedure with examples from nonlinear time series and fluid dynamics.
A broken system: the persistent patterns of reversals of death sentences in the United States

We collected data on the appeals process for all death sentences in U.S. states between 1973 and 1995. The reversal rate was high, with an estimated chance of at least two-thirds that any death sentence would be overturned by a state or federal appeals court. Multilevel regression models fit to the data by state and year indicate that high reversal rates are strongly associated with higher death-sentencing rates and lower rates of apprehending and imprisoning violent offenders. In light of our empirical findings, we discuss potential remedies including “streamlining” the appeals process and restricting the death penalty to the “worst of the worst” offenders.
Parameterization and Bayesian Modeling

Progress in statistical computation often leads to advances in statistical modeling. For example, it is surprisingly common that an existing model is reparameterized, solely for computational purposes, but then this new configuration motivates a new family of models that is useful in applied statistics. One reason why this phenomenon may not have been noticed in statistics is that reparameterizations do not change the likelihood. In a Bayesian framework, however, a transformation of parameters typically suggests a new family of prior distributions. We discuss examples in censored and truncated data, mixture modeling, multivariate imputation, stochastic processes, and multilevel models.
Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper)

Various noninformative prior distributions have been suggested for scale parameters in hierarchical models. We construct a new folded-noncentral-t family of conditionally conjugate priors for hierarchical standard deviation pa- rameters, and then consider noninformative and weakly informative priors in this family. We use an example to illustrate serious problems with the inverse-gamma family of "noninformative" prior distributions. We suggest instead to use a uni- form prior on the hierarchical standard deviation, using the half-t family when the number of groups is small and in other settings where a weakly informative prior is desired. We also illustrate the use of the half-t family for hierarchical modeling of multiple variance parameters such as arise in the analysis of variance.
Reliability of the Hach kit to test groundwater for arsenic in Bangladesh

A comparison of field and laboratory measurements of arsenic in groundwater of Araihazar, Bangladesh, indicates that the most widely-used field kit correctly determined the status of 88% of 799 wells relative to the local standard of 50 µg/L As. Additional tests show that the inconsistencies, mainly underestimates in the 50-100 µg/L As range, can be avoided by increasing the reaction time from 20 to 40 minutes. Despite this limitation, the field data already compiled for millions of wells by the Bangladesh Arsenic Mitigation and Water Supply Project (BAMWSP), in combination with information on well location and depth, should prove to be extremely useful to prioritize interventions in thousands of affected villages.
What Does 'Do Campaigns Matter?' Mean?

Scholars disagree over the extent to which presidential campaigns activate predispositions in voters or create vote preferences that could not be predicted. When campaign related information flows activate predispositions, election results are largely predetermined given balanced resources. They can be accurately forecast well before a campaign has run its course. Alternatively, campaigns may change vote outcomes beyond forcing predispositions to some equilibrium level. We find most evidence for the former: opinion poll data are consistent with Presidential campaigns activating predispositions, with fundamental variables increasing in importance as a presidential election draws near.
Exploratory Data Analysis for Complex Models

“Exploratory” and “confirmatory” data analysis can both be viewed as methods for comparing observed data to what would be obtained under an implicit or explicit statistical model. For example, many of Tukey's methods can be interpreted as checks against hypothetical linear models and Poisson distributions. In more complex situations, Bayesian methods can be useful for constructing reference distributions for various plots that are useful in exploratory data analysis. This article proposes an approach to unify exploratory data analysis with more formal statistical methods based on probability models. These ideas are developed in the context of examples from fields including psychology, medicine, and social science.