DSGym/DSGym-Tasks huggingface dataset:
Context: Regression towards the mean represents an empirical phenomenon where an observation with a value of the predictor further away from the distribution’s mean tends to have a value of an outcome variable closer to that mean. This tendency can be explained by chance alone.

We will examine whether or not the US presidential election data exhibit the regression towards the mean phenomenon. To do this, we use Obama’s vote share in the 2008 election to predict his vote share in his 2012 reelection.

The CSV data file pres08.csv contains the election results by state of the 2008 US presidential election.
Variable Description
state: abbreviated name of the state
state.name: unabbreviated name of the state
Obama: Obama’s vote share (percentage)
McCain: McCain’s vote share (percentage)
EV: number of Electoral College votes for the state

In addition, we have the CSV file pres12.csv, which contains the election results by state of the 2012 US presidential election.
Variable Description
state: abbreviated name of the state
Obama: Obama’s vote share (percentage)
Romney: Romney’s vote share (percentage)
EV: number of Electoral College votes for the state

Question: We standardize vote shares across elections by computing their z-scores so that we can measure Obama’s electoral performance in each state relative to his average performance of that year. That is, we subtract the mean from Obama’s vote share in each election and then divide it by the standard deviation. We regress Obama’s 2012 standardized vote share on his 2008 standardized vote share. What is the estimated slope of the regression model? Please round to the nearest thousandth.
Answer: 0.983
Topics: Social science, Statistics, Correlation, Linear regression

Question: We standardize vote shares across elections by computing their z-scores so that we can measure Obama’s electoral performance in each state relative to his average performance of that year. That is, we subtract the mean from Obama’s vote share in each election and then divide it by the standard deviation.
What is the proportion of states where Obama received a greater share of standardized votes in 2012 than he did in 2008, in the bottom quartile of Obama’s 2008 (standardized) vote share? Please round to the nearest thousandth.
Answer: 0.571
Topics: Social science, Statistics, Quartile

Question: We standardize vote shares across elections by computing their z-scores so that we can measure Obama’s electoral performance in each state relative to his average performance of that year. That is, we subtract the mean from Obama’s vote share in each election and then divide it by the standard deviation.
What is the proportion of states where Obama received a greater share of standardized votes in 2012 than he did in 2008, in the top quartile of Obama’s 2008 (standardized) vote share? Please round to the nearest thousandth.
Answer: 0.461
Topics: Social science, Statistics, Quartile
Context: Consider the problem of predicting the 2000 US election results in Florida using the 1996 US election results from the same state at the county level. In Florida, there are 68 counties, and the CSV file florida.csv contains the number of votes cast for each candidate in those two elections.

Variable Description
county: county name
Clinton96: Clinton’s votes in 1996
Dole96: Dole’s votes in 1996
Perot96: Perot’s votes in 1996
Bush00: Bush’s votes in 2000
Gore00: Gore’s votes in 2000
Buchanan00: Buchanan’s votes in 2000

Question: We focus on libertarian candidates Ross Perot in 1996 and Pat Buchanan in 2000, using the votes for the former to predict the votes for the latter with a linear regression model. What is the Coefficient of determination of the model? Please round to the nearest thousandth.
The coefficient of determination is a measure of model fit and represents the proportion of variation in the outcome variable explained by the predictor. It is defined as one minus the ratio of the sum of squared residuals (SSR) to the total sum of squares (TSS).
Answer: 0.513
Topics: Social science, Statistics, Linear regression, Coefficient of determination

Question: We focus on libertarian candidates Ross Perot in 1996 and Pat Buchanan in 2000, using the votes for the former to predict the votes for the latter with a linear regression model. As the Palm Beach county is an outlier, we fit the model without Palm Beach county. What is the Coefficient of determination of the model? Please round to the nearest thousandth.
The coefficient of determination is a measure of model fit and represents the proportion of variation in the outcome variable explained by the predictor. It is defined as one minus the ratio of the sum of squared residuals (SSR) to the total sum of squares (TSS).
Answer: 0.851
Topics: Social science, Statistics, Linear regression, Coefficient of determination, Outlier
Context: Researchers conducted randomized policy experiment in India where, since the mid-1990s, one-third of village council heads have been randomly reserved for female politicians. The CSV data set women.csv contains a subset of this data from West Bengal. The policy was implemented at the level of government called Gram Panchayat or GP. Each GP contains many villages. For this study, two villages were selected at random within each GP for detailed data collection. Each observation in the data set represents a village and there are two villages associated with each GP.

Variable Description
GP: identifier for the Gram Panchayat (GP)
village: identifier for each village
reserved: binary variable indicating whether the GP was reserved for women leaders or not
female: binary variable indicating whether the GP had a female leader or not
irrigation: variable measuring the number of new or repaired irrigation facilities in the village since the reserve policy started
water: variable measuring the number of new or repaired drinking water facilities in the village since the reservation policy started

Question: To explore if female politicians are more likely to support policies about drinking water, please estimate the average causal effects of the reservation policy on the number of new or repaired drinking water facilities since the reserve policy started. Please round to the nearest hundredth.
Answer: 9.25
Topics: Social science, Causality, Average treatment effect, Interventional data
Context: Three social scientists conducted an RCT in which they investigated whether social pressure within neighborhoods increases participation. Specifically, during a primary election in the state of Michigan, they randomly assigned registered voters to receive different get-out-the-vote (GOTV) messages and examined whether sending postcards with these messages increased turnout. The researchers exploited the fact that the turnout of individual voters is public information in the United States. The GOTV message of particular interest was designed to induce social pressure by telling voters that after the election their neighbors would be informed about whether they voted in the election or not. The researchers hypothesized that such a namingand-shaming GOTV strategy would increase participation.
There are three treatment groups: voters who receive either the social pressure message, the civic duty message, or the Hawthorne effect message. The Hawthorne effect refers to the phenomenon where study subjects behave differently because they know they are being observed by researchers. The experiment also has a control group which consists of those voters receiving no message. The researchers randomly assigned each voter to one of the four groups and examined whether the voter turnout was different across the groups.

The data is in the file social.csv.
Variable Description
hhsize: household size of the voter
messages: GOTV messages the voter received (Civic Duty, Control, Neighbors, Hawthorne)
sex: sex of the voter (female or male)
yearofbirth: year of birth of the voter
primary2004: whether the voter voted in the 2004 primary election (1=voted, 0=abstained)
primary2006: whether the voter turned out in the 2006 primary election (1=voted, 0=abstained)

Question: What is the difference in the average causal effect of the Neighbors message on whether the voter voted in the 2006 primary election between those who voted in the 2004 primary election and those who did not? Please round the result to the nearest thousandth.
Answer: 0.027
Topics: Social science, Causality, Heterogenous treatment effects, Interventional data

Question: What is the average causal effect of the Neighbors message on whether the voter voted in the 2006 primary election if the voter's age was 25 in 2006? Please estimate a linear regression model with both message and age as predictors, and round the result to the nearest thousandth.
Answer: 0.064
Topics: Social science, Causality, Heterogenous treatment effects, Interventional data

Question: What is the average causal effect of the Neighbors message on whether the voter voted in the 2006 primary election if the voter's age was 65 in 2006? Please estimate a linear regression model with both message and age as predictors, and round the result to the nearest thousandth.
Answer: 0.089
Topics: Social science, Causality, Heterogenous treatment effects, Interventional data
Context: We consider how much politicians can increase their personal wealth due to holding office. Scholars investigated this question by analyzing members of Parliament (MPs) in the United Kingdom.6 The authors of the original study collected information about personal wealth at the time of death for several hundred competitive candidates who ran for office in general elections between 1950 and 1970.

The data are contained in the CSV file MPs.csv.
Variable Description
surname: surname of the candidate
firstname: first name of the candidate
party: party of the candidate (labour or tory)
ln.gross: log gross wealth at the time of death
ln.net: log net wealth at the time of death
yob: year of birth of the candidate
yod: year of death of the candidate
margin.pre: margin of the candidate’s party in the previous election
region: electoral region
margin: margin of victory (vote share)

Question: What is the average causal effect of becoming members of Parliament on the log net wealth for Tory candidates? The difference in predicted values at the point of discontinuity, i.e., a zero margin of victory, between the two regressions represents the average causal effect on personal wealth of serving as an MP. Please round the result to the nearest integer.
Answer: 255051
Topics: Social science, Causality, Average treatment effect, Discontinuity
Context: The CSV file, florentine.csv, contains an adjacency matrix whose entries represent the existence of relationships between two units (one unit represented by the row and the
other represented by the column). Specifically, there are 16 elite Florentine families in the data, leading to a 16×16 adjacency matrix. If the (i, j) entry of this adjacency matrix is 1, then it implies that the ith and jth Florentine families had a marriage relationship. In contrast, a value of 0 indicates the absence of a marriage.

Question: The closeness of a node is a measure of centrality in a network, calculated as the reciprocal of the sum of the length of the shortest paths between the node and all other nodes in the graph. What is the closeness of the GUADAGNI family? Please round the result to the nearest thousandth.
Answer: 0.022
Topics: Social science, Statistics, Network data, Centrality, Closeness

Question: The betweenness of a node is a measure of centrality in a network. According to this measure, a node is considered to be central if it is responsible for connecting other nodes. In particular, if we assume that communication between a pair of nodes occurs through the shortest path between them, a node that lies on many such shortest paths may possess special leverage within a network. For a given node v, we calculate betweenness in three steps. First, compute the proportion of the shortest paths between two other nodes, t and u, that contain v. For example, two shortest paths occur between the Albizzi family and Tornabuon family, but we want only the proportion that contain v. Second, calculate this proportion for every unique pair of nodes t and u in the graph, excluding v. Third, sum all proportions.
What is the betweenness of the BISCHERI family? Please round the result to the nearest thousandth.
Answer: 9.500
Topics: Social science, Statistics, Network data, Centrality, Betweenness
Context: We analyze Twitter-following data among US senators as directed network data. In this data set, an edge represents an instance of a senator following another senator’s Twitter account. The data consist of two files, one listing pairs of the Twitter screen names of the following and followed politicians, twitter-following.csv, and the other containing information about each politician, twitter-senator.csv.

Variable Description
Twitter-following data
following: Twitter screen name of the following senator
followed: Twitter screen name of the followed senator
Twitter senator data
screen_name: Twitter screen name
name: name of senator
party: party (D = Democrat, R = Republican, I = Independent)
state: state abbreviation

Question: What is the indegree of the senator with the greatest value of indegree? Please output an integer.
Answer: 63
Topics: Social science, Statistics, Network data, Centrality, Degree
Context: Here is a random sample of 10,000 registered voters from Florida contained in the CSV file FLVoters.csv.
Variable Description
surname: surname
county: county ID of the voter’s residence
VTD: voting district ID of the voter’s residence
age: age
gender: gender m = male and f = female
race: self-reported race

The US Census Bureau releases a list of common surnames with their frequency. For example, the most common surname was “Smith” with 2,376,206 occurrences, followed by “Johnson” and “Williams” with 1,857,160 and 1,534,042, respectively. This data set is quite comprehensive, including a total of more than 150,000 surnames that occurred at least 100 times. In addition, the census provides the relative frequencies of individual race within each surname, using a six-category self-reported race measure: non-Hispanic white, non-Hispanic black, non-Hispanic Asian and Pacific Islander, Hispanic origin, non-Hispanic American Indian and Alaskan Native, and non-Hispanic of two or more races. This census name list is contained in the CSV data file names.csv.
Variable Description
surname: surname
count: number of individuals with a specific surname
pctwhite: percentage of non-Hispanic whites among those who have a specific surname
pctblack: percentage of non-Hispanic blacks among those who have a specific surname
pctapi: percentage of non-Hispanic Asians and Pacific Islanders among those who have a specific surname
pcthispanic: percentage of Hispanic origin among those who have a specific surname
pctothers: percentage of the other racial groups among those who have a specific surname

Question: What proportion of voters whose races are black are correctly classified in their racial category? Race is considered correctly classified if the racial category with the greatest conditional probability is identical to the self-reported race. Please remove those voters who contain at least a missing value. Please round to the nearest hundredth.
Answer: 0.16
Topics: Social science, Statistics, Probability, Conditional probability

Question: What proportion of voters whose races are non-Hispanic Asians and Pacific Islanders are correctly classified in their racial category? Race is considered correctly classified if the racial category with the greatest conditional probability is identical to the self-reported race. Please remove those voters who contain at least a missing value. Please round to the nearest hundredth.
Answer: 0.56
Topics: Social science, Statistics, Probability, Conditional probability

Question: What is the false discovery rate of the white race? A voter is classified into the racial category with the greatest conditional probability P (race | surname). The false discovery rate represents the proportion of voters who are not white among those classified as white. Please remove those voters who contain at least a missing value. Please round to the nearest hundredth.
Answer: 0.20
Topics: Social science, Statistics, Probability, Conditional probability