Genre: Information/Explanation

Stylized Facts and Experimentation Charles Crabtree , Christopher J. Fariss University of Michigan Abstract : In this comment , we clarify and extend Hirschman's ( 2016 ) discussion of " stylized facts . " Our focus is on the relationship between stylized facts and experimentation . Given the continued increase in experimentation across the social sciences , we think that it is important to consider the exact role that experiments play in the production and testing of stylized facts . We make three related contributions here . First , we describe how experiments can both provide new evidence to support existing stylized facts as well as produce new stylized facts . Second , we argue that the stylized facts produced through experimentation differ from other stylized facts . Third , we extend Hirschman's ( 2016 ) definition of " stylized facts " so that it distinguishes between those that describe correlation relationships and those that describe causal relationships . Keywords : stylized facts ; experimentation ; philosophy of science ; validity ; causation ; mechanisms In this comment , we clarify and extend Hirschman's ( 2016 ) discussion on " stylized facts . " Our focus is on the relationship between stylized facts and experimentation . Given the continued increase in experimentation across the social sciences ( Jackson and Cox 2013 ) , we think that it is important to consider the exact role that experiments play in the production and testing of stylized facts . We make three related contributions here . First , we describe how experiments can both provide new evidence to support existing stylized facts as well as produce new stylized facts . Second , we argue that the stylized facts produced through experimentation differ from other stylized facts . Third , we extend Hirschman's ( 2016 ) definition of " stylized facts " so that it distinguishes between those that describe correlation relationships and those that describe causal relationships . Citation : Crabtree , Charles , and Christopher J. Fariss . 2016 . " Stylized Facts and Experimentation . " Sociological Science 3 : 910914 . Received : August 12 , 2016 Accepted : August 28 , 2016 Published : October 12 , 2016 Editor ( s ) : Jesper Sørensen , Sarah Soule DOI : 10.15195 / v3 . a39 c 2016 The AuCopyright : thor ( s ) . This open-access article has been published under a Creative Commons Attribution License , which allows unrestricted use , distribution and reproduction , in any form , as long as the original author and source have been credited . c b Creating Stylized Facts with Experiments Hirschman ( 2016 ) defines " stylized facts " as " simple empirical regularities in need of explanation " ( p. 606 ) . Although they might imply causal relationships , they are not " causal claims [ but ] typically simple associations that have yet to be ' explained , ' either through more complicated analysis ... or through some appeal to underlying mechanisms , or through some inferential technique relying on experimental or quasi-experimental manipulations " ( Hirschman 2016 : 607 ) . This suggests that the relationship between stylized facts and experimentation is unidirectional . In this conceptualization , scholars first produce stylized facts . They then conduct experiments to test whether the relationship described in the stylized fact is causal . For example , scholars might first note that voters tend to evaluate incumbent candidates based on economic conditions . They then might conduct experiments to test the causal claim implied in this stylized fact . 910 Crabtree and Fariss Robust Stylized Facts We argue , however , that the relationship between stylized facts and experimentation is frequently circular . In our conceptualization , scholars conducting experiments to test stylized facts can also contribute to the creation of new stylized facts . Continuing with the example above , this would mean that scholars conducting experiments to test whether changes in economic conditions cause changes in incumbent evaluations might also help produce new stylized facts about voting behavior . To further illustrate our point , we briefly discuss the growing experimental research program on racial bias in the United States . 1 Nearly all the studies in this program are ( implicitly or explicitly ) motivated by the stylized fact that racial minorities experience discrimination . This is a widely known empirical regularity based on decades of lived experiences , case studies , and correlational research . Until recently however , the causal effect of race on discrimination was largely unknown . In the last decade , researchers have tested this potential causal relationship with a staggering number of experiments across a dizzying range of contexts ( Butler and Crabtree 2015 ; Butler 2014 ; Bertrand and Mullainathan 2004 ; Broockman 2014 ; Grose 2011 ; Gaddis and Ghoshal 2015 ; Pager and Western 2012 ; Pager , Western , and Bonikowski 2009 ; Yinger 1995 ) . The results from each experiment , in isolation , provide additional empirical support for the stylized fact that originally motivated them – racial minorities in America are discriminated against . Taken together , the results from this experimental research program also create an additional empirical regularity . The content of this new stylized fact is being a racial minority causes discrimination . This discussion suggests that stylized facts produced by experimentation differ from other stylized facts . The key difference is in regards to their limitations . Hirschman ( 2016 ) describes two limitations of stylized facts . One is that we rarely know whether the described relationship is causal . This is why Hirschman ( 2016 ) refers to stylized facts as " claims of non-robust dependence " ( Hirschman 2016 : 607 ) . A second is that we rarely know why the relationship described exists . Stylized facts are often " explicitly agnostic as to why X is associated with Y " ( Hirschman 2016 : 607 ) . Stylized facts then are associations that invite additional empirical examination as well as theoretical explanation . Both limitations , however , do not apply to stylized facts produced by experimentation . For that subset of facts , the question of causation is settled . Questions about theoretical mechanisms , however , often still remain ( Clark and Golder 2015 ) . In other words , stylized facts produced by experiments can tell us whether X causes Y but not why it does . An Expanded Conceptualization of Stylized Facts To account for this and other differences , we propose a new conceptualization of stylized facts . In this conceptualization , " stylized facts " are " empirical regularities in search of theoretical , causal explanations " ( Hirschman 2016 : 604 ) . These facts , as discussed above , do not make causal claims and are typically produced by correlational studies . As we discuss below , these facts are not necessarily supported by internally valid findings . " Robust stylized facts , " on the other hand , are empirical sociological science | www.sociologicalscience.com 911 October 2016 | Volume 3 Crabtree and Fariss Robust Stylized Facts Table 1 : Conceptualization of stylized facts . Definition Stylized Facts Empirical regularities in search of theoretical , causal explanations Robust Stylized Facts Empirical regularities in search of theoretical explanations No No No Yes Yes Yes Makes causal claims Produced by experimentation Based on internally valid findings regularities in search of theoretical explanations . These facts make causal claims , are produced by experimentation , and are supported by internally valid findings . Table 1 compares these two types of facts to each other . 2 One additional way of thinking about these two types of stylized facts is in terms of validity . When we call something a " stylized fact , " we assume that it has external validity . We believe this because we can generalize this relationship to other persons , settings , and times ( Trochim and Donnelly 2008 : 22 ) . As Hirschman ( 2016 ) points out , we usually do not know , however , if stylized facts possess internal validity . That is , we might have expectations that X causes Y , but we do not have robust evidence that this is so . The lack of evidence for one type of validity or another can be a problem because the " validity types build on one another " ( Trochim and Donnelly 2008 : 22 ) . If we do not know that the relationship between X and Y is causal , as is usually the case with " stylized facts , " then we might gain very little by knowing that the perceived relationship generalizes across different contexts . The fact that the relationship generalizes could suggest that it is a good candidate for additional investigation , but it does not increase our confidence that the relationship is not spurious . In the case of " robust stylized facts , " however , we have strong evidence that the relationship is not spurious . This means that the generalization tells us more about the real world . Why Does This Matter ? We consider it very important to distinguish between these two types of facts . When developing theory , researchers often treat stylized facts as either explanada or empirical inputs ( Most and Starr 1989 ) . This means that in many cases , theories are either motivated by or based on claims that " are taken as empiricial truths without real demonstration " ( Snidal 2008 : 330 ) . While this might be acceptable in the early stages of theory development , researchers should eventually pursue " stylized facts that have more accurate empirical content " ( Snidal 2008 : 331 ) . Robust stylized facts can provide this internally valid content . In this brief comment , we have clarified and extended Hirschman ( 2016 ) ' s discussion of stylized facts . We have shown how experiments can be used to produce stylized facts , highlighted the particular characteristics of these facts , and proposed a new conceptualization that distinguishes between stylized facts based sociological science | www.sociologicalscience.com 912 October 2016 | Volume 3 Crabtree and Fariss Robust Stylized Facts on correlational research and stylized facts produced by experimentation . This distinction matters : stylized facts provide evidence of external validity but not the internal validity obtained from an experiment or the deductive validity obtained from a logically rigorous theory . Robust stylized facts , however , provide additional evidence that the empirical relationship is an internally valid one . Overall the validity of any inference ( conclusion validity ) is enhanced when evidence for each of these types of validity ( external , internal , construct , and deductive ) is obtained . 3 We hope that our remarks provide a useful complement to Hirschman ( 2016 ) and encourage others to produce robust stylized facts . Notes 1 We focus on this experimental research program only because we have both contributed to it . Our general claims apply to a broad swath of experimental research programs . 2 See Houy , Fettke , and Loos ( 2015 ) for a detailed comparison of stylized facts to scientific laws . 3 For a discussion of enhancing the validity of observational studies , see Fariss and Jones ( 2015 ) . References Bertrand , Marianne , and Sendhil Mullainathan . 2004 . " Are Emily and Greg More Employable Than Lakisha and Jamal ? A Field Experiment on Labor Market Discrimination . " American Economic Review 94 ( 4 ) : 991-1013 . http://dx.doi.org/10.1257/0002828042002561 Broockman , David E. 2014 . " Distorted communication , unequal representation : constituents communicate less to representatives not of their race . " American Journal of Political Science 58 ( 2 ) : 307-321 . http://dx.doi.org/10.1111/ajps.12068 Butler , Daniel M. 2014 . Representing the Advantaged : How Politicians Reinforce Inequality . Cambridge University Press . http://dx.doi.org/10.1017/CBO9781139871969 Butler , Daniel M. , and Charles Crabtree . 2015 . Moving Beyond Measurement : Adapting Audit Studies to Test Bias-Reducing Interventions . Working Paper . Clark , William Roberts , and Matt Golder . 2015 . " Big Data , Causal Inference , and Formal Theory : Contradictory Trends in Political Science ? " PS : Political Science & Politics 48 ( 01 ) : 6570 . http://dx.doi.org/10.1017/S1049096514001759 Fariss , Christopher J. , and Zachary M. Jones . 2015 . " Enhancing Validity in Observational Settings When Replication is Not Possible . " Political Science Research and Methods . Forthcoming . Gaddis , S Michael , and Raj Ghoshal . 2015 . " Arab American Housing Discrimination , Ethnic Competition , and the Contact Hypothesis . " The ANNALS of the American Academy of Political and Social Science 660 ( 1 ) : 282-299 . http://dx.doi.org/10.1177/0002716215580095 Grose , Christian R. 2011 . Congress in black and white : Race and representation in Washington and at home . Cambridge University Press . http://dx.doi.org/10.1017/CBO9780511976827 Hirschman , Daniel . 2016 . " Stylized Facts in the Social Sciences . " Sociological Science 3 : 604626 . http://dx.doi.org/10.15195/v3.a26 sociological science | www.sociologicalscience.com 913 October 2016 | Volume 3 Crabtree and Fariss Robust Stylized Facts Houy , Constantin , Peter Fettke , and Peter Loos . 2015 . " Stylized Facts as an Instrument for Literature Review and Cumulative Information Systems Research . " Information Systems Research 37 : 10 . Jackson , Michelle , and David R Cox . 2013 . " The principles of experimental design and their application in sociology . " Annual Review of Sociology 39 : 27-49 . http://dx.doi.org/10. 1146 / annurev-soc-071811-145443 Most , Benjamin A. , and Harvey Starr . 1989 . Inquiry , Logic , and International Politics . Columbia , SC : University of South Carolina Press . Pager , Devah , and Bruce Western . 2012 . " Identifying discrimination at work : The use of field experiments . " Journal of Social Issues 68 ( 2 ) : 221-237 . http://dx.doi.org/10.1111/j. 1540-4560 . 2012 . 01746 . x Pager , Devah , Bruce Western , and Bart Bonikowski . 2009 . " Discrimination in a low-wage labor market a field experiment . " American sociological review 74 ( 5 ) : 777-799 . http : // dx . doi.org / 10 . 1177 / 000312240907400505 Snidal , Duncan . 2008 . " Commentary on the special issue . " The Journal of Conflict Resolution 52 ( 2 ) : 326-333 . http://dx.doi.org/10.1177/0022002707313694 Trochim , William M.K. , and James P. Donnelly . 2008 . Research Methods Knowledge Base . 3rd ed . Mason , OH : Atomic Dog . Yinger , John . 1995 . Closed doors , opportunities lost : The continuing costs of housing discrimination . Russell Sage Foundation . Acknowledgements : We thank Volha Chykina and Rose McDermott for their helpful comments . Charles Crabtree : Department of Political Science , University of Michigan . E-mail : ccrabtr@umich.edu . Christopher J. Fariss : Department of Political Science , University of Michigan . E-mail : cjf0006@gmail.com . sociological science | www.sociologicalscience.com 914 October 2016 | Volume 3