Talk:Selection bias

message: Might be handy if this page included some information about removing selection bias - but I don't have the knowledge.     Cached 11:42, 5 December 2005 (UTC)
replies: text:  I just started a new section on this. Tobacman 20:08, 11 January 2006 (UTC)
replies: text:  What about using techniques like difference-in-difference and PSM to mitigate selection bias. --27.104.144.248 (talk) 02:58, 22 October 2020 (UTC)
thread_title: Removing Selection Bias?

message: I believe that much of this article as written so far pertains to sampling bias, not selection bias, and hence the article clamors for substantial revision.  Properly, selection bias refers to bias in the estimation of the causal effect of a treatment because of heterogeneous selection into the group that receives the treatment. Sampling bias refers to biases in the collection of data. (See also the article on bias (statistics).) Tobacman 20:08, 11 January 2006 (UTC)
replies: text: I agree that the difference between sample and selection bias is not clearly stated. A trusted reference when describing this difference is greatly appreciated. -- steffen (09/09/08)  —Preceding unsigned comment added by 85.115.15.50 (talk) 08:49, 9 September 2008 (UTC)
replies: text: I made a distinction, but it's rather the answers from lecturers I've asked than any definite definition, since I've found none so far. However, the following text below which was there before I came needs some clarification - can not unconscious manipulation also be accidental? Mikael Häggström (talk) 17:23, 30 October 2009 (UTC)

text:  Malmquist bias ia a sample bias, not a selection bias! You are not selecting your data, you get the bias from the sampling! Generally, you cannot avoid Malmiquist bias. Only being able to observe well fainter than the population faintest object. SkZ (talk) 02:26, 12 April 2023 (UTC)
thread_title: "Selection bias" vs "sampling bias"

message: I am changing one word in the first paragraph again. Conclusions drawn from statistical analysis are inductive, not deductive. As such, inductive arguments exhibit a valence of strength. The language of logic dictates that deductive arguments are either valid or invalid, and either sound or unsound. Since only deductive arguments have validity, it does not make sense to refer to statistical inductive arguments as valid. It makes sense to refer to statistical arguments that suffer from a selection bias as weak. Weak arguments are inductive in nature and are not likely to preserve truth. Kanodin 09:35, 9 July 2007 (UTC)
replies: text: I put invalid back. I'll explain later, when I have time.--Boffob 11:14, 9 July 2007 (UTC)
replies: text: OK, I'm back. The issue here is not the inductive nature of statistical arguments, it's the validity of the underlying assumptions of the statistical analysis. Statistical arguments are inductive in nature but they should tend to the truth as the sample size increases (particularly, if one could have the entire population as a sample and were able to observe all quantities of interest, then the truth would be known with probability 1). Ignoring selection bias means relying on plainly wrong assumptions, namely that the data are a random sample from the target population (when they are not). The statistics and probabilities computed under such invalid assumptions will hence not merely be weak, they will not represent reality, in the sense that you will not obtain consistent estimators for the quantities wanted (those that relate to the target population). The big issue with selection bias, like with any sampling bias, is that merely increasing sample size without correcting the sampling technique will not alleviate inferential problems.--Boffob 14:45, 9 July 2007 (UTC)
replies: text: I see your point that you want to show that statistics that fail to account for selection bias are likely to be wrong, but it is not enough to call an argument invalid simply because it relies on plainly wrong assumptions. My central point is that using 'validity' to describe an inductive argument is wrong. "Weak" captures the spirit of what we are saying. It is only appropriate to use validity to describe deductive arguments that meet the criteria of validity. If we were to use validity to describe an argument, we would be saying that it is impossible for the premises to be true and the conclusion false. Statistical arguments, by definition, make no such assertion--often times statisticians provide a 'p' value, which describes the likelihood that the results are a statistical artifact. In fact, all statistical arguments that sample less than the entire population are invalid, not merely the defective ones. Most statistical arguments are inductive. The inductive analogue for validity is strength. A strong argument is such that if its premises are true, then the conclusion is probably true. Weak arguments are the negation: If an argument is weak, it is not the case that if the premises are true, then the conclusion is probably true. In the verbiage of logic, I see no problem with describing defective inductive arguments as weak, but I can understand why you would want more strength behind the evaluation in the article. Perhaps it would be acceptable to word it like this: "Statistical arguments that do not take selection bias into account are inductively weak, and their conclusions are likely to be wrong because the tainted sample does not represent the population." I think it would be fine to use stronger language, but using 'invalid' in a place where it does not belong hurts the credibility of the article. Kanodin 19:18, 11 July 2007 (UTC)
replies: text: It seems our quarrel is about whose technical lingo we should use. I put "invalid" instead of "weak", under an layman definition (after all, some of the premises are known to be false in the presence of selection bias), because it is concise and conveys the importance of the possible distortions caused by selection bias better than "may be weak". Your argument is that, strictly using the logic definition of "validity", this is not appropriate because statistical arguments boil down to the expression of probability of a conclusion given a set of premises in one form or another (you give the example of p-values which are the probability of observing the data, or more extreme data, given that the null hypothesis is true). I have to say that reading the article in terms of logic verbiage appears to be déformation professionnelle (my own déformation would read "may be weak" as "may converge weakly"). While I appreciate the compromise you offer, I don't see why one should burden the article through strict adherence to the vocabulary of formal logic.--Boffob 23:13, 11 July 2007 (UTC)
replies: text: There is a good possibility that someone may read this article and assume that invalid is used from a logicians standpoint. While I am averse to using invalid in the way the article prints it, it is also not essential that the article uses the word 'weak'. So, maybe another compromise would be to use neither 'weak' nor 'invalid'. The sentence could then be simplified as: "Statistical arguments that do not take selection bias into account often produce wrong conclusions." Kanodin 23:37, 12 July 2007 (UTC)
replies: text: Now we are getting somewhere. How about just replacing "invalid" with "wrong"? Here, the expression "statistical arguments" is again, sort of a logician's vocabulary. More often, in natural and social sciences, we're talking about the statistical analysis of the results of an experiment or data collected from a study, the term "argument" is not common use as far as I know. And then the qualifier "often" may not be entirely correct either, as the amount of bias induced may be relatively small in many cases (possibly more often than not, who knows), but we still want to express the fact that selection bias could completely skew the conclusions to the opposite of what they should be (which "may be weak" does not convey, unless one is a logician).--Boffob 01:18, 13 July 2007 (UTC)
replies: text: ETA: I made the change. I think it reads OK, though that doesn't leave out the possibility of a more elegant rephrase.--Boffob 01:23, 13 July 2007 (UTC)
replies: text: Looks good. Kanodin 08:38, 13 July 2007 (UTC)
thread_title:  From "invalid" to "wrong". 

message: Sherry Seethaler gives the case of the Chicago Tribune headline, "Dewey Defeats Truman" which was based in part on a telephone survey. At the time, telephones were expensive items whose owners tended to be in the elite - who favored Dewey much more than the average voter. Where should this go in the article? In the Participants section? --Uncle Ed (talk) 20:10, 20 May 2009 (UTC)
thread_title: Truman vs. Dewey

message: I moved this line below to here from after Partitioning data with knowledge of the contents of the partitions, and then analyzing them with tests designed for blindly chosen partitions, because I think the examples need more description here in order to be self-explanatory in the article, without the reader having to read all those target articles to understand why they even should be read. Mikael Häggström (talk) 15:37, 24 September 2009 (UTC)
thread_title:  Unclear relation 

message: I started an article specifically on circular analysis/double dipping. In my field of fMRI this might occur where a researcher adjusts parameters in a classifier to improve its accuracy, or stops adding pre-processing steps when the result 'reaches significance'. I think it definitely needs a section or something on this page. Also a redirect from circular analysis and double dipping. Here's the initial version of the page:
Circular Analysis is the selection of parameters of an analysis using the data to be analysed. It is often referred to as double dipping, as one uses the same data twice. Circular analysis inflates the statistical strength of results and, at the most extreme can result in a strongly significant result from noise.
===Examples===
At its most simple, it can include the decision to remove outliers, after noticing this might help improve the analysis of an experiment. The effect can be more subtle. In fMRI data, for example, considerable amounts of pre-processing is often needed. These might be applied incrementally until the analysis 'works'. Similarly, the classifiers used in a multivoxel analysis of fMRI data require parameters, which could be tuned to maximise the classification accuracy.
===Solutions===
Careful design of the analysis one plans to perform, prior to collecting the data, means the analysis choice is not affected by the data collected. Alternatively, one might decide to perfect the classification on one or two participants, and then use the analysis on the remaining participant data. Regarding the selection of classification parameters, a common method is to divide the data into two sets, and find the optimum parameter using one set and then test using this parameter value on the second set. This is a standard technique used (for example) by the princeton MVPA classification library.
===References===
Kriegeskorte, Nikolaus, et al. "Circular analysis in systems neuroscience: the dangers of double dipping." Nature neuroscience 12.5 (2009): 535-540.
Lionfish0 (talk) 10:30, 21 March 2013 (UTC)
replies: text: To whom it might concern: There is presently a discussion concerning Circular analysis. The question is if it should stay as an article or be merged. Lova Falk talk 17:48, 12 April 2013 (UTC)
thread_title:  Circular Analysis and Double Dipping 

message: Kriegeskorte, Nikolaus, et al. "Circular analysis in systems neuroscience: the dangers of double dipping." Nature neuroscience 12.5 (2009): 535-540.
Lionfish0 (talk) 10:30, 21 March 2013 (UTC)
replies: text: To whom it might concern: There is presently a discussion concerning Circular analysis. The question is if it should stay as an article or be merged. Lova Falk talk 17:48, 12 April 2013 (UTC)
thread_title: References

message: Dr. Nosenzo has reviewed this Wikipedia page, and provided us with the following comments to improve its quality:
We hope Wikipedians on this talk page can take advantage of these comments and improve the quality of the article accordingly.
We believe Dr. Nosenzo has expertise on the topic of this article, since he has published relevant scholarly research:
*Reference : Anderson, Jon E. & Burks, Stephen V. & Carpenter, Jeffrey P. & Gotte, Lorenz & Maurer, Karsten & Nosenzo, Daniele & Potter, Ruth & Rocha, Kim & Rustichini, Aldo, 2010. "Self Selection Does Not Increase Other-Regarding Preferences among Adult Laboratory Subjects, but Student Subjects May Be More Self-Regarding than Adults," IZA Discussion Papers 5389, Institute for the Study of Labor (IZA).
ExpertIdeasBot (talk) 16:34, 2 August 2016 (UTC)
thread_title:  Dr. Nosenzo's comment on this article 

message: Dr. Turon has reviewed this Wikipedia page, and provided us with the following comments to improve its quality:
We hope Wikipedians on this talk page can take advantage of these comments and improve the quality of the article accordingly.
We believe Dr. Turon has expertise on the topic of this article, since he has published relevant scholarly research:
*Reference : Dickson, Matt & Postel-Vinay, Fabien & Turon, Helene, 2014. "The Lifetime Earnings Premium in the Public Sector: The View from Europe," IZA Discussion Papers 8159, Institute for the Study of Labor (IZA).
ExpertIdeasBot (talk) 02:35, 6 September 2016 (UTC)
thread_title:  Dr. Turon's comment on this article 

message: As others have noted above, this page confounds multiple concepts. I'm particularly concerned with the section on "observer bias" and reference to "anthropic reasoning" I'm commenting on the talk page of Anthropic Principle too. I haven't yet read the Tegmark or Bostrom books sited there, but I don't think such WP:FRINGE metaphysical arguments should be presented without caveat on this page. DolyaIskrina (talk) 16:06, 2 July 2019 (UTC)
replies: text: I attributed the statement to Philosopher Nick Bostrom, which at least gives the reader some heads up that were are suddenly not talking about selection or sampling bias, but something philosophical (also some would argue cosmological). I propose moving the "observer bias" down to ==See also== I would also cut the paragraph about meteor impacting the earth, which I don't find very informative and as described prima facia wrong.DolyaIskrina (talk) 00:46, 11 July 2019 (UTC)
thread_title: Cognitive bias, statistical bias, Metaphysical bias

message: Vaccines causing autism might be a better example of Types/susceptibility bias.  — Preceding unsigned comment added by 2606:A000:825A:500:1D9:45B5:6018:DA31 (talk) 04:12, 10 November 2019 (UTC)
thread_title:  Better example?