Talk:Degrees of freedom (statistics)

message: The formula given under "Satterthwaite Approximation" does not occur in the reference provided ("Nonparametric Simple Regression: Smoothing Scatterplot"). Started discussion in https://stats.stackexchange.com/questions/590288/why-does-degrees-of-freedom-frac-operatornametrhh2-operatornametr to track it down.
--Yaroslavvb (talk) 04:19, 28 September 2022 (UTC)
thread_title: Missing Reference for Satterthwaite Approximation

message: As a beginner of statistics, I still don't understand what this means.. can anyone explain it in a more layman term? -- —Preceding unsigned comment added by 219.79.235.199 (talk • contribs) 
I still don't get it.. --Blakeops 19:57, 4 June 2006 (UTC)
thread_title: Confusion

message: The absolute simplest context in which degrees of freedom appears is in the problem of estimating a small finite collection of discrete densities.   For examples if you have a random variable that can take on integer values 0,1,2,3 and nothing else.  So you have 4 prior probabilities P(X=0), P(X=1), P(X=2), P(X=3)  and you know that these four must sum to 1.  The last unknown probability can be calculated from the first three if they are estimated so the fourth is not free to change on its own. But take care to note which prior can be "the fourth" is free to change in this context. We say then that this estimation problem has 3 degrees of freedom. 
Degrees of freedom is analogous to asking how many independent variables are in the equation  "a + b + c + d = 1".  There arnt 4 independent variables in that equation if it is going to remain true for real numbers.  If the domain for those variables is all the reals then there are 3 independent variables and one dependent variable.  But for example ahead of time I dont demand that "a" be the dependent variable.  I only demand that there be 3 independent variables and one dependent variable.  We call that 3 degrees of freedom.
Jeremiah Rounds
There's a really good visual demonstration of degrees of freedom in "Statistics: An Introduction using R" by Michael J. Crawley (Wiley, ISBN 13:978-0-470-02298-6) p36-37. To paraphrase:
Suppose we had a sample of 6 numbers with an average was 5. The sum of these numbers must be 30 otherwise the mean would not be 5.
|_| |_| |_| |_| |_| |_|
Fill each box in turn with a positive or negative real number. The first could be any number, for example 3.
|3| |_| |_| |_| |_| |_|
The next could be anything, say 9.
|3| |9| |_| |_| |_| |_|
The next could also be anything, say 4, 0 and 6.
|3| |9| |4| |0| |6| |_|
However, the last value can't be any number, it has to be 8 because the numbers must add to 30. There is total choice in selecting the first five numbers but none in selecting the sixth. There are five degrees of freedom when selecting six numbers. In general there are (N-1) degrees of freedom when estimating the mean from a sample of size N.
I think Crawley's description is excellent for stats beginners (like me) to be able to visualize what degrees of freedom means in a common context. This example is based on his example rather than directly copied; maybe someone with legal knowledge can decide whether this is allowed to be included! Chris--138.38.152.186 15:32, 11 June 2007 (UTC)
thread_title:  simplest context 

message: This page is a really horrible mess.  I'll be back ... next week, I hope. Michael Hardy 17:51, 8 August 2005 (UTC)
thread_title: Cleanup

message: Some of the confusion here is due to the term "Degrees of freedom" being used in different ways, with nuanced differences in meaning, in statistical theory and application.
The link [2] makes sense in that it alludes to this issue. Degrees of freedom is used to indicate the number of statistically independent pieces of information available with which to make inference. In statistical terms this is often stated as follows: Let Yi be independent and identically distributed random variables, i = 1, 2, ..., N; arising from a Normal distribution with mean 0 and variance 1 (the Standard Normal distribution).
In this special case, the sum of the squares of the Yi can be shown to have a Chi-square distribution with 'degrees of freedom' equal to N. (See e.g. 3 Thus historically in the early days of the development of statistical theory, 'degrees of freedom' was used to refer to the dimension of the space containing the data of interest.
Hopefully an expert in information theory can chime in here and solidify this use of the term 'degrees of freedom'.
I offer the following to start clarifying the different ways in which 'degrees of freedom' is used. I'll return and re-edit this over time, it's a bit complex to do in one sitting.
I would thus propose that the opening sentence be modified to read something like:
In statistics the term degrees of freedom (df) is used in three ways:  (1) to indicate the
amount of information available with which to make inference about a characteristic of
interest, (2) to indicate the complexity of a mathematical model used in the inference
process, and (3) a term referring to a characteristic of various probability distributions.
(1) Information content:  'Degrees of freedom' is a measure of the number of independent pieces of
information with which to make inference about a characteristic of interest.  Typically the pieces 
of information are called random variables and the characteristic of interest is the distribution
of those random variables, or some mathematical quantity associated with that distribution.  
More specifically, the degrees of freedom of a set of random variables is the dimension of the space 
containing the set.
One of the most important distributions is the Normal distribution (the so-called Bell Curve), 
and its mean (the point about which the random variables cluster) and standard deviation (the degree to 
which the random variables deviate from the mean) are typically the mathematical quantities of
interest.
(2) Model complexity:  'Degrees of freedom' refers to the number of parameters needed to completely
specify a mathematical model of interest.  For example, if the random variables at hand represent
some measured quantity of people such as height or weight, the model of interest might be the 
Bell Curve that best represents the measured quantities, and two parameters are needed to completely
specify the location and spread of that Bell Curve; namely the mean and the standard deviation.
More specifically, the mean and standard deviation are a pair of real numbers; thus they can be
represented by a point in two-dimensional space.  The dimension of the space that can represent
the parameters that specify a model of interest is called the 'degrees of freedom' of that model.
(3) Probability distributions:  'Degrees of freedom' refers to a characteristic of a probability
distribution, such as the first central moment of the Chi-square distribution.  Historically this
naming is related to definition (1) above, for if Y1, Y2, ..., Yn are a collection of independent
random variables each having a Standard Normal distribution, then the sum of the squares of the
random variables has a Chi-square distribution with n 'degrees of freedom'. 
In fitting a statistical model to data in N dimensions, the vector of residuals is constrained to lie in a subspace
of N-P dimensions, where P is the dimension of the space spanned by the parameters of the statistical model.
The total degrees of freedom is N.  The model degrees of freedom is P.  The residual degrees of freedom is N-P.
The ratio of the model degrees of freedom to the residual degrees of freedom is an indication of the amount
of information content of a model.  This ratio is inversely related to information content - as the amount of
information increases, the ratio of the model degrees of freedom to the residual degrees of freedom tends to zero.
This condition is important in statistical distribution theory, and is a necessary condition in many theorems
concerning the convergence of parameter estimates to the true underlying parameter value.
Rambling potential example perhaps more understandable to laypersons:
Statistical models are chosen typically to summarize or simplify the description of a complex set of information. Simplified descriptions are often extremely useful and allow decisions to be made that otherwise might be too complex. Summarizing a collection of weights of hundreds or thousands of people by modeling those weights as a Bell Curve allows many decisions to be made based on two numbers - the mean and standard deviation of the Bell Curve that best fits those hundreds or thousands of weights. For example, airplane manufacturers need to model the amount of weight that airline passengers will add to an airplane. The manufacturer can either keep a database containing the weights of all potential passengers, a very complex set of information, or they can keep two numbers, the mean and standard deviation of typical travelers. The 'degrees of freedom' of the weights of passengers is large, and complex to assess and maintain. The 'degrees of freedom' of the Bell Curve model is small and manageable: two degrees of freedom representing the mean and standard deviation of the Bell Curve. In order to estimate the mean and standard deviation, measurements of weight of some subset of potential travelers must be collected. No one would put much faith in a mean estimated from the weight of one potential traveler. Collecting the weights of several hundred or several thousand potential travelers will give a better idea of what an airplane manufacturer should plan for - thus the more data that goes in to an estimate of a distribution's parameters, the more 'degrees of freedom' or information content the estimate contains.
I offer the above to help clarify the confusion around 'degrees of freedom', but clearly much editing and refining is needed.
Smckinney2718 18:52, 3 November 2006 (UTC)
thread_title:  Degrees of Freedom confusion 

message: Who's written that fractional degrees of freedom are possible??  Why add confusion to an already confused article?  --Gak 20:59, 8 February 2007 (UTC)
replies: text: I don't know who added that, but I just added a bit on what they're used for. Michael Hardy 00:13, 9 February 2007 (UTC)
thread_title: Fractional values?

message: I couldn't work out why it said that "It has been suggested that Residual Mean Square be merged into this article or section". There is no mention of this on either discussion page, the two articles don't seem like sensible candidates for merging, and there is a proposal on the Residual Mean Square article that it should be merged into errors and residuals in statistics, which seems much more appropriate. I have therefore removed the merging proposal. If anyone objects feel free to reinstate it! Missdipsy 17:58, 13 August 2007 (UTC)
thread_title: Removed proposal to merge [[Residual Mean Square]] into this article

message: This page is a mess.  It will take time to fix.  The statement that d.f. is always one less than the sample size is nonsense. the "n − r − 1" identity is just as bad.
: 
I'll be back..... Michael Hardy 04:22, 4 October 2007 (UTC)
replies: text: I agree that the page is a mess; it isn't even close to being correct.  I was going to link to it as a reference in a discussion, but removed the link in order to avoid confusion.  Unless somebody intends to fix this page, I suggest deleting it.  — DAGwyn (talk) 00:43, 16 February 2008 (UTC)
replies: text: I agree it needs a lot of work. I think several of us have it on our list of things to do, but this must be one about the hardest concept in statistics to explain in a way that is both accessible and correct. Suggesting deleting it is a bit extreme though – there should clearly be an article on this, and at present it has a flag at the top warning the reader that it's in need of attention. Ultimately it's all Ronald Fisher's fault for introducing the term into statistics without defining it... Clearly he knew it from studying (and teaching) physics and personally I wish this article had never been split from degrees of freedom (physics and chemistry), but i wasn't around when that happened and i don't wish to revisit that decision as it would be another distraction from improving this article. Qwfp (talk) 10:05, 16 February 2008 (UTC)
thread_title:  What a mess!! 

message: I don't want to edit it outright, but there is a an error under the section "Residuals" on the line:
2. df = n-3 for cubic regression, because three points are needed to draw a cubic curve, this means 3 out of n points are lying on the curve, the rest n-3 are lying around the curve, this means standard error or aggregate fluctuations around the curve are because of n-2 points, so df = n -3 for cubic regression
You really need 4 point to draw a cubic, 3 points are only sufficient for a quadratic polynomial.
For this and other reasons, I agree with Michael Hardy that this page is in dire need of work, I just don't consider myself knowledgeable enought on this topic to do it myself.  —Preceding unsigned comment added by Colinshep (talk • contribs) 18:00, 29 November 2007 (UTC)
thread_title:  Error on Page 

message: I'm familiar with statistics, but struggle with the concept of degrees of freedom for some of the above mentioned reasons.  In particular, as nobody ever seems to clearly define it, I struggle when the term dimension is introduced.  Mathematically, dimension is a measure of linear independence (or presumably, transformed linear independence).  I can see how this plays with the mean, but when you start looking at other moments, or parameters, the term dimension seems to lose its value.  I would recommend that usage of the term dimension is done carefully, in a way that addresses these confusions directly, or alternatively, not used at all.  —Preceding unsigned comment added by 192.91.173.42 (talk) 20:04, 12 March 2008 (UTC)
thread_title:  Novice Thoughts 

message: I think that one of the places someone might come across "degrees of freedom" is in analysis of variance tables of various types, so it might be worth mentioning these specifically. I note that there is an old redirect from Analysis of variance/Degrees of freedom to Analysis of variance but the latter doesn't really mention degrees of freedom and nor does it really discuss such tables... are there any articles on this?
A second point relates to degrees of freedom in the context of likelihood ratio tests, where there is often an equivalence between the degrees of freedom of the test statistic and the number of extra parameters. This would at least allow some expansion beyond the linear-model content of the current article.
:Melcombe (talk) 13:26, 12 June 2008 (UTC)
replies: text: It might be possible to put in some R or Splus output and walk through it, talking about the significance of the df with regards to hypothesis testing etc. I've just tossed in a bit about the chi-square's df. I'm not sure how much we should be worrying about this, though, relative to getting the core of the article right. Is the article's explanation of df good as it is, now? (I think the real big problem is that as far as I can tell, even experts don't have that rigorous an idea about this, and I think there's been some tricks people have played with introducing new definitions over the years, fortunately all overlapping over the linear model bit.)--Fangz (talk) 15:24, 14 June 2008 (UTC)
thread_title:  Additions? 

message: I've started a new section on the geometric interpretation of degrees of freedom; in my view this is how d.f. should be defined, since it unifies all other definitions and interpretations. The section isn't complete yet (need to add more general examples, and relate to t and F distributions) -- will continue to work on it.
For now, leaving other sections untouched (even though they are poorly written and wrong in some places), will work on better integration later.
--Zaqrfv (talk) 23:58, 11 August 2008 (UTC)
thread_title:  Geometric interpretation, definition. 

message: I've proposed a Splitsection for Effective degrees of freedom. Reasons:
# This isn't degrees of freedom. It's about convenient distributional approximations and similar.
# This section could, and probably should, be significantly expanded, which I'm reluctant to do when it's part of a larger (and important) page. Effective d.f. are used much more widely than is discussed here, for example in variance components models and Welch's t test. --Zaqrfv (talk) 21:55, 30 August 2008 (UTC)
replies: text: Perhaps at some point this would be a good idea, but I don't see it best now, as the section is short and the article not overlong.  That said, if you or others do expand the section, perhaps it should be revisited.  Baccyak4H (Yak!) 20:09, 5 September 2008 (UTC)
replies: text: But the argument isn't about length; it's about writing good, focused, scientific articles (which has very different requirements than, say, most articles in Special:Contributions/Baccyak4H). Effective d.f. has very little to do with the main topic of this page, and so expansion before splitting is inappropriate. --Zaqrfv (talk) 08:54, 10 September 2008 (UTC)
replies: text: I'd support splitting for the reasons given, and also because this section is necessarily at a higher technical level than rest of the article, e.g. it requires an understanding of hat matrix and trace (linear algebra). (I'd prefer to keep this talk page for discussions of how to improve the article, however, rather than parenthetical asides about the general contributions of specific editors.) Qwfp (talk) 13:17, 10 September 2008 (UTC)
replies: text: The section still seems pretty short, and nothing has been done for two years. Is it OK to remove the split notice? -- Eraserhead1 <talk> 17:36, 26 July 2010 (UTC)
replies: text: So I went ahead and removed the notice.Fgnievinski (talk) 08:48, 2 November 2010 (UTC)
thread_title:  Splitsection: Effective degrees of freedom. 

message: Its nice to have a proper theoretical definition but it is useless to many readers.  Very few people relate to "dimension of the domain of a random vector" yet many people do need to work with basic statistics and know about degrees of freedom.  Lete's write for the common reader too.
As a start: "Statisticians use the terms "degrees of freedom" to describe the number of values in the final calculation of a statistic that are free to vary."   We could add: "Estimates of parameters can be based upon different amounts of information. The number of independent pieces of information that go into the estimate of a parameter is called the degrees of freedom (df). In general, the degrees of freedom of an estimate is equal to the number of independent scores that go into the estimate minus the number of parameters estimated as intermediate steps in the estimation of the parameter itself."
Please help develop a definition and discussion which is useful to the rest of us. Rlsheehan (talk) 01:57, 19 September 2008 (UTC)
replies: text: As discussed in Wikipedia:Manual of Style (mathematics)#Article introduction, an introduction for general readers has been added.  Rlsheehan (talk) 13:45, 22 September 2008 (UTC)
replies: text: What on god's earth is the following supposed to mean??
In general, the degrees of freedom of an estimate is equal to the number of independent scores that go into the estimate minus the number of parameters estimated as intermediate steps in the estimation of the parameter itself.  —Preceding unsigned comment added by 132.170.163.177 (talk) 23:39, 5 August 2009 (UTC)
thread_title: Workable definition

message: I once asked a prof who was teaching a grad level stat class, "so what are degrees of freedom anyway?" He said, "blah blah, the number of cases free to vary. I said, "but what does that mean??" and he just kept on repeating it over and over. I walked out and never went back. After 20 years I finally get it -- the vector space discussion was very helpful.
----  —Preceding unsigned comment added by 128.97.244.19 (talk) 20:54, 14 January 2009 (UTC)
thread_title: Thanks, I get it!

message: The degrees of freedom problem is often advanced as a critique of qualitative, small-n research. Case-study researchers often test a range of independent variables with a very limited number of cases. Therefore, the degrees of freedom, it is argued, are almost inevitably negative. George and Andrew Bennett (2005, "Case studies and theory development in the social sciences", Cambridge: MIT Press), however, point out that this overlooks the fact that a case study tends to consist of a lot more than just a single empirical observation. Therefore, small-n research allows for the testing of a much larger number of independent than commonly assumed. 
Would anyone mind if I added this to the article?  —Preceding unsigned comment added by Niclas M. (talk • contribs) 16:04, 4 April 2010 (UTC)
thread_title:  Degrees of freedom in qualitative small-n research 

message: I've been teaching this stuff long enough to know that a simple example is the best place to start. So, simply start with x+y=4, and I choose x to be 1, then y must 3, or if I chose x to be 2, then y must be 2. In other words, I can choose x, but once x is chosen, y is immediately determined. So I have 1 degree of freedom. The same reasoning can follow with respect to y.  — Preceding unsigned comment added by Towsiak (talk • contribs) 05:20, 2 February 2013 (UTC)
thread_title:  Why not start with something much simpler like x+y=4? 

message: The second example for understanding degrees of freedom for error in linear regression (see: http://en.wikipedia.org/wiki/Degrees_of_freedom_%28statistics%29#Linear_regression), is good but I think it is missing a crucial step. The relevant part currently reads (as of March 5, 2013):
Then the residuals
:e_i=y_i-(\widehat{a}+\widehat{b}x_i)\,
are constrained to lie within the space defined by the two equations
:e_1+\cdots+e_n=0,\,
:x_1 e_1+\cdots+x_n e_n=0.\,
One says that there are n − 2 degrees of freedom for error.
The problem is that this is the most important part of the explanation but does not in fact explain why the residuals are constrained by these two equations... The fact that the residuals add up to 0 make sense to me (the first equation). But where does the second equation come from? Why does the sum of the product of x's and residuals add up to 0? How is that derived? Finally, for the sake of a layman, perhaps it should be explicitly stated why, given two constraining equations, it follows that there are n-2 degrees of freedom. 
I would gladly help make the train of thought of this example easier to follow, but I fear my understanding is not good enough to do it. — Preceding unsigned comment added by Firth m (talk • contribs) 03:42, 5 March 2013 (UTC)
replies: text: I agree a clarification would be helpful. The origin for the second equation is described in Least_squares#Solving_the_least_squares_problem, where r = e and x_i = \partial e / \partial b. It sums to zero because you're seeking the minimum of the sum of squared residuals (see Maxima_and_minima). Fgnievinski (talk) 06:14, 5 March 2013 (UTC)
thread_title:  Linear regression example: simple? Perhaps needs further elaboration... 

message: It took me a while to wade through the https://en.wikipedia.org/wiki/Degrees_of_freedom_(statistics)#Regression_effective_degrees_of_freedom section of the article, and I think I now understand, but I'm not confident enough to make the edit myself.  I therefore submit the following text as a proposed introduction to this section for your review and approval / edit / rejection.
 In the case of smoothing matrices (e.g. a Gaussian blur function) which are more complex than simple polynomial fit, computing the effective degrees of freedom of the fit function is not straight-forward.  In these cases, it is important to estimate the Degrees of Freedom permitted by the $ H $ matrix so that the residual degrees of freedom can then be used to estimate statistical tests such as $ \chi^2 $.
Muskr (talk) 14:13, 10 March 2016 (UTC)
replies: text: Did you consult any book or article that could be cited in Wikipedia, e.g., about the "Gaussian blur" you mentioned? Other than that, your proposed sentence is innocuous as a general introduction, so please go ahead and add it to the article. Thanks. fgnievinski (talk) 15:24, 10 March 2016 (UTC)
thread_title:  Clarification of Regression Effective Degrees of Freedom 

message: The second residual vector is the least-squares projection.. Wfwps (talk) 12:26, 13 March 2023 (UTC)
thread_title:  The term least-squares projection might better be introduced before applying it or linked to a reference 

message: It is always  (X_1-\bar X)+(X_2-\bar X)+(X_3-\bar X)=0. So let  (X_1-\bar X)=2 and (X_2-\bar X)=5 then X3 is forced to be:  (X_3-\bar X)=-7. But we still don't know the values of X_1, X_2 and X_3 since \bar X is still unknown. And in order to know \bar X you need to know all the X_i. This confuses me. So how exactly does the sum being equal to zero help us??? 2003:E5:1F18:548E:289E:7610:CC0C:B301 (talk) 13:24, 14 May 2024 (UTC)
thread_title: Of random vectors