Talk:Covariance matrix

message: I have found use for the "pooled within-group covariance matrix."  See for example http://people.revoledu.com/kardi/tutorial/LDA/Numerical%20Example.html or "Analyzing Multivariate Data" by Lattin, Carroll, and Green.  This page seems a natural place to put a definition for such a thing, but it doesn't exactly fit into the flow of the page.  Suggestions?  Otherwise I'll just jump in at a future date. dfrankow (talk) 16:54, 29 December 2008 (UTC)
thread_title:  Pooled within-group covariance matrix 

message: This article starts with a big mess of slashes and sigmas and brackets... not sure if this is just my browser rendering something in a strange manner (though I'm using firefox, so I expect I'm not the only one seeing this) or quite what it is.  I don't know how to fix it either; maybe somebody else does?
—The preceding unsigned comment was added by 129.169.10.56 (talk) 16:32, 5 December 2006 (UTC)
thread_title:  New problem...

message: Okay I try to explain the idea more clearly:
if you have some set of vector measueremnts you can consider it as a cloud of points in n-dimensions. If you want to find something interesting about your data set you can look a the data from different directions, or what is  essentially the same perform a projection into 1,2, or 3 dimensions.
But there are many projections possible, which one to take ? Life is not enough to try them all ;-) One criterion you can apply (not the best one but is better than nothing and it works sometimes...) is to look at directions for which the data have large variance (this makes sense if you want to find the most "energetic" components...) I tried to explain that the covariance matrix is a tool (at least can be interpreted in this way) to represent the data variances in all possible drections in a effective and compact way. If you once understand this it is immediatly clear why it is useful to look for the eigenvalues and eigenvectors of the covariance matrix.
I can not expect to see your version of this ! ;-)
=== Rephrased ===
: I'll try my hand at the above:
: If you have many measurements of similar things (such as GNP and airplane passengers for several years), such lists of information can be manipulated with linear algebra as vectors of numbers.  If graphs are drawn of all useful combinations of these numbers, the result is many separate points on many graphs.  Comparisons between the numerous graphs can be done if all the numbers are placed within one graph with as many dimensions as there are data items, thus creating a "cloud" of points in this space.
: For example, GNP of one country for several years can be shown on a page as a graph of years by values.  That has only two dimensions.  Airline passenger numbers can be shown to the side of that graph in a three dimensional display.  Adding another measurement, such as electricity consumed, requires a fourth dimension which is often hard to visualise.
: Finding interesting information about such combinations can be difficult.  One issue to examine is which part of that cloud of points has the greatest difference.  The covariance matrix is the result of calculating distances between all the data points in all directions, and producing the direction through the cloud which has the greatest differences.  Recording the distances between different types of data within the cloud produces the covariance matrix, where larger numbers suggest greater relationships.
: (should eigenvalues and eigenvectors be introduced here, or are those considered to be parts of calculations other than covariance?) SEWilco 18:55, 15 Jan 2004 (UTC)
thread_title:  Another explanation 

message: : I'll try my hand at the above:
: If you have many measurements of similar things (such as GNP and airplane passengers for several years), such lists of information can be manipulated with linear algebra as vectors of numbers.  If graphs are drawn of all useful combinations of these numbers, the result is many separate points on many graphs.  Comparisons between the numerous graphs can be done if all the numbers are placed within one graph with as many dimensions as there are data items, thus creating a "cloud" of points in this space.
: For example, GNP of one country for several years can be shown on a page as a graph of years by values.  That has only two dimensions.  Airline passenger numbers can be shown to the side of that graph in a three dimensional display.  Adding another measurement, such as electricity consumed, requires a fourth dimension which is often hard to visualise.
: Finding interesting information about such combinations can be difficult.  One issue to examine is which part of that cloud of points has the greatest difference.  The covariance matrix is the result of calculating distances between all the data points in all directions, and producing the direction through the cloud which has the greatest differences.  Recording the distances between different types of data within the cloud produces the covariance matrix, where larger numbers suggest greater relationships.
: (should eigenvalues and eigenvectors be introduced here, or are those considered to be parts of calculations other than covariance?) SEWilco 18:55, 15 Jan 2004 (UTC)
thread_title:  Rephrased 

message: People, label yourself in your comments so we know who is talking.  Also be a little more specific about what you are pointing at.  There are too many "that", "I", and "you" for it to be clear who is talking about what.   I suggest the four-tilde signature so the date is included.  SEWilco 17:11, 15 Jan 2004 (UTC)
thread_title:  Please label your comments 

message: This article was starting to become an examplar of crackpothood.  Someone who apparently didn't like the opening paragraphs, instead of replacing them with other material, simply put the other material above those opening paragraphs, so that the article started over again, saying, in a later paragraph, "In statistics, a covariance matrix is ..." etc., and giving the same elaborate definition again, with stylistic differences.  And that eventually became the second of FOUR such iterations, with stylistic differences!  Other things were wrong too.  Why, for example was there a "stub" notice??  There should have been a "cleanup" notice right at the top, instead of a "stub" notice at the bottom.  
Inline TeX often gets misaligned or appears far too big, or both, on many browsers, but it looks as if someone went through and put perfectly good-looking non-TeX inline mathematical notation with TeX (e.g., "an n × n matrix" ---> "an n \times n matrix").  (Tex generally looks very good when "displayed", however.  And when TeX used in the normal way, as opposed to its use on Wikipedia, there's certainly no problem with inline math notation.)  Using lower-case letters for random variables is jarring, since in many cases one wants to write such things as
:FX(x) = Pr(X ≤ x)
and it is crucial to be careful about which of the "x"s above are capital and which are lower-case.  The cleanup isn't finished yet .... Michael Hardy 19:16, 28 Apr 2005 (UTC)
thread_title: Horrible mess!!

message: I don't see how var(X,Y) and cov(X,Y) "conflict".  We don't say that gamma(x) and x! or asin and arcsin "conflict" or "jar".  It is perfectly OK to have var(X)=cov(X)=cov(X,X). The fact that there is a one-argument function cov doesn't exclude there being a two-argument, related, function.  Consider, say, Γ(x)=Γ(x,0). --Macrakis 21:51, 21 December 2005 (UTC)
replies: text: Hear, hear!!  "Conflicting" notation is things like "is 0 in R^+" and "where does the 2\pi go in the Fourier Transform?".  I've toned down the expression of conflict. LachlanA 00:47, 22 January 2007 (UTC)
thread_title: Conflict of var and cov?

message: It would be nice to have a section on computational methods for calculating covariance matrices.  I am unfortunately not competant to write it.... Any volunteers? --Macrakis 21:51, 21 December 2005 (UTC)
replies: text: Basically by computation, you mean estimation, so this belongs in estimation of covariance matrices or sample mean and sample covariance Prax54 (talk) 14:43, 10 August 2012 (UTC)

text: However, it would be helpful to many readers if this article pointed out the difference between a covariance matrix for random variables and a sample covariance matrix  ( and perhaps a matrix of covariance estimators) and at least gave a link to an article on sample covariance matrices.  I think all the articles on statistical items should perform a similar service for their respective topics - e.g. sample variance vs variance of a random variable, sample mean vs mean of a random variable.  This would be extremely redundant and unnecessarily if we regard the Wikipedia as a big textbook.  From that point of view, these distinctions should be made in one introductory chapter.  But given that most readers consult invidual articles, I think it's a reasonable approach. 
Tashiro (talk) 16:43, 30 September 2012 (UTC)
thread_title: Calculation of covariance matrix

message: What are the restrictions on what matrices can be covariance matrices?  I guess the matrix has to be symmetric; is any symmetric matrix a possible covariance matrix? --Trovatore 23:11, 19 June 2006 (UTC)
replies: text: A square matrix with real entries is a covariance matrix if and only if it is non-negative definite.  If X is a column vector-valued random variable, then the expected value of XXT is the covariance matrix of the scalar compoments of X, so it should be clear why that has to be non-negative definite.  By the spectral theorem in its finite-dimensional version, every non-negative definite real matrix M has a non-negative definite real square root, which let us call M1/2.  Then let X be any column vector of the right size whose entries are random variables the variance of each of which is 1 and the covariance between any two of which is 0.  Then the covariance matrix of the entries in M1/2X is M.  So any non-negative definite matrix is a covariance matrix. Michael Hardy 17:09, 20 June 2006 (UTC)
replies: text: Suppose you know all diagonal entries and some off-diagonal, and you want to generate a nonnegative-definite symmetric matrix having those entries. Is there an efficient way to generate such a thing, in the large-finite-dimensional case? --Trovatore 22:48, 23 June 2006 (UTC)
thread_title: Possible covariance matrices

message: In this definition, 'expected value operator' (mu)is used. Per my Excel program explanation of covariance, mu is just the average. Isn't it simpler to just say that mu is just the average value of X rather than the 'expected value'? —The preceding unsigned comment was added by Steve 10-Jan-0771.121.7.79 (talk) 03:41, 11 January 2007 (UTC)
replies: text: Also, terms like mean and covariance can be used for the estimators as well as for the parameters they estimate, whereas expected value has no such ambiguity. Btyner (talk) 18:58, 13 January 2008 (UTC)
thread_title:  Expected value operator 

message: "In statistics and probability theory, the covariance matrix is a matrix of covariances between elements of a vector. It is the natural generalization to higher dimensions of the concept of the variance of a scalar-valued random variable.
If X is a column vector with n scalar random variable components, and μk is the expected value of the kth element of X, i.e., μk = E(Xk), then the covariance matrix is defined as:"
I guess many people, like me, come to visit this article because they want to do some kind of statistical analysis within their studies or other related work. Of course I can only speak for myself but the explanation that a covariance matrix is "a matrix of covariances" did not really help. Also the fact that it's a natural generalization to higher dimension of some concept didn't improve my understanding (and looking at the other comments I'm appearently not alone). Maybe somebody could add an introductory explanation or even a section in the text where this concept is explained for the uninitiated.
84.168.17.109 10:43, 7 February 2007 (UTC)
thread_title:  Introduction 

message: Apart from mathematica fans, no one - imho - would really find the Wolfram link useful. I don't see it as my place to modify it (in case there ARE lots of Mathematica fans), but perhaps the next Admin who reads this could ponder my suggestion Wrude bouie 14:02, 7 May 2007 (UTC)
replies: text: You're mistaken: many thousands of Wikipedia articles have external links to MathWorld articles, and I have found some of them useful, and I've never become skilled with mathematica software.  Why did you single out this particular article from among the thousands on Wikipedia that have external links to MathWorld articles? Michael Hardy 16:43, 7 May 2007 (UTC)
thread_title:  External Links 

message: Although the Wolfram site can be useful, it seems as though this article should point to at least a few good texts which discuss covariance matrices, their properties, their applications, in full glory. I personally only have seen them in one text, which was not a very good text for the ins and outs of covariance matrices (van Kampen); I am unenthusiastically adding it to the Further Reading section. Does anyone have anything better?
Also, it seems as though at least a little bit more development on the Wikipedia page would be nice: e.g., the mathematical and physical meaning of the "generalized variance" (determinant of Cov Matrix).
Alex Roberts 19:46, 20 May 2007 (UTC)
thread_title:  Extended References and Deeper Development 

message: You may be interested in this discussion on the relationship between the covariance matrix and the moment-of-inertia matrix. —Ben FrantzDale (talk) 20:29, 27 August 2008 (UTC)
thread_title:  FYI: Moment-of-inertia discussion 

message: The section "Which matrices are covariance matrices" mentions the existence of the matrix square root of a covariance matrix. Is there a name for this? It seems like "standard deviation matrix" or something similar would be appropriate. That is, aren't the eigenvalues of M^{1/2} the standard deviations in the principal directions and so if you wanted to draw a "confidence ellipsoid", you'd transform a sphere by M^{1/2}, just like you might draw confidence intervals at ±σ for a 1-D distribution? —Ben FrantzDale (talk) 14:37, 4 September 2008 (UTC)
replies: text: I don't think there is a general name. Note that in this context the square root is not unique and there are names associated with varios ways of defining the square root being used. For example "Cholesky decompostion" or "LU decomposition" for one computationally convenient approach, or "symmetric square root" for one based on the eigenvalue decomposition. Melcombe (talk) 13:33, 8 September 2008 (UTC)
replies: text:  Interesting. I hadn't thought about the square root being non-unique. I was assuming the root based on eigenvalue decomposition. Would Cholesky decomposition or LU decomposition even produce a meaningful square root in this context? (What would their square root mean?) —Ben FrantzDale (talk) 16:01, 12 September 2008 (UTC)
replies: text:  The "standard deviation matrix" is commonly called a whitening matrix or whitening transformation. When the covariance matrix is positive definite the Cholesky decomposition is defined, and can be used as a square-root matrix. Otherwise, when the covariance matrix is positive semi-definite (PSD), using the Singular Value Decomposition (SVD) you can produce a symmetric (numerically almost symmetric) square root matrix. See the book "An Introduction to Multivariate Statistical Analysis". Prax54 (talk) 14:21, 10 August 2012 (UTC)
thread_title:  Standard deviation matrix? 

message: How does the Hessian of the log likelihood function for a zero-mean multivariate normal distribution relate to the covariance matrix? They appear to be matrix inverses of each other, but Wikipedia has no mention of it. —Ben FrantzDale (talk) 22:11, 29 December 2008 (UTC)
replies: text: What is stated explicitly in multivariate normal distribution is the way in which the inverse of the covariance matrix occurs in the probability density function. Michael Hardy (talk) 01:34, 30 December 2008 (UTC)
thread_title:  Relation to Hessian of log likelihood? 

message: Is there any reference to the fact that covariance matrices for complex random vectors are Hermitian positive definite, could they be Hermitian positive semidefinite? Does anybody have a citation or at least an explanation of why this is?
Most of the books in probability only deal with real vectors and when they do talk about complex, they don't go into stating the properties of the covariance matrix, just define the special case when vectors are complex.
So, please, anybody with some insight about this, please.
Felipe (talk) 19:41, 7 July 2010 (UTC)
thread_title:  Complex random vectors 

message: Hmmm.. As far as I know the product X * Transpose(X) is not defined if X is a column matrix. It should be the opposite: Transpose(X) * X, shouldn't it? Yet, throughout the article, the first "notation" is used.  —Preceding unsigned comment added by 217.211.151.32 (talk) 14:19, 6 March 2011 (UTC)
replies: text: The article is correct as is.  If X is n×1 then XXT is n×n, and its ij element is xixj.  Duoduoduo (talk) 17:31, 6 March 2011 (UTC)

text: Duoduoduo is right. You just follow regular matrix-multiplication rules and wind up with a big square matrix. This operation is also known as an outer product, as opposed to X^\top X, which is an inner product or scalar product, which results in just a scalar result. —Ben FrantzDale (talk) 13:39, 7 March 2011 (UTC)
thread_title: XX<sup>T</sup>

message: Hi all, 
Since i am not a registred user i write here:
The definition of a covariance matrix as it is now suggests that X1 etc. are single elements, as the article starts with a random variable explanation.
This is misleading in the rest of the article as well as X1 being single element is not true. 
Note that COV(X,Y) is the sum  over all i in X and Y (Xi-mean(X))*(Yi-mean(Y)). Please not that the number of elements within X and Y have to be the same.
In plain english: If you have n observations from 2 variables X and Y then there is just one covariance. If you have n observations of m variables (i.e. your data has m dimensions), you will have m!/(2(m-2)!) covariances. 
To have variances and covariances convieniently in one matrix the variance-covariance matrix displays all those covariances and variances in one matrix.
I think this is a very important article being looked up a lot and it should be corrected.
Good reference on the topic, with cristal clear explanation can be found here: 
http://users.ecs.soton.ac.uk/hbr03r/pa037042.pdf
Cheers Ben  —Preceding unsigned comment added by 129.31.217.14 (talk) 23:05, 12 April 2011 (UTC)
replies: text: The article's definition is correct. Xi is indeed a scalar random variable. X1 is a different random variable from X2. Both these random variables, and all the other ones, have kmax observations  The expectations in the article are taken over the observations k=1,...,kmax  Duoduoduo (talk) 00:10, 13 April 2011 (UTC)

text: With my edit to the lede earlier today, it does mention that each of the variables Xi has a certain number of either observations or potential observations. I think that should make it clear. As for using the same letters for vectors and scalars, it simply uses boldface unsubscripted X and Y to refer to vectors, and unboldfaced subscripted X and Y to refer to scalars. This is standard, and I don't think there's any non-standard alternative that would be workable. However, I will put in something alerting the reader to the notational scheme.  Duoduoduo (talk) 19:30, 13 April 2011 (UTC)
thread_title:  Definition contains errors 

message: There are now some articles about matrix-valued random variables: e.g. matrix normal distribution, matrix t distribution. Some of these refer to a covariance matrix for these matrix random variables. It would be good to have something here about how such a thing is defined, and if there is a standard way of doing this. Clearly one can re-arrange the matrix into a vector and get a covariance matrix for this, but there must be a standard for working by rows or columns etc., and possibly a different way of treating symmetric=matrix-random-variables. Melcombe (talk) 12:26, 5 July 2012 (UTC)
thread_title:  Matrix random variables 

message: The article states "The inverse of this matrix, Σ−1 is the inverse covariance matrix, also known as the concentration matrix or precision matrix;[1] see precision (statistics). The elements of the precision matrix have an interpretation in terms of partial correlations and partial variances.[citation needed]"
It's vague as to what is meant by "an interpretation". If the elements are the partial correlations, then please just say so. I'm not certain myself, but it seems that this indeed is the interpretation given by users of graphical models, as in Friedman, Hastie, & Tibshirani, 2007:
"The basic model for continuous data assumes that the observations have a multivariate Gaussian distribution with mean μ and covariance matrix Σ. If the ijth component of Σ−1 is zero, then variables i and j are conditionally independent, given the other variables."
161.130.188.94 (talk) 15:46, 14 April 2014 (UTC)
thread_title:  Inverse precision matrix 

message: The current Wikipedia article on "Covariance Matrix" has a section on its "properties".  It would be useful to know how many of these properties are relevant and correct for the sample covariance matrix.  The sample covariance matrix could be regarded as the covariance matrix of a population consisting exactly of the sample, except for  1/(n-1) factor in the estimator.  Is that difference important for "properties"?
Tashiro (talk) 19:15, 30 November 2014 (UTC)
replies: text: The properties still hold for the covariance matrix of the sample and for that matrix adusted by the factor 1/(n–1). Loraof (talk) 17:44, 5 July 2018 (UTC)
thread_title:  Properties of the sample covariance matrix 

message: It seems to me that the second paragraph in the introduction is about something other than what I understand by "covariance matrix". By "covariance matrix" I understand a matrix where the (i,j)th element is the covariance between variables X_i and X_j, but in this paragraph there are two sets of variables and the (i,j)th element is the covariance between X_i and Y_j. This is different from a covariance matrix in many ways, and in particular, it need not be symmetric, positive definite or even square. 
From a brief glance the rest of the article seems to be in line with what I'd expect. This includes the first and third paragraphs of the introduction, which thus seem to contradict the second one. Is this use of multiple definitions intentional or should it be changed? Nathaniel Virgo (talk) 05:42, 10 October 2016 (UTC)
thread_title:  Second paragraph in the introduction 

message: I am not familiar with the E operator being used for expectation value, but that may follow from my background in physics rather than statistics. Even so, there's a structural problem with the E notation. It's not really an operator because nothing is actually being operated on. X_i itself doesn't contain any information about its own mean value, so structurally, E(X_i) is a single token variable that represents some known mean value of X_i. It certainly doesn't look like a single token, and that can lead to serious confusion on the part of the reader while peppering the page with numerous E's. Wolfram Alpha uses \left\langle\mathrm{angle brackets}\right\rangle, which I also find (slightly less) confusing for the same reason.
I suggest we use the \overline{\mathrm{overline}}, which clearly indicates that \overline{X_i} is a different variable than X_i and that you can't get \overline{X_i} by doing \sum_i^N\frac{X_i}{N}, which is tempting and wrong. It also greatly shortens several formulae. This was standard notation in my mathematical education, and I'm sure I could dig up a few textbooks that use it.
Please discuss before we make any changes. Acronymsical (talk) 15:57, 3 August 2017 (UTC)
thread_title:  Notation in definition 

message: Section "Which matrices are covariance matrices?" reads:
"From the symmetry of the covariance matrix's definition it follows that only a positive-semidefinite matrix can be a covariance matrix."
This is false: symmetry (even with nonnegative diagonal entries) does not imply positive semi-definiteness, e.g.
\begin{bmatrix}1 & 1 \\ \end{bmatrix} \begin{bmatrix}1 & -2 \\-2 & 1 \\ \end{bmatrix} \begin{bmatrix}1 \\ 1 \\ \end{bmatrix} = -2<0.
The correct justification  for the positive semi-definiteness of covariance matrices is:
w^{\rm T} \operatorname{E} \left[(\mathbf{X} - \operatorname{E}[\mathbf{X}]) (\mathbf{X} - \operatorname{E}[\mathbf{X}])^{\rm T}\right] w
= \operatorname{E} \left[w^{\rm T}(\mathbf{X} - \operatorname{E}[\mathbf{X}]) (\mathbf{X} - \operatorname{E}[\mathbf{X}])^{\rm T}w\right]
= \operatorname{E} \left[(w^{\rm T}(\mathbf{X} - \operatorname{E}[\mathbf{X}]))^2\right] \geq 0
I made this change. I found the proof in SEhttps://math.stackexchange.com/questions/114072/what-is-the-proof-that-covariance-matrices-are-always-semi-definite, but I don't know if I should cite SE. Also see 3.21 in Wasserman.  — Preceding unsigned comment added by 2001:16A2:87EF:4900:782C:159B:2B2D:753B (talk) 14:48, 22 May 2018 (UTC)
thread_title:  Explanation for positive semi-definiteness 

message: The typography of the Covariance matrix#Block matrices section is inconsistent with the other sections, e.g. Covariance matrix#Definition. I see three ways how to restore consistency:
1. Edit bold and italic fonts in Covariance matrix#Block matrices to be consistent with Eq. 1, which is 
2. Unfortunately, the typography in above equation itself is inconsistent since K is a matrix and μ is a vector, so they should be bold and roman (and T also should be roman):
3. However, subscripts X should be regarded as names rather than vectors, since using a vector as a subscript is an ill-defined concept. Therefore these subscripts should not be bold:
I think convention 3 is the best but most Wikipedia articles on statistics use convention 2. Before I add a couple of sections to this article (see User_talk:Sandstein#Covariance_mapping_redirection) I would like to correct the typography but do not want to make it inconsistent with other articles. What do you think? Is important to keep it consistent across several articles? Or only within any given article? Or its OK to mix conventions even withing the same article?
FizykLJF (talk) 18:59, 3 August 2019 (UTC)
thread_title:  Consistent typography 

message: As described in the previous post, I have added a section on "Covariance mapping" and corrected some inconsistent typography and notation in other sections. . FizykLJF (talk) 14:05, 18 August 2019 (UTC)
thread_title:  Covariance mapping section 

message: The section links to a definition of "data matrices" where a data matrix is defined to contain one experiment per row. But in the same sentence two data matrices are defined with one experiment per column. This is confusing. The easiest solution would be to just remove the link to the "data matrix" article. A better solution might be to actually _use_ the "one experiment per row" definition, and adapt the formulas in the section on estimation accordingly. Ernstkl (talk) 09:57, 12 July 2021 (UTC)
thread_title:  Estimation, reference to "data matrices" 

message: The variable rho is used without definition in section "Inverse of the covariance matrix." Jaguarmountain (talk) 17:43, 30 July 2023 (UTC)
thread_title:  Variable rho used without definition