Talk:Estimation of covariance matrices

message: Shouldnt we also note that the MLE is biased, and include a section on an unbiased estimator?
:Perhaps, but I don't see any urgency about it.  Most reasonable estimators, including most MLEs, are biased. Michael Hardy 22:00, 2 May 2005 (UTC)
replies: text: Guys, please at least put some links for unbiased discussions. I got some strange phenomenon when I was doing a project, so I adjust the diagonal eleme nts to unbiased variance estimates. It seems work better. Jackzhp 12:48, 21 October 2006 (UTC)
thread_title:  Biased vs. Unbiased estimate 

message: I'd like to point out that the symbol \Sigma starts out denoting a parameter, but then is later used as an estimator. Btyner 19:40, 9 February 2007 (UTC)
thread_title:  notation 

message: I have rewritten the summary, putting the maximum likelihood estimator in perspective.
The use X and x should be made consistent throughout the article.
I am not sure about the purpose of the statement
Although no one is surprised that the estimator of the population covariance matrix is [closely related to] the sample covariance matrix, the mathematical derivation is perhaps not widely known and is surprisingly subtle and elegant.
for the following reasons:
* The form with 1/n is not what is commonly called sample covariance matrix. For now I have added the text "closely related to" but that weakens the surprise part.
* Subtlety, elegance, and surprise are subjective, and the information contents of the statement is zero.
* The argument is nice but not all that subtle and elegant for that to be notable. (In my opinion, see the note above about subjective.)
Jmath666 14:34, 25 March 2007 (UTC)
thread_title:  New summary and further proposed changes 

message: In the beginning of the article, where we explaining way we divide by the factor n-1 rather than with n (in the unbiased estimator for the cov matrix), it is said that the reason is because the mean is not known and is replaced by the sample mean.
Of course this is true. And more then that, it directs us to the notion of degrees of freedom. and thus should link to the article on Degrees of freedom (statistics).
What do you think ?
Talgalili 13:04, 25 August 2007 (UTC)
thread_title:  linking to the article on "degrees of freedom" 

message: What is known about the general, non-gaussian case? In what sense does the sample covariance estimate/converges to the covariance matrix then? What if the random vectors are not independent but only approximately so? Jmath666 (talk) 05:45, 9 November 2008 (UTC)
replies: text: Interesting question.  One could wonder if something like the Gauss–Markov theorem could be done. Michael Hardy (talk) 20:24, 6 May 2009 (UTC)
thread_title: Convergence in the general case

message: I have some issues with the new section recently added.  I've commented it out in the article for now.
: 
===Maximum likelihood estimation: general case===
The first-order conditions for a MLE of parameter θ are that the first derivative of the log-likelihood function should be null at θMLE. Intuitively, the second derivative of the log-likelihood function indicates its curvature : the higher it is, the better identified θMLE since the likelihood function will be inverse-V-shaped around θMLE.
Formally, it can be proved that 
:\sqrt{T}(\theta_\text{MLE}-\theta) \rightarrow \mathcal{N}(0,\Omega) \, 
where \Omega can be estimated by
:\left(-\frac{1}{T}\sum_{t=1}^T \frac{\partial^2 \ell_t}{\partial \theta \, \partial \theta '} (\theta_\text{MLE})\right)^{-1}.
: 
=== (end of new section) ===
: 
In this case "general case" appears not to mention covariance matrices at all, but as applied to covariance matrices being estimated, it seems to say that the sample covariance matrix has approximately a normal distribution with a variance Ω.  I'm not sure I know what is meant by the variance of a matrix-valued random variable.  I may have seen it defined at some point in the past, but I think if one writes about such a thing here, the article should explain what it is.  And the parameter θ would in this case be a positive-definite matrix.  How do you compute those partial derivatives with respect to a positive-definite matrix-valued random variable?  I don't know for sure whether I've seen such a thing, but I think if that is to be done here, the article should explain what it is.
: 
Maybe the new paragraph belongs in the main maximum likelihood article or one of the other related articles. Michael Hardy (talk) 20:23, 6 May 2009 (UTC)
replies: text: My interpretation is that this new stuff is about estimating the covariance matrix of an estimate of a parameter vector obtained my maximum likelihood ... so not the covariance matrix of a population of which there is a direct sample as is the apparent intention of the rest of the article so far. Whether the article content should be extended to cover this topic is, I suppose, open. The article Unbiased estimation of standard deviation, which might be thought parallel to this one, presently covers both the population stdev and the stdev of the sample mean. Melcombe (talk) 08:48, 7 May 2009 (UTC)
thread_title:  Issues with a new section 

message: : 
In this case "general case" appears not to mention covariance matrices at all, but as applied to covariance matrices being estimated, it seems to say that the sample covariance matrix has approximately a normal distribution with a variance Ω.  I'm not sure I know what is meant by the variance of a matrix-valued random variable.  I may have seen it defined at some point in the past, but I think if one writes about such a thing here, the article should explain what it is.  And the parameter θ would in this case be a positive-definite matrix.  How do you compute those partial derivatives with respect to a positive-definite matrix-valued random variable?  I don't know for sure whether I've seen such a thing, but I think if that is to be done here, the article should explain what it is.
: 
Maybe the new paragraph belongs in the main maximum likelihood article or one of the other related articles. Michael Hardy (talk) 20:23, 6 May 2009 (UTC)
replies: text: My interpretation is that this new stuff is about estimating the covariance matrix of an estimate of a parameter vector obtained my maximum likelihood ... so not the covariance matrix of a population of which there is a direct sample as is the apparent intention of the rest of the article so far. Whether the article content should be extended to cover this topic is, I suppose, open. The article Unbiased estimation of standard deviation, which might be thought parallel to this one, presently covers both the population stdev and the stdev of the sample mean. Melcombe (talk) 08:48, 7 May 2009 (UTC)
thread_title:  (end of new section) 

message: Donald Andrews and others have written on median-unbiased estimators, with application to time series analysis.  Kiefer.Wolfowitz  (Discussion) 14:59, 13 April 2011 (UTC)
thread_title:  [[Median-unbiased estimation]]