Talk:Least-squares estimation of linear regression coefficients

message: After the last round of edits, it is still completely unclear what is to be proved in this article, and highly implausible that it proves anything. Michael Hardy 00:51, 9 February 2006 (UTC)
replies: text: In this article, we provide a proof for the general expression of this estimator (as seen for example in the article regression analysis):
\widehat{\theta}_n^{LS}=(X^t X)^{-1}X^t Y
----
What does that mean?  Does it mean that the least-squares estimator actually is that particular matrix product?  If so, the proof should not involve probability, but only linear algebra.  Does it mean that the least-squares estimator is the one that satisfies some list of criteria?  If so which criteria?  The Gauss-Markov assumptions?  If it's the Gauss-Markov assumptions, then this would be a proof of the Gauss-Markov theorem.  But I certainly don't think that's what it is.  In the present state of the article, the reader can only guess what the writer intended to prove! Michael Hardy 03:19, 9 February 2006 (UTC)
thread_title: Recent edits

message: I'm going to disect this slowly.  The following is just the first step.  The article says:
::(\Omega,\mathcal{A}, P) will denote a probability space and n\in\mathbb{N}^* (called number of observations). \mathcal{B}_n will be the n-dimensional Borel algebra. \Theta\subseteq\mathbb{R} is a set of coefficients.
::The response variable (or vector of observations) Y is a random variable, i.e. a measurable function Y:(\Omega,\mathcal{A})\rightarrow(\mathbb{R}^n,\mathcal{B}_n).
::Let p\in\mathbb{N}^*. p is called number of factors. \forall i\in \{1,\cdots,p\}, X_i:(\Omega,\mathcal{A})\rightarrow(\mathbb{R}^n, \mathcal{B}_n) is called a factor.
:::\forall\theta\in\Theta^{p+1}, let \eta(X;\theta):=\theta^0 + \sum_{j=1}^p \theta^j X_j.
::We define the errors \varepsilon(\theta):=Y-\eta(X;\theta) with \theta:=(\theta_0,\cdots,\theta_p)\in\Theta^{p+1}. We can now write:
::\forall \theta\in\Theta, Y=\theta^0 + \sum_{j=1}^p \theta^j X_j+\varepsilon(\theta)
In simpler terms, what this says is the following:
::Let Y be a random variable taking values in Rn, whose components we call observations, and having expected value
:::\eta=\theta_0 \mathbf{1}_n + \sum_{j=1}^p \theta_j X_j,
::where 
::* Xj ∈ Rn for j = 1, ..., p is a vector called a factor,
::* 1n is a column vector whose n components are all 1, and
::* θj is a scalar, for j = 0, ..., p.
::Define the vector of errors to be ε = Y − η.
The first version is badly written because
* Explicit mention of the underlying probability space, and Borel measureability, are irrelevant clutter, occupying the readers attention but not giving the reader anything.  When, in the study of statistics, do you ever see a random vector that is not Borel-measurable?  Will the fact of measurability be used in the succeeding argument?  A link to expected value is quite relevant to the topic; a link to measurable function is not.
* Saying "\Theta\subseteq\mathbb{R} is a set of coefficients" makes no sense.  The coefficients are the individual components of a vector θ somewhere within this parameter space.  If anything, Θ must be a subset of Rp in which the unobserved vector θ is known to lie.  If that subset is anything other than the whole of Rp, then I think you'll have trouble making the case that least-squares estimation of θ is appropriate, since the estimate presumably should be within the parameter space;
* The column vector of n "1"s is missing;
* It alternates between subscripts and superscripts on the letter θ, for no apparent reason;
* Why in the world is ε asserted to depend on θ?  Later the article brings the Gauss-Markov assumptions, which would conflict with that.
* One should use mathematical notation when it serves a purpose, not just whenever one can.  It is clearer to say "For every subset A of C" than to say "\forall A\in\mathcal{P}(C), where \mathcal{P}(C) is the set of all subsets of C."
OK, this is just one small point; the article has many similar problems, not the least of which is that its purpose is still not clear.  I'll be back. Michael Hardy 00:25, 20 February 2006 (UTC)
thread_title: one bit at a time...

message: OK, this makes sense: I'll correct the article. Except for the "having expected value" part. The way I present it, you can always write y=\eta+\varepsilon. What the Gauss-Markov assumptions add is that there exists an optimal parameter \overline{\theta} for which \varepsilon has an expectation of 0 and that its components are independant. The advantage is that you do not have to suppose that the X_j's are constants. In the case of randomized designs, this is important. Deimos 28 12:10, 20 February 2006 (UTC)
thread_title:  Thanks