HDTenEightyP/NetNews huggingface dataset:
From: thompson@atlas.socsci.umn.edu (T. Scott Thompson)
Subject: Re: Standard Deviation.
Keywords: (n) versus (n-1)
Reply-To: thompson@atlas.socsci.umn.edu
Organization: University of Minnesota
Date: Mon, 17 Aug 1992 16:52:03 GMT

psrdj@warwick.ac.uk (G M Collis) writes:

>What intrigues me is that the most elementary stats texts make a big
>fuss about using n-1 for an unbiased estimate of the variance, but ignore
>the fact that this gives a biased estimate for the SD. I recall
>that n - 1.5 is nearer the target for the SD when the sample is
>from a normally distributed population. I gather that minimising
>the bias when estimating the SD is rather sensitive to the population
>distribution - I'd like to know more about this.  But my big puzzle
>remains - why is the biasedness of the usual SD estimator (with N-1)
                                        ^^^^^ ^^ ^^^^^^^^^  ^^^^^^^^
>so rarely mentioned, in stark contrast to the case of the variance.

The lack of mention probably arises because general results are
unavailable due to the dependence on distributional shape that you
mention.  By contrast, distribution-free results for the sample
variance are well known.

I assume that you intend

   "usual" SD estimate == sqrt("usual" variance estimate)

This is biased because (1) the usual (unbiased) variance estimate
is itself random and (2) the sqrt( ) function is nonlinear.  Generally
for any random variable x, if Var[x] > 0 then

         E[ sqrt(x) ] < sqrt( E[x] )

(due to the concavity of the sqrt( ) function).  Plugging in

        x == "usual" variance estimate

so that E[x] = population variance, and using

        population SD == sqrt( population variance )

we get

        E[ "usual" SD estimate ] < population SD.

That is, the usual estimate is downward biased.  The bias comes from
the fact that the usual variance estimate has some variation around
the true variance.  This variation interacts with the curvature of the
sqrt( ) function to give the bias.  The amount of bias depends on
exactly how the variation in the usual population variance estimator
is distributed around the mean value.  Hence the sensitivity to
distributional assumptions.

The factor n - 1.5 may produce better results for the normal
distribution.  I haven't checked.  Keep in mind, that the bias
disappears fairly quickly with increases in sample size.  In fact,
using n - <arbitrary constant> will work just fine for most purposes
provided n is large enough.  This is because

     (n-1)/(n-<arbitrary constant>) -> 1

as n increases, and also because the variance in the "usual estimate"
decreases with the sample size.

Xref: sparky sci.math.stat:1640 sci.math.num-analysis:2450
From: selby@ptdca7.intel.com (Rick Selby)
Newsgroups: sci.math.stat,sci.math.num-analysis
Subject: Looking for MATLAB equivalent for SVR4/386
Date: 12 Aug 92 14:27:54 GMT
Organization: Portland Technology Development, Intel Corp., Aloha, OR
Originator: selby@ptdca0

Does anyone out there know of a package similar to MATLAB which provides a nice
interface to EISPAC and LINPAC (sp?) matrix math libraries. Please bear with any
ignorance that I may display in this request or followups. As I am a software 
engineer, I am not going to be the user of the package, but am helping a
process engineer to find MATLAB-equivalent functionality for our UNIX SVR4/386 
systems which are used in wafer fabrication station control applications. Email 
replies (selby@ptdca7.intel.com) would be most appreciated since I don't get to 
regularly keep up on the math groups. Thanx for any help.
Newsgroups: sci.math.stat
From: Andrew.Robinson@anu.edu.au
Subject: Re: Degrees of Freedom  Was: Re: Standard Deviation.
Organization: Australian National University
Date: Fri, 21 Aug 92 00:03:14 GMT

In article <thompson.714338397@kiyotaki.econ.umn.edu>
thompson@atlas.socsci.umn.edu (T. Scott Thompson) writes:
>hougen@uirvlh.csl.uiuc.edu (Darrell Roy Hougen) writes:
>
[deletia]
>
>I have never been very satisfied with this notion of "degrees of
>freedom" being equivalent to "pieces of independent information".  The
>correspondence works well enough for linear models and independent
>observations, as in the example given, but there are alternative
>interpretations (in terms of the dimension of orthogonal subspaces)
>that do just as well in this setting.
>
>To put out a concrete example that illustrates the nature of my
>qualms, consider the following.  Suppose my data consist of a single
>number, say X.  Suppose that I am given two pieces of information
>about X (but not X itself):
>
>           (1)  I can observe Y := X**2
>           (2)  I can observe Z := {  1 if X > 0
>                                   { -1    otherwise
>
>How many "pieces of information" do I now have?  Clearly I can recover
>X from Y and Z by the formula X = Z * <positive square root of Y>, but
>neither Y nor Z alone suffices (excluding the special case where 
>X = Y = 0).  Thus Y and Z must each "use up" (at least) one degree of
>freedom in the usual accounting.  Thus I must conclude that
>
>       <degrees of freedom in X> .GE. 1 + 1 = 2.
>
>For obvious reasons I do not find this to be an acceptable answer!
>What do you think?  What am I missing?  Is there any meaningful
>interpretation to "degrees of freedom" in a general theory of
>inference?

I'm going to go _way_ out on a limb here and say that I think that you have
_one_ piece of infomation.  If you don't have X then your data doesn't consist
of a single number, it consists of a single observation from which you have
measured a two variable vector - [Y,Z].  You know  that there is an X and you
want to use Y and Z to estimate it using a predetermined model, which you have
specified.  This seems to me to be a non-straightforward case of a multivariate
regression, with two dependent variables and one observation.

Comments?

regardless,
Andrew

now hit d
From: gerald@i80s20.ira.uka.de (Gerald)
Newsgroups: sci.math.stat
Subject: Literature wanted
Date: 20 Aug 1992 09:17:29 GMT
Organization: University of Karlsruhe, FRG
Distribution: world
NNTP-Posting-Host: i80s20.ira.uka.de

Hi netters,

I am looking for a statistics book. The only information I've got is
 
    H. Ventsel, "Theory of Probability"

Any help in further information is appreciated.

--------------------------------------------------------------------
UNIVERSITY of KARLSRUHE (Germany)         Phone (+49) (721) 608 4227 
Institute of Computer Design              FAX   (+49) (721) 370  455 
and Fault Tolerance                        E-Mail: gerald@ira.uka.de
--------------------------------------------------------------------
Newsgroups: sci.math.stat
From: thompson@atlas.socsci.umn.edu (T. Scott Thompson)
Subject: Re: linear covariance estimate for max likelihood
Keywords: parameter estimation, maximum likelihood, covariance estimate
Reply-To: thompson@atlas.socsci.umn.edu
Organization: Economics Department, University of Minnesota
Date: Fri, 21 Aug 1992 16:22:13 GMT

juber@uceng.UC.EDU (james uber) writes:

>I obtain parameter estimates via maximum likelihood where
>my model is in the standard reduced form y = f(p), y are the
>data and p are the parameters. I assume that the distribution
>of the model + measurement errors is normal with zero mean
>and known covariance matrix Ve. Thus i am solving the optimization
>problem:

>	min Tr(y - f(p))Inv(Ve)(y - f(p))
>         p

[rest of post deleted]

I do not understand the question.  If y = f(p) (where f is presumably
a known and fixed function of the parameters) and y is observed then
there is no measurement error.  Perhaps you meant y = f(p) + e where e
is a vector of measurement errors.  (This seems implicit in your
description of the nonlinear least squares procedure.)

But you also refer to "model errors".  What are these and how do they
fit in?  If the model is really

              y = f(p) + <error>

and f(p) is known (up to the parameters p) and fixed, then
Var(y) = Var(<error>) regardless of the source of the errors.

Please clarify.
Newsgroups: sci.math.stat
From: N020BA@tamvm1.tamu.edu
Subject: Principal Component Analysis For Autocorrelated Data
Organization: Texas A&M University
Date: Fri, 14 Aug 1992 23:27:32 GMT

    Hello! I have autocorrelated data (brainwaves.) I need to know how to
do principal component analysis (PCA) this data. References, explanations
and code in C or BASIC is welcome. You can respond by E-mail or post the
message to this group. E-mail to N020BA@tamvm1.tamu.edu.
    Thank you for your help.
 
========================================================================
The truth may hurt, but lies hurt more.  | My opinions are mine and mine
The difference between left-wing incomp- | alone. I speak for myself and
etance and right-wing incompetance is    | no-one else.
the difference between cat #$!& and dog  |______________________________
#$!&. Reply to N020BA@tamvm1.tamu.edu.        Bill and Opus in '92!!
Newsgroups: sci.math.stat
From: dmurdoch@mast.queensu.ca (Duncan Murdoch)
Subject: Re: Standard Deviation.
Keywords: (n) versus (n-1)
Organization: Queen's University
Date: Mon, 17 Aug 1992 14:35:16 GMT

In article <c48nbgtf@csv.warwick.ac.uk> psrdj@warwick.ac.uk (G M Collis) writes:
>What intrigues me is that the most elementary stats texts make a big
>fuss about using n-1 for an unbiased estimate of the variance, but ignore
>the fact that this gives a biased estimate for the SD. I recall
>that n - 1.5 is nearer the target for the SD when the sample is
>from a normally distributed population. I gather that minimising
>the bias when estimating the SD is rather sensitive to the population
>distribution - I'd like to know more about this.  But my big puzzle
>remains - why is the biasedness of the usual SD estimator (with N-1)
>so rarely mentioned, in stark contrast to the case of the variance.

I think it's just a tradition - introductory statistics texts aren't 
supposed to explain things correctly, they're just supposed to present a 
cookbook of methods, with handwaving and often incorrect justifications.

As others have pointed out, getting an unbiased variance estimate is hardly
essential.  I have trouble even thinking of a rigged artificial example 
where bias of the variance estimate would matter.  In many cases, the 
standard deviation estimate is just used to give a rough idea of the 
variability of a population, e.g. mean +/- s.d.,  or the precision of an 
estimate, e.g. mean +/- s.e.  In these cases it almost never matters whether 
you use N or N-1 --- they give essentially the same answer, unless N is very 
small.

The other place to use a standard deviation is in the construction of 
confidence intervals or tests.  There the denominator doesn't matter a 
bit:  the formula for the CI or test statistic will compensate.

I've argued so far that it doesn't matter whether you use N or N-1.  So, 
which do I use?  Generally N-1, because it tends to make the Normal theory 
formulas simpler, e.g. an F test based on the ratio of two variance 
estimates depends on the degrees of freedom, not the sample size.  

Make it a general rule in doing variance estimates to divide by degrees of 
freedom, not sample size, and you'll find you have simpler formulas to 
remember.  

Duncan Murdoch
dmurdoch@mast.queensu.ca
Newsgroups: sci.math.stat
From: jsvrc@rc.rit.edu (J A Stephen Viggiano)
Subject: Re: Fwd: Standard Deviation.
Organization: RIT Research Corp
Date: Sun, 16 Aug 1992 14:57:24 GMT

In article <seX2yRq00Uh785H2EB@andrew.cmu.edu> dr3u+@andrew.cmu.edu (Daniel Read) writes:
>---------- Forwarded message begins here ----------
>
>Can someone explain why calculating the Standard Deviation (SD),
>for small samples, with (n-1) in the denominator is better than
>doing so with (n) in the denominator?  I'm sure that there's
>a perfectly good reason for doing so.  But we, lowly engineers
>aren't usually told the reason.  Thanks now, for your response later.;-)

This is done in only in the case of the _sample_ standard deviation. The
reason for this is that the _sample_ mean is used in the formula, rather
than the population mean. Using a denominator of (n-1) results in the sample
_variance_ being unbiased -- the expected value of the sample variance will
be the population variance. (Unfortunately, the sample standard deviation
will be slightly biased. It is difficult to come up with an unbiased sample
standard deviation; I imagine that the square root make the results
dependent on the underlying distribution. So the goal is to make the sample
_variance_ unbiased.)

For a proof of this, check any introductory text on mathematical statistics.
If you'd like to try it yourself, take the expected value of the sample
variance. Expand the (x - xbar)**2 term inside.
From: chadster@statBerkeley.EDU (Chad Heilig)
Newsgroups: sci.math.stat
Subject: Re: Standard Deviation.
Keywords: (n) versus (n-1)
Date: 17 Aug 92 22:15:41 GMT
Organization: Statistics Dept., U. C. Berkeley

It is true that the `usual estimator' for the standard deviation is biased
for the population SD.  However, it might remain popular for the following
reason:

Suppose we want to test the hypothesis  Null:  mean=0  vs.  Alt:  mean != 0, 
but we do not know the population variance.  Then our test takes the form:

 	reject the null if Xbar/sqrt(S^2) > some constant

where S^2 is the unbiased estimator for the variance and the constant is
chosen so that we reject with a certain probability.
So where am I going with this?  Well, under the assumption that the
data are normally distributed with mean 0, the test statistic above (sometimes
called T squared) has a t-distribution with n-1 degrees of freedom; this is 
only true, however, under the additional contstaint that we use the unbiased
estimator for the variance in forming the test-statistic.  Thus the "N-1".

That said, let me make two more notes:
(1)  S^2 is unbiased for population variance, regardless of the underlying
dsitribution (as long as we have finite variance).
(2)  The distribution of the T-statistic is somewhat robust against non-
normal underlying distributions.

I realize that this gets awat a bit from the original question, but I think
it rounds out the defense for using `N-1' in the SD estimate.

Chad
Newsgroups: sci.math.stat
From: jja@fnbc.com (Joseph Alotta)
Subject: chicago area tutor wanted for stochastic methods
Keywords: stochastic, tutor
Organization: First National Bank Of Chicago, Chicago IL, USA
Date: Mon, 17 Aug 92 18:25:35 GMT

i am looking for someone to help me with stochastic methods.

i have a book that i'd like to use as a text.  i understand some
of it, but i feel i am missing alot.

i was thinking about meeting once every week or two weeks and going over  
the book and the exercises given.

the west suburbs of chicago are best.

this will, of course, be in exchange for money.

please, email me, if you are interested.

cheers,

joe.


--
==============================================================
! joe alotta                                    jja@fnbc.com !                                               
!                                             (312) 732-3439 !
!                                                            !
! "The main thing is to keep the main thing the main thing." !
! Dr. George Sweeney                                         !
==============================================================
From: maj@waikato.ac.nz
Newsgroups: sci.math.stat
Subject: Re: Standard Deviation.
Date: 18 Aug 92 15:17:41 +1200
Organization: University of Waikato, Hamilton, New Zealand

I'd just like to add a couple of points to the many that have
been made. (More in the direction of people who are mathematically
sophisticated, but with little exposure to statistics. Unfortunately
this class remains sizeable.)

Firstly the ' n - 1 ' is really the dimension of a vector space:
the vector space of contrasts in the observations. In more complex
situations other numbers will appear in place of n -1. For example
if the observations come from several groups we are often 
interested in the within-group contrasts. Here we head towards the
idea of 'degrees of freedom'.

Secondly the ' n ' divisor is the one which makes the standard
deviation a functional on the space of empirical distribution
functions: if you take each observation twice the e.d.f. does not
change, and neither does the n-definition standard deviation.

For a lot of work in the theory of robust statistics and diagnostics
it is convenient to have statistics as functionals.

Both definitions are here to stay, as each leads to convenient
theory in different contexts.

-- 
Murray A. Jorgensen [ maj@waikato.ac.nz ]    University of Waikato
Department of Mathematics and Statistics     Hamilton, New Zealand      
__________________________________________________________________
'Tis the song of the Jubjub! the proof is complete,
if only I've stated it thrice.'
Newsgroups: sci.math.stat
From: juber@uceng.UC.EDU (james uber)
Subject: linear covariance estimate for max likelihood
Keywords: parameter estimation, maximum likelihood, covariance estimate
Organization: College of Engineering, University of Cincinnati
Date: Thu, 20 Aug 1992 14:23:53 GMT

I obtain parameter estimates via maximum likelihood where
my model is in the standard reduced form y = f(p), y are the
data and p are the parameters. I assume that the distribution
of the model + measurement errors is normal with zero mean
and known covariance matrix Ve. Thus i am solving the optimization
problem:

	min Tr(y - f(p))Inv(Ve)(y - f(p))
         p

where Tr() is the transpose and Inv() is the inverse. I have a question
about the mathematics of deriving the standard linear estimate of the
covariance matrix of the sampling distribution of the parameter estimates, p*.

To form my question, let me go briefly into my interpretation of the
covariance matrix derivation. First of all, the optimal value function of
the max. like. problem depends on data, that is p* = p*(y). The definition of
the covariance matrix of the sampling distribution, Vp, is:

	Vp = E[(dp*)Tr(dp*)]

where the perturbation of the optimal parameters, dp* is estimated by the
linear terms of a Taylor series:

	dp* ~ Tr(dp*/dy)dy

where dp*/dy are the partial derivatives of the function p*(y) and are 
evaluated at (y,p*). Substituting this into the definition of Vp gives:

	Vp ~ Tr(dp*/dy)E[(dy)Tr(dy)](dp*/dy)
	Vp ~ Tr(dp*/dy)Vy(dp*/dy)


Now the rest of the derivation continues with the evaluation of the partial
derivatives (dp*/dy), often requiring additional approximations based on
small residuals and other assumptions. The true dp*/dy would certainly 
require second partials of the log likelihood function, whereas certain
approximations (the most common, i believe) use only first partials. In any
case, the partials dp*/dy will certainly involve the combined error covariance
matrix Ve.

My question can now (finally) be stated: Is the covariance matrix Vy above
one and the same with Ve, the covariance of the model + measurement errors?
I have been really confused by this for a long time. Just intuitively,
but perhaps naively, i'd have thought that Vy only includes the *measurement*
errors, and not the model errors. All of the derivations i've seen, however,
seem to at some stage treat both Vy and Ve as the same. For example, one
standard final results is the following:

	Vp ~ Inv[Tr(df/dp)Inv(V)(df/dp)]

But, i must confess, i really am unsure how to interpret the covariance
V in the above equation.

Thank you for your time and assistance in helping me with this problem.    

jim uber
dept. of civil & env. engineering
univ. of cincinnati
juber@uceng.uc.edu

-- 
--
james uber
juber@uceng.uc.edu
Newsgroups: sci.math.stat
From: news@massey.ac.nz (USENET News System)
Subject: Re: Adjusted r-squared as analogous to sample variance estimator
Organization: Massey University
Date: Fri, 21 Aug 92 00:31:48 GMT

In article <Aug20.212648.51218@yuma.ACNS.ColoState.EDU>, mglacy@lamar.ColoState.EDU writes:
> 
> After following the thread about the approporiate divisor for the
> sample variance, and the comments about the misconceptions and
> mistakes concerning it that are propogated in introductory textbooks,
> I was struck by an inconsistency.  While most intro. books make
> considerable noise about the need to use n-1 to make the sample
> variance an unbiased estimator, virtually none of them consider
> the analogous correction to make the sample r-squared an unbiased
> estimator of the population parameter.
> 
> Does anyone have any explanation for this inconsistency, other
> than simple error on the part of textbook writers?
>
It's not a case of making considerable noise, but of pointing out why
the "obvious" divisor is not used. It is not important anyway if only a
point estimate is required. But, for inference, the t-distribution was
calculated on the basis of the df as the divisor, so, unless you want to
use non-standard tables, you have to stick to that.

It is also convenient to have the trouble at this point rather than when
studying regression. Otherwise there would be an inconsistency: use
n for single samples, n-p when fitting p-location parameters.

Having said that, Moore & McCabe "Introduction to the Practice of
Statistics" p199, do use n-1 as the divisor when calculating r - strictly
speaking, when calculating the covariance. (They just don't make
"considerable noise" about it). (Also true of most books).
The t-test statistic for testing rho = 0, r sqrt((n-2)/(1-r^2)) relies on
this definition of r. Of course the n-1 cancels when the correlation is
written in terms of sums of squares and sums of products -
(sum (x-xbar)(y-ybar)/(sqrt(sum (x-xbar) sum(y-ybar))) -
so you may not have noticed that it _was_ incorporated into the
formula.

Incidently, at least one book on sampling theory uses n as a divisor
because it leads to simpler formulae when the finite sampling
correction is taken into account.

Terry Moore
Newsgroups: sci.math.stat
From: micja@ida.liu.se (Michael Jansson)
Subject: Weibull distribution
Keywords: Weibull
Organization: University of Linkoping, Dep't of Computer Science
Date: Mon, 17 Aug 1992 16:57:30 GMT


Hello, this question is really from a friend of mine who currently
doesn't receive this group so please replay with E-mail (too?) to
zap@indic.se! (I can forward messages though.)

--------------- Included question -----------------

I would appreciate information on how to estimate the parameters
of a Weibull distribution using a sample from a population,
assumed to be Weibull distributed.

A hint on a suitable book is really all I need, but other help
is welcome.

		Thanks in advance / Jonas (zap@indic.se)

--------------- End of inclusion ------------------
-- 
     __________________________________________________________________________
     Michael Jansson	  mij@IDA.LIU.SE, uunet!liuida!mij, mij@SELIUI51.BITNET