HDTenEightyP/NetNews huggingface dataset:
From: thompson@atlas.socsci.umn.edu (T. Scott Thompson)
Subject: Re: Testing for Normality
Reply-To: thompson@atlas.socsci.umn.edu
Organization: Economics Department, University of Minnesota
Date: Sat, 5 Sep 1992 19:22:39 GMT

bateman@nsslsun.nssl.uoknor.edu (Monte Bateman) writes:

>I would like to test data for normality.  The books I have
>access to talk about graphing frequencies on "probability paper".

Look up the Kolmogorov-Smirnov (sp?) statistic in a more advanced
statistics text than the ones you have now.  This statistic is
implemented in many of the major commercial stat packages, and is not
hard to compute.  (It is basically a software implementation of using
probability paper.)  Be forewarned, however, that the test is not
very powerful against many alternative distributions, especially the
ones that are "normal-like", so you might need a very large sample to
do a reliable test.

--
T. Scott Thompson              email:  thompson@atlas.socsci.umn.edu
Department of Economics        phone:  (612) 625-0119
University of Minnesota        fax:    (612) 624-0209
Newsgroups: sci.math.stat
From: william@kub.nl (William Verkooijen)
Subject: Does anyone know the software package XploRe?
Date: Tue, 15 Sep 92 07:18:24 GMT
Organization: KUB, The Netherlands

Hello,

Does anyone know the mathematical software package XploRe (eXploratory
Regression) developed by Wolfgang Haerdle? (TURBO PASCAL 5.O)

I understand that there is also a SUN version (in C) written by David Scott,
Rice University, Houston, Texas.

If anyone can tell me how and where I can get XploRe, I will be very
pleased.


Any keys are welcome!

Thanks,
 
William Verkooijen
E-mail: william@kub.nl
From: hrubin@pop.stat.purdue.edu (Herman Rubin)
Newsgroups: sci.math.stat
Subject: Re: Testing for Normality
Date: 6 Sep 92 13:44:34 GMT
Organization: Purdue University Statistics Department

In article <1992Sep5.064647.15570@constellation.ecn.uoknor.edu> bateman@nsslsun.nssl.uoknor.edu (Monte Bateman) writes:
>I would like to test data for normality.  The books I have
>access to talk about graphing frequencies on "probability paper".

>First off, I don't have any.

>Second, it seems that there should be software available 
>to do this test.

>Pointers?

>Any/all help appreciated!


There is no shortage of tests for normality.  One problem is that the
behavior of the test statistic, even the size, is generally not computable
except by simulation, even approximately.  The problem is also to get a
reasonably good test against the most likely alternatives.

The Kolmogorov-Smirnov test with the parameters, assuming normality, 
estimated from the data is not difficult to calculate.  There are tables
of the significance levels.
-- 
Herman Rubin, Dept. of Statistics, Purdue Univ., West Lafayette IN47907-1399
Phone: (317)494-6054
hrubin@pop.stat.purdue.edu (Internet, bitnet)  
{purdue,pur-ee}!pop.stat!hrubin(UUCP)
From: jerry@nutmeg.hnrc.tufts.edu (Jerry Dallal)
Newsgroups: sci.math.stat
Subject: Re: Least Square Errors
Date: 9 Sep 92 19:27:22 GMT
Organization: USDA HNRC at Tufts University

In article <1992Sep9.142202.352@nutmeg.hnrc.tufts.edu>, I wrote:

<some stuff . . .>

As a post script, let me add that if the data were within sampling variability
of the fitted curve, that is, I were confident that this was not a case of
model failure, I would seriously consider fitting the constrained model.
The consideration would revolve around whether the response should be *exactly*
0 at X=20.  I don't know enough about your problem to make that judgement by
myself.
Newsgroups: sci.math.stat
From: charlie@umnstat.stat.umn.edu (Charles Geyer)
Subject: Re: Simple Proof that c=median minimizes E[ |X-c| ] needed.
Organization: School of Statistics, University of Minnesota
Date: Sun, 13 Sep 1992 00:30:11 GMT

In article <3SEP199213440863@utkvx2.utk.edu> menees@utkvx2.utk.edu
(Menees, Bill) writes:

> I'm a senior math major taking my first p&s course, and this problem
> has come up and it intrigues me.  My prof. has a proof for it, but he said it
> was way over my head.  Does anyone know of a proof suitable for a senior 
> undergrad?  Thanks in advance.

In article <1992Sep5.224529.12961@news2.cis.umn.edu> I replied:

> Because this might be a homework problem, I won't post a proof right now,
> but there is a one-line proof that requires no calculus and no probability
> beyond the fact that expection is a positive linear operator, ... [and]
> It works for a general probability distribution using the definition that
> c is a median if
>
>   1/2 <= P(c <= X)   and   1/2 <= P(X <= c)
>
> I'll post the proof in a week or so.

The promised proof, if anyone is still interested.

Suppose c is a median and c < d (the case c > d follows by symmetry).

  E{|X - d| - |X - c|} >= (d - c)[P(X <= c) - P(X > c)} >= 0

the first inequality following because |x - d| - |x - c| is equal to d - c
for x <= c and greater than - (d - c) elsewhere, and the second inequality
following by the definition of "median".

There is a similar proof that the mean minimizes expected squared deviation
-- that shouldn't use any calculus either.

-- 
Charles Geyer
School of Statistics
University of Minnesota
charlie@umnstat.stat.umn.edu
Newsgroups: sci.math.stat
From: dos105@rsphy2.anu.edu.au (Darren O'Shaughnessy)
Subject: Re: Darts
Followup-To: sci.math.stat
Organization: Australian National University
Date: Wed, 16 Sep 92 02:00:20 GMT

In article <00960A78.F915B0E0@uinpla.npl.uiuc.edu>,
tayloe@uinpla.npl.uiuc.edu (Rex Tayloe) wrote:
>  
>      Try a gaussian in X and 1 in Y. (not necessarily of the same width). I
> have had a similar experience when trying to monte carlo particle trajectories.
> 
>                                             Good Luck,
> 

  But surely if you treat the X and Y directions as independent, which is
what you are suggesting, the 2D distribution will be somewhat square- or
rectangular- shaped? From experience, the vertical and horizontal errors
have to be (anti-)
correlated to some degree.

      Signed,
              Pat Rourke  (you may not have heard of me)
From: feh@duke.edu (Frank Harrell)
Newsgroups: sci.math.stat
Subject: comparing 2 groups in presence of competing event
Keywords: survival analysis
Date: 14 Sep 92 22:49:56 GMT

I am involved in a study of critically ill patients for whom we
wish to compare two groups with regard to the timing of
"Do Not Resussitate" (DNR) orders. Not all patients had DNRs
and many have died. The risk of DNR increases as the risk of
death increases, hence there is informative censoring.
In the spirit of the Mantel-Haenszel-logrank statistic (and
related to the work of Pepe in Stat. in Med. and JASA), I can
conceive of a cascade of 2x2 tables of the form:

	For the n patients still alive at time t,
	what is the fraction, in each treatment group,
	who had DNR before time t.

Vary t from 0 to the largest follow-up time.
I need assistance in combining these tables into a test
statistic, or a pointer to an existing statistic suitable
for comparing non-fatal endpoints in the presence of informative
censoring (a terminating event, death).

Frank Harrell
Division of Biometry
Duke U.
Durham NC USA
feh@biostat.mc.duke.edu (our mail header gives an erroneous address)


-- 
----------------------------------------------------------------------------
Frank E Harrell Jr			feh@biostat.mc.duke.edu
Associate Professor of Biostatistics
Division of Biometry                    Duke University Medical Center
Newsgroups: sci.math.stat
From: alanm@cognos.com (Alan Myrvold)
Subject: Re: Testing for Normality
Organization: Cognos Incorporated, Ottawa CANADA
Date: Thu, 10 Sep 1992 12:43:12 GMT

In article <Bu5suB.HxM@mentor.cc.purdue.edu> hrubin@pop.stat.purdue.edu (Herman Rubin) writes:
>In article <1992Sep5.064647.15570@constellation.ecn.uoknor.edu> bateman@nsslsun.nssl.uoknor.edu (Monte Bateman) writes:
>>I would like to test data for normality.  The books I have
>>access to talk about graphing frequencies on "probability paper".
>
>>First off, I don't have any.
>
>>Second, it seems that there should be software available 
>>to do this test.
>
>>Pointers?
>
>>Any/all help appreciated!
>
>
>There is no shortage of tests for normality.

Yes, there are many test statistics that can be calculated. The probability
plot, as originally asked for, has the advantage of being GRAPHICAL, so you
actually see your data, and you see HOW it differs from normality.

It is fairly easy to do without the special paper - one way is to :

   1) Order your data from smallest to largest
   2) Determine the expected values of the order statistics from
      a Normal(0,1) population the same size as your sample.
   3) Plot your values against the corresponding order statistics.
   4) Assess normality : if your sample is normal, the data will fall 
      roughly along a straight line.

Step #2 can be done by consulting tables, doing some icky numerical 
integration, or conveniently and approximately if you have access to
an implementation of the inverse of the Normal CDF (call it G(y)).

Just apply the inverse CDF to uniform order statistics :

      r(i) = G((i-1/2)/n)       for i from 1 to n

A slightly better approximation to the normal order statistics is (if
my memory serves me right) :

      r(i) = G((i-3/8)/(n+1/4))  for i from 1 to n

You'll find the inverse normal CDF implemented in surprisingly many
spreadsheets, math librarys, and math&stats packages.

Most statistics packages do perform this plot for you!  
I suggest you find one.


---
Alan Myrvold          3755 Riverside Dr.    
Cognos Incorporated   P.O. Box 9707          alanm@cognos.com
(613) 738-1440 x3317  Ottawa, Ontario       
                      CANADA  K1G 3Z4       
Newsgroups: sci.math.stat
From: rizzo@cbnewsf.cb.att.com (anthony.r.rizzo)
Subject: Re: Testing for Normality
Organization: AT&T
Date: Tue, 8 Sep 1992 14:38:40 GMT

In article <1992Sep5.064647.15570@constellation.ecn.uoknor.edu> bateman@nsslsun.nssl.uoknor.edu (Monte Bateman) writes:
>I would like to test data for normality.  The books I have
>access to talk about graphing frequencies on "probability paper".
>
>First off, I don't have any.
>
>Second, it seems that there should be software available 
>to do this test.
>
>Pointers?
>
>Any/all help appreciated!
>
>
>Monte

I can offer one little pointer, for what its worth.
Box, Hunter, and Hunter, in "Statistics for Experimenters" (I'm guessing
at the title) give the scale factors with which you can create
your own normal probability paper.

Tony
From: wvenable@algona.stats.adelaide.edu.au (Bill Venables)
Newsgroups: sci.math.stat
Subject: Re: Testing for Normality
Date: 14 Sep 92 13:13:53 GMT
	<WVENABLE.92Sep12234508@algona.stats.adelaide.edu.au>
	<BuJ7v8.EKM@mentor.cc.purdue.edu>
Organization: Department of Statistics, University of Adelaide
NNTP-Posting-Host: algona.stats.adelaide.edu.au
In-reply-to: hrubin@pop.stat.purdue.edu's message of 13 Sep 92 19:37:55 GMT

>>>>> "Herman" == Herman Rubin <hrubin@pop.stat.purdue.edu> writes:

Herman> Even from the classical standpoint, the traditional chi-squared
Herman> test is wrong.  The distribution of the chi-squared statistic
Herman> should not have the full reduction in degrees of freedom for
Herman> estimated parameters.  This was proved by Chernoff and Lehmann in
Herman> 1953 for fixed partitions, and by A. R. Roy in his dissertation
Herman> under me in 1954 for data-defined partitions; the theory using
Herman> "good" estimators under the null hypothesis shows that it does not
Herman> matter how the partitions are defined.  Some of the papers of D. H.
Herman> Moore about 1970 provide a readable summary.

That's an interesting point, and I would just like to add a small comment.
If you reduce the data to panel frequencies and estimate the parameters
from the resulting multinomial likelihood then the asymptotic chi-squared
distribution must indeed be correct.  The catch is, of course, that no-one
ever does this, and the reduction to frequencies is not a sufficiency
reduction, so it would be inefficient so to do.

Herman> For a discussion of testing from a parametric robust Bayesian
Herman> viewpoint, readers may wish to look at my paper with Sethuraman in
Herman> Sankhya, 1965, on the subject.

I shall.  But I seem dimly to remember a paper by Anscombe on the subject
in the JRSS(?) yonks ago.  Does anyone else recall this?  (It may be
referenced in Herman's paper, of course; I think Anscombe was pre 1965.)

Herman> But one thing to keep in mind--the higher the dimension, the more
Herman> important the assumptions.

Amen.
--
___________________________________________________________________________
Bill Venables, Dept. of Statistics, | Email: venables@stats.adelaide.edu.au
Univ. of Adelaide, South Australia. | Tel: +61 8 228 5412  Fax: ...232 5670
From: chaas@coe.drexel.edu (Chuck Haas)
Newsgroups: sci.math.stat
Subject: Re: Least Square Errors
Date: 9 Sep 92 19:24:31 GMT
Organization: Drexel University
X-Useragent: Nuntius v1.1b2

What you propose to do is reasonable, and amounts to constrained least
squares.  There is a fair body of literature on constrained least
squares.  Your proposed method is also a degenerate case of a weighted
least squares problem, in which not all dependent values are measured
with the same lack of precision, and so that a poorer fit is tolerated at
those more imprecisely measured points.

[Don't apologize for being an engineer; I am one also -- who has heavily
delved into the statistical arena]

CN Haas
Environmental Engineering
Drexel University
Philadelphia
From: tayloe@uinpla.npl.uiuc.edu (Rex Tayloe)
Newsgroups: sci.math.stat
Subject: Re: Darts
Date: 15 Sep 92 16:15:19 GMT
Reply-To: tayloe@uinpla.npl.uiuc.edu (Rex Tayloe)
Organization: Nuclear Physics Laboratory, University of Illinois, Urbana-Champaign

In article <dos105-150992153659@150.203.49.43>, dos105@rsphy2.anu.edu.au (Darren O'Shaughnessy) writes:
>In article <dos105-130992174724@150.203.49.43>, dos105@rsphy2.anu.edu.au
>(Pat Rourke) wrote:
>> 
>>    I'm working on a darts simulation program, and so have studied the.....
>
>    For the sake of clarity, the following:
>     I am, in fact, treating the radial distance as the normally
>distributed variable, with the angle uniformly randomly chosen. I then have
>
>              Pat Rourke  (you may not have heard of me)

     Try a gaussian in X and 1 in Y. (not necessarily of the same width). I
have had a similar experience when trying to monte carlo particle trajectories.

                                            Good Luck,


   __    __    ____     __
 /__/\ /__/| /____/ \ /__/|     Rex Tayloe           TAYLOE@uinpla.npl.uiuc.edu
 |   \\| | ||  __ \ /|| | |     Nuclear Physics Lab  TAYLOE@uiucnpl.bitnet 
 | |\ \| | || |__) |/ | | |__   U of Illinois        UINPLA::(47165::)TAYLOE
 | | \   | ||  ___/   | |/__ /| Champaign Il 61820   (317)333-3190
 |_|/ \ _|/ |_|/      |_____|/
From: chaas@coe.drexel.edu (Chuck Haas)
Newsgroups: sci.math.stat
Subject: Re: Testing for Normality
Date: 7 Sep 92 18:18:47 GMT
     <1992Sep6.110952.349@ginger.hnrc.tufts.edu>
Organization: Drexel University
X-Useragent: Nuntius v1.1b2

RE: Testing for Normality

see the following reference for a good review of the various normality
tests, including comparative power calculations.  This is also a nice
review of the various methods for drawing class intervals in the
chi-squared tests for distributional goodness of fit:

"How to Test Normality and other Distributional Assumptions", SS Shapiro,
American Society for Quality Control - Statistics Division, Volume 3-ASQC
Basic References in Quality Control (1990).
From: flim@weber.ucsd.edu (Francis Lim)
Newsgroups: sci.math.stat
Subject: Predictive splines
Date: 15 Sep 1992 23:08:39 GMT
Organization: University of California, San Diego
NNTP-Posting-Host: weber.ucsd.edu

I need a program which will estimate a spline transformation
for a variable and subsequently give *predictions* for additional
observations based on the estimated spline.

I've checked SAS, S, BLSS, IMSL Fortran library, etc. with no success.

A professor pointed me to netlib@research.att.com and to look
in the "pppack" directory. However, I wonder if there are
statistical packages or programs already written out there
that will provide what I need?

-- 
==============================================================================
     FRANCIS W. LIM / email: flim@ucsd[.edu] / dept phone: (619) 534-3383
     Dept of Economics, Univ of Cal at San Diego, La Jolla, CA 92093-0508
==============================================================================
Newsgroups: sci.math.stat
From: thompson@atlas.socsci.umn.edu (T. Scott Thompson)
Subject: Re: Question on ratio estimate
Reply-To: thompson@atlas.socsci.umn.edu
Organization: Economics Department, University of Minnesota
Date: Sat, 5 Sep 1992 19:16:28 GMT

mpp@cs.brown.edu (Michael P. Perrone) writes:

>T. Scott Thompson writes:
>>To apply this to your problem let K = 2, J = 1 and set 
>>Z_i = (X_i,Y_i).  Set
>>
>>	g(z) = z(1) / z(2)  so that G = [ 1 / m(y),  -m(x)/m(y)^2 ].

>but i thought that z(1) and z(2) may be correlated
>thus the i.i.d. assumption may be bad in general.

You misunderstood my assumption.  I was assuming that the _vectors_
Z_i are i.i.d.  This does not mean that Z_i(1) and Z_i(2) are
independent from each other, nor that they are identically
distributed.  It means that the vector Z_i is independent from Z_j and
that these vectors have the same distribution for i != j.  (And
likewise for any finite collection of indices.)

--
T. Scott Thompson              email:  thompson@atlas.socsci.umn.edu
Department of Economics        phone:  (612) 625-0119
University of Minnesota        fax:    (612) 624-0209
Newsgroups: sci.math.stat
From: rob@med.umich.edu (Robert Strawderman)
Subject: Re: two pairs
Date: Tue, 15 Sep 92 08:19:43 EDT
Organization: University of Michigan Health Sciences, Ann Arbor
Distribution: umich
Reply-To: rob@med.umich.edu (Robert Strawderman)
Originator: rob@gaspra


You can think of the problem as follows:

You have 13 sets of 4 identical cards, and you first choose two of those sets.  Then,
to get the two pair, you must choose 2 from the 4 in each set.  The remaining card
may be chosen from the 11 remaining sets of 4 cards.  Thus, the numerator is

13C2 * (4C2)^2 * (11 * 4C1)

and the denominator is 52C5.

Rob Strawderman
Dept. of Biostatistics
University of Michigan
From: gagnon@buma15.bu.edu (David Gagnon)
Newsgroups: sci.math.stat
Subject: Re: Testing for Normality
Date: 8 Sep 92 21:46:50 GMT
Organization: Department of Mathematics, Boston University, Boston


   The reference for D'Agostino's test of normality is:

A Suggestion for Using Powerful and Informative Tests of Normality
  by D'Agostino, Belanger and D'Agostino Jr.
The American Stastician, Nov 1990, Vol 44, No. 4, 316-321

   I found it a good overview of the problem.

						>David<
From: popat@image.mit.edu (Kris Popat)
Newsgroups: sci.math.stat
Subject: min ave self-information inference
Date: 15 Sep 92 02:20:00 GMT
Organization: MIT Advanced Television Research Program


Suppose you have a parametric model for the probability density
function of a discrete random variable, and a set of observed values.
Call the model pdf p(x), and call the observed values x_1,...x_N.  The
goal is to find parameter values that make the model pdf p(x_i)
approximate the "true" but unknown pdf q(x_i).

One way to fit the model to the data would be to find parameter values
that minimize

                    -sum(log p(x_i))

i.e., to minimize the total self-information of the observed points
with respect to the model.  This "works" because for a given true pdf
q(x_i) and for all valid model pdfs p(x_i),

         E[-log(p(x_i))] = -sum( q(x_i) log p(x_i) )

is minimized when p = q.

I'd like a pointer to any papers that discuss this or similar
approaches to parametric pdf fitting.

Kris Popat
MIT Rm E15-391  Cambridge, MA  02139
From: jms@vision.sarnoff.com (Jerry Shapiro x2420)
Newsgroups: sci.math.stat
Subject: Re: min ave self-information inference
Date: 15 Sep 92 21:02:21 GMT
Reply-To: jms@vision.sarnoff.com
Organization: David Sarnoff Research Center

In article 92Sep14212000@image.mit.edu, popat@image.mit.edu (Kris Popat) writes:
>
>Suppose you have a parametric model for the probability density
>function of a discrete random variable, and a set of observed values.
>Call the model pdf p(x), and call the observed values x_1,...x_N.  The
>goal is to find parameter values that make the model pdf p(x_i)
>approximate the "true" but unknown pdf q(x_i).
>
>One way to fit the model to the data would be to find parameter values
>that minimize
>
>                    -sum(log p(x_i))
>
>i.e., to minimize the total self-information of the observed points
>with respect to the model.  This "works" because for a given true pdf
>q(x_i) and for all valid model pdfs p(x_i),
>
>         E[-log(p(x_i))] = -sum( q(x_i) log p(x_i) )
>
>is minimized when p = q.
>
>I'd like a pointer to any papers that discuss this or similar
>approaches to parametric pdf fitting.
>
>Kris Popat
>MIT Rm E15-391  Cambridge, MA  02139


This is the best approach. You're actually minimizing the "sample" relative
entropy D(p||q) which is has many properties of a distance metric between pdf's.
D(p||q) is defined as sum(p(x) log[p(x)/q(x)]). See Cover's information theory text.

For example for a two-sided Laplacian density q(x;A) = 0.5Aexp(-A|x|),
you end up with A = 1/E[|x|], where E[|x|] is the sample mean of the absolute values.

Jerry Shapiro
Newsgroups: sci.math.stat
From: ppc1@cec1.wustl.edu (Peter Pui Tak Chiu)
Subject: two pairs
Organization: Washington University, St. Louis MO
Date: Tue, 15 Sep 1992 02:49:45 GMT



hi,

can anyone point out what is wrong with my calculation of getting a hand
of two pairs from a shuffled deck of cards?

    the probability i have is:

	(13 * 4C2) (12 * 4C2) (4 * 11)
	------------------------------ = 0.095078
		      52C5

which is exactly two times the correct value (0.047539).

pete