HDTenEightyP/NetNews huggingface dataset:
From: hrubin@pop.stat.purdue.edu (Herman Rubin)
Newsgroups: sci.math.stat
Subject: Re: Testing for Normality
Date: 13 Sep 92 19:37:55 GMT
Organization: Purdue University Statistics Department

In article <WVENABLE.92Sep12234508@algona.stats.adelaide.edu.au> wvenable@algona.stats.adelaide.edu.au (Bill Venables) writes:
>>>>>> "Charles" == Charles Packer <packer@amarna.gsfc.nasa.gov> writes:

>Charles> Why not use a traditional chi-square test?

>I can think of two possible reasons:

>1. It requires an arbitrary partition of the range into panels, *before*
>   the sample comes to hand.  (In fact it really does not test normality as
>   such, but rather that the grouped sample distribution agrees with a
>   similarly grouped normal.)  Arbitrariness always comes at some cost.

>2. In seeking to get some power against a very wide class of alternatives
>   it manages only to achieve low power against any subclass, including the
>   subclass of practically important alternatives.  In this sense it is not
>   well focused enough.

>[BTW I would be interested in a Bayesian reaction to this question.  It always
>seemed to me that tests of fit could be rather an embarrassment to a Bayesian.]

Even from the classical standpoint, the traditional chi-squared test is 
wrong.  The distribution of the chi-squared statistic should not have the
full reduction in degrees of freedom for estimated parameters.  This was
proved by Chernoff and Lehmann in 1953 for fixed partitions, and by A. R. Roy
in his dissertation under me in 1954 for data-defined partitions; the theory
using "good" estimators under the null hypothesis shows that it does not 
matter how the partitions are defined.  Some of the papers of D. H. Moore
about 1970 provide a readable summary.

The power is VERY low against any reasonable alternative.  The reason for
this is that the test fails to consider that adjacent intervals are likely
to differ from the null in the same manner.  Such tests as the Kolmogoroff-
Smirnoff or Kuiper do a far better job.

For a discussion of testing from a parametric robust Bayesian viewpoint, 
readers may wish to look at my paper with Sethuraman in Sankhya, 1965, on
the subject.  For at least one "real" problem, that of looking for concentrated
contamination to a distribution, this can be approximately carried out in the
infinite-dimensional version.  But one thing to keep in mind--the higher the
dimension, the more important the assumptions.
-- 
Herman Rubin, Dept. of Statistics, Purdue Univ., West Lafayette IN47907-1399
Phone: (317)494-6054
hrubin@pop.stat.purdue.edu (Internet, bitnet)  
{purdue,pur-ee}!pop.stat!hrubin(UUCP)
Newsgroups: sci.math.stat
From: ppc1@cec1.wustl.edu (Peter Pui Tak Chiu)
Subject: FULL HOUSE
Organization: Washington University, St. Louis MO
Date: Sun, 13 Sep 1992 14:33:29 GMT


hi everyone,

i am trying to find the probability of getting a hand of full house from a
shuffled deck of 52 cards.

theoretically, the probability (i think) is:

        (13 * 4C2) * (12 * 4C3)
        -----------------------  =  0.001440576
                 52C5

but i wrote a c program to calculate this probability and the results turn
out to be very different.

here is part of the results i have:

* the first column denotes the number of trials
* the second column denotes the probability obtained so far


5400000 0.00420815
5410000 0.00420758
5420000 0.00420867
5430000 0.00420829
5440000 0.00420937
5450000 0.00421046
5460000 0.00421154
5470000 0.00420987
5480000 0.00420967
5490000 0.00420765
5500000 0.004208  ..... this means after 5500000 trials, the prob. is 0.004208


from these results, i conclude that the probability would be pretty much around
0.00421 which is very different from the theoretical value.

does anyone know why???

i have included the program i wrote.  it is written in C++ and is very crude...
please try it and give me some feedback.

thanx a lot!

pete


// To calculate the probability of getting a full house

#include <iostream.h>
#include <time.h>
#include <stdlib.h>
#include <math.h>

int BAD,hand[5];		// BAD when the hand dealt is illegal
				// e.g. 5 of the same kind.
				// hand is an array holding the cards dealt
long COUNTER=0, GOOD=0;		// COUNTER = # of trials
				// GOOD = # of full house dealt


void init(){
for(int i=0;i<5;i++)hand[i]=rand()%13;	// deal the cards
}

int fullhouse(){			// determine if fullhouse or illegal
int counter[13],flag3=0,flag2=0;
BAD=0;
for(int i=0;i<13;i++)counter[i]=0;
for(i=0;i<5;i++)counter[hand[i]]++;
for(i=0;i<13;i++){
	if(counter[i]==3)flag3++;	// detect for 3 of a kind
	if(counter[i]==2)flag2++;	// detect for a pair
	if(counter[i]>4)BAD=1;}		// detect for a bad hand
return((flag3==1) && (flag2==1));
}

main(){
srand((unsigned)time(NULL));		// initialize random seed
while(1){
init();					// deal the cards
if(fullhouse())GOOD++;			// if a FULLHOUSE is dealt, increment
					// GOOD
if(!BAD)COUNTER++;			// if the hand dealt is legal
					// increment COUNTER
if(!fmod(COUNTER,10000))cerr<<COUNTER<<" "<<double(GOOD)/COUNTER<<"\n";
	// print out the result every 10000 trials in the format:
	// [ # of trials ] [ probability obtained so far to get full house ]
}}
Newsgroups: sci.math.stat
From: charlie@umnstat.stat.umn.edu (Charles Geyer)
Subject: Re: Simple Proof that c=median minimizes E[ |X-c| ] needed.
Organization: School of Statistics, University of Minnesota
Date: Sat, 5 Sep 1992 22:45:29 GMT

In article <3SEP199213440863@utkvx2.utk.edu> menees@utkvx2.utk.edu
(Menees, Bill) writes:

>	I'm a senior math major taking my first p&s course, and this problem
> has come up and it intrigues me.  My prof. has a proof for it, but he said it
> was way over my head.  Does anyone know of a proof suitable for a senior 
> undergrad?  Thanks in advance.

Because this might be a homework problem, I won't post a proof right now,
but there is a one-line proof that requires no calculus and no probability
beyond the fact that expection is a positive linear operator, i. e.

  E(aX + Y) = a E(X) + E(Y)

and

  0 <= X   implies   0 <= E(X)

It works for a general probability distribution using the definition that
c is a median if

  1/2 <= P(c <= X)   and   1/2 <= P(X <= c)

I'll post the proof in a week or so.

-- 
Charles Geyer
School of Statistics
University of Minnesota
charlie@umnstat.stat.umn.edu
From: ph@physiology.oxford.ac.uk (Patrick Haggard)
Newsgroups: sci.math.stat
Subject: Test for bimodality
Date: 11 Sep 92 13:25:40 GMT
Organization: Physiology Department, Oxford University, Oxford, UK.
Originator: ph@galen.physiol


How should I test whether the distribution of a variable is bimodal?
Patrick
Newsgroups: sci.math.stat
From: smith@ms.uky.edu (Brian Smith)
Subject: Repeated measures problem
Expires: Mon, 5 Oct 1992 04:00:00 GMT
Distribution: na
Date: Tue, 15 Sep 1992 22:34:48 GMT
Organization: University Of Kentucky, Dept. of Math Sciences

I have an expeiment where each experimental unit has been measured
dayly for something like 40 consecutive days.  I realize this is a 
problem and have been told that ideas exist as to how to handle this
problem.  My limited time in digging through journals has not been
very successful.  If any of you can point me in the right direction,
I would be very thankful.
                                                        Sincerely
                                                        Brian Smith

Newsgroups: sci.math.stat
From: rizzo@cbnewsf.cb.att.com (anthony.r.rizzo)
Subject: Re: Testing for Normality
Organization: AT&T
Date: Wed, 9 Sep 1992 14:19:05 GMT

I AM normal, damn it!                                                 Me too!

Newsgroups: sci.math.stat
From: lafleur@VAX309.NOSC.MIL
Subject: Re: Testing for Normality
Reply-To: lafleur@VAX309.NOSC.MIL
Organization: Naval Health Research Center, San Diego, CA
Date: Wed, 9 Sep 1992 15:16:31 GMT

To: Bateman@nsslsun.nssl.uoknor.edu
I believe that SYSTAT has the ability to perform Lilifors test...but it
is easy to do by hand as well.
Newsgroups: sci.math.stat
Subject: Re: likelihood ratios from detection data..
From: ml@icarus.lis.pitt.edu (Michael Lewis)
Date: 7 Sep 92 22:07:08 GMT
Distribution: sci
Organization: University of Pittsburgh
Summary: contrasting ideal observers & subjects using likelihood ratio functions


     I have some data to analyze & thought someone in this group might have
a good suggestion.  The data comes from an experiment comparing two displays.
When the system is unfailed its output is z = .5x + .5y + n when it fails
the displayed output z_i = z_i-1 + n.  x and y are displayed inputs which are
driven in a random walk by a zero mean gaussian disturbance.  The noise is
also gaussian. Failures are introduced randomly with a fixed probability over
each half second interval.  The subject's task is to monitor the display and 
press a button when she believes the process has "failed".
     The problem is a typical signal detection one.
I want to reference subjects' performance to various ideal observers which
attend to particular features of the display.  The "ideal" ideal observer,
for example, would be a Wald observer with memory.  My problem involves dealing
with subjects' response biases.  I would like to find some measure of agreement
between a subject & observer which does not "overly" penalize the subject
for choice of cutoff.  One way of looking at the situation is that over the
course of the experiment the subject's responses estimate the
likelihood ratio function being used.  ^
                                     p(F|X)      p(F|X)
                                     ------  vs. ------
                                       ^_          _
                                     p(F |X)     p(F|X)

The subject's "error" wrt observer is then something like the integral 
of the difference between these functions from the first to last cutoff.  
Although this measure seems intuitively to do the right sort of thing, I can't 
seem to figure out the cover story that justifies it.. (distortion in 
apprehending X with fixed criterion and/or accurate X with shifting criterion)..
It really seems to be a measure of scaling departures for the distortion or 
criterion shifts..

My alternative is to pick fixed cutoffs so I get
a reference to a Neyman-Pearson observer with the same alpha (punishes the
subject's variability in cutoff) or a Max-P(C) observer (punishes for distance
from the saddlepoint) neither of which quite capture what I want to measure..
which is: does one display provide subjects with more accurate evidence of
failures than the other...  

If any of this sounds familiar.. particularly estimating likelihood ratio
functions from detection data please point me in the right direction...

            -Mike Lewis  ml@icarus.lis.pitt.edu
From: jerry@ginger.hnrc.tufts.edu (Jerry Dallal)
Newsgroups: sci.math.stat
Subject: Re: Testing for Normality
Date: 6 Sep 92 16:09:52 GMT
Organization: USDA HNRC at Tufts University

In article <thompson.715720959@hermes.socsci.umn.edu>, thompson@atlas.socsci.umn.edu (T. Scott Thompson) writes:
> bateman@nsslsun.nssl.uoknor.edu (Monte Bateman) writes:
> 
>>I would like to test data for normality.  The books I have
>>access to talk about graphing frequencies on "probability paper".
> 
> Look up the Kolmogorov-Smirnov (sp?) statistic in a more advanced
> statistics text than the ones you have now.  

A clarification:  Using the K-S statistic to test for normality, 
calculated after plugging in the sample mean and variance in place of the 
population parameters, is known as
Lilliefors' [Strunk and White be damned!] test.

A Shapiro-Wilk test or a D'Agostino test would probably serve you better.
Lilliefors' test tends to be more sensitive to departures from normality
in the center of the distribution.  (One look at the test statistic will
reveal why.)  THe other tests tend to be more sensitive to departures
from normality in the tails, which is where non-normality is most often a
problem.  

This various tests have been compared most recently in, of all places, 
the Stata Technical Bulletin.  I dislike mentioning articles in a computer 
program's documentation/house organ because they are typically neither peer 
reviewed nor widely circulated, but it's there, so what can I say?
(I have heard rumors that Royston is preparing his contributions for formal 
publication, but they are only second hand rumors.)
From: bakera@topaz.ucq.edu.au (Alan Baker)
Newsgroups: sci.math.stat
Subject: Regression - Partition Inverse Problem
Date: 14 Sep 92 11:19:42 AET
Organization: University of Central Queensland, Australia

Hi,
I have a problem that I have not been able to sort out and would 
certainly appreciate some help.

In the regression model y = X*B + e
by writing y = x*B1 + Xs*Bs + e
where x is a T-vector, B1 a scalar, Xs a T*(K-1) matrix and Bs is 
a (K-1) vector.

The result var(B1) = sigma^2 / summation(x - xbar)^2(1-R^2)
can be derived (but how it is derived is the problem I cannot solve)
by applying partitioned inversion to X'X 
which comes from 
var-cov(b) = sigma^2(X'X)^-1

R^2 is the coefficient of determination obtained by regressing x on 
Xs.


This is a problem that has annoyed me for some days and if any one can 
shed some light on the matter I would certainly appreciate their help.
================================================================================
Alan Baker                             AARNET:  BAKER@COVAX.COMMERCE.UQ.OZ.AU
Department of Commerce                 TELEPHONE: Work  07 365 6686
University of  Queensland, QLD 4072    International  +617 365 6686
Australia	                                   Home 07 371 1386
================================================================================
From: fishwick@fish.cis.ufl.edu (Paul Fishwick)
Newsgroups: sci.math.stat
Subject: Time Series ftp Sites
Date: 7 Sep 92 18:51:50 GMT
Organization: Univ. of Florida CIS Dept.

Is there a location for freely obtaining time series data for several 
different domains (population, stock market, etc.)? An anonymous ftp site is
preferable. Thanks for any help on this.

-paul

+-------------------------------------------------------------------------+
| Paul A. Fishwick                                                        |
| Computer Simulation and Animation Group                                 |
| Dept. of Computer Science            EMAIL: fishwick@cis.ufl.edu        |
| University of Florida                PHONE: (904) 392-1414              |
| Bldg. CSE, Room 301                  FAX:   (904) 392-1220              |
| Gainesville, FL 32611                UUCP:  gatech!uflorida!fishwick    |
+-------------------------------------------------------------------------+
Organization: Queen's University at Kingston
Date: Tue, 15 Sep 1992 18:24:02 EDT
From: Jeff Boyd <BOYDJ@QUCDN.QueensU.CA>
Newsgroups: sci.math.stat
Subject: Re: two pairs


Does anyone else here feel that counting is akin to witchcraft?

I took a slightly different approach to this one, treating each pair
-AND- the odd card as selections from sets of 4.  The result is the
same, but I think it's a bit more rigid in its logic.


   1)  select 3 of 13 sets of 4 (two pair slots, one singleton slot)

               13c3

   2)  select 1 pair from one of the 3 sets of 4

                4c2

   3)  select 1 pair from another of the 3 sets of 4

                4c2

   4)  select 1 singleton from the last of the 3 sets of 4

                4c1

   5)  select 2 of the 3 sets of 4 for the pair slots, -OR-
       select 1 of the 3 sets of 4 for the singleton slot

                3c2  or  3c1

Then

    P[2 pair]  =  (13c3)(4c2)(4c2)(4c1)(3c1) / (52c5)

               =  0.047539
From: maj@waikato.ac.nz
Newsgroups: sci.math.stat
Subject: Re: Simple Proof that c=median minimizes E[ |X-c| ] needed.
Date: 7 Sep 92 09:32:35 +1200
Organization: University of Waikato, Hamilton, New Zealand

In article <3SEP199213440863@utkvx2.utk.edu>, menees@utkvx2.utk.edu (Menees, Bill) writes:
> 	I'm a senior math major taking my first p&s course, and this problem
> has come up and it intrigues me.  My prof. has a proof for it, but he said it
> was way over my head.  Does anyone know of a proof suitable for a senior 
> undergrad?  Thanks in advance.
This reminds me of an interesting approach to several
common location statistics. I'll state it for samples
to avoid technicalities.

Suppose we have a sample x_1, x_2, . . .,x_n.
Given a positive real number p define m to minimise

  Sum |x_i - m|**p

then we get

  p --> 0         mode
  
  p = 1           median
  
  p = 2           mean
  
  p --> infinity  midrange
  
Neat, eh?  
-- 
Murray A. Jorgensen [ maj@waikato.ac.nz ]    University of Waikato
Department of Mathematics and Statistics     Hamilton, New Zealand      
__________________________________________________________________
'Tis the song of the Jubjub! the proof is complete,
if only I've stated it thrice.'
Newsgroups: sci.math.stat
From: bateman@nsslsun.nssl.uoknor.edu (Monte Bateman)
Subject: Testing for Normality
Originator: usenet@kittyhawk.ecn.uoknor.edu
Date: Sat, 5 Sep 1992 06:46:47 GMT
Organization: National Severe Storms Laboratory

I would like to test data for normality.  The books I have
access to talk about graphing frequencies on "probability paper".

First off, I don't have any.

Second, it seems that there should be software available 
to do this test.

Pointers?

Any/all help appreciated!


Monte

--
===============================================================================
Monte Bateman, WB5RZX
Physics Ph.D. Candidate at The University of Oklahoma 
& The National Severe Storms Laboratory, Norman, OK! 

InterNet: bateman @ nsslsun.nssl.uoknor.edu
Newsgroups: sci.math.stat
Subject: Re: FULL HOUSE (REVISED)
From: a_rubin@dsg4.dse.beckman.com (Arthur Rubin)
Date: 15 Sep 92 21:27:25 GMT

In <1992Sep13.185343.26823@wuecl.wustl.edu> ppc1@cec1.wustl.edu (Peter Pui Tak Chiu) writes:



>// this previous version of this program assumes that we are drawing from
>// an infinite deck.  (which is a bug).

>// now this version has no such problem...
>// but still the probability is around 0.004...
>// which is four times the theoretical value 0.001441

The program looks more or less correct, although I only know C rather than
C++ so I can't verify the output line, but...

The low order bits of random numbers are freqently systematic.  If, for
instance, the low 2 bits of rand move systematically among the suits with
period 4, and, assuming the remaining bits are sufficiently random
(possibly true), we would have the probablity of a full house

(2/13)^3 = .00364
--
Arthur L. Rubin: a_rubin@dsg4.dse.beckman.com (work) Beckman Instruments/Brea
216-5888@mcimail.com 70707.453@compuserve.com arthur@pnet01.cts.com (personal)
My opinions are my own, and do not represent those of my employer.
My interaction with our news system is unstable; please mail anything important.
Newsgroups: sci.math.stat
From: nichols@spss.com (David Nichols)
Subject: Re: Testing for Normality
Organization: SPSS Inc.
Date: Wed, 9 Sep 1992 22:33:34 GMT

The Lilliefors version of the K-S test is in the SPSS EXAMINE procedure.

--
 David Nichols           Statistical Support Specialist            SPSS, Inc.
 Phone: (312) 329-3684     Internet: nichols@spss.com     Fax: (312) 329-3657
*******************************************************************************
 Any correlation between my views and those of SPSS is strictly due to chance.
Newsgroups: sci.math.stat
Subject: Re: two pairs
From: a_rubin@dsg4.dse.beckman.com (Arthur Rubin)
Date: 15 Sep 92 21:26:11 GMT

In <1992Sep15.024945.17690@wuecl.wustl.edu> ppc1@cec1.wustl.edu (Peter Pui Tak Chiu) writes:



>hi,

>can anyone point out what is wrong with my calculation of getting a hand
>of two pairs from a shuffled deck of cards?

>    the probability i have is:

>	(13 * 4C2) (12 * 4C2) (4 * 11)
>	------------------------------ = 0.095078
>		      52C5

>which is exactly two times the correct value (0.047539).

>pete

You can't distinguish between the pairs, so you have to divide by 2.
--
Arthur L. Rubin: a_rubin@dsg4.dse.beckman.com (work) Beckman Instruments/Brea
216-5888@mcimail.com 70707.453@compuserve.com arthur@pnet01.cts.com (personal)
My opinions are my own, and do not represent those of my employer.
My interaction with our news system is unstable; please mail anything important.
From: ECOGK@LURE.LATROBE.EDU.AU (KOROSI,Gabor)
Newsgroups: sci.math.stat
Subject: FORTRAN routine for fast Fourier Transform?
Date: 10 Sep 92 05:33:23 GMT
Organization: La Trobe University
X-News-Reader: VMS NEWS 1.22

I need a fortran subroutine which computes the fast Fourier transform of time
series. I would prefer a public domain routine, but pointers to standard
libraries would also be helpful. 


G\'abor K\H or\"osi

La Trobe University,          on leave from: Institute of Economics
Department of Econometrics                   Hungarian Academy of Sciences
Bundoora (Melbourne),                        Budapest
Victoria, Australia, 3083                    Hungary
Newsgroups: sci.math.stat
From: packer@amarna.gsfc.nasa.gov (Charles Packer)
Subject: Re: Testing for Normality
News-Software: VAX/VMS VNEWS 1.4-b1  
Organization: Dept. of Independence
Date: Fri, 11 Sep 1992 10:53:00 GMT

Why not use a traditional chi-square test?
From: desj@ccr-p.ida.org (David desJardins)
Newsgroups: sci.math.stat
Subject: Re: Darts
Date: 13 Sep 92 17:46:01 GMT
Organization: IDA Center for Communications Research, Princeton

Darren O'Shaughnessy <dos105@rsphy2.anu.edu.au> writes:
>   I would have expected the distribution of the darts to be approximately
>normal, but it appears that the distribution is flatter on top than a
>Gaussian curve with the same standard deviation, hence with relatively more
>darts falling around the one s.d. area.

Perhaps you should define what you mean by "the distribution of the
darts."  The distribution is obviously two-dimensional, but your other
comments sound like you are dealing with a one-dimensional distribution.
I don't know, for example, what "around the one s.d. area" means for
your two-dimensional distribution.

If you are looking at just the distance from the center as the
statistic, then if the distribution of the dart locations were Gaussian,
the probability distribution of the distance would definitely not be
Gaussian (nor half-Gaussian).

                                        David desJardins
From: manderse@orion.oac.uci.edu (Mark Andersen)
Subject: Re: CART methods and software?
Newsgroups: sci.math.stat
Organization: University of California, Irvine
Date: 15 Sep 92 18:38:20 GMT
Distribution: na

Many thanks to all who replied to my earlier request for information
on CART methods and software. I am very grateful. Once, again, the net
comes through.

thanks

Mark Andersen                   manderse@orion.oac.uci.edu
Dept. of Ecology and Evolutionary Biology
UC Irvine, Irvine, CA, 92717
Newsgroups: sci.math.stat