HDTenEightyP/NetNews huggingface dataset:
From: perlman@moose.cis.ohio-state.edu (Gary Perlman)
Subject: Re: |STAT now available via anonymous ftp
Organization: Computer & Info Sci  Ohio State Univ  Columbus, OH 43210
Date: Sun, 16 Aug 1992 03:07:17 GMT

In article <WVENABLE.92Aug15142816@algona.stats.adelaide.edu.au> wvenable@algona.stats.adelaide.edu.au (Bill Venables) writes:

>uuencoding is not necessary for files transferred by remote ftp, so long as
>"binary" mode is set during transmission.

I would not like the lack of access to uudecode to be a problem,
so I have placed a non-uuencoded file, stat.tar.Z.crypt, in
the pub/stat directory at ftp.cis.ohio-state.edu.

>I suppose it is well known that the "crypt" facility is still an illegal
>export from the US and (in theory) should not exist on any machine outside
>the US.  So persons like me outside the US intending to get |STAT this way
>will need first to get an account somewhere inside the US, remote login to
>it, ftp it there first, decrypt it there, re-ftp it home and so on.  But if
>that's the deal, then I guess that's the deal.

crypt is based on WWII technology and does not include controlled
encryption methods.  I use crypt in the MKS toolkit, imported into
the US from Canada.  Still, if you don't have crypt, then you don't have it.

The "deal" is that I do not want |STAT placed on a bulletin board or
and ftp site without some assurance that people will abide by its restrictions
(those basically being that you can't distribute modified copies).

I am willing to send stat.tar.Z.uu (without encryption) to those who
send me the line:
	I AGREE TO ADHERE TO THE CONDITIONS OF USING |STAT.
The message would be a little over 500 Kbytes.

Some people have sent me some messages about problems compiling |STAT
on their machines.  I have made some changes in the declarations of
functions to overcome these, but if you find more, then send me a
transcript of the compiler messages and I will try to address them.
-- 
Name:  Gary Perlman               | Computer and Information Science Department
Email: perlman@cis.ohio-state.edu | Ohio State University, 228 Bolz Hall
Phone: 614-292-2566               | 2036 Neil Avenue Mall
Fax:   614-785-9837 or 292-9021   | Columbus, OH 43210-1277  USA
Newsgroups: sci.math.stat
From: maj@waikato.ac.nz
Newsgroups: sci.math.stat
Subject: Re: Standard Deviation.
Date: 21 Aug 92 13:20:28 +1200
Organization: University of Waikato, Hamilton, New Zealand

In article <thompson.714333068@kiyotaki.econ.umn.edu>, thompson@atlas.socsci.umn.edu (T. Scott Thompson) writes:
> 
> ---------------------------------------------------------------------------
> Note:  (1) A brief lesson on the geometry of a sample mean;


If you enjoyed the brief lesson you may like to have a look at

Statistical Methods: The Geometric Approach
by Dave Saville and Graham Wood, Springer (1991)

> 
> T. Scott Thompson                 thompson@atlas.socsci.umn.edu
> Department of Economics
> University of Minnesota
-- 
Murray A. Jorgensen [ maj@waikato.ac.nz ]    University of Waikato
Department of Mathematics and Statistics     Hamilton, New Zealand      
__________________________________________________________________
'Tis the song of the Jubjub! the proof is complete,
if only I've stated it thrice.'
Newsgroups: sci.math.stat
From: dhinds@leland.Stanford.EDU (David Hinds)
Subject: Re: Fwd: Standard Deviation.
Organization: DSG, Stanford University, CA 94305, USA
Date: Sat, 15 Aug 92 00:00:49 GMT

In article <1992Aug14.231916.23479@magnus.acs.ohio-state.edu> regeorge@magnus.acs.ohio-state.edu (Robert E George) writes:
>
>More intuitively, if we take a very sample, we are less likely to get
>extreme values and so our notion of what the population variance is (note
>that I am not proposing some particular estimator) will be unrealistic.
>For instance, I give an exam to two students. Their scores are 67 and
>71. I think, "Gee, there's not a lot of variability in these scores."
>But then 8 more students take the exam:
>     60 78  100 38  50 88  99 39
>
>and it now is clear there *is* more variability in these scores.
>
>But let me reiterate that T will *always* have a negative bias for the
>population variance whatever the sample size is

This is wrong.  If this set of 8 values is the "population", sure, for
your particular sample, the predicted variance happens to be less than
the population variance.  However, the mean variance of all samples
from this population will equal the population variance (i.e., the
sample variance is an unbiased estimator).  Note that most samples of
size 2 from this population will have much larger variances than the
one you picked, and many will have variances larger than the parent
population (i.e., you are just as likely to get [100 38] as [67 71]).

The reason the N-weighted variance is biased, is because it measures
variability around the sample mean rather than the population mean,
and this estimate absorbs some of the variability in the sample.  The
N-weighted variance of the sample around the population mean, and the
N-1 weighted variance of the sample around the sample mean, are both
unbiased estimates of the variance in the population.

        - David Hinds
          dhinds@allegro.stanford.edu
Newsgroups: sci.math.stat
From: dmurdoch@mast.queensu.ca (Duncan Murdoch)
Subject: Re: Generating a set of random bits
Organization: Queen's University
Date: Thu, 20 Aug 1992 20:48:18 GMT

In article <1992Aug20.195901.2259@cc.tut.fi> vattu@cc.tut.fi (Vattulainen Ilpo) writes:
>>nmm@cl.cam.ac.uk (Nick Maclaren) writes:
>>Standard warnings:
>>
>>    2) Don't trust Numerical Recipes - about half of its techniques are
>>reliable, but the others contain serious traps.  
>
>	I agree.
 ...
>	are. But - do not use RAN3 (in Numerical Recipes ), if you 
>	want random bit-wise properties... This is due to my own 
>	experiences :-/

I think both people I've quoted above are being somewhat unfair to Numerical 
Recipes.  Perhaps it has changed in later editions, but in the original 
edition, it plainly says that none of the routines in the book are 
applicable to this very problem, and refers the reader to Knuth.  If you 
were burned by using RAN3 for a vector of random bits, you can't really blame
Press et al.

More generally, I think it's true of advice from most books (and certainly 
of that received over the net) that at least half of it contains traps for 
the unwary.  Saying this about Numerical Recipes in particular doesn't 
really convey any information.  

Thanks to both of the above for posting useful answers, as well as what 
I've picked out above.

Duncan Murdoch
dmurdoch@mast.queensu.ca
From: psrdj@warwick.ac.uk (G M Collis)
Newsgroups: sci.math.stat
Subject: Re: Standard Deviation.
Keywords: (n) versus (n-1)
Date: 17 Aug 92 09:35:38 GMT
Organization: Dept of Psychology, Warwick University, UK

What intrigues me is that the most elementary stats texts make a big
fuss about using n-1 for an unbiased estimate of the variance, but ignore
the fact that this gives a biased estimate for the SD. I recall
that n - 1.5 is nearer the target for the SD when the sample is
from a normally distributed population. I gather that minimising
the bias when estimating the SD is rather sensitive to the population
distribution - I'd like to know more about this.  But my big puzzle
remains - why is the biasedness of the usual SD estimator (with N-1)
so rarely mentioned, in stark contrast to the case of the variance.
Glyn M. Collis.
Newsgroups: sci.math.stat
From: jerry@ginger.hnrc.tufts.edu (Jerry Dallal)
Newsgroups: sci.math.stat
Subject: <None>
Date: 19 Aug 92 17:35:58 GMT
Organization: USDA HNRC at Tufts University

In article <57241@mentor.cc.purdue.edu>, hrubin@pop.stat.purdue.edu (Herman Rubin) writes:
> 
> As for statistical significance, neither I nor anyone else has been able
> to come up with any good reason to use it; it is a leftover from the time
> when it was misinterpreted.  
> 

I agree whole-heartedly.  The real problem is that many people have come up
with good *ways* to *mis*use it, from requiring statistical significance as a
prerequisite for publication to concentrating on statistical significance while
ignoring practical significance.  I see no quick way out of these traps,
although things seem to be improving ever so slightly over the long haul.  (Like
the hour hand on the clock, you don't see it move, but it moves.)
From: wvenable@algona.stats.adelaide.edu.au (Bill Venables)
Newsgroups: sci.math.stat
Subject: Re: Standard Deviation.
Date: 18 Aug 92 08:30:02 GMT
Organization: Department of Statistics, University of Adelaide
In-reply-to: psrdj@warwick.ac.uk's message of 17 Aug 92 09:35:38 GMT

>>>>> "Glyn" == G M Collis <psrdj@warwick.ac.uk> writes: 

Glyn> What intrigues me is that the most elementary stats texts make a big
Glyn> fuss about using n-1 for an unbiased estimate of the variance, but
Glyn> ignore the fact that this gives a biased estimate for the SD. I
Glyn> recall that n - 1.5 is nearer the target for the SD when the sample
Glyn> is from a normally distributed population. I gather that minimising
Glyn> the bias when estimating the SD is rather sensitive to the population
Glyn> distribution - I'd like to know more about this.  But my big puzzle
Glyn> remains - why is the biasedness of the usual SD estimator (with N-1)
Glyn> so rarely mentioned, in stark contrast to the case of the variance.

What surprises me is how this quaint little thread got going at all.  The
elementary books are wrong if they make a big issue of unbiasedness, period.

In this context the *two* important quantities are (a) the sum of squares,
since it is the squared length of the orthogonal projection of the
observation vector onto the residual space, and (b) the degrees of freedom,
which is the dimension of the residual space.  This latter number is
sometimes n-1, but more often n-p where p is somewhat larger than 1.  These
two quantities, *separately*, are what you need for virtually all
inferential procedures, like testing and confidence intervals.  Whether you
divide one by the other to give an estimate of the variance is up to you.
Incidently, if you do, it turns out to be unbiased, but "so what?", really.

In my opinion statistical inference is all about reliably capturing
information from data (and elsewhere if you are a Bayesian); it's not
really about coming up with a number from a data set that you can show will
be "close" to an unknown parameter value, in some special sense of "close".
The trouble with many elementary books is that they get hung up on a narrow
definition of "estimation" and elevate unbiasedness to an importance far in
excess of what is warranted, at the same time not mentioning sufficiency,
say, a far more important concept, (but harder to describe, of course).
--
___________________________________________________________________________
Bill Venables, Dept. of Statistics, | Email: venables@stats.adelaide.edu.au
Univ. of Adelaide, South Australia. | Tel: +61 8 228 5412  Fax: ...232 5670
Newsgroups: sci.math.stat
From: mokaba@binah.cc.brandeis.edu
Subject: LISREL factor scores
Reply-To: mokaba@binah.cc.brandeis.edu
Organization: Brandeis University
Date: Mon, 17 Aug 1992 16:12:03 GMT

A few weeks ago someone posted a question on the use or method of
calculating factor score when using LISREL. could he/she updateus on what
the answer to the question was. E-mail would be great.
thanks
From: rizzo@cbnewsf.cb.att.com (anthony.r.rizzo)
Newsgroups: sci.math.stat
Subject: THANKS! to all.
Date: 17 Aug 92 14:29:24 GMT
Organization: AT&T


                    #     #                   ###
                    #  #  #   ####   #    #   ###
                    #  #  #  #    #  #    #   ###
                    #  #  #  #    #  #    #    #
                    #  #  #  #    #  # ## #
                    #  #  #  #    #  ##  ##   ###
                     ## ##    ####   #    #   ###

My most sincere and most numerous thanks to ALL who've responded
to my request for an explanation (n versus n-1).
I've had so many responses, some brief and some quite involved,
that I won't be able to read them all at one sitting.
I've glanced at them all already, but quite a few of them
require STUDY.  To all those who obviously spent a good deal
of time preparing their responses, I thank you sincerely.
Rest assured that your efforts to educate me will not be wasted!


         end
        ----
        |
        |   Study(net responses)dt = Rizzo knows more!  ;-)
        |
     ----
     now

Thanks again to all!

Tony Rizzo (att.com!homxc!rizzo)

(There's no shortage of engineers.  There's only a shortage of engineering.)
Newsgroups: sci.math.stat
From: mokaba@binah.cc.brandeis.edu
Subject: Re: Fwd: Standard Deviation.
Reply-To: mokaba@binah.cc.brandeis.edu
Organization: Brandeis University
Date: Mon, 17 Aug 1992 21:57:53 GMT

In article <1992Aug14.231916.23479@magnus.acs.ohio-state.edu>, regeorge@magnus.acs.ohio-state.edu (Robert E George) writes:
>
>
.>> (LOTS OF INTERESTING STAFF DELETED TO PRESERVE SPACE)

>I'm not certain what "underestimate" means here: if you are speaking of
>bias (bias being the difference between the estimand and the expected
>value of the estimator), then
>
>        ---  _        _
>        \   | X - Xbar |2
>T =     /   |_        _|
>        ---
>       ------------------
>               n
>
>will "underestimate" the population variance *for any sample size n*.
>T will always have a negative bias *regardless of how large a sample is
>used to compute T*.
>
>More intuitively, if we take a very sample, we are less likely to get
>extreme values and so our notion of what the population variance is (note
>that I am not proposing some particular estimator) will be unrealistic.
>For instance, I give an exam to two students. Their scores are 67 and
>71. I think, "Gee, there's not a lot of variability in these scores."
>But then 8 more students take the exam:
>     60 78  100 38  50 88  99 39
>
>and it now is clear there *is* more variability in these scores.
>
>But let me reiterate that T will *always* have a negative bias for the
>population variance whatever the sample size is

MY IGNORAMUS QUESTION? Does't that negative bias decrease as the sample
size approaches population size? 

thanks in anticipation for answering my question.
Mokaba
From: rizzo@cbnewsf.cb.att.com (anthony.r.rizzo)
Newsgroups: sci.math.stat
Subject: Standard Deviation.
Keywords: (n) versus (n-1)
Date: 14 Aug 92 17:28:33 GMT
Article-I.D.: cbfsb.1992Aug14.172833.11844
Organization: AT&T

Can someone explain why calculating the Standard Deviation (SD),
for small samples, with (n-1) in the denominator is better than
doing so with (n) in the denominator?  I'm sure that there's
a perfectly good reason for doing so.  But we, lowly engineers
aren't usually told the reason.  Thanks now, for your response later.;-)

Tony Rizzo (att.com!homxc!rizzo)
Newsgroups: sci.math.stat
From: POKORNY@rzmain.rz.uni-ulm.de (POKORNY DAN)
Subject: Anybody from Dubrovnik/Croatia to COMPSTAT ?
Organization: University of Ulm, Germany
Date: Wed, 19 Aug 1992 21:09:28 GMT
X-News-Reader: VMS NEWS 1.23

Dobar dan,

I wish to contact anybody from Dubrovnik (or Dalmatia, or Croatia at least),
where the last statistical symposium COMPSTAT has been held,
who intends to go to the current COMPSTAT in Neuchatel/Schwitzerland
next week.
I will greatly appretiate her/his opinion to a certain idea I have got.

Thanks

Dan Pokorny, Dept. Pychotherapy, Univ. of Ulm
pokorny@rzmain.rz.uni-ulm.de
From: nmm@cl.cam.ac.uk (Nick Maclaren)
Newsgroups: sci.math.stat
Subject: Re: Generating a set of random bits
Date: 19 Aug 92 12:23:45 GMT
Reply-To: nmm@cl.cam.ac.uk (Nick Maclaren)
Organization: U of Cambridge Computer Lab, UK

In article <36984@sdcc12.ucsd.edu>, whart@cs.ucsd.edu (Bill Hart) writes:
|> I have an application for which I need to generate a random 'vector' of 
|> bits of length N.  I have looked at stuff in Numerical Recipies for
|> generating a random sequence of bits, but it remains unclear whether
|> such a generator would be appropriate for generating a random
|> vector of bits.
|> 
|> I need/want to have the distribution of vectors remain 'random'.  Right
|> now I'm generating a random float in [0,1> and testing to see
|> if it is less than 0.5 or not.  This works, but I'm sure there is
|> a better way of doing this.

Standard warnings:

    1) Your current technique is OK, but do NOT break up a random float
into multiple bits to save on efficiency.  Some techniques will survive
this but others will not.

    2) Don't trust Numerical Recipes - about half of its techniques are
reliable, but the others contain serious traps.  In this case it is not
too bad, though your reservations about whether its techniques extend to
vectors are fully justified.

Perhaps the best method for your application is a Tausworthe (also called
shift-register, primitive polynomial over GF(2), etc.) with a generating
polynomial of order at least N.  Knuth and others have details, but finding
a good one is non-trivial and dependent on the size of N, whether it varies
and what degree of randomness you need.  Much of the theory of this sort of
generator is in hardware-related journals.

I am afraid that there is no simple solution to the general case of random
vectors of bits, but there are several fairly reliable ones.


Nick Maclaren
University of Cambridge Computer Laboratory,
New Museums Site, Pembroke Street,
Cambridge CB2 3QG, England.
Email:  nmm@cl.cam.ac.uk
Tel.:   +44 223 334761
Fax:    +44 223 334679
Newsgroups: sci.math.stat
From: ted@dgbt.doc.ca (Ted Grusec)
Subject: HELP - WITHIN-S ANOVA
Organization: The Communications Research Centre
Date: Thu, 20 Aug 92 14:55:49 GMT

My CSS:Statistica (Statsoft) program will not analyze a 2 X 10 X 13
within-subject design. It indicates a 252 dependent variable limit,
while this experiment has 260.  All of this despite a recent update
to release 3.1 which I got just days ago, the accompanying literature
to which says ridiculous things like "....the new architecture of
this module [referring to ANOVA/MANOVA] allows it to break
practically all design limitations and now supports analysis designs
which previously could not be computed on any microcomputer or
mainframe platform."!!! [Tech support at Statsoft tells me they were
really referring to Between- subjects designs!!!!]

I've had Statistica for several years and have very frequently run
into size limits which are not related to my generous-enough
computing resources (500 meg hard disk, 16 meg memory, 386, etc.) but
are strictly a function of the program. I usually wind up using the
truly marvellous |Stat program of Gary Perlman which at least gives
me a complete ANOVA table from which I can do the various internal
comparisons I need, albeit with some computational labour on my part.

Is there any commercial package that will do "large" within-subject
designs? Can someone share experiences, make recommendations?
-- 
==========================================================================
Ted Grusec    Communications Research Centre    (Govt. of Canada, DOC)
 3701 Carling Ave., Ottawa, Ont.K2H 8S2, CANADA  Internet: ted@dgbt.doc.ca
  Compuserve: 73607,1576  (613) 998 2762   Fax (613) 993 8657
Newsgroups: sci.math.stat
From: choup@bionette.cgrb.orst.edu (Ping Chou)
Subject: Distribution of a function of Normal distributions
Organization: Biological Computing Consortium, OSU, Corvallis, OR.
Distribution: usa
Date: Tue, 18 Aug 1992 20:39:23 GMT


Quiz of the day:

Draw a random number, P, from a Normal distribution of mean=0, variance=Vp,
and a random number, Q, from a Normal distribution of mean=0, Variance=Vq,
and let R=P/Q.
What kind of distribution R will follow?
What is the probability density function of this distribution?

Ok, this is not a quiz, this is my headache. Any answer, pointer or hint
are greatly appreciated.

Ping
Newsgroups: sci.math.stat
From: vattu@cc.tut.fi (Vattulainen Ilpo)
Subject: Re: Generating a set of random bits
Organization: Tampere University of TeXnology
Date: Thu, 20 Aug 92 19:59:01 GMT


>In article <36984@sdcc12.ucsd.edu>, whart@cs.ucsd.edu (Bill Hart) writes:
>|> I have an application for which I need to generate a random 'vector' of 
>|> bits of length N.  
>
>nmm@cl.cam.ac.uk (Nick Maclaren) writes:
>Standard warnings:
>
>    2) Don't trust Numerical Recipes - about half of its techniques are
>reliable, but the others contain serious traps.  

	I agree.

>Perhaps the best method for your application is a Tausworthe (also called
>shift-register, primitive polynomial over GF(2), etc.) with a generating
>polynomial of order at least N.

	Indeed, although one cannot generalize the properties of 
	shift-register generators, especially many GFSR (generalized feedback
	shift-register) generators have been noticed to have good 
	statistical bit-wise properties. The results, however, depend 
	strongly on initialization and the choice of exponents in the 
	primitive polynomial (like p and q in a polynomial 
	x**p + x**q + 1 ). Hence it might be a quite good choice to 
	call such a generator with integers, and transform them to 
	bits. These words of bits you could use for your purpose. Notice, 
	however, that this rng should not be called to get real numbers 
	distributed [0,1) and then multiplied with some integer (like 
	2**p - 1, where p is the number of bits in a particular cpu). 
	This way you will get only 24 random bits out of 31 (on a 
	32-bit machine), due to 24-bit mantissas in real mode...

>I am afraid that there is no simple solution to the general case of random
>vectors of bits, but there are several fairly reliable ones.

	And one quite reliable method is to take only few most 
	significant bits out of random numbers. Usually all the bits 
	are not random, but at least 10 of the most significant ones 
	are. But - do not use RAN3 (in Numerical Recipes ), if you 
	want random bit-wise properties... This is due to my own 
	experiences :-/

	vattu@ee.tut.fi
From: vattu@cc.tut.fi (Vattulainen Ilpo)
Subject: Random matrices
Organization: Tampere University of TeXnology
Date: Thu, 20 Aug 92 21:07:30 GMT



	Lately I've been studing the properties of several random 
	number generators (rng), and sort of thing, and one 
	interesting application is the use of random bits in 
	cryptography. Now I'd need information of random matrices, 
	statistics of binary vectors and so on. Therefore, I would be 
	very grateful if anyone would be able to mention good references 
	concerning these subjects, especially random matrices. 
	References of books, articles etc. are all welcome.

	Thanks in advance.

-----------------------------------------------------------------------------
	Ilpo Vattulainen
	Department of Physics			vattulai@convex.csc.fi
	Tampere univ. of tech.			vattu@ee.tut.fi
	Finland
-----------------------------------------------------------------------------
Newsgroups: sci.math.stat