Talk:Algorithms for calculating variance

message: Most all the tests I've seen of these algorithms add some unrealistic constant (i.e. 10^6 or larger) to the dataset to demonstrate that the suggested algorithm on this page is indeed better. I naively used this algorithm in my own work, to horrible effect. My dataset consists of a large number of discrete values, perhaps with the values -1, 0, or 1, and with an average usually between -1 and 1. I wrote the following simple test program to demonstrate the difference in results between what I'll call METHOD1 (using a running sum and sum of squares of the dataset) and METHOD2 (using a running computation of the average and the variance, which the current wiki strongly recommends).
#include <iostream>
#include <iomanip>
#include <cmath>
#include <cstdlib>
using namespace std;
int main( )
{
	srand( time( NULL ) );
	const double target = 0.95;
	float sum = 0;
	float average = 0;
	float sumsq = 0;
	float qvalue = 0;
	double sumd = 0;
	double averaged = 0;
	double sumsqd = 0;
	double qvalued = 0;
	int numtrials = 0;
	const int width = 15;
	cout << setw( width ) << left << "numtrials"
		<< setw( width ) << "float avg 2"
		<< setw( width ) << "float avg 1"
		<< setw( width ) << "double avg 2"
		<< setw( width ) << "double avg 1"
		<< setw( width ) << "float std 2"
		<< setw( width ) << "float std 1"
		<< setw( width ) << "double std 2"
		<< setw( width ) << "double std 1"
		<< endl;
	while( true )
	{
		for( int i = 0; i < 1000000; i++ )
		{
			const int sample = ( static_cast< double >( rand( ) ) / RAND_MAX < target ? 1 : 0 );
			numtrials++;
			sum += sample;
			sumd += sample;
			const float delta = sample - average;
			average += delta / numtrials;
			const double deltad = sample - averaged;
			averaged += deltad / numtrials;
			sumsq += sample * sample;
			sumsqd += sample * sample;
			qvalue += delta * ( sample - average );
			qvalued += deltad * ( sample - averaged );
		}
		cout << fixed << setprecision( 6 );
		cout << setw( width ) << left << numtrials 
			<< setw( width ) << average 
			<< setw( width ) << sum / numtrials 
			<< setw( width ) << averaged
			<< setw( width ) << sumd / numtrials 
			<< setw( width ) << sqrt( qvalue / ( numtrials - 1 ) )
			<< setw( width ) << sqrt( ( sumsq - ( sum / numtrials ) * sum ) / ( numtrials - 1 ) )
			<< setw( width ) << sqrt( qvalued / ( numtrials - 1 ) )
			<< setw( width ) << sqrt( ( sumsqd - ( sumd / numtrials ) * sumd ) / ( numtrials - 1 ) )
			<< endl;
	}
	return 0;
}
And here sample output:
numtrials      float avg 2    float avg 1    double avg 2   double avg 1   float std 2    float std 1    double std 2   double std 1
1000000        0.948275       0.950115       0.950115       0.950115       0.218147       0.217707       0.217707       0.217707
2000000        0.941763       0.949966       0.949966       0.949966       0.217107       0.218015       0.218015       0.218015
3000000        0.922894       0.949982       0.949982       0.949982       0.217433       0.217982       0.217982       0.217982
4000000        0.909789       0.950044       0.950044       0.950044       0.215531       0.217854       0.217854       0.217854
5000000        0.899830       0.950042       0.950042       0.950042       0.219784       0.217859       0.217859       0.217859
6000000        0.890922       0.950006       0.950006       0.950006       0.218891       0.217933       0.217933       0.217933
7000000        0.884997       0.950047       0.950047       0.950047       0.215908       0.217848       0.217848       0.217848
8000000        0.879075       0.950082       0.950082       0.950082       0.213635       0.217776       0.217776       0.217776
9000000        0.873134       0.950091       0.950091       0.950091       0.214217       0.217758       0.217758       0.217758
10000000       0.868035       0.950095       0.950095       0.950095       0.219110       0.217749       0.217749       0.217749
11000000       0.865048       0.950076       0.950076       0.950076       0.220991       0.217788       0.217788       0.217788
12000000       0.862079       0.950086       0.950086       0.950086       0.218815       0.217768       0.217768       0.217768
13000000       0.859129       0.950118       0.950118       0.950118       0.216916       0.217701       0.217701       0.217701
14000000       0.856129       0.950086       0.950086       0.950086       0.215379       0.217768       0.217768       0.217768
15000000       0.853163       0.950096       0.950096       0.950096       0.213971       0.217746       0.217746       0.217746
16000000       0.850167       0.950074       0.950074       0.950074       0.212786       0.217793       0.217793       0.217793
17000000       0.847186       0.950069       0.950069       0.950069       0.211621       0.217803       0.217803       0.217803
18000000       0.844209       0.932068       0.950068       0.950068       0.210246       0.251630       0.217805       0.217805
19000000       0.841231       0.883011       0.950066       0.950066       0.209009       0.321407       0.217808       0.217808
20000000       0.838260       0.838861       0.950071       0.950071       0.207879       0.367659       0.217798       0.217798
21000000       0.835285       0.798915       0.950072       0.950072       0.206857       0.400811       0.217797       0.217797
22000000       0.832305       0.762601       0.950069       0.950069       0.205931       0.425489       0.217803       0.217803
23000000       0.829313       0.729444       0.950057       0.950057       0.205096       0.444247       0.217828       0.217828
24000000       0.826340       0.699051       0.950059       0.950059       0.204305       0.458671       0.217823       0.217823
25000000       0.823366       0.671089       0.950061       0.950061       0.203576       0.469818       0.217819       0.217819
26000000       0.820379       0.645278       0.950055       0.950055       0.203316       0.478429       0.217832       0.217832
27000000       0.817389       0.621378       0.950047       0.950047       0.202405       0.485044       0.217849       0.217849
28000000       0.816234       0.599186       0.950046       0.950046       0.201543       0.490063       0.217849       0.217849
29000000       0.816234       0.578525       0.950056       0.950056       0.200723       0.493795       0.217829       0.217829
30000000       0.816234       0.559241       0.950035       0.950035       0.200000       0.496478       0.217872       0.217872
31000000       0.816234       0.541201       0.950036       0.950036       0.199291       0.498300       0.217871       0.217871
32000000       0.816234       0.524288       0.950039       0.950039       0.198619       0.499410       0.217864       0.217864
33000000       0.816234       0.508400       0.950038       0.950038       0.197993       0.499929       0.217867       0.217867
34000000       0.816234       0.493448       0.950034       0.950034       0.197406       0.499957       0.217876       0.217876
35000000       0.816234       0.479349       0.950037       0.950037       0.196839       0.499573       0.217868       0.217868
36000000       0.816234       0.466034       0.950038       0.950038       0.196306       0.498845       0.217866       0.217866
37000000       0.816234       0.453438       0.950037       0.950037       0.195804       0.497827       0.217868       0.217868
38000000       0.816234       0.441506       0.950036       0.950036       0.195328       0.496567       0.217871       0.217871
39000000       0.816234       0.430185       0.950032       0.950032       0.194878       0.495102       0.217878       0.217878
Note how the computed average using float's and method 2 fails to six digits accuracy before even 1 million trials, while method 1 using floats reproduces the double results all the way out to 17 million trials.  — Preceding unsigned comment added by 173.219.85.45 (talk) 19:41, 19 October 2012 (UTC)
replies: text: Method 1's sum += sample is exact (with sample values 1 and 0) until it has no effect, so method 1 fails catastrophically after about 18 million.  Method 2's average += delta / numtrials is generally imprecise, and you can't expect single precision to be great here for numtrials in 6 or 7 digits, but it's not too far off on average, so even with single floats and these large numbers of samples the estimated variances are not too unreasonable. Dicklyon (talk) 21:18, 22 May 2018 (UTC)
thread_title:  Online algorithm in testing yields horrible results when mean=~0 

message: I'm pretty sure this is wrong. From Welford's Paper:
S[n]=S[n-1]+(n-1)/n*(x[n]-mean[n-1])^2
--cfp 21:58, 15 August 2006 (UTC)
replies: text: I'm talking rubbish it's fine as x[n]-mean[n] = (n-1)/n*(x[n]-mean[n-1]). Sorry! --cfp 22:20, 15 August 2006 (UTC)
thread_title:  Algorithm III 

message: OSFTactical (talk) 22:59, 6 December 2007 (UTC)
thread_title: Minor spelling and grammar fixes

message: On 17/11/2007
Hi, 
I suspect that the two end formulaes given for the third algorithm :
  variance_n = sqrt(M2/n)
  variance = sqrt(M2/(n - 1))
are not the "variance" but the "standard deviation" (Ecart-type in French)
Can someone check it ?
Pascal P.
90.37.121.130 (talk) 13:52, 17 November 2008 (UTC)
thread_title: Suspecting a formulea error

message: Hi all,
I was looking at Murray Spiegel's Probability and Statistics (1994), and it lists this formulation for variance:
Var(X) = E[X^2]-(E[X])^2.
Mathworld had a similar formulation for standard deviation. This all means that, as long as you keep running tabs of:
- sum of all samples
- sum of squares of all samples
- number of samples
then you have an easy way to compute the variance as you go. You also have an easy way to combine the variances of two populations. Just add the sums (and sums of squares) and the counts. No divide-by-zeros. Very few math operations. What more can you ask for? :)
: That is already in the article (the first algorithm), and the problem with it is stated there.  McKay (talk) 02:14, 2 May 2009 (UTC)
replies: text:  Ah, well the connection between the two wasn't spelled out to the satisfaction of my (possibly caffeine-deprived) mind. And, as far as I can tell, the online algorithm does not easily merge statistics from multiple populations, but rather updates the statistic one sample at a time. Correct me if I am wrong, or please add a suitable method to the page. 209.237.23.36 (talk) 22:56, 13 May 2009 (UTC)
replies: text:  The connection to the second equation of the "naive algorithm" follows immediately from the definition of E[X^2] and E[X]. The "parallel algorithm" is an online algorithm that easily merges statistics from multiple population (as noted in the text), though a better heading might make this clear. The key point is that a parallel algorithm can be implemented as a repeated serial algorithm, and the merging part is identical in both. Markjamesabraham (talk) 22:45, 26 May 2010 (UTC)
thread_title:  Easiest online algorithm yet. 

message: The numerically stable mean and variance computation works, but attempting something similar for skewness and kurtosis using the incremental moments computation code in the later section seems to fail. Is something missing in it ? Are there any reference implementations ? Shyamal (talk) 16:17, 14 July 2009 (UTC)
replies: text: I've implemented the Terriberry moment-based method for skewness and kurtosis in the incremental case (i.e. B={x}) and it worked fine. Markjamesabraham (talk) 22:36, 26 May 2010 (UTC)
replies: text: Some published code is at http://lib.stat.cmu.edu/apstat/52 , which also indicates its original source. Melcombe (talk) 12:13, 27 May 2010 (UTC)
replies: text: Actually I subsequently found that Chan's online-merging algorithm does fine for higher-order moments, but calculates the mean very badly. When A and B have large and comparable size, \bar x_X = \bar x_A + \delta\cdot\frac{n_B}{n_X} for \delta\! = \bar x_B - \bar x_A is not as numerically stable as when one of A and B are very much larger than the other. The error contribution from the subtraction that forms \delta is scaled only by about 1/2, whereas when n_A >> n_B = 1 the error contribution from the subtraction is divided by n_A + 1. Instead, prefer to use \bar x_X = \frac{n_A \bar x_A + n_B \bar x_B}{n_A + n_B}. Main page updated accordingly Markjamesabraham (talk) 14:54, 30 May 2010 (UTC)
thread_title: Pseudocode

message: I cannot make sense of the "compensated" algorithm.  It does not appear to be using ordinary Kahan summation.
Bkalafut (talk) 01:19, 9 July 2010 (UTC)
thread_title:  Compensated variant 

message: Actually I just wanted to ask that on the page  :Algorithms for calculating variance is the pseudocode that is given for calculating variance correct?
variance = E(X**2) - (E(X))**2 
Computational formula for the variance
Should'nt it be mean*mean as opposed to Sum * mean? 
Thanks...
````  —Preceding unsigned comment added by Nk28 (talk • contribs) 06:59, 12 October 2010 (UTC)
thread_title:  Hey 

message: The weighted variance pseudocode seems correct to me; I just did a back-of-the-envelope check (but feel free to check it more thoroughly!). I've removed the misleading comment in the pseudocode (#[WARNING] This seems wrong. It should be moved before M2 assignment). This comment doesn't seem to make much sense; one can easily show that it yields the wrong result for the trivial case in which the weights are all equal to one. 
Theaucitron (talk) 11:07, 21 August 2012 (UTC)
thread_title:  Weighted variance pseudocode correctness 

message: Is this algorithm right? I'm not an expert, but when I compare it to the online variance computation, it's different, and if I replace y with x, I should get cov(x,x) = var(x), so could anyone have a look please? 131.227.95.222 (talk) 18:03, 27 February 2013 (UTC)
replies: text: Alright, I found a paper talking about this and now I understand that the formula is talking about co-moments and not covariances, I will add the note to the article. 131.227.95.222 (talk) 10:31, 28 February 2013 (UTC)
thread_title:  Online algorithm for covariance 

message: The wikipedia page for kurtosis claims that unbiased estimation of kurtosis is possible for normally distributed inputs. However, when I run the code my results are clearly biased.
[hs4 ~] 30$ ~/tmp/simple_kurt.py
Average kurtosis: -0.545012387836
[hs4 ~] 30$ ~/tmp/simple_kurt.py 
Average kurtosis: -0.536851508033
[hs4 ~] 30$ ~/tmp/simple_kurt.py
Average kurtosis: -0.55578863384
[hs4 ~] 30$ ~/tmp/simple_kurt.py
Average kurtosis: -0.546190322949
Here's the code I'm running:
#!/usr/bin/python2.7
from __future__ import division
from random import gauss
def kurtosis(data):
    n = 0
    mean = 0
    M2 = 0
    M3 = 0
    M4 = 0
 
    for x in data:
        n1 = n
        n = n + 1
        delta = x - mean
        delta_n = delta / n
        delta_n2 = delta_n * delta_n
        term1 = delta * delta_n * n1
        mean = mean + delta_n
        M4 = M4 + term1 * delta_n2 * (n*n - 3*n + 3) + 6 * delta_n2 * M2 - 4 * delta_n * M3
        M3 = M3 + term1 * delta_n * (n - 2) - 3 * delta_n * M2
        M2 = M2 + term1
 
    kurtosis = (n*M4) / (M2*M2) - 3
    return kurtosis
max_datasets = 10000
sum_kurtosis = 0
for lv in range(max_datasets):
    data = [gauss(0, 1) for i in range(10)]
    sum_kurtosis += kurtosis(data)
print "Average kurtosis:", sum_kurtosis/max_datasets
208.77.214.129 (talk) 21:02, 3 June 2013 (UTC)
thread_title:  Kurtosis algorithm biased for normal distribution? 

message: def weighted_incremental_variance(dataWeightPairs):
...
        M2 = M2 + sumweight * delta * R  # Alternatively, "M2 = M2 + weight * delta * (x−mean)"
...
I did the math to see if the stated update formula is indeed correct with the alternative mentioned in the comment; I have found that it is not. As such, I propose the alternative comment be removed.
If anyone would like me to do show the math of why there are not equivalent, let me know. Basically, the alternative confuses the sumweight of the previous iteration and the sumweight of the current iteration (which is updated after M2 is calculated).  — Preceding unsigned comment added by Anishpatel0 (talk • contribs) 13:11, 29 April 2014 (UTC)
thread_title:  Weighted Incremental Algorithm Alternative Incorrect 

message: There are algorithms in the article which appear to be in some computer language or logical/mathematical notation.   What is the language?  Its syntax differs from programming languages I studied back in the day. It is not self-evident and self-explanatory.  A link to some article dealing with the conventions and assumptions of the language would be helpful. None of the examples in Algorithm or in Algorithm examples are quite like the ones here, whose syntax appears to require they start with "DEF" and end with "RETURN."  These algorithms have "do loops" which do not include an index telling it how many times to execute the statement in the loop (such as "for i = 1 to n", or a test at the end of each loop execution to determine if all the data items have been processed. Edison (talk) 18:01, 26 May 2014 (UTC)
replies: text: Looks like Python_(programming_language) to me -- RoySmith (talk) 00:56, 30 May 2014 (UTC)
replies: text:  Python - style pseudocode. The details you mention are omitted here to highlight the main points, and are omitted in practice in many programming languages. --Hroðulf (or Hrothulf) (Talk) 13:32, 23 July 2014 (UTC)
thread_title: What language are the algorithms in?

message: There was a merge tag added at Standard deviation#Rapid calculation methods to this article. I think it is quite reasonable to have that section be a summary of this article. I must admit I was rather surprised I couldn't find a version of the \operatorname{E}\left[(X-\operatorname{E}(X))^2\right] = \operatorname{E}\left[X^2 \right] - (\operatorname{E}[X])^2 formula in the standard deviation article, I've put a bit in its talk page about that and that that article is slow referring to variance and never links to it.
For this article as old speedup calculation method is at assumed mean. Dmcq (talk) 09:19, 17 May 2013 (UTC)
thread_title:  Merge discussion 

message: Revisions of this page up to  referred to algorithms by number.  Next revision improved the titles like so:
* I. Naïve algorithm
* II. Two-pass algorithm
* III. On-line algorithm
* IV. Weighted incremental algorithm
Revision of  further "corrected" titles by eliminating roman numerals.  Today's revision of  replaced roman numerical references from the text according to the above map. ale (talk) 10:32, 29 February 2016 (UTC)
replies: text: 2. I think the comment "These formulas suffer from numerical instability, as they repeatedly subtract a small number from a big number which scales with n" is actually wrong. Maybe there is an accuracy issue, but I don't see why there should be an instability here. A reference would be most welcome. Natchouf (talk) 14:08, 20 March 2023 (UTC)
thread_title:  Algorithms by (uppercase roman) number mapped to names 

message: The second shown formula in this section does not compute the population variance \sigma^2, it rather computes the sample variance s^2. This can easily be seen when comparing the (wrong) formula divisor n-1 with the respective divisor n as used in the section "Naive algorithm" just above (where capital N is used instead of n).
The comments in the corresponding code snippet below makes the situation a bit clearer. Maybe one could write the code explicitly as
 ...
 // for sample variance use
 variance = (Ex2 - Ex**2 / n) / (n - 1)
 // for population variance use
 // variance = (Ex2 - Ex**2 / n) / n
 ...
141.249.133.134 (talk) 06:17, 10 April 2024 (UTC)
thread_title:  Typo in "Computing shifted data" section