Inconsistency between PyTorch and TensorFlow's variance function's results and how PyTorch implements it using the summation function?

username: rahulbhalley
cooked: <p>I’m working on a vision model and it requires using variance function.</p>
<h2>
<a name="inconsistency-between-pytorch-and-tensorflow-variance-function-outputs-1" class="anchor" href="#inconsistency-between-pytorch-and-tensorflow-variance-function-outputs-1"></a>Inconsistency between PyTorch and TensorFlow variance function outputs</h2>
<p>For some reasons I am testing the TensorFlow and PyTorch’s variance functions and I find that the results for same input values are different. And it sounds weird because variance is a parameterless function, hence, there are no values to set that might affect the output value.</p>
<pre><code class="lang-python"># TensorFlow Example
import tensorflow as tf
x = tf.constant([[1., 2.], [3., 4.]])

print(tf.math.reduce_variance(x))
print(tf.math.reduce_variance(x, 0))
print(tf.math.reduce_variance(x, 1))

# Outputs
# tf.Tensor(1.25, shape=(), dtype=float32)
# tf.Tensor([1. 1.], shape=(2,), dtype=float32)
# tf.Tensor([0.25 0.25], shape=(2,), dtype=float32)
</code></pre>
<pre><code class="lang-python"># PyTorch Example
import torch as t
x = t.tensor([[1., 2.], [3., 4.]])

print(t.var(x))
print(t.var(x, 0))
print(t.var(x, 1))

# Outputs
# tensor(1.6667)
# tensor([2., 2.])
# tensor([0.5000, 0.5000])
</code></pre>
<h2>
<a name="implementation-help-with-pytorchs-var-using-sum-function-2" class="anchor" href="#implementation-help-with-pytorchs-var-using-sum-function-2"></a>Implementation help with PyTorch’s <code>var()</code> using <code>sum()</code> function</h2>
<p>Since my model is trained in PyTorch I want to know how to write the <code>var()</code> function using <code>sum()</code> function. The implementation I tried is as follows but the result doesn’t match with that of PyTorch’s official implementation:</p>
<pre><code class="lang-python">import torch as t
_ = t.manual_seed(0)

def var(x, dim):
  x_mean = t.mean(x)
  sq_devs = t.square(x - x_mean)
  return t.mean(sq_devs, dim=dim)

x = t.randn(5, 4, 3)
x_var1 = t.var(x, dim=0)
x_var2 = var(x, dim=0)

print("x:", x)
print("x_var1:", x_var1)
print("x_var2:", x_var2)
</code></pre>
<p>Output</p>
<pre><code class="lang-auto">x: tensor([[[-1.1258, -1.1524, -0.2506],
         [-0.4339,  0.8487,  0.6920],
         [-0.3160, -2.1152,  0.3223],
         [-1.2633,  0.3500,  0.3081]],

        [[ 0.1198,  1.2377,  1.1168],
         [-0.2473, -1.3527, -1.6959],
         [ 0.5667,  0.7935,  0.5988],
         [-1.5551, -0.3414,  1.8530]],

        [[ 0.7502, -0.5855, -0.1734],
         [ 0.1835,  1.3894,  1.5863],
         [ 0.9463, -0.8437, -0.6136],
         [ 0.0316, -0.4927,  0.2484]],

        [[ 0.4397,  0.1124,  0.6408],
         [ 0.4412, -0.1023,  0.7924],
         [-0.2897,  0.0525,  0.5943],
         [ 1.5419,  0.5073, -0.5910]],

        [[-0.5692,  0.9200,  1.1108],
         [ 1.2899, -1.4959, -0.1938],
         [ 0.4455,  1.3253, -1.6293],
         [-0.5497, -0.4798, -0.4997]]])
x_var1: tensor([[0.5831, 1.0012, 0.4474],
        [0.4593, 1.6567, 1.5645],
        [0.3082, 1.8627, 0.9353],
        [1.5127, 0.2319, 0.9604]])
x_var2: tensor([[0.4835, 0.8038, 0.5476],
        [0.4048, 1.3637, 1.2851],
        [0.2938, 1.5346, 0.7877],
        [1.3801, 0.2065, 0.8126]])
</code></pre>
<p>I’d really appreciate anyone willing to help me here. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>Thanks in advance,<br>
Rahul Bhalley</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think what you want is to pass <code>unbiased=False</code> to the <code>var</code> function in pytorch: <a href="https://pytorch.org/docs/stable/generated/torch.var.html?highlight=var#torch.var" class="inline-onebox">torch.var — PyTorch 1.8.1 documentation</a><br>
By default, pytorch compute the unbiased variance but I think Tensorflow computes the biased version.</p>

username: rahulbhalley
cooked: <p>Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"> It helps.</p>

username: rahulbhalley
cooked: <p>Hey <a class="mention" href="/u/alband">@albanD</a>! How can I implement the <code>var()</code> function using PyTorch’s <code>sum()</code> function?</p>

username: rahulbhalley
cooked: <p>I found the solution myself.</p>
<p>Following is an unbiased estimator implementation of variance using summation function, if anyone needs that:</p>
<pre><code class="lang-python">def var(input, dim, keepdim=False):
  input_means = t.mean(input, dim=dim, keepdim=True)
  difference = input - input_means
  squared_deviations = t.square(difference)
  return t.mean(squared_deviations, dim=dim, keepdim=keepdim)
</code></pre>