Use tensor.mean() on column ignoring certain numbers

jerron: <p>This is revisit this <a href="https://discuss.pytorch.org/t/use-tensor-mean-method-but-ignore-0-values/60170">old question</a>: How about mean on the columns for 2D array? torch.mean can take parameter dim to return mean for each column. Can we do so with mask filtering out certain bad values? Although we can loop through each column like following, is there better way?</p>
<pre><code class="lang-auto">       for i in  range(y['train'].shape[1]):
          mask=y['train'][:,i]!=bad_value
          masked_y=y['train'][:,i][mask]
          y_mean = torch.mean(masked_y)
          y_std = torch.std(masked_y)
          y['train'][:,i][mask]=(y['train'][:,i][mask]- y_mean) / y_std
          mask=y['val'][:,i]!=bad_value
          y['val'][:,i][mask]=(y['val'][:,i][mask]- y_mean) / y_std
          mask=y['test'][:,i]!=bad_value
          y['test'][:,i][mask]=(y['test'][:,i][mask]- y_mean) / y_std
</code></pre>

KFrank: <p>Hi Jerron!</p>
<aside class="quote no-group" data-username="jerron" data-post="1" data-topic="186680" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/48db29/48.png" class="avatar"> jerron:</div>
<blockquote>
<p>Can we do so with mask filtering out certain bad values?</p>
</blockquote>
</aside>
<p>This looks like one of the use cases for which pytorch’s <em>prototpye</em> <a href="https://pytorch.org/docs/2.0/masked.html" rel="noopener nofollow ugc">MaskedTensor</a><br>
was developed.</p>
<p>Here’s a simple example:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.0.1'
&gt;&gt;&gt; t = torch.arange (3.)
&gt;&gt;&gt; t
tensor([0., 1., 2.])
&gt;&gt;&gt; m = t != 2.
&gt;&gt;&gt; m
tensor([ True,  True, False])
&gt;&gt;&gt; mt = torch.masked.masked_tensor (t, m)
&lt;path_to_pytorch_install&gt;\torch\masked\maskedtensor\core.py:156: UserWarning: The PyTorch API of MaskedTensors is in prototype stage and will change in the near future. Please open a Github issue for features requests and see our documentation on the torch.masked module for further information about the project.
  warnings.warn(("The PyTorch API of MaskedTensors is in prototype stage "
&gt;&gt;&gt; mt
MaskedTensor(
  [  0.0000,   1.0000,       --]
)
&gt;&gt;&gt; t.mean()
tensor(1.)
&gt;&gt;&gt; mt.mean()
MaskedTensor(  0.5000, True)
</code></pre>
<p>Does this do what you want?</p>
<p>Best.</p>
<p>K. Frank</p>

jerron: <p>Thanks! It is exactly what I’m looking for. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>

jerron: <p>Hi Frank, May I follow up with the operation on the masked_tensor?<br>
So I got the mean for each columns of the masked matrix:</p>
<pre><code class="lang-auto">  mask=y!=bad_value
  my = torch.masked.masked_tensor (y, mask)
  y_mean,y_std=torch.mean(my,dim=0),torch.std(my,dim=0)
  y_mean,y_std
  (MaskedTensor(
   [  0.0643,   0.2096,  -0.0998]
 ),
 MaskedTensor(
   [  0.4513,   0.4316,   0.4149]
 ))
</code></pre>
<p>How to use it, for example, to get normalized value? It can be substracted and divided from normal tenor. But not with two masked tenor:<br>
This succeed:</p>
<pre><code class="lang-auto">my.shape, y.shape, (y-y_mean).shape,(y/y_std).shape
(torch.Size([9933, 3]), torch.Size([9933, 3]), torch.Size([9933, 3]), torch.Size([9933, 3]))
</code></pre>
<p>and this failed:</p>
<pre><code class="lang-auto">(y-y_mean)/y_std
ValueError                                Traceback (most recent call last)

&lt;ipython-input-17-7d6c5321b00a&gt; in &lt;cell line: 1&gt;()
----&gt; 1 (y-y_mean)/y_std

5 frames

/usr/local/lib/python3.10/dist-packages/torch/masked/maskedtensor/binary.py in _binary_helper(fn, args, kwargs, inplace)
     83 
     84     if not _masks_match(*args[:2]):
---&gt; 85         raise ValueError(
     86             "Input masks must match. If you need support for this, please open an issue on Github."
     87         )

ValueError: Input masks must match. If you need support for this, please open an issue on Github.
</code></pre>
<p>This also failed:</p>
<pre><code class="lang-auto">my-y_mean
ValueError                                Traceback (most recent call last)

&lt;ipython-input-20-b1b0b5b17e0f&gt; in &lt;cell line: 1&gt;()
----&gt; 1 my-y_mean

5 frames

/usr/local/lib/python3.10/dist-packages/torch/masked/maskedtensor/binary.py in _binary_helper(fn, args, kwargs, inplace)
     83 
     84     if not _masks_match(*args[:2]):
---&gt; 85         raise ValueError(
     86             "Input masks must match. If you need support for this, please open an issue on Github."
     87         )

ValueError: Input masks must match. If you need support for this, please open an issue on Github.
</code></pre>
<p>Thanks again!</p>

KFrank: <p>Hi Jerron!</p>
<aside class="quote no-group" data-username="jerron" data-post="4" data-topic="186680" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/48db29/48.png" class="avatar"> jerron:</div>
<blockquote>
<p>How to use it, for example, to get normalized value? It can be substracted and divided from normal tenor. But not with two masked tenor:</p>
</blockquote>
</aside>
<p>The short story is that it appears that <code>MaskedTensor</code>s don’t (currently)<br>
broadcast against other <code>MaskedTensor</code>s.</p>
<p>The somewhat longer story is that (for legitimate semantic reasons) pytorch<br>
doesn’t like to perform, say, element-wise addition on <code>MaskedTensor</code>s whose<br>
masks differ.  This probably bleeds over into the broadcasting case because<br>
you would have to sort through what you want the semantics of broadcasting<br>
the masks to be.</p>
<p>One straightforward approach to your use case:  Use <code>MaskedTensor</code>s to<br>
compute the <code>.mean()</code> and <code>.std()</code> and then convert them to regular <code>Tensors</code><br>
to perform the normalization with broadcasting.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.0.1
&gt;&gt;&gt;
&gt;&gt;&gt; bad_value = 3.0
&gt;&gt;&gt; y = torch.arange (15.).reshape (5, 3)
&gt;&gt;&gt;
&gt;&gt;&gt; mask = y != bad_value
&gt;&gt;&gt; my = torch.masked.masked_tensor (y, mask)
&lt;path_to_pytorch_install&gt;\torch\masked\maskedtensor\core.py:156: UserWarning: The PyTorch API of MaskedTensors is in prototype stage and will change in the near future. Please open a Github issue for features requests and see our documentation on the torch.masked module for further information about the project.
  warnings.warn(("The PyTorch API of MaskedTensors is in prototype stage "
&gt;&gt;&gt; y_mean, y_std = torch.mean (my, dim = 0), torch.std (my, dim = 0)
&gt;&gt;&gt;
&gt;&gt;&gt; my
MaskedTensor(
  [
    [  0.0000,   1.0000,   2.0000],
    [      --,   4.0000,   5.0000],
    [  6.0000,   7.0000,   8.0000],
    [  9.0000,  10.0000,  11.0000],
    [ 12.0000,  13.0000,  14.0000]
  ]
)
&gt;&gt;&gt; y_mean, y_std
(MaskedTensor(
  [  6.7500,   7.0000,   8.0000]
), MaskedTensor(
  [  5.1235,   4.7434,   4.7434]
))
&gt;&gt;&gt;
&gt;&gt;&gt; my.shape, y.shape, (y - y_mean).shape, (y / y_std).shape
(torch.Size([5, 3]), torch.Size([5, 3]), torch.Size([5, 3]), torch.Size([5, 3]))
&gt;&gt;&gt;
&gt;&gt;&gt; y - y_mean    # MaskedTensor y_mean broadcasts against regular Tensor y
MaskedTensor(
  [
    [ -6.7500,  -6.0000,  -6.0000],
    [ -3.7500,  -3.0000,  -3.0000],
    [ -0.7500,   0.0000,   0.0000],
    [  2.2500,   3.0000,   3.0000],
    [  5.2500,   6.0000,   6.0000]
  ]
)
&gt;&gt;&gt;
&gt;&gt;&gt; # my - y_mean   # fails with ValueError -- y_mean doesn't broadcast against MaskedTensor my
&gt;&gt;&gt;
&gt;&gt;&gt; # one way to do this -- convert to regular Tensors for broadcasting ...
&gt;&gt;&gt;
&gt;&gt;&gt; nan = float ('nan')   # use nans to track masked values
&gt;&gt;&gt;
&gt;&gt;&gt; y_norm = (my.to_tensor (nan) - y_mean.to_tensor (nan)) / y_std.to_tensor (nan)   # regular Tensors will broadcast
&gt;&gt;&gt; my_norm = torch.masked.masked_tensor (y_norm, ~y_norm.isnan())   # convert back to MaskedTensor, if desired
&gt;&gt;&gt;
&gt;&gt;&gt; y_norm
tensor([[-1.3175, -1.2649, -1.2649],
        [    nan, -0.6325, -0.6325],
        [-0.1464,  0.0000,  0.0000],
        [ 0.4392,  0.6325,  0.6325],
        [ 1.0247,  1.2649,  1.2649]])
&gt;&gt;&gt; my_norm
MaskedTensor(
  [
    [ -1.3175,  -1.2649,  -1.2649],
    [      --,  -0.6325,  -0.6325],
    [ -0.1464,   0.0000,   0.0000],
    [  0.4392,   0.6325,   0.6325],
    [  1.0247,   1.2649,   1.2649]
  ]
)
&gt;&gt;&gt;
&gt;&gt;&gt; # check normalization
&gt;&gt;&gt; y_norm.mean (dim = 0)
tensor([nan, 0., 0.])
&gt;&gt;&gt; y_norm.std (dim = 0)
tensor([nan, 1., 1.])
&gt;&gt;&gt;
&gt;&gt;&gt; my_norm.mean (dim = 0)
MaskedTensor(
  [ -0.0000,   0.0000,   0.0000]
)
&gt;&gt;&gt; my_norm.std (dim = 0)
MaskedTensor(
  [  1.0000,   1.0000,   1.0000]
)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>