Logit transform of dataset

username: kirk86
cooked: <p>Someone suggested me to do a logit transform of the dataset before passing it to the model. Has anyone heard of it before?</p>
<p>Is the following correct in the sense of performing logit transform to the dataset? Or should I do it before any other transformation occurs.</p>
<pre><code class="lang-auto"> transform_train = tr.Compose(
            [tr.Resize((32, 32)),  # &lt;-- should I put logit transform here?
             tr.Pad(4, padding_mode="reflect"),
             tr.RandomCrop(im_sz),
             tr.RandomHorizontalFlip(),
             tr.ToTensor(),
             tr.Normalize((.5, .5, .5), (.5, .5, .5)),
             lambda x: x + args.sigma * torch.randn_like(x),
             lambda x: -torch.log(1/x - 1)]  # &lt;-- logit transform
        )
</code></pre>

username: KFrank
cooked: <p>Hello Kirk!</p>
<aside class="quote no-group" data-username="kirk86" data-post="1" data-topic="83351" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kirk86/48/151_2.png" class="avatar"> kirk86:</div>
<blockquote>
<p>Someone suggested me to do a logit transform of the dataset before passing it to the model. Has anyone heard of it before?</p>
</blockquote>
</aside>
<p>I am somewhat skeptical of the idea.  I certainly wouldn’t do it without<br>
also building an equivalent model without the logit transformation and<br>
checking that the logit transformation improves things.</p>
<aside class="quote no-group">
<blockquote>
<p>Is the following correct in the sense of performing logit transform to the dataset?</p>
<pre><code class="lang-auto"> transform_train = tr.Compose(
            [tr.Resize((32, 32)),  # &lt;-- should I put logit transform here?
...
             tr.Normalize((.5, .5, .5), (.5, .5, .5)),
             lambda x: x + args.sigma * torch.randn_like(x),
             lambda x: -torch.log(1/x - 1)]  # &lt;-- logit transform
        )
</code></pre>
</blockquote>
</aside>
<p>Some comments:</p>
<p>Your logit function looks right.</p>
<p>I think you have to wrap your <code>lambda</code> in a pytorch <code>tr.Lambda</code><br>
transformation.</p>
<p>The logit function maps a probability in <code>[0.0, 1.0]</code> to <code>[-inf, inf]</code>.<br>
There is no reason to think that your data reside in <code>[0.0, 1.0]</code>,<br>
and, even if they did, the <code>tr.Normalize</code> could well push them outside<br>
of <code>[0.0, 1.0]</code>.  (The mean of 0.5 and standard deviation of 0.5 don’t<br>
mean that the result will lie strictly withing <code>[0.0, 1.0]</code>; deviations<br>
of one-and-a-half and two standard deviations are quite likely, so after<br>
<code>tr.Normalize</code>, you are likely to have values outside of <code>[0.0, 1.0]</code>.)</p>
<p>If your raw input data were naturally probabilities I think it would make<br>
sense to apply the logit transformation.  But otherwise my intuition is<br>
that doing so is not likely to be helpful.  In any event, the raw data, or<br>
a transformed version of it, had better lie within <code>[0.0, 1.0]</code>; otherwise<br>
the logit transformation will mix things up (it’s not monotone), and will<br>
risk (unnecessary) divergences.</p>
<p>Best.</p>
<p>K. Frank</p>

username: kirk86
cooked: <p>Hi K. Frank,</p>
<p>thanks for the reply. Let me give a bit of context which I think might help clarify things.<br>
Basically I’m trying what ppl call an “energy based model” (disclaimer: I hate this name, it’s just a probabilistic model), on fashion mnist. Basically since fmnist has only one channel because I didn’t wanted to change the underlying model the author was using, I just used <code>tr.Grayscale(num_output_channels=3)</code> to transform it having 3 channels.</p>
<p>Training the model I can only get as high as 83% in terms of accuracy. So, I reached to the original authors for any tips and one of the recommended tips was the logit transform.</p>
<p>Now that I’ve tried it I can tell with certainty that it gave worse result than without it.</p>