How to prevent inf while working with exponential

Bled_Clement: <p>Hi there,</p>
<p>I’m trying to create  a function in a network with trainable parameters. In my function I have an exponential that for large tensor values goes to infinity. What would the best way to avoid this be?</p>
<p>The function is as follows:</p>
<pre><code class="lang-auto">step1 = Pss-(k*Pvv)
step2 = step1*s
step3 = torch.exp(step2)
step4 = torch.log10(1+step3)
step5 = step4/s

#or equivalently
# train_curve = torch.log(1+torch.exp((Pss-k*Pvv)*s))/s
</code></pre>
<p>If it makes it easier to understand, the basic function is log10(1+e^(x-const)*10)/10. The exponential inside the log gets too big and goes to inf.</p>
<p>I think I might have to normalize my tensor x, and this would mean normalizing the constants and the rest of the function also. Would someone have any thoughts on the best way to go about this?</p>
<p>Thanks so much.</p>

KFrank: <p>Hi Clement!</p>
<aside class="quote no-group" data-username="Bled_Clement" data-post="1" data-topic="74622" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bled_clement/48/21076_2.png" class="avatar"> Bled_Clement:</div>
<blockquote>
<p>The exponential inside the log gets too big and goes to inf.</p>
</blockquote>
</aside>
<p>Please adapt the <a href="https://en.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_for_log-domain_calculations" rel="noopener nofollow ugc">log-sum-exp trick</a> to your specific use case.</p>
<p>(Also, of course, although related, <code>log10()</code> is not the same as<br>
<code>log()</code> (“natural log,” log base-e").</p>
<p>Best.</p>
<p>K. Frank</p>

Bled_Clement: <p>I tried this but couldn’t get it to work for my application. I’m going to give it another go, and possibly modify the source code to use base 10. Should it be as simple as replacing torch.log with <code>torch.</code> <code>logsumexp()</code>?</p>

KFrank: <p>Hello Clement!</p>
<aside class="quote no-group" data-username="Bled_Clement" data-post="3" data-topic="74622" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bled_clement/48/21076_2.png" class="avatar"> Bled_Clement:</div>
<blockquote>
<p>I tried this but couldn’t get it to work for my application.</p>
</blockquote>
</aside>
<p>Please feel free to post a short, complete, runnable script<br>
that has the original code you posted (that fails when the<br>
tensor values become to large), together with your improved<br>
attempt (using <code>logsumexp()</code> or whatever).  If you show us<br>
what fails or what the discrepancy is, someone here will<br>
probably be able to help you out.</p>
<aside class="quote no-group">
<blockquote>
<p>Should it be as simple as replacing torch.log with <code>torch.</code> <code>logsumexp()</code>?</p>
</blockquote>
</aside>
<p>In isolation, such a direct replacement won’t work.</p>
<p>First, note that <code>log10 (x) = log (x) / log (10)</code>.</p>
<p>Second, you wish to calculate<br>
<code>torch.log (1 + torch.exp (step2))</code>.  Because<br>
<code>torch.exp (0) = 1</code>, this can be written as<br>
<code>torch.log (torch.exp (0) + torch.exp (step2))</code>,<br>
for which you can use <code>torch.logsumexp()</code>.</p>
<p>Since you are working with tensors, I imagine that you would<br>
add a new dimension of length 2 to your tensor.  Along this<br>
dimension, the first element would be that of your original<br>
tensor, and the second element would be 0.  You would then<br>
apply <code>torch.logsumexp()</code> along the new dimension, e.g.,<br>
<code>torch.logsumexp (my_new_tensor, dim = 0)</code>, where<br>
the extra new dimension added to create <code>my_new_tensor</code> is,<br>
in this example, <code>dim = 0</code>.</p>
<p>Good luck!</p>
<p>K. Frank</p>

Bled_Clement: <p>Hi K. Frank,</p>
<p>Thank you very much for your response. I’ve actually learned a bit from some of your other responses on the forum in previous work. I’ll give some more context to what I’m doing and more code. You can save time by skipping to the end to see what I’ve attempted. I think it could be too much to share everything needed to run a test of this module as an image and other modules are involved.</p>
<p>I’m training an autoencoder to denoise an image by</p>
<ol>
<li>Using the autoencoder/UNET to output a blurry mask/residual image.</li>
<li>Subtracting this mask from the original image to get a noisy texture map.</li>
<li>Filtering this noisy texture map in the frequency domain with a Wiener filter.</li>
<li>Adding the output of (3) back onto the output of the autoencoder to restore detail to the final image. Basically, a two-part denoiser where the network implicitly learns to creates a mask over some parts of the image and lets the wiener filter take over.</li>
</ol>
<p>I’m trying to make the Wiener filter learn a mapping for attenuating certain frequency magnitudes, instead of just making it a function of noise variance<code> Pvv</code> to signal power spectrum <code>Pss</code>.</p>
<p>Here is what I had before, where H is the filter vector to be applied to each value in the frequency block tensor:</p>
<pre><code class="lang-auto">            freq_block = torch.rfft(win_data_block, win_data_block.ndim, onesided=False)
            Pss = torch.abs(freq_block)**2
            Pss = torch.sum(Pss, 2)
            Pss = Pss.double()
            H = torch.max((Pss-Pvv), torch.zeros(Pss.size(), dtype=torch.double)) / Pss

            H = H.unsqueeze(2).repeat(1, 1, 2)

            filt_freq_block = H*freq_block
</code></pre>
<p>This is comparing the power spectrum magnitude of the signal to the noise power and penalises the signal for a high noise. Pss represents a 16x16 window (lets say between 0 and 30) and Pvv is the constant noise power (lets say 1.5). I want to make a new frequency coring function based on a function with trainable parameters   Here is the filter I am working with, with the problem we are discussing. Most of the code is dealing with indexing and figuring out boundaries so I’ll put the part I’m struggling with in bold:</p>
<pre><code class="lang-auto">

import numpy as np
import torch

def add_noise(img, std):
    noise = torch.randn(img.size()) * (std)
    noisy_image = img + noise
    return noisy_image


def wiener_3d(I, noise_std, block_size,k,s):

    width = I.shape[1]
    height = I.shape[0]
    IR = torch.zeros(height, width, dtype=torch.float64)
    # if(len(list(I.shape)) &gt;= 3):
    #    frames = I.shape[2]
    # else:
    #   bt = 1
    bt = 1
    bx = block_size
    by = block_size

    hbx = bx/2
    hby = by/2
    hbt = bt/2

    sx = (width + hbx - 1)/hbx
    sy = (height + hby - 1)/hby

    win = torch.ones(by, bx, bt)
    win1x = torch.cos((torch.arange(-hbx + .5, hbx - .5 + 1)/bx) * np.pi)
    win1y = torch.cos((torch.arange(-hby + .5, hby - .5 + 1)/by) * np.pi)
    win1t = torch.cos((torch.arange(-hbt + .5, hbt - .5 + 1)/bt) * np.pi)

    for x in range(bx):
        for y in range(by):
            for t in range(bt):
                win[y, x, t] = win1y[y]*win1x[x]*win1t[t]

    if(bt == 1):
        win = torch.squeeze(win)

    Pvv = torch.mean(torch.pow(win, 2))*torch.numel(win)*(noise_std**2)
    Pvv = Pvv.double()
    bx0 = torch.range(0, bx-1)
    by0 = torch.range(0, by-1)

    for x in range(0, int((hbx*sx)), int(hbx)):
        for y in range(0, int((hby*sy)), int(hby)):
            # print(x,y)
            tx = np.arange(x-hbx+1, x+hbx+1)
            validx = np.arange(np.maximum(-tx[0], 0), bx - np.maximum((tx[-1]-width+1), 0))
            cx = np.minimum(np.maximum(tx, 0), width-1)
            validx = validx.astype(int)
            rcx = torch.as_tensor(tx[validx], dtype=torch.long)
            bcx = torch.as_tensor(bx0[validx], dtype=torch.long)

            ty = np.arange(y-hby+1, y+hby+1)
            validy = np.arange(np.maximum(-ty[0], 0), by - np.maximum((ty[-1]-width+1), 0))
            cy = np.minimum(np.maximum(ty, 0), width-1)
            validy = validy.astype(int)
            rcy = torch.as_tensor(ty[validy], dtype=torch.long)
            bcy = torch.as_tensor(by0[validy], dtype=torch.long)

            cy = torch.as_tensor(cy, dtype=torch.long)
            cx = torch.as_tensor(cx, dtype=torch.long)
            data_block = torch.index_select(I, 0, cy)
            data_block = torch.index_select(data_block, 1, cx)

            mean_block = torch.mean(data_block)
            win_data_block = (data_block - mean_block)*win

            freq_block = torch.rfft(win_data_block, win_data_block.ndim, onesided=False)
            Pss = torch.abs(freq_block)**2
            Pss = torch.sum(Pss, 2)
            Pss = Pss.double()

            step1 = Pss-(k*Pvv)
            step2 = step1*s
            step3 = torch.exp(step2)
            zeros = torch.zeros_like(step3)
            step4 = torch.logsumexp([step3, zeros], 0)
            step5 = step4/s

            # train_curve = torch.log(1+torch.exp((Pss-k*Pvv)*s))/s
            H = torch.max(step5, torch.zeros(Pss.size(), dtype=torch.double))  
            #H = H / normalised_Pss
            # H[H != H] = 0
            H = H.unsqueeze(2).repeat(1, 1, 2)

            filt_freq_block = H*freq_block
            filt_data_block = torch.irfft(filt_freq_block, win_data_block.ndim, onesided=False)
            filt_data_block = (filt_data_block + mean_block*win) * win
            # hbt = torch.round(hbt)

            filt_data_block = torch.index_select(filt_data_block, 0, bcy)
            filt_data_block = torch.index_select(filt_data_block, 1, bcx)
            IR[rcy[0]:rcy[-1] + 1, rcx[0]:rcx[-1] + 1] = IR[rcy[0]:rcy[-1] + 1, rcx[0]:rcx[-1] + 1] + filt_data_block
    return IR

</code></pre>
<p><strong>and again the part I’m struggling with</strong></p>
<pre><code class="lang-auto">
            step1 = Pss-(k*Pvv)
            step2 = step1*s
            step3 = torch.exp(step2)
            zeros = torch.zeros_like(step3)
            step4 = torch.logsumexp([step3, zeros], 0)
            step5 = step4/s

            # train_curve = torch.log(1+torch.exp((Pss-k*Pvv)*s))/s
            H = torch.max(step5, torch.zeros(Pss.size(), dtype=torch.double))  
            H = H.unsqueeze(2).repeat(1, 1, 2)

            filt_freq_block = H*freq_block
</code></pre>
<p><strong>I’m not calling the function properly but I thought I’d show you what I’ve done so far. I don’t 100% understand what you mean when you say add a new dimension of length 2.</strong></p>
<p>Thanks a million,<br>
Clément</p>

KFrank: <p>Hello Clement!</p>
<aside class="quote no-group" data-username="Bled_Clement" data-post="5" data-topic="74622">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bled_clement/48/21076_2.png" class="avatar"> Bled_Clement:</div>
<blockquote>
<p>I think it could be too much to share everything needed to run a test of this module</p>
</blockquote>
</aside>
<p>I’m suggesting that you post a test script just for your log-exp<br>
calculation.</p>
<p>Something like:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

sample_tensor =  something_like_torch_randn()

original_result = torch.log (1 + torch.exp (sample_tensor))

new_result = something_with_torch_logsumexp (sample_tensor)
</code></pre>
<p>This should be a complete, runnable script (that is simple,<br>
totally self-contained, and doesn’t depend on any of your<br>
other modules or outside data).  If your <code>new_result</code> doesn’t<br>
run, we can see the errors and make suggestions.  If it does<br>
run, but doesn’t give the right answer, we can look for the<br>
cause of the discrepancy.</p>
<p>(Your <code>original_result</code> will be our baseline for comparison.<br>
It should be correct for moderate tensor values.  It’s okay if it<br>
becomes <code>inf</code> for large tensor values because that’s the issue<br>
we are trying to address.)</p>
<p>Best.</p>
<p>K. Frank</p>

Bled_Clement: <p>Hi Frank, apologies I never got back to you, I was nearing a deadline at the time. Your answer did help me understand the maths, thank you!!</p>