Beginner question about sine estimation

Silk: <p>Dear all,</p>
<p>I’m fairly new to PyTorch and neural networks but out of curiosity, I have started experiencing with PyTorch.</p>
<p>This might be a dummy question given my level of understanding but I’m trying (and failing miserably) to use PyTorch for a simple matter of sine wave phase and amplitude estimation. I feed the network with a sine wave over one period only so estimating its amplitude is simply the max of the input value whereas its phase can be derived from the max found and the first value feeded.</p>
<p>I started with the <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html" rel="noopener nofollow ugc">sine example of PyTorch</a>.</p>
<p>What I was hoping was that once the network is trained, it would manage to readily estimate amplitude and phase of a sine wave given a sine wave output.</p>
<p>Here is the code:</p>
<blockquote>
<p>import torch<br>
import math<br>
import matplotlib.pyplot as plt</p>
<p>x = torch.linspace(-math.pi, math.pi, 2000)</p>
<p>model = torch.nn.Sequential(<br>
torch.nn.Linear(2000, 15),<br>
torch.nn.Linear(15, 2),<br>
)<br>
loss_fn = torch.nn.MSELoss(reduction=‘sum’)</p>
<p>amp = 2;<br>
phase = 2<em>3.14</em>torch.rand(1)<br>
y = amp * torch.sin(x + phase)</p>
<p>plt.ion()<br>
fig = plt.figure()<br>
ax = fig.add_subplot(111)<br>
line1, = ax.plot(x, y)<br>
line2, = ax.plot(x, y, ‘r’)<br>
fig.canvas.draw()<br>
fig.canvas.flush_events()</p>
<p>learning_rate = 1e-3<br>
optimizer = torch.optim.SGD(model.parameters(), lr=5.5e-8, momentum=0.95)<br>
for t in range(200000):</p>
<pre><code>if t%50 == 0:
    amp = 2*torch.rand(1);
    phase = 2*3.14*torch.rand(1)
    y = amp * torch.sin(x + phase)
    line1.set_ydata(y.detach().numpy())

amp_phase = model(y)

y_pred = amp_phase[0] * torch.sin(x + amp_phase[1])

# Compute and print loss.
#loss = loss_fn(y_pred, y)
loss = torch.sum((y_pred - y) * (y_pred - y))
if t % 100 == 99:
    print(t, loss.item())
    print('true',amp,phase)
    print('estimated',amp_phase)
    line2.set_ydata(y_pred.detach().numpy())
    fig.canvas.draw()
    fig.canvas.flush_events()


optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
</blockquote>
<p>Is it a correct approach ?</p>

KFrank: <p>Hi Silk!</p>
<p>I haven’t looked at all of your code, but I do see one clear problem (below).</p>
<aside class="quote no-group quote-modified" data-username="Silk" data-post="1" data-topic="112270" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/silk/48/32591_2.png" class="avatar"> Silk:</div>
<blockquote>
<blockquote>
<p>model = torch.nn.Sequential(<br>
torch.nn.Linear(2000, 15),<br>
torch.nn.Linear(15, 2),<br>
)</p>
</blockquote>
</blockquote>
</aside>
<p>Your <code>Sequential</code> chains two <code>Linear</code>s together <em>without</em> any intervening<br>
non-linear “activation.”  This becomes equivalent to a single layer, namely<br>
<code>Linear (2000, 2)</code>.</p>
<p>You want something like:</p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.nn.Sequential (
    torch.nn.Linear (2000, 15),
    torch.nn.ReLU(),
    torch.nn.Linear (15, 2),
)
</code></pre>
<p>(or <code>Tanh</code> or other non-linear activation).</p>
<p>I’m only guessing here, but you probably also want a deeper network,<br>
that is, one with more layers.  So you might want three <code>Linear</code>s (or maybe<br>
more), again separated non-linear activations.</p>
<p>Best.</p>
<p>K. Frank</p>

Silk: <p>Hi Franck,</p>
<p>Thanks for your reply. I will try your solutions.<br>
I still have another “beginner” question regarding neural network in general. It seems that the process behind the network are linear (at lest most of the time) so how can a network learn something that is not directly related to the weighted sum of the input (through one or more layer).</p>
<p>Let say, is it possible to build a network that would do nothing more than find the maximum value of an input vector built from random Gaussian noise ? How would you do that ?</p>

KFrank: <p>Hi Silk!</p>
<aside class="quote no-group" data-username="Silk" data-post="3" data-topic="112270" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/silk/48/32591_2.png" class="avatar"> Silk:</div>
<blockquote>
<p>It seems that the process behind the network are linear (at lest most of the time)</p>
</blockquote>
</aside>
<p>This really isn’t true.  And if it were true, networks would be much less<br>
broadly useful.</p>
<p>I think it might look like networks are “mostly linear” because we<br>
have these big, multidimensional, fully-connected layers with lots of<br>
parameters, whereas the nonlinearities (usually) occur in small,<br>
“simple,” element-wise nonlinear “activations.”  That is, we use<br>
linear layers to mix elements together (by adding them), and only<br>
introduce nonlinearities element-wise.</p>
<p>But the linear layers and the activations work together to produce<br>
much more general nonlinearities.</p>
<p>You might ask why we don’t “mix” elements together by multiplying<br>
them instead of adding them.  But in fact we do.  Consider the<br>
nonlinearity <code>f (x) = x**2</code>.  If we mix <code>a</code> and <code>b</code> by adding them,<br>
<code>a + b</code>, then passing the mixture through the nonlinearity gives<br>
<code>f (a + b) = a**2  +  2 * a * b  +  b**2</code>.  Note the <code>a * b</code> term.<br>
We <em>have</em> mixed <code>a</code> and <code>b</code> together by multiplying them – we just<br>
used a linear layer together with an element-wise nonlinearity to<br>
do so.</p>
<p>One could imagine a neural-network framework where the “mixing”<br>
layers also included cross-products as well as sums.  I don’t really<br>
know the history of neural networks, but I imagine that making such<br>
a cross-product layer general would require many more parameters<br>
and that it turns out to be more manageable and efficient to rely on<br>
the linear-layer / element-wise-nonlinearity division of labor to<br>
generate those cross-products (and higher-order terms).</p>
<aside class="quote no-group">
<blockquote>
<p>so how can a network learn something that is not directly related to the weighted sum of the input (through one or more layer).</p>
</blockquote>
</aside>
<p>Because the “weighted sums” <em>together with</em> the nonlinear activations<br>
are able to create products of elements (as well as higher-order terms).</p>
<aside class="quote no-group">
<blockquote>
<p>Let say, is it possible to build a network that would do nothing more than find the maximum value of an input vector built from random Gaussian noise ? How would you do that ?</p>
</blockquote>
</aside>
<p>Build a network with at least one “hidden layer” (and probably more),<br>
but, of course, with nonlinear activations in between the linear layers.</p>
<p>However, this problem is not a good example.  If the goal, to choose<br>
something precise, is to find the index of the largest value in the<br>
input vector (compare to predicting the class label in a multi-class<br>
classification), then you just use an “empty” network that does nothing,<br>
and calculate the <code>argmax()</code> of the “output” vector (which is equal to<br>
the input vector).</p>
<p>We could set up a really artificial problem where we try to forbid the<br>
network somehow from using this obvious information, but doing so<br>
would just confuse the issue.</p>
<p>Best.</p>
<p>K. Frank</p>

Silk: <p>Hi Franck,</p>
<p>Thanks a lot for your very clear answers to my question ! You are really helping me out understanding neural network.</p>
<p>Silk</p>