Converting tensorflow model to pytorch

username: itaymr
cooked: <p>Greetings,</p>
<p>My data consists of time-series samples with 100 steps, each containing 2 features. In other words, my data is shaped as <code>(samples, steps, features)</code>.</p>
<p>The model I’m currently implementing works in TensorFlow, but I’m having trouble properly implementing it in PyTorch.</p>
<pre><code class="lang-python">class KnownDetector(Model):
    def __init__(self):
        super(KnownDetector, self).__init__()
        self.TCP = tf.keras.Sequential([
            layers.Conv1D(filters=32, kernel_size=3, activation="relu", input_shape=(100, 2)),  # 100 packets, 2 features
            layers.MaxPool1D(pool_size=3, padding='same'),
            layers.Conv1D(filters=32, kernel_size=3, activation="relu"),  # 100 packets, 2 features
            layers.MaxPool1D(pool_size=3, padding='same'),
            layers.Conv1D(filters=32, kernel_size=3, activation="relu"),  # 100 packets, 2 features
            layers.Flatten(),
            layers.Dense(128),
            layers.Dense(num_classes, activation='softmax') # num_classes = 36 in this example
        ])

    def call(self, x):
      return self.TCP(x)
fx = KnownDetector()
fx.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(), optimizer='adam', metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])
fx.fit(train_data, train_labels, epochs=30, validation_data=(val_data, val_labels))
</code></pre>
<p>My understanding is that the input should be reshaped from <code>(steps, input_dim)</code> to <code>(input_dim, steps)</code>.</p>
<p>PyTorch equivalent:</p>
<pre><code class="lang-python">class ModelKnown(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.TCP = torch.nn.Sequential(torch.nn.Conv1d(in_channels=2, out_channels=32, kernel_size=3),
                                       torch.nn.ReLU(),
                                       torch.nn.MaxPool1d(kernel_size=3),
                                       torch.nn.Conv1d(in_channels=32, out_channels=32, kernel_size=3),
                                       torch.nn.ReLU(),
                                       torch.nn.MaxPool1d(kernel_size=3),
                                       torch.nn.Conv1d(in_channels=32, out_channels=32, kernel_size=3),
                                       torch.nn.Flatten(),
                                       torch.nn.Linear(in_features=256, out_features=128),
                                       torch.nn.Linear(in_features=128, out_features=36),
                                       torch.nn.Softmax(1))
    def forward(self, x):
        return self.TCP(x)
</code></pre>
<p>PyTorch doesn’t have a compile the same way TF does, so I’ve been doing my best following pytorch documentation:</p>
<pre><code class="lang-python">x = torch.from_numpy(data)
y = torch.from_numpy(labels.to_numpy())
x.requires_grad=True

# Construct our model by instantiating the class defined above
model = ModelKnown()

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-6)
for t in range(20):
    # Forward pass: Compute predicted y by passing x to the model
    y_pred = model(x)
    # Compute and print loss
    loss = criterion(y_pred, y)
    print(t, loss.item())

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>This outputs:</p>
<pre><code class="lang-auto">0 3.5832407474517822
1 3.5832395553588867
... # and so on
</code></pre>
<ol>
<li>What am I doing incorrectly here?</li>
<li>In TensorFlow, I’m passing data for evaluation, is there a way to do so for PyTorch?</li>
</ol>
<p>Thanks in advance for any help!</p>

username: ptrblck
cooked: <p>You are missing an <code>nn.ReLU()</code> module in PyTorch after the last conv layer and should also remove the <code>nn.Softmax(1)</code> layer as <code>nn.CrossEntropyLoss</code> expects raw logits and will internally apply <code>F.log_Softmax</code> and <code>F.nll_loss</code>.</p>

username: itaymr
cooked: <p>Thanks! I added the missing reLU layer and removed the softmax.<br>
If it does the loss internally is there no point computing the loss? I’m a bit confused there. I’ve changed the classes to only 8 to see if the results are accurate(should &gt;95%)</p>
<p>Model looks like this:</p>
<pre><code class="lang-python">class ModelKnown(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.TCP = torch.nn.Sequential(torch.nn.Conv1d(in_channels=2, out_channels=32, kernel_size=3),
                                       torch.nn.ReLU(),
                                       torch.nn.MaxPool1d(kernel_size=3),
                                       torch.nn.Conv1d(in_channels=32, out_channels=32, kernel_size=3),
                                       torch.nn.ReLU(),
                                       torch.nn.MaxPool1d(kernel_size=3),
                                       torch.nn.Conv1d(in_channels=32, out_channels=32, kernel_size=3),
                                       torch.nn.ReLU(),
                                       torch.nn.Flatten(),
                                       torch.nn.Linear(in_features=256, out_features=128),
                                       torch.nn.Linear(in_features=128, out_features=8))

    def forward(self, x):
        return self.TCP(x)
import torch.optim as optim

model = ModelKnown()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.1)

x = torch.from_numpy(data)
y = torch.from_numpy(labels.to_numpy())
# x.requires_grad=True

for epoch in range(5):  # loop over the dataset multiple times

    optimizer.zero_grad()
    outputs = model(x)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer.step()
</code></pre>
<p>I’m fairly certain I’m doing something wrong, as I’m not getting accurate results, and my softmax values are negative?</p>
<pre><code class="lang-auto">preds = model(x)
preds
tensor([[-0.5422, -0.6145,  0.7050,  ...,  0.4405, -0.3194, -0.3350],
        [-0.5422, -0.6145,  0.7050,  ...,  0.4405, -0.3194, -0.3350],
        [-0.5422, -0.6145,  0.7050,  ...,  0.4405, -0.3194, -0.3350],
        ...,
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="itaymr" data-post="3" data-topic="136226">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/itaymr/48/43758_2.png" class="avatar"> itaymr:</div>
<blockquote>
<p>If it does the loss internally is there no point computing the loss?</p>
</blockquote>
</aside>
<p>No, <code>nn.CrossEntropyLoss</code> uses <code>F.log_softmax</code> and <code>F.nll_loss</code> internally and thus calculates the loss in:</p>
<pre data-code-wrap="python"><code class="lang-python">criterion = nn.CrossEntropyLoss()
...
loss = criterion(output, target)
</code></pre>
<p>However, it expects logits (unbound values) instead of probabilities (outputs of softmax) since the log probabilities will be calculated internally via <code>F.log_softmax</code>.</p>
<aside class="quote no-group" data-username="itaymr" data-post="3" data-topic="136226">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/itaymr/48/43758_2.png" class="avatar"> itaymr:</div>
<blockquote>
<p>I’m fairly certain I’m doing something wrong, as I’m not getting accurate results, and my softmax values are negative?</p>
</blockquote>
</aside>
<p>These are not softmax outputs (probabilities), but logits and are expected to have postive as well as negative values.</p>
<p>You are also using a very high learning rate, so I would probably lower it and use the same as you are using in TF/Keras.</p>

username: itaymr
cooked: <p>As I understood it, logits are the raw predictions that are fed into the softmax function which produces the probabilities for each class - if so, why is the final output of the model logits?</p>
<p>When you say <code>F.log_softmax</code> is calculated internally, is that done automatically?</p>
<p>I would really appreciate it if you could give me a working example, following the <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#train-the-network" rel="noopener nofollow ugc">docs</a> for a classifier, they did calculate the criterion &amp; loss in the same fashion as I did while also using cross-entropy.</p>
<p>Initially, I expected <code>model.train()</code> to train the model, but given that it is done manually I’m having difficulties understanding the process.</p>
<p>Edit: I lowered my learning rate.</p>

username: 1chimaruGin
cooked: <p>Your learning rate is high.<br>
TF/keras use learning 0.001 as default.<br>
You should change learning rate.</p>

username: itaymr
cooked: <p>I definitely did, I was just experimenting at the time - changed it.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="itaymr" data-post="5" data-topic="136226">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/itaymr/48/43758_2.png" class="avatar"> itaymr:</div>
<blockquote>
<p>As I understood it, logits are the raw predictions that are fed into the softmax function which produces the probabilities for each class - if so, why is the final output of the model logits?</p>
</blockquote>
</aside>
<p>You can pick between these approaches:</p>
<ul>
<li>model output as raw logits + <code>nn.CrossEntropyLoss</code></li>
<li>model output as log probabilities (via <code>nn.LogSoftmax</code> or <code>F.log_sotmax</code>) + <code>nn.NLLLoss</code></li>
<li>numerically unstable approach by applying <code>torch.log</code> on probabilities (via <code>torch.log(F.softmax(output, dim=1))</code>)</li>
</ul>
<p>Here is a code snippet for the valid approaches (I skipped the numerically unstable approach, as you shouldn’t use it):</p>
<pre data-code-wrap="python"><code class="lang-python"># setup
batch_size = 2
nb_classes = 10
output = torch.randn(batch_size, nb_classes) # initialize as logits
target = torch.randint(0, nb_classes, (batch_size,))

# nn.CrossEntropyLoss with logits
criterion = nn.CrossEntropyLoss()
# pass raw logits to the criterion
loss_ce = criterion(output, target)

# nn.NLLLoss with log probabilities
criterion = nn.NLLLoss()
log_probs = F.log_softmax(output, dim=1)
loss_nll = criterion(log_probs, target)

# same but with functional API, which is applied in `nn.CrossEntropyLoss` internally
loss_nll_func = F.nll_loss(F.log_softmax(output, dim=1), target)

# compare
print((loss_ce - loss_nll).abs().max())
&gt; tensor(0.)

print((loss_ce - loss_nll_func).abs().max())
&gt; tensor(0.)
</code></pre>
<p>As you can see, the different is zero since all approaches are using the same operations.<br>
<a href="https://github.com/pytorch/pytorch/blob/0a07488ed2c47765e337e290bd138c0e6e459cbd/aten/src/ATen/native/LossNLL.cpp#L595-L600">Here</a> you can also see the code showing that <code>nn.CrossEntropyLoss</code> is using <code>F.nll_loss(F.log_softmax(...))</code> internally (the names are difference as it’s written in the C++ backend:</p>
<pre><code class="lang-auto">    ret = at::nll_loss_nd(
        at::log_softmax(self, 1, self.scalar_type()),
        target,
        weight,
        reduction,
        ignore_index);
</code></pre>
<aside class="quote no-group" data-username="itaymr" data-post="5" data-topic="136226">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/itaymr/48/43758_2.png" class="avatar"> itaymr:</div>
<blockquote>
<p>When you say <code>F.log_softmax</code> is calculated internally, is that done automatically?</p>
</blockquote>
</aside>
<p>Yes, <code>F.log_softmax</code> is applied automatically in <code>nn.CrossEntropyLoss</code> on the model outputs.</p>
<aside class="quote no-group" data-username="itaymr" data-post="5" data-topic="136226">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/itaymr/48/43758_2.png" class="avatar"> itaymr:</div>
<blockquote>
<p>I would really appreciate it if you could give me a working example, following the <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#train-the-network">docs </a> for a classifier, they did calculate the criterion &amp; loss in the same fashion as I did while also using cross-entropy.</p>
</blockquote>
</aside>
<p>The linked example is correct as it returns the logits from the model (output of the last linear layer without any activation function applied on it) and passes them to <code>nn.CrossEntropyLoss</code>.<br>
Your first example used <code>nn.Softmax</code> as the last activation function, which was wrong.</p>

username: abushaim
cooked: <p>Sorry for re-opening the thread after so long, but I came across this and figured I’d share a potentially useful tool for people that may still encounter code conversion issues. While I’m working with Unify, the team behind Ivy, an open-source tool, I genuinely believe it could be a solution to the problem you’re facing. It’s worth exploring for its compatibility features. For more information, you can check out Ivy’s <a href="https://unify.ai/docs/ivy/" rel="noopener nofollow ugc">documentation</a> or <a href="https://github.com/unifyai/ivy" rel="noopener nofollow ugc">repository</a>.</p>