How to assemble two models into one big model

HJX-zhanS: <p>I am currently trying to build the following model.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0ca96b81da90d3a5013977abaa51274a5621729c.png" data-download-href="https://discuss.pytorch.org/uploads/default/0ca96b81da90d3a5013977abaa51274a5621729c" title="1658283160152"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0ca96b81da90d3a5013977abaa51274a5621729c_2_690x402.png" alt="1658283160152" data-base62-sha1="1O0I240YXh0FVZKM4JcHQMAbUdK" width="690" height="402" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0ca96b81da90d3a5013977abaa51274a5621729c_2_690x402.png, https://discuss.pytorch.org/uploads/default/original/3X/0/c/0ca96b81da90d3a5013977abaa51274a5621729c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/c/0ca96b81da90d3a5013977abaa51274a5621729c.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1658283160152</span><span class="informations">982×573 83.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My code is as following shows:</p>
<pre><code class="lang-auto">class CNN_1(torch.nn.Module):
	def __init__(self):
		super(CNN_1, self).__init__()
		...

	def forward(self, X):
		...

		return X


class CNN_2(torch.nn.Module):
	def __init__(self):
		super(CNN_2, self).__init__()
		...

	def forward(self, X):
		...

		return X

class CNN_3(torch.nn.Module):
	def __init__(self):
		super(CNN_3, self).__init__()

		self.cnn1 = CNN_1()
		self.cnn2 = CNN_2()

		self.fc=torch.nn.Linear(... , ...)

	def forward(self, X_1, X_2):
		
		X_1 = self.cnn1(X_1)
		X_2 = self.cnn2(X_2)

		X = torch.cat([X_1, X_2])

		X = self.fc(X)

		return X
</code></pre>
<p>The training code is as follows:</p>
<pre><code class="lang-auto">model = CNN_3()
criterion = CrossEntropyLoss()
optimizer = torch.optim.Adam(params=model.parameters(), lr=1e-3)

X_1 = ...
X_2 = ...
labe l= ...

model.train()

preds = model(X_1, X_2)

loss = criterion(preds, label)

loss.requires_grad_(True)

optimizer.zero_grad()

loss.backward()
optimizer.step()
</code></pre>
<p>However, the performance of the model is not good. I checked the parameters of the model before and after training, and found that after training, the model parameters did not change, and <code>grad_value</code> is None.</p>
<p>I don’t know what’s wrong, please help me, thank you.</p>

thecho7: <p>The structure is correct.<br>
Would you check that <code>optimizer.step()</code> is correctly called?</p>
<p>This is the basic of how your model is updated.</p>
<pre><code class="lang-auto">optimizer.zero_grad()
out = model(input)
loss = loss_function(out, gt)
loss.backward()
optimizer.step()
</code></pre>

HJX-zhanS: <p>Thank you for your answer.</p>
<p>The <code>optimizer.step()</code> is correctly called. But the model parameters did not change, and <code>grad_value</code> is None.</p>

thecho7: <p>Would you give us a code snippet?</p>

HJX-zhanS: <p>Here is the code snippet.</p>
<pre><code class="lang-auto">class ConvBlock(torch.nn.Module):
    def __init__(self, kernel_h, emb_size, max_line):
        super(ConvBlock, self).__init__()

        self.cnn = torch.nn.Conv1d(in_channels=emb_size, out_channels=10, kernel_size=kernel_h)

        self.max_pool = torch.nn.MaxPool1d(kernel_size=(max_line - kernel_h + 1))

    def forward(self, X):
        X = self.cnn(X.squeeze(1).permute(0, 2, 1))

        X = F.relu(X)

        # X = X.squeeze(-1)
        X = self.max_pool(X)

        X = X.squeeze(-1)

        return X


class MyTextCNN(torch.nn.Module):
    def __init__(self, emb_size, max_line):
        super(MyTextCNN, self).__init__()

        self.block2 = ConvBlock(3, emb_size, max_line)
        self.block3 = ConvBlock(4, emb_size, max_line)
        self.block4 = ConvBlock(5, emb_size, max_line)

    def forward(self, X):
        X = X.unsqueeze(1)
        X_2 = self.block2(X)
        X_3 = self.block3(X)
        X_4 = self.block4(X)

        X = torch.cat([X_2, X_3, X_4], dim=1)

        return X

class AssembleModel(torch.nn.Module):
    def __init__(self):
        super(AssembleModel, self).__init__()

        self.cnn_1 = MyTextCNN(300, 200)
        self.cnn_2 = MyTextCNN(300, 500)

        self.fc = torch.nn.Linear(30, 2)

    def forward(self, X_1, X_2):
        X_1 = self.cnn_1(X_1)
        X_2 = self.cnn_2(X_2)

        X = torch.cat([X_1, X_2], dim=1)

        out = self.fc(X)

        return out

# train

model = AssembleModel()
criterion = CrossEntropyLoss()
optimizer = torch.optim.Adam(params=model.parameters(), lr=1e-3)

X_1 = ...
X_2 = ...
label= ...

model.train()

optimizer.zero_grad()

preds = model(X_1, X_2)

loss = criterion(preds, label)

loss.requires_grad_(True)

loss.backward()
optimizer.step()
</code></pre>

thecho7: <ol>
<li>
<code>loss.requires_grad_(True)</code> is unnecessary.</li>
<li>How did you check <code>grad_value</code>?</li>
</ol>

HJX-zhanS: <p>I use this code to check <code>grad_value</code>:</p>
<pre><code class="lang-auto">    for name, parms in model.named_parameters():
        print('--&gt;name:', name)
        print('--&gt;para:', parms)
        print('--&gt;grad_requirs:', parms.requires_grad)
        print('--&gt;grad_value:', parms.grad)
</code></pre>
<p>As for <code>loss.requires_grad_(True)</code>, if I remove it, I get this error:</p>
<blockquote>
<p>Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
</blockquote>

ptrblck: <p>I don’t see any operation which would detach the computation graph, so could you post the input shapes to make the code executable?</p>

HJX-zhanS: <p>The shape of <code>X_1</code> is <code>(batch_size, 200, 300)</code>, the shape of <code>X_2</code> is <code>(batch_size, 500, 300)</code>.</p>

ptrblck: <p>Thanks! The shapes don’t work as you would be running into a shape mismatch in <code>self.fc</code> After fixing it by setting <code>in_features=60</code> the model works correctly:</p>
<pre><code class="lang-python">model = AssembleModel()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(params=model.parameters(), lr=1e-3)

X_1 = torch.randn(10, 200, 300)
X_2 = torch.randn(10, 500, 300)
label = torch.randint(0, 2, (10,))

model.train()
optimizer.zero_grad()
preds = model(X_1, X_2)
loss = criterion(preds, label)

loss.backward()
optimizer.step()

for name, param in model.named_parameters():
    print(name, param.grad.abs().sum())
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">cnn_1.block2.cnn.weight tensor(81.9196)
cnn_1.block2.cnn.bias tensor(0.0434)
cnn_1.block3.cnn.weight tensor(119.6594)
cnn_1.block3.cnn.bias tensor(0.0476)
cnn_1.block4.cnn.weight tensor(126.1779)
cnn_1.block4.cnn.bias tensor(0.0402)
cnn_2.block2.cnn.weight tensor(132.8887)
cnn_2.block2.cnn.bias tensor(0.0697)
cnn_2.block3.cnn.weight tensor(106.3185)
cnn_2.block3.cnn.bias tensor(0.0431)
cnn_2.block4.cnn.weight tensor(169.7578)
cnn_2.block4.cnn.bias tensor(0.0539)
fc.weight tensor(10.4102)
fc.bias tensor(0.1133)
</code></pre>
<p>So your actual use case seems to use another code.</p>

HJX-zhanS: <p>Thank you so much! I tested the code you provided and succeeded.</p>