How to cat layer with batch_size of 1 to a larger batch?

pumplerod: <p>I have a set of data which I’ve used to generate an image, which I then iterate through convolution layers and then ultimately flatten the features into one layer so it becomes a size: [1, 11552].</p>
<p>I now want to connect that layer to be included in the training for each sample in a batch of say [4000, 310].  I know that if I copied that layer into a size [4000, 11552] I could then use torch.cat() to combine the features, however that seems needlessly wasteful (Though I may be missing something important).</p>
<p>Is there a way to connect that layer to each sample from the larger batch without having to duplicate the data?  So that during training I would effectively be training a batch of size [4000, 11862]?</p>

ariG23498: <p>Hey <a class="mention" href="/u/pumplerod">@pumplerod</a><br>
Could you share a better flow chart of the shapes of things you are dealing with?</p>
<p>I am a little confused with the way the model is processing the images and your question.</p>

pumplerod: <p>Sorry for the confusion. Basically it boils down to this…  I have a dense layer which I would like to remain common to all samples from a batch…<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc34cdcf8f6ae65257e01a147a74780d7f9e9ddc.png" data-download-href="https://discuss.pytorch.org/uploads/default/cc34cdcf8f6ae65257e01a147a74780d7f9e9ddc" title="Screen Shot 2021-05-09 at 9.09.47 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc34cdcf8f6ae65257e01a147a74780d7f9e9ddc_2_450x500.png" alt="Screen Shot 2021-05-09 at 9.09.47 AM" data-base62-sha1="t8uxe2aMBdJd8YruD3XU74FyPGA" width="450" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc34cdcf8f6ae65257e01a147a74780d7f9e9ddc_2_450x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc34cdcf8f6ae65257e01a147a74780d7f9e9ddc_2_675x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc34cdcf8f6ae65257e01a147a74780d7f9e9ddc_2_900x1000.png 2x" data-dominant-color="E8E8ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-05-09 at 9.09.47 AM</span><span class="informations">1080×1200 78.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Ideally I would not want to duplicate that common layer n_batch times since it’s always the same.  I am currently creating a new tensor of ones the size of my batch of samples, and the width of the common layer.  Then multiplying by the common layer and using torch.cat() to join them, but this just feels wrong.</p>

ariG23498: <p>I think <a class="mention" href="/u/ptrblck">@ptrblck</a> has already solved this.</p><aside class="quote quote-modified" data-post="2" data-topic="20462">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/concatenate-layer-output-with-additional-input-data/20462/2">Concatenate layer output with additional input data</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Here is a small example for your use case: 
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.cnn = models.inception_v3(pretrained=False, aux_logits=False)
        self.cnn.fc = nn.Linear(
            self.cnn.fc.in_features, 20)
        
        self.fc1 = nn.Linear(20 + 10, 60)
        self.fc2 = nn.Linear(60, 5)
        
    def forward(self, image, data):
        x1 = self.cnn(image)
        x2 = data
        
        x = torch.cat((x1, x2…
  </blockquote>
</aside>

<p>Hope this helps! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning_face_with_smiling_eyes.png?v=12" title=":grinning_face_with_smiling_eyes:" class="emoji" alt=":grinning_face_with_smiling_eyes:" loading="lazy" width="20" height="20"></p>

pumplerod: <p>I spent some time going over the variations in the thread and I don’t quite see the situation I’m referring to.  Perhaps I’ve missed it.</p>
<p>To simplify my issue I’m including code with error and my “work around”.</p>
<p>Here is the code which breaks because the torch.cat() bumps on not having the proper dimensions…</p>
<pre><code class="lang-auto">layers = [ torch.nn.Linear( 15, 5), 
           torch.nn.ReLU( inplace=True), 
           torch.nn.Linear( 5, 3),
           torch.nn.ReLU( inplace=True),
           torch.nn.Linear( 3, 1),
           torch.nn.Sigmoid() ]

model = torch.nn.Sequential( *layers)

single_layer = torch.randn( (1,10))
batch = torch.randn( (100,5))

input_layer = torch.cat( (single_layer, batch), dim=1)
output = model( input_layer)
</code></pre>
<p>And here is what I’ve done to get around the issue…</p>
<pre><code class="lang-auto">layers = [ torch.nn.Linear( 15, 5), 
           torch.nn.ReLU( inplace=True), 
           torch.nn.Linear( 5, 3),
           torch.nn.ReLU( inplace=True),
           torch.nn.Linear( 3, 1),
           torch.nn.Sigmoid() ]

model = torch.nn.Sequential( *layers)

single_layer = torch.randn( (1,10))
print( f'Constant Layer Size:: { single_layer.size()}')

batch = torch.randn( (100,5))
print( f'Batch Size:: { batch.size()}')

replicated_single_layer = torch.ones( (batch.size(0), single_layer.size(1)))
replicated_single_layer *= single_layer
print( f'Replicated Layer Size: { replicated_single_layer.size()}')

input_layer = torch.cat( (replicated_single_layer, batch), dim=1)
print( f'Input Layer Size: { input_layer.size()}')
output = model( input_layer)
print( output.size())
</code></pre>
<p>Perhaps the thread you’ve suggested has the solution staring me in the face and I’m just lost on the variation of its implementation.</p>

ariG23498: <p>Hey <a class="mention" href="/u/pumplerod">@pumplerod</a><br>
I did some digging and I think this should be your best option:</p>
<pre><code class="lang-python">single_layer = torch.randn((1,10))
batch = torch.randn((100,5))

single = single_layer.repeat(100,10)
input_layer = torch.cat([batch,single],dim=1)
</code></pre>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/grinning_face_with_smiling_eyes.png?v=9" title=":grinning_face_with_smiling_eyes:" class="emoji" alt=":grinning_face_with_smiling_eyes:"></p>

pumplerod: <p>Oh, interesting.  Thank you so much.  Looks like it will certainly save a multiplication step.  I’m not sure what the implications on autograd that will have but hopefully some speed improvement, even though it looks like it will still consume memory where it ought not.</p>
<p>I believe the slight modification to your suggestion is that I need to make sure to only repeat along the batch axis.  Not repeat the single 10 nodes 10 times also.  So my final code looks like:</p>
<pre><code class="lang-auto">single_layer = torch.randn( ( 1,10))
batch = torch.randn( ( 100,5))

replicated_single = single_layer.repeat( batch.size(0),1)
input_layer = torch.cat( [ batch,replicated_single], dim=1)
</code></pre>