Convolutional Autoencoder

isfet: <p>Hi, im trying to train a convolutional autoencoder over a dataset composed by 20k samples. Each sample is an array of 65536 elements, each one is float value. i want to train the autoencoder to reduce the dimension of the dataset from 65536 → 1024 elements and than use the reduced dataset to train a DNN. The following is the model of the autoencoder</p>
<pre><code class="lang-auto">class Conv1dAE(nn.Module):
    def __init__(self):
        super(Conv1dAE, self).__init__()
        
        # Encoder
        self.encoder = nn.Sequential(
            # Conv1d Layer 1: Input Channels = 1, Output Channels = 16, Kernel Size = 4, Stride = 4
            nn.Conv1d(64, 16, kernel_size=2, stride=2),  
            nn.ReLU(),
            nn.MaxPool1d(2),
            # Output: [Batch, 16, 16384]
            
            # Conv1d Layer 2: Input Channels = 16, Output Channels = 32, Kernel Size = 4, Stride = 4
            nn.Conv1d(16, 32, kernel_size=2, stride=2), 
            nn.ReLU(),
            nn.MaxPool1d(2),
            # Output: [Batch, 32, 4096]
            
            # Conv1d Layer 3: Input Channels = 32, Output Channels = 64, Kernel Size = 2, Stride = 2
            nn.Conv1d(32, 64, kernel_size=2, stride=2), 
            nn.ReLU(),
            nn.MaxPool1d(2)
            # Output: [Batch, 64, 1024]
        )
        
        # Decoder
        self.decoder = nn.Sequential(
            # ConvTranspose1d Layer 1: Input Channels = 64, Output Channels = 32, Kernel Size = 2, Stride = 2
            nn.ConvTranspose1d(64, 32, kernel_size=4, stride=4),  
            nn.ReLU(),
            # Output: [Batch, 32, 4096]
            
            # ConvTranspose1d Layer 2: Input Channels = 32, Output Channels = 16, Kernel Size = 4, Stride = 4
            nn.ConvTranspose1d(32, 16, kernel_size=4, stride=4),  
            nn.ReLU(),
            # Output: [Batch, 16, 16384]
            
            # ConvTranspose1d Layer 3: Input Channels = 16, Output Channels = 1, Kernel Size = 4, Stride = 4
            nn.ConvTranspose1d(16, 64, kernel_size=4, stride=4),  
            nn.Sigmoid()
            # Output: [Batch, 1, 65536]
        )
        
    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x
</code></pre>
<p>and the following is the code that i use to reduce the original dataset</p>
<pre><code class="lang-auto">encoder = Conv1dAE()
    encoder.load_state_dict(torch.load('./results/autoencoder_conv_maxpool_1024_2024_06_19_00_07_28/autoencoder_conv_512512_2024_06_19_00_07_28.pth', map_location=torch.device('cpu')))
    encoder.to(device)

    encoder.eval()
    encoded_features_train = []
    with torch.no_grad():
        for batch in train_dataloader:
            inputs = batch[0]
            outputs = encoder.encoder(inputs)  # Applica solo l'encoder
            
            encoded_features_train.append(outputs)
            
    # Concatena le rappresentazioni a bassa dimensionalità in un unico tensore
    encoded_features_train = torch.cat(encoded_features_train, dim=0)

    print(f"Encoded train features shape: {encoded_features_train.shape}")
</code></pre>
<p>Actually i’m not sure about the “torch.cat” operation, that i use to obtain again a dataset in one-dimensional form, 'cause the output of the encoder from the autoencoder has more than one dimension.<br>
In case it’s proven wrong, how can i add a FC linear layer between encoder and decoder in order to obain a one dimension output of 1024 elements?</p>

KFrank: <p>Hi Michele!</p>
<aside class="quote no-group" data-username="isfet" data-post="1" data-topic="204924" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/isfet/48/66530_2.png" class="avatar"> isfet:</div>
<blockquote>
<p>Each sample is an array of 65536 elements, each one is float value.</p>
</blockquote>
</aside>
<p>But the code you posted below seems to expect 64 channels for each<br>
of your 65536 elements.</p>
<aside class="quote no-group">
<blockquote>
<p>i want to train the autoencoder to reduce the dimension of the dataset from 65536 → 1024 elements</p>
</blockquote>
</aside>
<p>First, by having your early layers be convolutions, you are saying that your<br>
input data has some spatial (albeit one-dimensional) structure – perhaps<br>
it’s a time series or audio sample.  Is this true?</p>
<aside class="quote no-group">
<blockquote>
<p>and than use the reduced dataset to train a DNN.</p>
</blockquote>
</aside>
<p>But then, do you want your reduced dataset to also have similar spatial<br>
structure?  When you say that the reduced dataset will be used to train<br>
a DNN (dense neural network), I imagine the input to your DNN to be a<br>
vector of 1024 “features” that don’t really have any particular spatial<br>
structure (and that the first layer of your DNN is a fully-connected <code>Linear</code>).<br>
True?</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            # Conv1d Layer 1: Input Channels = 1, Output Channels = 16, Kernel Size = 4, Stride = 4
            nn.Conv1d(64, 16, kernel_size=2, stride=2),
</code></pre>
</blockquote>
</aside>
<p>So which is it?  Does the first layer of your autoencoder (and your input data)<br>
have one channel or 64?</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            # Conv1d Layer 3: Input Channels = 32, Output Channels = 64, Kernel Size = 2, Stride = 2
            nn.Conv1d(32, 64, kernel_size=2, stride=2), 
            nn.ReLU(),
            nn.MaxPool1d(2)
            # Output: [Batch, 64, 1024]
</code></pre>
</blockquote>
</aside>
<p>If you want the output of the encoder section of your autoencoder to<br>
consist of 1024 features (without spatial structure), keep max-pooling<br>
(and convolving) down to a <em>spatial</em> extent of 1, but with 1024 channels.<br>
(You can then <code>squeeze()</code> away the singlenton spatial dimension to get<br>
input of shape <code>[Batch, 1024]</code> for your DNN.)</p>
<p>(On the other hand, if you want the output of your encoder to <em>have</em> spatial<br>
structure, then max-pool down to a spatial dimension of 1024, as you are<br>
doing, but have your <code>Conv1d</code>s reduce their <code>out_channels</code> to a final value<br>
of one.)</p>
<p>Best.</p>
<p>K. Frank</p>

isfet: <p>Hi, the dataset is extracted from bytes frequencies of different gile format. In this particular case from two format. So it is a “linear” dataset. Is it wrong to use 64 input channel in the first conv layer? I tried to use a single channel, by pytorch expected 64 instead (maybe due to batch_size i guess). I just want to achieve a sort of smart dimensionality reduction from 65536 element to 1024, no more. Could you show me a snippet with the last conv layer of the encoder?</p>

KFrank: <p>Hi Michele!</p>
<aside class="quote no-group" data-username="isfet" data-post="3" data-topic="204924" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/isfet/48/66530_2.png" class="avatar"> isfet:</div>
<blockquote>
<p>So it is a “linear” dataset.</p>
</blockquote>
</aside>
<p>When you say “linear” dataset, do you mean the following?</p>
<p>Consider a data sample that is a single vector, <code>v</code>, of length 65536, so of<br>
shape <code>[65536]</code> (no batch nor channel dimension).</p>
<p>Do you mean that some single element, say <code>v[100]</code>, is more closely related<br>
to <code>v[101]</code> (or <code>v[95]</code> or <code>v[110]</code>) than it is to, say, <code>v[10000]</code>, because it is<br>
closer to <code>v[101]</code> in “index space?”  (Or are the indices of the elements of<br>
<code>v</code> essentially arbitrary, so that <code>v[100]</code> is no more closely related to <code>v[101]</code><br>
than it is to <code>v[10000]</code>?)</p>
<aside class="quote no-group">
<blockquote>
<p>Is it wrong to use 64 input channel in the first conv layer?</p>
</blockquote>
</aside>
<p>If your input data doesn’t have 64 channels, then your first <code>Conv1d</code> layer<br>
should not have 64 channels.</p>
<aside class="quote no-group">
<blockquote>
<p>(maybe due to batch_size i guess).</p>
</blockquote>
</aside>
<p>Having a batch_size, but no explicit channels dimension could be your<br>
problem.</p>
<p>A <code>Conv1d</code> expects either a two-dimensional input (<code>[nChannels, length]</code>)<br>
or a three-dimensional input (<code>[nBatch, nChannels, length]</code>), but the<br>
input has to have an <code>nChannels</code> dimension, even if the number of channels<br>
is one.  If you pass in input of shape <code>[nBatch = 64, length = 65536]</code>,<br>
the <code>Conv1d</code> will treat your batch dimension as a channels dimension (and<br>
will raise an error unless the <code>Conv1d</code> has <code>in_channels = 64</code>).</p>
<p>To be clear, in such a case you should be passing in a three-dimensional<br>
input of shape <code>[nBatch = 64, nChannels = 1, length = 65536]</code>,<br>
with an explicit channels dimension (of size one).  Note, you can use<br>
<code>unsqueeze (1)</code> to add such a singleton channels dimension if your<br>
input doesn’t already have a channels dimension.</p>
<aside class="quote no-group">
<blockquote>
<p>I just want to achieve a sort of smart dimensionality reduction from 65536 element to 1024</p>
</blockquote>
</aside>
<p>To repeat the question I asked in my previous post, do you want the<br>
reduced-dimensionality output of your encoder (of length 1024) to be<br>
“linear” in the sense discussed above?  If you intend to pass that output<br>
into a fully-connected <code>Linear</code> layer of a DNN, then any “linear” structure<br>
it might have would be irrelevant because the ordering of the columns of<br>
the <code>weight</code> matrix of a <code>Linear</code> is essentially arbitrary.</p>
<p>That is, should the length-1024 output of the encoder reflect the “linear”<br>
structure of its length-65536 input, but just “reduced in dimensionality”<br>
by a factor of 64?  Or should it just consist of 1024 “non-spatial” features<br>
that have no particular “linear” structure?</p>
<p>Best.</p>
<p>K. Frank</p>

isfet: <p>Ok let me explain. Each array of 65536 elements contains the bigram of a chunk of a file. 65536 came from all the possible pair of bytes, from 00-00 to FF-FF. I compute the frequency of esch pair and then each pair is converted from hex to dec and this value represents the index in which i’ll store the frequency in the array. Given that, there is no relation between each value of the array. The encode should output a one dimension array of 1024 elements. Thats it</p>

KFrank: <p>Hi Michele!</p>
<aside class="quote no-group" data-username="isfet" data-post="5" data-topic="204924" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/isfet/48/66530_2.png" class="avatar"> isfet:</div>
<blockquote>
<p>there is no relation between each value of the array.</p>
</blockquote>
</aside>
<p>Okay, in that case you do <em>not</em> want to use convolution layers – that’s<br>
not how convolutional layers work.</p>
<p>I assume that your goal is to train your encoder somehow to get the<br>
length-1024 output and that you’re using an autoencoder so that you<br>
can train the encoder by using some sort of unsupervised training to<br>
train the whole autoencoder.</p>
<p>You can certainly work with a <em>non-convolutional</em> autoencoder.</p>
<aside class="quote no-group">
<blockquote>
<p>The encode should output a one dimension array of 1024 elements.</p>
</blockquote>
</aside>
<p>Your encoder could look something like:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.nn.Sequential (
    torch.nn.Linear (65536, 32768),
    torch.nn.ReLU(),   # could also be something like Sigmoid or Tanh
    torch.nn.Linear (32768, 16384),
    torch.nn.ReLU(),
    ...
    torch.nn.Linear (4096, 2048),
    torch.nn.ReLU(),
    torch.nn.Linear (2048, 1024)
)
</code></pre>
<p>It will take in a single sample of shape <code>[65536]</code> and output a single result<br>
of shape <code>[1024]</code> or take in a batch of shape <code>[nBatch, 65536]</code> and output<br>
a batch of results of shape <code>[nBatch, 1024]</code>.</p>
<p>Your decoder would basically be the same thing in reverse order with a<br>
sequence of <code>Linear</code>s that increase the number of features from 1024<br>
to 65536.  (There’s nothing magic about changing the number features<br>
by a factor of two with each <code>Linear</code> – you just need to get from 65536<br>
down to 1024 and then back up to 65536 with enough layers that you<br>
can perform an adequate encoding-decoding computation.)</p>
<p>Best.</p>
<p>K. Frank</p>

isfet: <p>Ok very clear! The only reason that move me from a linear autoencoder to a convolutional one is memory consumption. A linear autoencoder with such layers (actually double by encoder and decoder) took quite a lot of memory (gpu o ram) to train</p>

KFrank: <p>Hi Michele!</p>
<aside class="quote no-group" data-username="isfet" data-post="7" data-topic="204924" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/isfet/48/66530_2.png" class="avatar"> isfet:</div>
<blockquote>
<p>A linear autoencoder with such layers (actually double by encoder and decoder) took quite a lot of memory (gpu o ram) to train</p>
</blockquote>
</aside>
<p>Because the 65536 elements of an input data sample are independent<br>
of one another, you pretty much need your first <code>Linear</code> layer to have<br>
<code>in_features = 65536</code> so that you can train an independent set of weights<br>
for each independent input element.</p>
<p>But beyond that you have a lot of flexibility.</p>
<p>It would be logically reasonable to do something like:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.nn.Sequential (
    torch.nn.Linear (65536, 2048),
    torch.nn.ReLU(),
    torch.nn.Linear (2048, 2048),
    torch.nn.ReLU(),
    torch.nn.Linear (2048, 1024)
)
</code></pre>
<p>(or any such variations on this general theme).</p>
<p>By having your first <code>Linear</code> drop the number of features to a much<br>
smaller number, you will, obviously, have a much smaller model.</p>
<p>I don’t have any intuition about the trade-off between the size of your<br>
model and its trainability and how well your encoder will work, but I<br>
don’t have any argument that such an approach couldn’t work well.<br>
For example, for a similar total number of parameters, you could have<br>
fewer larger intermediate layers or more smaller intermediate layers.<br>
I don’t have an opinion about would be likely to work better for you.</p>
<p>(I view a lot of these kind of questions as the “black magic” of neural<br>
networks – sometimes you just have try a handful of such variations<br>
and see which trade-offs work best for your use case.)</p>
<p>Good luck!</p>
<p>K. Frank</p>

isfet: <p>Yeah i know. I just tried convolution and hoped it works, but this time is not the case. Thx!</p>