How to get custom dataset to return tuple of embedding sizes instead of tensors

Jordan_Howell: <p>Hello,</p>
<p>I hope everyone in the community is well.</p>
<p>I’m trying to get my custom data set to return the following:</p>
<ol>
<li>Image tensor</li>
<li>Policy (unique ID)</li>
<li>numerical columns tensor</li>
<li>categorical columns tensor</li>
<li>categorical embedding sizes tuple</li>
</ol>
<p>I have 1 through 4 coming back correctly.  However, when trying to return the embedding size tuples, I am not getting tuples but tensors and I’m not sure why.  This throws an error trying to instantiate my model.  The below is the code to my data set:</p>
<pre><code class="lang-auto">class image_Dataset(Dataset):
    '''
    image class data set   
    
    '''
    def __init__(self, data, transform = None):
        '''
        Args:
        ------------------------------------------------------------
            data = dataframe
            image = column in dataframe with absolute path to the image
            label = column in dataframe that is the target classification variable
            numerical_columns =  numerical columns from data
            categorical_columns = categorical columns from data
            policy = ID variable
            
        '''
        self.image_frame = data
        self.transform = transform
        
    def __len__(self):
        return len(self.image_frame)
    
    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
         
        label = self.image_frame.loc[idx, 'target']
        pic = Path(self.image_frame.loc[idx,'location'])
        img = Image.open(pic)
        policy = self.image_frame.loc[idx, 'policy']
        sample = {'image': img, 'policy': policy, 'label':label}
        numerical_data = self.image_frame.loc[idx, numerical_columns]

        if self.transform:
            image = self.transform(img)
            
        for category in categorical_columns:
            self.image_frame[category] = self.image_frame[category].astype('category')
            
        categorical_column_sizes = [len(self.image_frame[column].cat.categories) for column in categorical_columns]
        categorical_embedding_sizes = [(col_size, min(50, (col_size+1)//2)) for col_size in categorical_column_sizes]
        
        self.image_frame[category] = self.image_frame[category].astype('category').cat.codes.values
            
        categorical_data = self.image_frame.loc[idx, categorical_columns]
        categorical_data = torch.tensor(categorical_data, dtype = torch.int64)
        
        numerical_data = torch.tensor(numerical_data, dtype = torch.float)
            
        return image, label, policy, numerical_data, categorical_data, categorical_embedding_sizes
</code></pre>
<p>I’m not sure if this is needed, but this is the model object:</p>
<pre><code class="lang-auto">class Image_Embedd(nn.Module):

    def __init__(self, embedding_size):
        '''
        Args
        ---------------------------
        embedding_size: Contains the embedding size for the categorical columns
        num_numerical_cols: Stores the total number of numerical columns
        output_size: The size of the output layer or the number of possible outputs.
        layers: List which contains number of neurons for all the layers.
        p: Dropout with the default value of 0.5
        
        '''
        super(Image_Embedd, self).__init__()
        
        self.all_embeddings = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in embedding_size])
        self.embedding_dropout = nn.Dropout(p)
        
        
        self.cnn = models.resnet50(pretrained=False)
        
        self.cnn.fc = nn.Linear(self.cnn.fc.in_features, 1000)
        self.fc1 = nn.Linear(1000, 1017)
        self.fc2 = nn.Linear(1017, 128)
        self.fc3 = nn.Linear(128, 2)
        
        
    #define the foward method
    def forward(self, image, x_numerical, x_categorical):
        
        embeddings = []
        for i, e in enumerate(self.all_embeddings):
            embeddings.append(e(x_categorical[:,i]))
        
        x1 = self.cnn(image)
        x2 = numerical_data
        
        x = torch.cat((x1, x2), dim = 1)
        x = torch.cat(embeddings, 1)
        #x = F.relu(self.fc1(x))
        x = self.fc2(x)
        #x = F.relu(self.fc2(x))
        x = self.fc3(x)
        x = F.log_softmax(x)
        return x
</code></pre>
<p>And this is the error thrown when I try to instantiate:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-671-2a2bc631b997&gt; in &lt;module&gt;
----&gt; 1 combined_model = Image_Embedd(categorical_embedding_sizes)
      2 
      3 combined_model = combined_model.cuda()

&lt;ipython-input-669-6ab41a05624c&gt; in __init__(self, embedding_size)
     14         super(Image_Embedd, self).__init__()
     15 
---&gt; 16         self.all_embeddings = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in embedding_size])
     17         self.embedding_dropout = nn.Dropout(p)
     18 

&lt;ipython-input-669-6ab41a05624c&gt; in &lt;listcomp&gt;(.0)
     14         super(Image_Embedd, self).__init__()
     15 
---&gt; 16         self.all_embeddings = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in embedding_size])
     17         self.embedding_dropout = nn.Dropout(p)
     18 

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\sparse.py in __init__(self, num_embeddings, embedding_dim, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse, _weight)
     95         self.scale_grad_by_freq = scale_grad_by_freq
     96         if _weight is None:
---&gt; 97             self.weight = Parameter(torch.Tensor(num_embeddings, embedding_dim))
     98             self.reset_parameters()
     99         else:

TypeError: new() received an invalid combination of arguments - got (Tensor, Tensor), but expected one of:
 * (torch.device device)
 * (torch.Storage storage)
 * (Tensor other)
 * (tuple of ints size, torch.device device)
      didn't match because some of the arguments have invalid types: (!Tensor!, !Tensor!)
 * (object data, torch.device device)
      didn't match because some of the arguments have invalid types: (!Tensor!, !Tensor!)
</code></pre>
<p>Thank you for the help.</p>

ptrblck: <p>You could try to get the Python scalar values for <code>ni</code> and <code>nf</code> via:</p>
<pre><code class="lang-python">... nn.Embedding(ni.item(), nf.item()) ...
</code></pre>
<p>Let me know, if that works for you.</p>

Jordan_Howell: <p>Thank you. I am about to go out of town but will try when I get back. Thanks again. Happy new year.</p>
<p>Jordan</p>

Jordan_Howell: <p>I’ve tried a few different things (one post withdrawn).  It finally worked by bringing in the embedding sizes at the instantiation, not from the data loader.  However, if I want to test the model out, I want to use the test/validation data set’s embedding sizes.  So bringing in the embedding sizes as an argument doesn’t work for new data. I tried the method you suggested and I get the following error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
&lt;ipython-input-179-e5525e645ef2&gt; in &lt;module&gt;
      1 torch.manual_seed(101)
----&gt; 2 combined_model = Image_Embedd()
      3 criterion = torch.nn.NLLLoss().cuda()
      4 optimizer = torch.optim.Adam(combined_model.parameters(), lr=0.001)
      5 scheduler = ReduceLROnPlateau(optimizer, 'min', patience = 4, verbose = True, min_lr = .00000001)

&lt;ipython-input-178-54593bbab9b1&gt; in __init__(self)
     14         super().__init__()
     15 
---&gt; 16         self.all_embeddings = nn.ModuleList([nn.Embedding(ni.item(), nf.item()) for ni, nf in embedding_size])
     17         self.embedding_dropout = nn.Dropout(p = .04)
     18 

NameError: name 'embedding_size' is not defined
</code></pre>
<p>Here is the model object for reference:</p>
<pre><code class="lang-auto">class Image_Embedd(nn.Module):

    def __init__(self):
        '''
        Args
        ---------------------------
        embedding_size: Contains the embedding size for the categorical columns
        num_numerical_cols: Stores the total number of numerical columns
        output_size: The size of the output layer or the number of possible outputs.
        layers: List which contains number of neurons for all the layers.
        p: Dropout with the default value of 0.5
        
        '''
        super().__init__()
              
        self.all_embeddings = nn.ModuleList([nn.Embedding(ni.item(), nf.item()) for ni, nf in embedding_size])
        self.embedding_dropout = nn.Dropout(p = .04)
        
        
        self.cnn = models.resnet50(pretrained=False).cuda()
        
        self.cnn.fc = nn.Linear(self.cnn.fc.in_features, 1000)
        self.fc1 = nn.Linear(1000, 1077)
        self.fc2 = nn.Linear(1077, 128)
        self.fc3 = nn.Linear(128, 2)
        
        
    #define the foward method
    def forward(self, image, x_numerical, x_categorical):
        
        embeddings = []
        for i, e in enumerate(self.all_embeddings):
            embeddings.append(e(x_categorical[:,i]))
            
        x = torch.cat(embeddings, 1)
        x = self.embedding_dropout(x)
        x1 = self.cnn(image)
        x2 = numerical_data
        
        x3 = torch.cat((x1, x2), dim = 1)
        x4 = torch.cat((x, x3), dim = 1)
        x4 = F.relu(self.fc2(x4))
        x4 = F.relu(self.fc3(x4))
        x4 = F.log_softmax(x4)
        return x4
</code></pre>
<p>So the problem set is still how to get the embedding sizes in.</p>

ptrblck: <p>In your first code snippet, you were passing <code>embedding_size</code> as an argument to <code>__init__</code>, while you’ve removed it in your current code, which will throw this error.</p>

Jordan_Howell: <p>Yes.  I’ve added it back in.  I still haven’t comprehended how to bring new data in if the embedding sizes are different.  Do you know anyone that can recommend a solution for this problem?</p>

ptrblck: <p>An embedding is similar to a lookup table, where you provide indices as the input and get dense tensors (defined via <code>embedding_dim</code>) as the output.</p>
<p>Could you explain a bit, how your embedding sizes differ?</p>
<p>The input shape is flexible as seen here:</p>
<pre><code class="lang-python">
nb_words = 10
emb_dim = 100

emb = nn.Embedding(nb_words, emb_dim)

x = torch.randint(0, nb_words, (10,))
output = emb(x)
print(output.shape)
&gt; torch.Size([10, 100])

x = torch.randint(0, nb_words, (10, 2))
output = emb(x)
print(output.shape)
&gt; torch.Size([10, 2, 100])

x = torch.randint(0, nb_words, (10, 2, 3))
output = emb(x)
print(output.shape)
&gt; torch.Size([10, 2, 3, 100])
</code></pre>
<p>Do you want to add new (unknown) word indices to the embedding matrix?</p>

Jordan_Howell: <p>I have the training data, testing data and validation data separate.  My training/test data embedding sizes are:</p>
<pre><code class="lang-auto">[(8, 4), (21, 11), (3, 2), (48, 24), (5, 3), (9, 5), (5, 3), (21, 11), (10, 5)]
</code></pre>
<p>My validation embedding sizes are:</p>
<pre><code class="lang-auto">[(8, 4), (24, 12), (3, 2), (48, 24), (4, 2), (8, 4), (4, 2), (21, 11), (10, 5)]
</code></pre>
<p>One of those sizes represent states out of the US.  If I were to add a state I don’t currently have, wouldn’t this error out the model due to the difference in embedding sizes?  How would I solve for that type of problem set?</p>

ptrblck: <p>I assume the new state would have a new index?<br>
If so, it’s not easy to answer, as it would be comparable to an increased feature space for a linear layer.<br>
The weight matrix does not contain valid parameters for this index yet, so you could e.g. extend the <code>weight</code> parameter and initialize the new parameters randomly.</p>
<p>We had similar discussions in this forum about adding new feature dimensions to a linear or conv layer and I’m not sure, what the best approach was (initializing the new parameters randomly or copying from existing parameters).</p>

Jordan_Howell: <p>Sounds like a “user design” question for the team then.  Thank you for all the help.</p>