Fusing Additional input/features for Transfer Learning

abdullah05: <p>Hey! I am new to Pytorch. I am trying to do transfer learning and at some point, while doing the training I want to give some additional input for each data point. So let’s say image_1 has a feature like (1.01, 0.5, .06) and image_2 has a feature like (.9, .001, .023) and so on, I would like to bundle them together.</p>
<p>The notebook I am following: <a href="https://github.com/WillKoehrsen/pytorch_challenge/blob/master/Transfer%20Learning%20in%20PyTorch.ipynb" rel="nofollow noopener">https://github.com/WillKoehrsen/pytorch_challenge/blob/master/Transfer%20Learning%20in%20PyTorch.ipynb</a></p>
<p>So here it shows how we can change the <strong>fc</strong> layer at the end and insert whatever we want to. For my problem would I then need to change the very first <strong>Conv2D</strong> layer (for resnet18) and somehow pass in the features? For example, adding a <strong>fc</strong> layer there and fusing the data? Or are there other ways to do this like fusing on a tensor level?</p>
<p>In a programming sense, how should I go about doing this?</p>
<p>Thanks</p>

ptrblck: <p>What kind of additional data do you have?<br>
Is it also an image or do you have any other features (e.g. 1-dimensional) for each image?</p>

abdullah05: <p>Thank you so much for getting back!</p>
<p>Additional data is numeric. Features would be 2 to 4 dimensional, definitely not more than that.</p>
<p>So after going through the vision forum, I think I found some other posts that were very similar to what I was trying to do. I am referring to <a href="https://discuss.pytorch.org/t/concatenate-layer-output-with-additional-input-data/20462">this</a> post specifically. But then I have some other questions from there as well. I have a total of 250 something images over two classes. Classes are named, <strong>PD</strong> and <strong>Non_PD</strong>.</p>
<p>So let’s say I have csv file with the header: <strong>filename</strong>, <strong>feature_1</strong>, <strong>feature_2</strong>, <strong>feature_3</strong>, <strong>feature_4</strong>. Now the files in filename are all located in PD and Non_PD folders but there are no indications in the csv file which are in which class (I could add it if it would make my life easy).</p>
<p>That being said, for my problem what would <strong>additional_data_dimension</strong> be as referred to in the other post I just mentioned? 4? How would I read in the data here? Through DataLoader? Since right now my code is getting all the images from the train directory. How would the code look like for that?</p>

ptrblck: <p>Based on the file description, it seems your additional features are 1-dimensional with a size of 4.<br>
What is stored in the <code>filename</code> column? Are there your image file names (or paths)?<br>
If so, I would write a custom <code>Dataset</code>, index the <code>DataFrame</code> inside <code>__getitem__</code> to get the file path to the image, the additional features, and the current target (I would recommend to create the target in a separate column).</p>
<p>Let me know, if that would work for you.</p>

abdullah05: <p>Yup that’s how it should work I believe!</p>
<p>The custom Dataset class looks like this:</p>
<pre><code class="lang-auto">class PD_Dataset(Dataset):

	def __init__(self, csv_file, root_dir, transform=None):
		self.pd_features = pd.read_csv(csv_file)
		self.root_dir = root_dir
		self.transform = transform
		self.classes = ("PD", "Non-PD")

	def __len__(self):
		return len(self.pd_features)

	def __getitem__(self, idx):
		if torch.is_tensor(idx):
			idx = to_list()

		img_name = os.path.join("",self.pd_features.iloc[idx, 0])
		image = Image.open(img_name)
		features = self.pd_features.iloc[idx, 1]
		labels = self.pd_features.iloc[idx, 2]

		if self.transform:
			image = self.transform(image)
		return image, features, labels
</code></pre>
<p>Now one thing I have to figure out is changing the <strong>features</strong> variable here so that its shape follows <strong>image</strong> variable’s shape. <strong>features</strong> variable looks like <code>[(x1, x2, x3)]</code>, and I was told that I need to pad a tensor with the same values so that shape becomes the shape of <strong>image</strong>.</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
	def __init__(self):
		super(MyModel, self).__init__()
		self.cnn = models.resnet18(pretrained=True)
		self.cnn.fc = nn.Sequential(
			nn.Linear(self.cnn.fc.in_features, 256),
			nn.ReLU(),
			nn.Dropout(0.2),
			nn.Linear(256, 2),
			nn.LogSoftmax(dim=1))

	def forward(self, image, data):
		x1 = self.cnn(image)
		x2 = data

		x = torch.cat((x1, x2), dim=1)
		x = self.cnn.fc(x)
		return x

model = MyModel()
</code></pre>
<p>Now for the training part:</p>
<pre><code class="lang-auto">for epoch in range(n_epochs):

		train_loss = 0.0
		valid_loss = 0.0

		train_acc = 0
		valid_acc = 0

		model.train()
		start = timer()

		for ii, (data, target) in enumerate(train_loader):
			if train_on_gpu:
				data, target = data.cuda(), target.cuda()

			optimizer.zero_grad()

			output = model(data)

			target_tensor = torch.tensor(target, dtype=torch.long, device=torch.device('cuda'))
			loss = criterion(output, target_tensor)
			loss.backward()

			optimizer.step()

			train_loss += loss.item() * data.size(0)

			_, pred = torch.max(output, dim=1)
			correct_tensor = pred.eq(target.data.view_as(pred))
			accuracy = torch.mean(correct_tensor.type(torch.FloatTensor))
			train_acc += accuracy.item() * data.size(0)
</code></pre>
<p>Since my Dataset now will return in the format of  <code>image, features, data</code>, do you I have to enumerate the <code>train_loader</code> a different way? And manually handle it? The reason I ask is, I am not seeing where the <code>forward</code> function’s inputs are getting taken care of.</p>
<p>Would appreciate some help on this part!</p>

ptrblck: <p>Great to see that your data loading pipeline seems to work.</p>
<p>I’m a bit skeptical about your <code>forward</code> implementation.<br>
Currently you are reusing <code>self.cnn.fc</code>. The image tensor batch will use it once in <code>x1 = self.cnn(image)</code> and then again the concatenated tensor in <code>x = self.cnn.fc(x)</code>.<br>
Is this your workflow (it might make sense) or would you rather concatenate the penultimate resnet18 output of the image tensor with your additional data?</p>
<p>In the latter case, replace the last linear layer with <code>self.cnn.fc = nn.Identity()</code> and define the new classification block as <code>self.fc = nn.Sequential(...)</code>.<br>
If you are using this approach you won’t need to pad your additional feature tensor to an image size, and can just concatenate it in the feature dimension (dim1) as is already done in your code.</p>
<p>Take care of the scaling of the additional features, as we’ve seen some issue with this approach in the past, when the value ranges are quite different.<br>
E.g. if <code>x1</code> has values in <code>[0, 1]</code>, while <code>x2</code> has values in <code>[0, 100]</code>, the last block might just “focus” on the larger values and treat the smaller ones as noise.</p>
<p>Since you are now expecting three output tensors in your <code>DataLoader</code>, use:</p>
<pre><code class="lang-python">for ii, (data1, data2, target) in enumerate(train_loader):
</code></pre>
<p>and feed both data tensors to the model via <code>model(data1, data2)</code>.</p>

abdullah05: <p>Thanks for the quick response!<br>
x1 has values in <code>[600, 800]</code>, x2 in <code>[100, 200]</code> and x3 in <code>[10, 30]</code></p>
<p>So the model would become something like that if I didn’t wanna bother with padding:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
	def __init__(self):
		super(MyModel, self).__init__()
		self.cnn = models.resnet18(pretrained=True)
        self.cnn.fc = nn.Identity()
		self.fc = nn.Sequential(
			nn.Linear(self.cnn.fc.in_features, 256),
			nn.ReLU(),
			nn.Dropout(0.2),
			nn.Linear(256, 2),
			nn.LogSoftmax(dim=1))

	def forward(self, image, data):
		x1 = self.cnn.fc(image)
		x2 = data

		x = torch.cat((x1, x2), dim=1)
		x = self.fc(x)
		return x

model = MyModel()
</code></pre>
<p>Did I understand you correctly?</p>

ptrblck: <p>Some minor mistakes:</p>
<ul>
<li>Change <code>x1 = self.cnn.fc(image)</code> to <code>x1 = self.cnn(image)</code>.</li>
<li>
<code>self.cnn.fc.in_features</code> will be undefined, as it’s not an <code>nn.Identity</code> module. You could flatten <code>x1</code> after the cnn forward pass, print out the shape and define it in your model definition.</li>
</ul>
<pre><code class="lang-python"># Define the in_features of the linear layer with a random number, as we want to throw an error
def forward(self, image, data):
    x1 = self.cnn(x1)
    x1 = x1.view(x1.size(0), -1)
    print(x1.shape)
    x2 = data
    print(x2.shape)
    x = torch.cat((x1, x2), dim=1)
    x = self.fc(x) # will throw an error, but we got the shapes!
    return x
</code></pre>
<p>Use a single forward pass with your input tensors, look at the printed shapes, and define the <code>in_features</code> accordingly.</p>

abdullah05: <p>Ah gotcha! I will try this out tomorrow when I am near the GPU machine.</p>
<p>After this project I really need to learn PyTorch ground up, I will have to work on a more heavy duty vision project than what I am currently doing. All these lessons will come in very handy then!</p>