RuntimeError: multi-target not supported

username: gerber
cooked: <p>Hi, I’m training an RNN for intent prediction. There are 11 actions and 29 slots for each action.<br>
for example:<br>
I’d like 2 tickets to see Zoolander 2 tomorrow at Regal Meridian 16 theater in Seattle at 9:25 PM<br>
request(ticket;moviename=Zoolander 2;date=tomorrow;theater=Regal Meridian 16;city=Seattle;starttime=9:25 PM;numberofpeople=2)</p>
<p>I used TfidfVectorizer to transform the text into vectors. Below is the code for training.</p>
<p>Code:</p>
<pre><code class="lang-auto">if __name__ == '__main__':
	parser = argparse.ArgumentParser()
	parser.add_argument('--slots', dest='run_mode', type=int, default=0, help='only predict action')

	args = parser.parse_args()
	params = vars(args)

	run_mode = params['run_mode']

	raw = load_file()
	if(run_mode == 1):
		features_numpy, targets_numpy = Data_Preproccess(raw)
	elif(run_mode == 0):
		features_numpy, targets_numpy = Data_Preproccess2(raw)
	
	# train test split. Size of train data is 80% and size of test data is 20%. 
	features_train, features_test, targets_train, targets_test = train_test_split(features_numpy, targets_numpy, test_size = 0.2, random_state = 42)
	# create feature and targets tensor for train set. As you remember we need variable to accumulate gradients. Therefore first we create tensor, then we will create variable
	featuresTrain = torch.from_numpy(features_train)
	targetsTrain = torch.from_numpy(targets_train).type(torch.LongTensor) # data type is long
	print(featuresTrain.shape)
	print(targetsTrain.shape)
	# create feature and targets tensor for test set.
	featuresTest = torch.from_numpy(features_test)
	targetsTest = torch.from_numpy(targets_test).type(torch.LongTensor) # data type is long

	# batch_size, epoch and iteration
	batch_size = 100
	n_iters = 2500
	num_epochs = n_iters / (len(features_train) / batch_size)
	num_epochs = int(num_epochs)

	# Pytorch train and test sets
	train = torch.utils.data.TensorDataset(featuresTrain,targetsTrain)
	test = torch.utils.data.TensorDataset(featuresTest,targetsTest)

	# data loader
	train_loader = torch.utils.data.DataLoader(train, batch_size = batch_size, shuffle = False)
	test_loader = torch.utils.data.DataLoader(test, batch_size = batch_size, shuffle = False)
		
	# Create RNN
	#print(len(train))
	input_dim = 4773   # input dimension
	hidden_dim = 100  # hidden layer dimension
	layer_dim = 2     # number of hidden layers
	output_dim = 11*29   # output dimension

	model = RNNModel(input_dim, hidden_dim, layer_dim, output_dim)

	# Cross Entropy Loss 
	error = nn.CrossEntropyLoss()

	# SGD Optimizer
	learning_rate = 0.05
	optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
	
	seq_dim = 1 
	loss_list = []
	iteration_list = []
	accuracy_list = []
	count = 0
	for epoch in range(num_epochs):
		for i, (images, labels) in enumerate(train_loader):

			train  = Variable(images.view(-1, seq_dim, input_dim))
			labels = Variable(labels)
				
			# Clear gradients
			optimizer.zero_grad()
			
			# Forward propagation
			outputs = model(train.float())
			
			# Calculate softmax and cross entropy loss
			loss = error(outputs, labels)
			
			# Calculating gradients
			loss.backward()
			
			# Update parameters
			optimizer.step()
			
			count += 1
			
			if count % 250 == 0:
				# Calculate Accuracy         
				correct = 0
				total = 0
				# Iterate through test dataset
				for images, labels in test_loader:
					images = Variable(images.view(-1, seq_dim, input_dim))
					
					# Forward propagation
					outputs = model(images)
					
					# Get predictions from the maximum value
					predicted = torch.max(outputs.data, 1)[1]
					
					# Total number of labels
					total += labels.size(0)
					
					correct += (predicted == labels).sum()
				
				accuracy = 100 * correct / float(total)
				
				# store loss and iteration
				loss_list.append(loss.data)
				iteration_list.append(count)
				accuracy_list.append(accuracy)
				if count % 500 == 0:
					# Print Loss
					print('Iteration: {}  Loss: {}  Accuracy: {} %'.format(count, loss.data[0], accuracy))
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 85, in &lt;module&gt;
    loss = error(outputs, labels)
  File "/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/anaconda3/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 862, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1550, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1407, in nll_loss
    return torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: multi-target not supported at /Users/soumith/miniconda2/conda-bld/pytorch_1532623076075/work/aten/src/THNN/generic/ClassNLLCriterion.c:21
</code></pre>
<p>I can see that the problem is at <code>loss = error(outputs, labels)</code>, since outputs should be 1D. However, I printed its shape and the result is torch.Size([100, 319]). How should I fix this?<br>
Thankyou.</p>

username: ptrblck
cooked: <p>This error is usually thrown when you try to pass a target tensor having an invalid shape, e.g. <code>[batch_size, 1]</code>. Could you check it and <code>squeeze()</code> it if necessary?</p>

username: gerber
cooked: <p>May I ask what the input size should be when I throw my data into the model？</p>

username: ptrblck
cooked: <p>The data you pass into the model depends on the layers you are using.<br>
In case you are using a linear layer, your data shape should be <code>[batch_size, num_features]</code>.</p>
<p><code>nn.CrossEntropyLoss</code> expects logits in the sshape of <code>[batch_size, number_of_classes]</code> and a target tensor of <code>[batch_size]</code> with class indices as its values.</p>

username: NGH
cooked: <p>I faced the same problem. The label of my input is a vector of size 8, ex [1,0.2,0.5,0.33,0.666,1,0.25,0.133], and each of my input has a label of size eight with different values,  I want to learn this vector for each of my input images, I tried to use CrossEntropy but I got this error <strong>RuntimeError: multi-target not supported</strong>.<br>
bach_size = 3<br>
target: torch.Size([3, 8])<br>
predict: torch.Size([3, 8])<br>
Do you have any suggestions on what type of loss function I should use to train my model?</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> is used for a multi-class classification/segmentation use case and thus uses class indices in the target tensor.<br>
Based on your description your target contains values in the range <code>[0, 1]</code>, so you could try to use e.g. <code>nn.BCEWithLogitsLoss</code> or <code>nn.MSELoss</code>. I’m unsure what your exact use case is based on “I want to learn this vector for each of my input images”. If it’s a regression task, use <code>nn.MSELoss</code>, if it’s a “classification” task, you could probably try to use <code>nn.BCEWithLogitsLoss</code>.</p>

username: NGH
cooked: <p>Thank you so much. the task is mulilabel multiclass classification. I will try the BCEwithLgitsLoss.</p>

username: NGH
cooked: <p>In my task, I have eight classes in the output in which each class for each image in my data set has a different label, so instead of having only zero and one in my target vector to show that image belongs to which classes, for each class I have a score for each class.  e.x<br>
for <strong>img1_ label = [1, 0.2, 0.5, 0.33, 0.666, 1, 0.25, 0.133]</strong>, and <strong>img2_label = [0.666, 1, 0.5, 0.33, 1, 0.5, 0.25, 0.133]</strong>. So for img1 the class 1 and 6 has a score one, and class 3 has score 0.5 and so on.<br>
In this case, is still <strong>nn.BCEWithLogitsLoss</strong> a good option for loss function? and if I used this loss function do I need to have Sigmoid as an activate function on the last layer?</p>

username: ptrblck
cooked: <p>Yes, you could still use <code>nn.BCEWithLogitsLoss</code> as the criterion.</p>
<aside class="quote no-group" data-username="NGH" data-post="8" data-topic="27637">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ngh/48/26804_2.png" class="avatar"> NGH:</div>
<blockquote>
<p>and if I used this loss function do I need to have Sigmoid as an activate function on the last layer?</p>
</blockquote>
</aside>
<p>No, you would have to pass the raw logits to this loss function.<br>
You could use <code>sigmoid</code> + <code>nn.BCELoss</code>, but this would yield worse numerical stability.</p>

username: NGH
cooked: <p>Thanks for your help.</p>

username: Sam_Shamsan1
cooked: <p>I am doing multi-label classification, when using CrossEntropyLoss I get this error:<br>
RuntimeError: 1D target tensor expected, multi-target not supported<br>
and when change it to BCEWithLogitsLoss I get this error:<br>
ValueError: Target size (torch.Size([1, 1])) must be the same as input size (torch.Size([1, 18]))</p>
<p>The label indices look as follow for the first 10 examples:<br>
[[0, 1], [0], [0], [0, 1], [0, 2], [0, 1], [0], [0, 1], [0], [0]]</p>
<p>I have 18 classes. Would you please let me know what I am doing wrong.</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>A multi-label classification is used if each sample can belong to zero, one, or multiple classes.<br>
For this to work, the output tensor of the model as well as the target should have the shape <code>[batch_size, nb_classes]</code>. In your current example of the target tensor it seems that it’s using a variable shape, which won’t work, as it should contain the target values for each class.<br>
Assuming you are working with 18 classes, both tensors are expected to have the shape <code>[batch_size, 18]</code>, where the target contains values in <code>[0, 1]</code> for each class.</p>