Confused about tensor dimensions and batches

AndrewKirillo: <p>Hey guys.<br>
So I’m very new to PyTorch and Neural Networks in general, and I’m having some problems creating a Neural Network that classifies names by gender.<br>
I based this off of the PyTorch tutorial for RNNs that classify names by nationality, but I decided not to go with a recurrent approach… <em>Stop me right here if this was the wrong idea!</em><br>
However, whenever I try to run an input through the network it tells me:<br>
<code>RuntimeError: matrices expected, got 3D, 2D tensors at /py/conda-bld/pytorch_1493681908901/work/torch/lib/TH/generic/THTensorMath.c:1232</code><br>
I know this has something to do with how PyTorch always expects there to be a batch size or something, and I have my tensor set up that way, but you can probably tell by this point that I have no idea what I’m talking about.<br>
Here’s my code:<br>
from <strong>future</strong> import unicode_literals, print_function, division<br>
from io import open<br>
import glob<br>
import unicodedata<br>
import string<br>
import torch<br>
import torchvision<br>
import torch.nn as nn<br>
import torch.optim as optim<br>
import random<br>
from torch.autograd import Variable<br>
import matplotlib.pyplot as plt<br>
import matplotlib.ticker as ticker</p>
<pre><code>"""------GLOBAL VARIABLES------"""

all_letters = string.ascii_letters + " .,;'"
num_letters = len(all_letters)
all_names = {}
genders = ["Female", "Male"]

"""-------DATA EXTRACTION------"""

def findFiles(path):
    return glob.glob(path)

def unicodeToAscii(s):
    return ''.join(
        c for c in unicodedata.normalize('NFD', s)
        if unicodedata.category(c) != 'Mn'
        and c in all_letters
    )

# Read a file and split into lines
def readLines(filename):
    lines = open(filename, encoding='utf-8').read().strip().split('\n')
    return [unicodeToAscii(line) for line in lines]

for file in findFiles("/home/andrew/PyCharm/PycharmProjects/CantStop/data/names/*.txt"):
    gender = file.split("/")[-1].split(".")[0]
    names = readLines(file)
    all_names[gender] = names

"""-----DATA INTERPRETATION-----"""

def nameToTensor(name):
    tensor = torch.zeros(len(name), 1, num_letters)
    for index, letter in enumerate(name):
        tensor[index][0][all_letters.find(letter)] = 1
    return tensor

def outputToGender(output):
    gender, gender_index = output.data.topk(1)
    if gender_index[0][0] == 0:
        return "Female"
    return "Male"

"""------NETWORK SETUP------"""

class Net(nn.Module):
    def __init__(self, input_size, output_size):
        super(Net, self).__init__()
        #Layer 1
        self.Lin1 = nn.Linear(input_size, int(input_size/2))
        self.ReLu1 = nn.ReLU()
        self.Batch1 = nn.BatchNorm1d(int(input_size/2))
        #Layer 2
        self.Lin2 = nn.Linear(int(input_size/2), output_size)
        self.ReLu2 = nn.ReLU()
        self.Batch2 = nn.BatchNorm1d(output_size)
        self.softMax = nn.LogSoftmax()

    def forward(self, input):
        output1 = self.Batch1(self.ReLu1(self.Lin1(input)))
        output2 = self.softMax(self.Batch2(self.ReLu2(self.Lin2(output1))))
        return output2

NN = Net(num_letters, 2)

"""------TRAINING------"""

def getRandomTrainingEx():
    gender = genders[random.randint(0, 1)]
    name = all_names[gender][random.randint(0, len(all_names[gender])-1)]
    gender_tensor = Variable(torch.LongTensor([genders.index(gender)]))
    name_tensor = Variable(nameToTensor(name))
    return gender_tensor, name_tensor, gender

def train(input, target):
    loss_func = nn.NLLLoss()

    optimizer = optim.SGD(NN.parameters(), lr=0.0001, momentum=0.9)

    optimizer.zero_grad()

    output = NN(input)

    loss = loss_func(output, target)
    loss.backward()
    optimizer.step()

    return output, loss

all_losses = []
current_loss = 0

for i in range(100000):
    gender_tensor, name_tensor, gender = getRandomTrainingEx()
    output, loss = train(name_tensor, gender_tensor)
    current_loss += loss

    if i%1000 == 0:
        print("Guess: %s, Correct: %s, Loss: %s" % (outputToGender(output), gender, loss.data[0]))

    if i%100 == 0:
        all_losses.append(current_loss/10)
        current_loss = 0

# plt.figure()
# plt.plot(all_losses)
# plt.show()
</code></pre>
<p>Please help a newbie out!</p>

Jordan_Campbell: <p>Hi,</p>
<p>The input to a linear layer should be a tensor of size <code>[batch_size, input_size]</code> where <code>input_size</code> is the same size as the first layer in your network (so in your case it’s <code>num_letters</code>).</p>
<p>The problem appears in the line:</p>
<pre><code>tensor = torch.zeros(len(name), 1, num_letters)
</code></pre>
<p>which should actually just be:</p>
<pre><code>tensor = torch.zeros(len(name), num_letters)
</code></pre>
<p>As an easy example:</p>
<pre><code>input_size = 8
output_size = 14
batch_size = 64

net = nn.Linear(input_size, output_size)
input = Variable(torch.FloatTensor(batch_size, input_size))

output = net(input)

print("Output size:", output.size())
</code></pre>
<blockquote>
<p>Output size: (64, 14)</p>
</blockquote>
<p>Hope this helps,<br>
Jordan</p>

AndrewKirillo: <p>I’ve thought about this, but when I change the input dimensions, I get hit with this batch size error, which I can’t make heads or tails of:<br>
Traceback (most recent call last):<br>
File “/home/andrew/PyCharm/PycharmProjects/CantStop/Cant-Stop/Python Files/TestNet.py”, line 119, in <br>
output, loss = train(name_tensor, gender_tensor)<br>
File “/home/andrew/PyCharm/PycharmProjects/CantStop/Cant-Stop/Python Files/TestNet.py”, line 108, in train<br>
loss = loss_func(output, target)<br>
File “/home/andrew/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 206, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/andrew/anaconda3/lib/python3.6/site-packages/torch/nn/modules/loss.py”, line 36, in forward<br>
return backend_fn(self.size_average, weight=self.weight)(input, target)<br>
File "/home/andrew/anaconda3/lib/python3.6/site-packages/torch/nn/<em>functions/thnn/auto.py", line 41, in forward<br>
output, *self.additional_args)<br>
RuntimeError: Assertion `THIndexTensor</em>(size)(target, 0) == batch_size’ failed.  at /py/conda-bld/pytorch_1493681908901/work/torch/lib/THNN/generic/ClassNLLCriterion.c:50</p>

Jordan_Campbell: <p>I’m a bit confused by your code, previously I thought <code>len(name)</code> was your batch size, is this correct? That error message suggests the batch size of your target and output are different, which does seem plausible if the batch dimension of your input tensor changes according to the name it’s given (without the target tensor also changing).</p>
<p>If it helps here is a self-contained example:</p>
<pre><code>from __future__ import print_function
import torch
import torch.nn as nn
from torch.autograd import Variable

batch_size = 10
input_size = 2
output_size = 2

net = nn.Linear(input_size,output_size)
x = Variable(torch.FloatTensor(batch_size,input_size).normal_(0,1))
criterion = nn.NLLLoss()
target = Variable(torch.LongTensor(batch_size).fill_(1))

output = nn.LogSoftmax()(net(x))

loss = criterion(output, target)

print(loss)

print(x)
print(target)
print(output)
</code></pre>
<p>Also you should move the lines</p>
<pre><code>loss_func = nn.NLLLoss()
optimizer = optim.SGD(NN.parameters(), lr=0.0001, momentum=0.9)
</code></pre>
<p>outside of your <code>train()</code> function, as these only need to be set once.</p>

AndrewKirillo: <p>Yeah, i guess my batch size is equal to the number of letters in the name… from a conceptual standpoint, is that okay?<br>
My idea was that you pass in a tensor representing the name (where every row is a one-hot vector describing the letter), and it outputs a 1x2 tensor that represents the probability that said name was either a boy or a girl. However, I now see the output’s size is batch_sizex2… does this mean that it shows a probability of each letter in the name being indicative of gender? And is there any way to condense these values into a 1x2 tensor like the current target tensor, or is this batch_sizex2 output format okay and I should change what the target tensor looks like, in which case how should I model it?<br>
I’m sorry for the barrage of questions, I just really want to wrap my head around this <img src="//discuss.pytorch.org/images/emoji/apple/grimacing.png?v=5" title=":grimacing:" class="emoji" alt=":grimacing:"></p>

Jordan_Campbell: <p>Generally, an MLP expects to produce an output vector for every input example. I.e for every name it will produce a single gender label.</p>
<p>In your case you are treating <em>every letter in the name as an input example</em>. This is what it means to set the batch size equal to the length of a name. In this case the network expects to produce an output label for every letter in the input, which is not very helpful.</p>
<p>Instead, you should take your one-hot encodings and concatenate them into a single vector, then treat this as a single input example. Your network would then take a <em>batch of names</em> as input, and produce a <em>batch of labels</em> as output.</p>
<p>I think your confusion lies in the definition of the terms: <em>feature, instance, batch</em>. A feature is a single pixel or a single letter, an instance is a single name (a collection of features), while a batch is a collection of names. How you choose to represent these is up to you, but generally a feature is a scalar value, an instance is a vector, while a batch is a matrix. If you are dealing with images (and convolutional networks) then an instance is a matrix and a batch is a tensor (4 dimensions).</p>
<p>Hope this is helpful!</p>
<p>And plenty of questions is fine! Everyone has to learn at some point.</p>

ZHEN_LIU1: <aside class="quote no-group" data-username="Jordan_Campbell" data-post="4" data-topic="4761" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jordan_campbell/48/91_2.png" class="avatar"> Jordan_Campbell:</div>
<blockquote>
<pre><code>target = Variable(torch.LongTensor(batch_size).fill_(1))
</code></pre>
</blockquote>
</aside>
<p>Hey, what does <code>.fill_(1)</code> do in your target? As far as I’m understanding, the target should have a shape of <code>(batch_size, output_size)</code>.</p>