foldl/torch-forum huggingface dataset:
JoinTable like operation when combining CNNs

pchandar: <SCODE>class Net(nn.Module):
def __init__(self):
    super(Net, self).__init__()
    self.embed = nn.Embedding(vocab.size(), 300)
    #self.embed.weight = Parameter( torch.from_numpy(vocab.get_weights().astype(np.float32)))        
    self.conv_3 = nn.Conv2d(1, 50, kernel_size=(3, 300),stride=(1,1))
    self.conv_4 = nn.Conv2d(1, 50, kernel_size=(4, 300),stride=(1,1))
    self.conv_5 = nn.Conv2d(1, 50, kernel_size=(5, 300),stride=(1,1))
    self.decoder = nn.Linear(50 * 3, len(labels))


def forward(self, x):
    e1 = self.embed(x)
    x = F.dropout(e1, p=0.2)
    x = e1.view(x.size()[0], 1, 50, 300)
    cnn_3 = F.relu(F.max_pool2d(self.conv_3(x), (maxlen - 3 + 1, 1)))
    cnn_4 = F.relu(F.max_pool2d(self.conv_4(x), (maxlen - 4 + 1, 1)))
    cnn_5 = F.relu(F.max_pool2d(self.conv_5(x), (maxlen - 5 + 1, 1)))
    x = torch.cat([e.unsqueeze(0) for e in [cnn_3, cnn_4, cnn_5]]) 
    x = x.view(-1, 50 * 3)
    return F.log_softmax(self.decoder(F.dropout(x, p=0.2)))<ECODE>

apaszke:
Simple L2 regularization?

Kalamaya: Hi, does simple L2 / L1 regularization exist in pyTorch? I did not see anything like that in the losses. I guess the way we could do it is simply have the data_loss + reg_loss computed, (I guess using nn.MSEloss for the L2), but is an explicit way we can use it without doing it this way? Thanks

fmassa: Hi, <SCODE>l1_crit = nn.L1Loss(size_average=False)
reg_loss = 0
for param in model.parameters():
    reg_loss += l1_crit(param)

factor = 0.0005
loss += factor * reg_loss
<ECODE> Note that this might not be the best way of enforcing sparsity on the model though.

Kalamaya: 

apaszke: Yeah, that’s been added there as an optimization, as L2 regularization is often used.

Kalamaya: 

ecolss: what do you recommend which would be a better way to enforce sparsity instead of L1?

fmassa: 

ecolss: 

ncullen93: example: <SCODE>xx = nn.Parameter(torch.from_numpy(np.ones((3,3))))
l1_crit = nn.L1Loss()
l1_crit(xx)
<ECODE>

fmassa: <SCODE>xx = nn.Parameter(torch.from_numpy(np.ones((3,3))))
target = Variable(torch.from_numpy(np.zeros((3,3))))
l1_crit = nn.L1Loss()
l1_crit(xx, target)
<ECODE>

Ja-Keoung_Koo: I have two questions about L1 regularization: 
How do we backpropagate for L1 regularization? I wonder it because the term is not differentiable.
 
Where can I see the implementation of L1 regularization? In the following link, there is only pass.


smth: 

gwg: 

fmassa: If you add the L1 regularization to the loss as I explained in a previous post, the gradients will be handled by autograd automatically.

yibo: 

Brando_Miranda: <SCODE>batch_loss = loss(input=y_pred,target=batch_ys)
batch_loss += lambda*R
<ECODE> ?

hughperkins: 

Neta_Zmora: 

shirui-japina: <SCODE>weight_p, bias_p = [],[]
for name, p in model.named_parameters():
  if 'bias' in name:
    bias_p += [p]
  else:
    weight_p += [p]

optim.SGD(
  [
    {'params': weight_p, 'weight_decay':1e -5},
    {'params': bias_p, 'weight_decay':0}
  ],
  lr=1e-2, momentum=0.9
)
<ECODE>

Deepak: We do regularization to handle the high variance problem (overfitting). It’s good if the regularization includes all the learnable parameters (both weight and bias). But since bias is only a single parameter out of the large number of parameters, it’s usually not included in the regularization; and exclusion of bias hardly affects the results.
Why use nn.Conv2D over nn.functional.conv2d?

Kalamaya: Hi, I stumbled upon this question because I am trying to have control over how my convolutional weights are initialized. The API for both of those however seems different. For the former: torch.nn.functional.conv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1) For the latter: class torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True) So, my questions are the following: Thanks

smth:
Are GPU and CPU random seeds independent?

rasbt: <SCODE>torch.manual_seed(args.seed)
if args.cuda:
    torch.cuda.manual_seed(args.seed)
<ECODE> torch.manual_seed(args.seed) sufficient?

apaszke: It is sufficient for CPU determinism, but it won’t affect the GPU PRNG state. We’ve been thinking about merging these two, and we’ll probably do so in the future.

rasbt: Thanks! And merging them in future sounds like a good idea.

Alex_Choy: Agree, less chance for mysterious inconsistency for programmers who’re not aware of this.
Equivalent of np.reshape() in pyTorch?

Kalamaya: 

rasbt: <SCODE>>>> import torch
>>> t = torch.ones((2, 3, 4))
>>> t.size()
torch.Size([2, 3, 4])
>>> t.view(-1, 12).size()
torch.Size([2, 12])<ECODE>

Kalamaya: 

apaszke: A caveat is that it has to be contiguous, but that matches numpy as far as I know.

tachim: 

jekbradbury: Since people have asked about this several times – yes. You just have to define the module version yourself: <SCODE>class View(nn.Module):
    def __init__(self, *shape):
        super(View, self).__init__()
        self.shape = shape
    def forward(self, input):
        return input.view(*shape)

sequential_model = nn.Sequential([Linear(10, 20), View(-1, 5, 4)])
<ECODE>

mderakhshani: 

SherlockLiao: You can use permute in pytorch to specify its order of reshaping. <SCODE>t = torch.rand((2, 3, 4))
t = t.permute(1, 0, 2)
<ECODE> this can reshape its order

cosmozhang1988: 

colesbury: 

cosmozhang1988: Thank you!!!

Brando_Miranda: why is this not called reshape?

rasbt: My guess is that it’s been done to be consistent with torch rather than numpy, which makes sense. However, yeah, some of the naming conventions are a bit annoying for people coming from NumPy, not Torch

Eric_K: I think the torchTensor.view() is same as to np.reshape() after my experiment. And torchTesnor.permute is same as to np.transpose. But I have a question about what is the use of np.reshape. Because in image processing domain, reshape will change the structure information of the image and that is fatal.

Eric_K: In image processing field, we should use permute in my opinion then what is the meaning of view()'s existence? Thank you!

jpeg729: <SCODE>linear_input = conv_output.view(batch_size, channels_out*height_out*width_out)
<ECODE>

Elias_Vansteenkiste:
Visual watcher when training/evaluating or tensorboard equivalence?

pengsun: Hi, As leveraging python ecosystem seems one of the main reasons for switching to pytorch from Torch 7, would you provide guidance to visually overseeing training/evaluating with 3rd party Python libraries that is hard to do in pure Torch 7 ? For example,

smth: 

Atcold: 

Atcold: 

pengsun: 

edgarriba: 

AjayTalati: Hi, noob question here - might be especially relevant for people moving over from TensorFlow? I just wonder if anyone has got Crayon working for multi-threads? I can’t figure it out? I keep getting this error <SCODE>Traceback (most recent call last):
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ajay/PythonProjects/PyT_Neural_Arch_Search_v1_2/train_v1.py", line 201, in train
    foo.get_scalar_values("Mean Reward")
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/site-packages/pycrayon/crayon.py", line 167, in get_scalar_values
    return json.loads(r.text)
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
<ECODE> It works fine through when I run the following from the interpreter, <SCODE>from pycrayon import CrayonClient
cc = CrayonClient( hostname="http://127.0.1.1" , port=8889)
foo = cc.create_experiment("train_" + str(rank))

foo.add_scalar_value("Mean Reward", mean_reward, step = episode_count)
foo.add_scalar_value("Max Reward" , max_reward,  step = episode_count)

foo.get_scalar_values("Mean Reward")
foo.get_scalar_values("Max Reward")
<ECODE>

AjayTalati: Nevermind, it works without the extra lines, <SCODE>foo.get_scalar_values("Mean Reward")
foo.get_scalar_values("Max Reward")
<ECODE> Also I needed to give distinct date time’s for each of the threads, <SCODE>cc = CrayonClient( hostname="http://127.0.1.1" , port=8889)
exp_name = "train_" + str(rank) + "_" + datetime.now().strftime('train_%m-%d_%H-%M') # prevents server errors ??
foo = cc.create_experiment(exp_name)<ECODE>

ibadami:
Variable Can not compare with other parameter?

cumttang: When I try to find out some Variable result who is the biggest. However, I found that “Variable” can not compare with other parameters, such as ndarray or FloatTensor. Thus, is there any thing wrong with Variable, or what should I do?

apaszke: 

cumttang: Oh, thanks. Maybe there is another way to solve this problem for transforming Variable into Ndarray. However, I think to use comparison operators for variable is more convenient.

Atcold: 

cumttang:
Weight initilzation

Hamid: How I can initialize variables, say Xavier initialization?

Atcold: 

fmassa: 

michael_k: 

Kalamaya: 

Atcold: Does it make any sense what I am saying? If it does not, I can try better and with an example.

Kalamaya: 

Atcold: You first define your name check function, which applies selectively the initialisation. <SCODE>def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        xavier(m.weight.data)
        xavier(m.bias.data)
<ECODE> <SCODE>net = Net() # generate an instance network from the Net class
net.apply(weights_init) # apply weight init
<ECODE>

apaszke: A less Lua way of doing that would be to check if some module is an instance of a class. This is the recommended way: <SCODE>def weights_init(m):
    if isinstance(m, nn.Conv2d):
        xavier(m.weight.data)
        xavier(m.bias.data)
<ECODE>

Hamid: 

Atcold: <SCODE>size = m.weight.size() # returns a tuple
fan_out = size[0] # number of rows
fan_in = size[1] # number of columns
<ECODE>

apaszke: 

Atcold: 

Hamid: my code: <SCODE>def weight_init(m): 
	if isinstance(m, nn.Linear):
		size = m.weight.size()
		fan_out = size[0] # number of rows
		fan_in = size[1] # number of columns
		variance = np.sqrt(2.0/(fan_in + fan_out))
		m.weight.data.normal_(0.0, variance)


class Residual(nn.Module):
	def __init__(self,dropout, shape, negative_slope, BNflag = False):
		super(Residual, self).__init__()
		self.dropout = dropout
		self.linear1 = nn.Linear(shape[0],shape[1])
		self.linear2 = nn.Linear(shape[1],shape[0])
		self.dropout = nn.Dropout(self.dropout)
		self.BNflag = BNflag
		self.batch_normlization = nn.BatchNorm1d(shape[0])
		self.leakyRelu = nn.LeakyReLU(negative_slope = negative_slope , inplace=False)

	def forward(self, X):
		x = X
		if self.BNFlag:
			x = self.batch_normlization(x)
		x = self.leakyRelu(x)
		x = self.dropout(x)
		x = self.linear1(x)
		if self.BNFlag:
			x = self.batch_normlization(x)
		x = self.leakyRelu(x)
		x = self.dropout(x)
		x = self.linear2(x)
		x = torch.add(x,X)
		return x
		
		
class FullyCN(nn.Module):
	def __init__(self, args):
		super(FullyCN, self).__init__()
		self.numlayers = arg.sm-num-hidden-layers
		self.learning-rate= args.sm-learning-rate
		self.dropout = arg.sm-dropout-prob
		self.BNflag = args.sm-bn
		self.shape = [args.sm-input-size,args.sm-num-hidden-units]		
		self.res =  Residual(self.dropout,self.shape,args.sm_act_param,self.self.BNflag)
		self.res(weight_init)
		self.res-outpus = []

	def forward(self,X):
		self.res-outpus.append(self.res(X))
		for i in range(self.numlayers):
			self.res-outpus.append(self.res(self.res-outpus[-1]))
		return self.res-outpus[-1]<ECODE>

Hamid: sorry about confusion

Atcold: Correct.

Hamid: But I have called weight_init once for the class while I call linear layers in a for loop (i.e., there are multiple sets of variables).

Atcold: <SCODE>net = Residual() # generate an instance network from the Net class
net.apply(weights_init) # apply weight init
<ECODE>

apaszke: 

Hamid:
High memory usage while training

NgPDat: <SCODE>class testNet(nn.Module):
    def __init__(self):
        super(testNet, self).__init__()
        self.rnn = nn.RNN(input_size=200, hidden_size=1000, num_layers=1)
        self.linear = nn.Linear(1000, 100)

    def forward(self, x, init):
        x = self.rnn(x, init)[0]
        y = self.linear(x.view(x.size(0)*x.size(1), x.size(2)))
        return y.view(x.size(0), x.size(1), y.size(1))

net = testNet()
init = Variable(torch.zeros(1, 4, 1000))
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.9)

total_loss = 0.0
for i in range(10000): #10000 mini-batch
    input = Variable(torch.randn(1000, 4, 200)) #Seqlen = 1000, batch_size = 4, feature = 200
    target = Variable(torch.LongTensor(4, 1000).zero_())

    optimizer.zero_grad()
    output = net(input, init)
    loss = criterion(output.view(-1, output.size(2)), target.view(-1))
    loss.backward()
    optimizer.step()
    total_loss += loss[0]

print(total_loss)
<ECODE> I expect memory usage not increasing per mini-batch. What might be the problem? (Correct me if my script is wrong)

smth: I’m trying to reproduce your results. So far, my run is pretty stable at around 105MB, after 400 mini-batches, I will wait for some time.

apaszke: 

NgPDat: 

NgPDat:
Train Neural network problem in reinforcement learning

cumttang: <SCODE>self.value_optimizer.zero_grad()
preValue = self.value_net(state).data
nextValue = self.value_net(next_state).data
expectedValue = (self.gamma * nextValue) + reward

preValue = Variable(preValue)
expectedValue = Variable(expectedValue)
loss = F.smooth_l1_loss(preValue, expectedValue)
loss.backward()
self.value_optimizer.step()

**self._execution_engine.run_backward((self,), (gradient,), retain_variables)**
**RuntimeError: there are no graph nodes that require computing gradients**
<ECODE>

smth: 

apaszke: <SCODE>self.value_optimizer.zero_grad()
# Here, when you unpack the data, you detach the data from the graph
# No backpropagation through the model is possible, because you got rid
# of the reference to the grpah.
preValue = self.value_net(state).data
nextValue = self.value_net(next_state).data
expectedValue = (self.gamma * nextValue) + reward

# Here, you repack the tensors in Variables, but the history of
# operations is not retained - they are leaf Variables.
# Also you didn't specify that they require gradients (they don't
# by default).
preValue = Variable(preValue)
expectedValue = Variable(expectedValue)
loss = F.smooth_l1_loss(preValue, expectedValue)
# At this point your while graph looks like this - no model there:
#  preValue        expectedValue
#         \          /
#        smooth_f1_loss
#              |
#            loss
loss.backward()
self.value_optimizer.step()
<ECODE>

cumttang: I have tried it, but it seems not work.

cumttang: 

apaszke: Try this: <SCODE>self.value_optimizer.zero_grad()
preValue = self.value_net(state)
nextValue = self.value_net(next_state).detach() # don't backprop this way
expectedValue = (self.gamma * nextValue) + reward
loss = F.smooth_l1_loss(preValue, expectedValue)
loss.backward()
self.value_optimizer.step()
<ECODE>

cumttang: I have tried it, but this problem occurs. <SCODE>**expectedValue = (self.gamma * nextValue) + reward**
<ECODE>

apaszke:
Sharing optimizer between processes

delgado: I am wondering if it is possible to share optimizer between different threads. To be specific, when optimizer.step() is applied, modified state of the optimizer should be available for all processes.

smth: i cant think of a way to do this, because the optimizer also has python scalars. What are you trying to do?

delgado: 

apaszke: No, I think most of the optimizer state is kept in Tensors, so it should be possible to share it. I’m not 100% now, I’ll need to take a look at RMSprop and can confirm it tomorrow.

delgado: Thanks. I will also look at the code and see if I can find some hack.

ypxie: 

apaszke: It’s not. You need to guard it with a mutex yourself

mimoralea: Just a quick follow-up for those wanting to implement A3C. A3C uses a Hogwild! update style, which means updates are made lock-free. Workers possibly overwrite each other updates at times, but that’s OK. But other than that, it is doable, and not too hard.
RuntimeError: expected a Variable argument, but got function

Hamid: Hello, I get this weird error while running my code. <SCODE>class Residual(nn.Module):
	def __init__(self,dropout, shape, negative_slope, BNflag = False):
		super(Residual, self).__init__()
		self.dropout = dropout
		self.linear1 = nn.Linear(shape[0],shape[1])
		self.linear2 = nn.Linear(shape[1],shape[0])
		self.dropout = nn.Dropout(self.dropout)
		self.BNflag = BNflag
		self.batch_normlization = nn.BatchNorm1d(shape[0])
		self.leakyRelu = nn.LeakyReLU(negative_slope = negative_slope , inplace=False)

	def forward(self, X):
		x = X
		if self.BNflag:
			x = self.batch_normlization(x)
		x = self.leakyRelu(x)
		x = self.dropout(x)
		x = self.linear1(x)
		if self.BNflag:
			x = self.batch_normlization(x)
		x = self.leakyRelu(x)
		x = self.dropout(x)
		x = self.linear2(x)
		x = torch.add(x,X)
		return x

res =  Residual(0.5,[100,200],0.2,False)
<ECODE> I haven’t passed any data yet.

Hamid: Never mind. problem  Solved
How to check if Model is on cuda

napsternxg: 

smth: As replied on the github issues, an easy way is: <SCODE>next(model.parameters()).is_cuda # returns a boolean
<ECODE>

mikolchon: 

thadar: I tested it right now, and it works even in pytorch v0.4

justusschock: But why is it even necessary? If a model is on cuda and you call model.cuda() it should be a no-op and if the model is on cpu and you call model.cpu() it should also be a no-op.

mikolchon: 

rasbt: Personally, I develop and debug 99% of the code on macOS, and then sync it over to a headless cluster, which is why this pattern is useful to me, for example. Yes, e.g., you can now specify the device 1 time at the top of your script, e.g., <SCODE>device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 
<ECODE> and then for the model, you can use <SCODE>model = model.to(device)
<ECODE> The same applies also to tensors, e.g,. <SCODE>for features, targets in data_loader:
    features = features.to(device)
    targets = targets.to(device)<ECODE>

chiragraman: 

Martin_Ennemoser: You can get the device by: <SCODE>next(network.parameters()).device
<ECODE>

Brando_Miranda: why doesn’t: <SCODE>self.device
<ECODE> <SCODE>(Pdb) self.device
*** torch.nn.modules.module.ModuleAttributeError: 'MyNN' object has no attribute 'device'
<ECODE>

ptrblck: 

Brando_Miranda: 

ptrblck: Yes, that’s correct.
Dropout functional API, advantages/disadvantages?

rasbt: I saw in one of the examples that the functional API was used to implement dropout for a conv layer but not for the fully connected layer. Was wondering if this has a specific reason? I pasted the code below <SCODE>class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = F.relu(self.fc2(x))
        return F.log_softmax(x)
<ECODE> Am I correct that this is the same as the following code? <SCODE>class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc1_drop = nn.Dropout(p=0.5) # added this line
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = self.fc1_drop(x) # added this line
        #x = F.dropout(x, training=self.training) # removed this line
        x = F.relu(self.fc2(x))
        return F.log_softmax(x)
<ECODE>

apaszke: No, they’re equivalent and they expand to calls to the same autograd functions. The same applies for all other layers - both functional and non-functional versions work exactly the same, it’s only a matter of personal taste.

apaszke: In that tutorial I probably missed the dropout module and that’s why I didn’t change it to functional. I find using the functional API for parameter-less operations and modules for everything containing parameters the best.

surag: 

rasbt: After I got back into PyTorch for some research projects lately, I adopted the habit of using the functional API for everything that does not have parameters (in the sense of weights, biases, etc.) Regarding dropout, I think there’s no issue with that as you can specify the training/eval mode via e.g., <SCODE>x = F.dropout(out, p=dropout_prob, training=self.training)
<ECODE> PS: I don’t use nn.Sequential, though, because I usually like to be able access intermediate states conveniently

two_four:
How to get the graph of your DNN drawn?

Kalamaya: Hi all, I am wondering if there is an easy plug-and-play way in order for me to have a nice visualization of my DNN that I have designed? Im after something simple to be honest, just something where I can see the various layers connected to each other to get a sense of what I have coded. Thanks

fmassa: 

ido4848: 

fmassa: For the moment, we need to wait until the new autograd is merged into master (which should happen next week). Once that’s done, one can start working on a tensorboard integration, as the underlying graph representation from PyTorch shouldn’t change much

varun-suresh: Any updates on the tensorboard integration? How can I contribute?

hhsecond: 

fmassa: 

hhsecond: Sounds good, i’ll wait for that then. Thanks

hhsecond: Do we have any plans to build own visualization tool based on jit.trace or ONNX? I and my colleagues had built one basic visualizer and we were planning to make a better one. If somebody is working on it already, would like to discuss.
Why is input used as var name in examples and pytorch convention!

napsternxg: A discussion is needed for adopting common python specific convention to pytorch.

apaszke: I know that many people will consider that a “bad practice”, but I seriously don’t think that giving up readability for the sake of satisfying a rule that doesn’t make a lot of sense in this context is worth it.

emanjavacas: 

apaszke: 

emanjavacas: Sure, I was just suggesting that not overwriting reserved symbols will bring an increase in the quality of the examples (however small that increase is).

apaszke: If someone sends a PR I’ll accept it, but it’s not a priority for us now, so we’ll do it later otherwise.
Data set loader questions

gsp-27: I have downloaded and split the CIFAR 10 data using the given torch.datasets api, now I want to separate out training and validate data, say I want out 50000 sample 49000 as training example and 1000 as validation examples, how do I achieve this?? Also say I keep the batch size as 64, then in the case where I have 50000 training samples last batch will not have the same number of samples as the other batches, how is this case handled??

gsp-27: