Question: I'm training a model with tensorflow 2.0. The images in my training set are of different resolutions. The Model I've built can handle variable resolutions (conv layers followed by global averaging). My training set is very small and I want to use full training set in a single batch.  Since my images are of different resolutions, I can't use model.fit(). So, I'm planning to pass each sample through the network individually, accumulate the errors/gradients and then apply one optimizer step. I'm able to compute loss values, but I don't know how to accumulate the losses/gradients. How can I accumulate the losses/gradients and then apply a single optimizer step? Code: for i in range(num_epochs): print(f'Epoch: {i + 1}') total_loss = 0 for j in tqdm(range(num_samples)): sample = samples[j] with tf.GradientTape as tape: prediction = self.model(sample) loss_value = self.loss_function(y_true=labels[j], y_pred=prediction) gradients = tape.gradient(loss_value, self.model.trainable_variables) self.optimizer.apply_gradients(zip(gradients, self.model.trainable_variables)) total_loss += loss_value epoch_loss = total_loss / num_samples print(f'Epoch loss: {epoch_loss}') 

Answer: If I understand correctly from this statement:  How can I accumulate the losses/gradients and then apply a single optimizer step?  @Nagabhushan is trying to accumulate gradients and then apply the optimization on the (mean) accumulated gradient. The answer provided by @TensorflowSupport does not answers it. In order to perform the optimization only once, and accumulate the gradient from several tapes, you can do the following: for i in range(num_epochs): print(f'Epoch: {i + 1}') total_loss = 0 # get trainable variables train_vars = self.model.trainable_variables # Create empty gradient list (not a tf.Variable list) accum_gradient = [tf.zeros_like(this_var) for this_var in train_vars] for j in tqdm(range(num_samples)): sample = samples[j] with tf.GradientTape as tape: prediction = self.model(sample) loss_value = self.loss_function(y_true=labels[j], y_pred=prediction) total_loss += loss_value # get gradients of this tape gradients = tape.gradient(loss_value, train_vars) # Accumulate the gradients accum_gradient = [(acum_grad+grad) for acum_grad, grad in zip(accum_gradient, gradients)] # Now, after executing all the tapes you needed, we apply the optimization step # (but first we take the average of the gradients) accum_gradient = [this_grad/num_samples for this_grad in accum_gradient] # apply optimization step self.optimizer.apply_gradients(zip(accum_gradient,train_vars)) epoch_loss = total_loss / num_samples print(f'Epoch loss: {epoch_loss}')  Using tf.Variable() should be avoided inside the training loop, since it will produce errors when trying to execute the code as a graph. If you use tf.Variable() inside your training function and then decorate it with "@tf.function" or apply "tf.function(my_train_fcn)" to obtain a graph function (i.e. for improved performance), the execution will rise an error. This happens because the tracing of the tf.Variable function results in a different behaviour than the observed in eager execution (re-utilization or creation, respectively). You can find more info on this in the tensorflow help page.