explain this in plain english with numerical examples:

Improving training time with packing
When building this model, we used a batch size of 1. This is not a very efficient way to
train a model. What happens if you change the batch size to 2? Go on, try it.
You should get an error message. The problem is that each name is a different length,
so by default, it is very hard to represent as a tensor. Tensors need all dimensions to
be consistent and the same, but our time dimension (in this case, how many characters
long each name is) is not consistent. This is what causes the error: we have two different
sequences with different lengths.
We saw in chapter 2 that training on batches of data can reduce the training time
significantly by making better use of our GPU’s compute, so we have a real reason to
increase the batch size. Yet we appear to be stuck using an inefficient batch size of 1
due to inputs with differing sizes.
This is an abstraction problem because fundamentally, nothing prevents an RNN from
working with inputs of varying lengths of time. To fix this, PyTorch provides the packed
sequence abstraction (http://mng.bz/4KYV). All types of RNNs available in PyTorch
support working on this class.
Figure 4.13 shows conceptually what is happening when we pack six different
sequences of varying lengths. PyTorch organizes them by length and begins by including the first time step of all sequences in one batch over time. As time progresses,
the shorter sequences reach their end, and the batch size is reduced to the number of
sequences that have not reached their end.134 CHAPTER 4 Recurrent neural networks
b e n
s a m
e d w a r d
f r a n
l u c y
k
Batch of 5
Batch of 3
Batch of 2
Batch of 1
PAD PAD
PAD
PAD
PAD PAD
PAD PAD PAD
Batch of 5 items and 6 steps, but
not everything has the same length
After 3 steps, the shortest
2 sequences are done. So
we stop processing them,
“shrinking” the batch size.
Figure 4.13 Example of packing a batch of five items with sequence lengths of 3, 3, 4, 5, and 6. For
the first three steps, we process all five sequences. On the fourth step t = 4, two of the sequences have
run out, so we drop them and continue the batch on only the sequences that have a length ≥ 4. Packing
organizes the data by length to make this fast and efficient.
Let’s talk through what is happening in this example. We are trying to train on a
batch of five names: “ben,” “sam,” “lucy,” “frank,” and “edward.” The packing process
has sorted them from shortest to longest. There will be a total of T = 6 steps in this
sequence because “edward” is the longest name at six characters. This means our RNN
will perform six iterations in total. Now let’s see what happens at each step:
1 In the first iteration, all five items are processed together in one large batch.
2 All five items are again processed together in one large batch.
3 All five items are again processed, but we have reached the end of the first two
items, “ben” and “sam.” We record their final hidden states from this step h3, since
that’s when they were finished being processed.
4 A batch of only three items is processed, containing “lucy,” “frank,” and “edward,”
because the first two items are done. PyTorch adaptively shrinks the effective batch
size to what remains. “lucy” is finished, and its final state saved as h4.
5 A batch of only two items is processed. “frank” is finished after this step, so h5 is
saved for “frank.”
6 The last step processes only one item, “edward,” bringing us to the end of the
batch and the final hidden state: h6.Improving training time with packing 135
Processing this way ensures that we get the correct hidden state hT for each item even
though they have different lengths. It also makes PyTorch run faster. We do more
efficient batched computation as much as possible and shrink that batch as needed so
we only work on the still-valid data.
4.3.1 Pad and pack
Packing actually involves two steps: padding (make everything the same length) and
packing (store information about how much padding was used). To implement both
padding and packing, we need to override the collate function used by the DataLoader.
This function’s job is to collate lots of independent data points into one larger batch of
items. The default collate function can handle tensors as long as all tensors have the
same shape. Our data comes in as a tuple (xi; yi) with shapes (Ti) and (1), respectively.
Ti can be different for each item, which is normally a problem, so we need to define a
new function called pad_and_pack. It’s a two-step process that is outlined in figure 4.14.
def pad_and_pack(batch):
List of tuples where
1. Store lengths
2. Create list
3. Create list
4. Create padded version of from
5. Create packed version of from
6. Return tuple
Figure 4.14 Steps of packing and padding input so that an RNN can be trained on batches of data that
have different lengths
The input is passed to our function as list objects. Every item in the tuple comes directly
from the Dataset class. So if you alter the Dataset to do something unique, this is how
you update the DataLoader to work with it. In our case, the Dataset returns a tuple of
(input; output), so our pad_and_pack function takes in a list of tuples. The steps are as
follows:
1 Store the length of every item.
2 Create new lists of just the inputs and just the output labels.136 CHAPTER 4 Recurrent neural networks
3 Create a padded version of the input list. This makes all the tensors the same size,
with a special token appended to the shorter items. PyTorch can do this using the
pad_sequence function.
4 Create the packed version of the input using PyTorch’s pack_padded_sequence
function. This takes the padded version as input, along with the list of lengths, so
that the function knows how long each item was originally.
That’s the high-level idea. The following code implements this process:
def pad_and_pack(batch):
input_tensors = [] Organizes the batch
input lengths, inputs, and
outputs as separate lists
labels = []
lengths = []
for x, y in batch:
input_tensors.append(x)
labels.append(y)
lengths.append(x.shape[0]) Assumes shape is (T, *)
x_padded = torch.nn.utils.rnn.pad_sequence(
Creates the padded
input_tensors, batch_first=False) version of the input
x_packed =
Creates the packed
version from the
padded and lengths
torch.nn.utils.rnn.pack_padded_sequence(x_padded, lengths,
batch_first=False, enforce_sorted=False)
y_batched = torch.as_tensor(labels, dtype=torch.long)
Converts the lengths
into a tensor
return x_packed, y_batched
Returns a tuple of
the packed inputs
and heir labels
Notice two things about this code. First, we set an optional argument batch_first=
False because our input data does not yet have a batch dimension; it just has the length.
If we had a batch dimension, and it was stored as the first dimension (the norm for
most code), we would set this value to True.
Second, for the packing step, we have the flag enforce_sorted=False because we
have chosen not to presort the input batches by length. If we set enforce_sorted=True,
we would get an error because our inputs are not sorted. Older versions of PyTorch
required you to do this sorting yourself, but the current version works fine on unsorted
inputs. It still avoids unnecessary computations and is generally just as fast, so we chose
this easier option.