Repository: pytorch/text

Issue:
Unintuitive behavior of Iterator when sort is False
Currently, the following line is executed regardless of `sort` value.
https://github.com/pytorch/text/blob/2980f1bc39ba6af332c5c2783da8bee109796d4c/torchtext/data/iterator.py#L162

It could result in a counter-intuitive behavior when `sort is False`, since one would probably expect that the order of data is kept intact when `sort` is `False`.

I think this should be executed only when `sort` is `True`.
Is it by design, or a bug?


Discussion:
Hmm it seems that it can't be solved simply --- I think currently sorting is kind of tangled in the implementation.
For example when BucketIterator is used, sorting is done within a batch, whether sort is True or not.
(But with the plain Iterator, sort-within-batch does not occur thus we can't use packed_padded_sequence unless we perform some explicit sorting.)

I think just calling "reverse" function in the iterator cannot completely solve the issue wrt "packed sequence". Currently it works only with BucketIterator.

I suggest the following modification:
1) "sort" flag is used only for indicating "sort the entire data".
2) Add the additional "sort_batch" (or more appropriate name) flag to let the iterator sort the batch.
3) Sort order (increasing or decreasing) should be designated only by sort_key function of a dataset. Implanting "reverse" operation directly into the core Dataset implementation does not seem to be a good way to do this.

For example, if one wants to sort the batch, not the entire data, in the decreasing order of length, they should set `sort=False`, `sort_batch=True`, and `sort_key=lambda ex: -len(ex.text)`.
I agree with most of that, but I still think it's helpful for the order within a batch to default to the opposite of the provided sort_key order (which is used directly as the order between batches). That packed sequences are sorted in decreasing order is essentially a cuDNN implementation detail, while reversing the batch later is difficult to do performantly in the absence of negative stride support in TH/THC core and I think it's strange to ask users to manually provide a reversed sort_key if they want to use packed sequences (since I see sort_key as largely a dataset property that says by which field's length/combination of field lengths a dataset is most naturally sorted).
Ideally I want to satisfy everyone, but this sorting question has been causing issues in OpenNMT for a few weeks, so I think we should come to a decision and release 0.2 to pypi? Personally I'd lean towards a) making the change outlined in the OP and b) also adding a separate flag to allow sorting the batch even when the data isn't otherwise sorted, but retaining the `reverse`. Any thoughts @nelson-liu?
Whoops, this seems to have skipped my inbox. I agree with the conclusion that seems to have been reached (make change described in OP, add flag to allow sorting batch when data isn't otherwise sorted but retaining the reverse)...

Resolving patch:
diff --git a/torchtext/data/iterator.py b/torchtext/data/iterator.py
--- a/torchtext/data/iterator.py
+++ b/torchtext/data/iterator.py
@@ -63,19 +63,28 @@ class Iterator(object):
         sort: Whether to sort examples according to self.sort_key.
             Note that repeat, shuffle, and sort default to train, train, and
             (not train).
+        sort_within_batch: Whether to sort (in descending order according to
+            self.sort_key) within each batch. If None, defaults to self.sort.
+            If self.sort is True and this is False, the batch is left in the
+            original (ascending) sorted order.
         device: Device to create batches on. Use -1 for CPU and None for the
             currently active GPU device.
     """
 
     def __init__(self, dataset, batch_size, sort_key=None, device=None,
                  batch_size_fn=lambda new, count, sofar: count, train=True,
-                 repeat=None, shuffle=None, sort=None):
+                 repeat=None, shuffle=None, sort=None,
+                 sort_within_batch=None):
         self.batch_size, self.train, self.dataset = batch_size, train, dataset
         self.batch_size_fn = batch_size_fn
         self.iterations = 0
         self.repeat = train if repeat is None else repeat
         self.shuffle = train if shuffle is None else shuffle
         self.sort = not train if sort is None else sort
+        if sort_within_batch is None:
+            self.sort_within_batch = self.sort
+        else:
+            self.sort_within_batch = sort_within_batch
         if sort_key is None:
             self.sort_key = dataset.sort_key
         else:
@@ -157,9 +166,14 @@ def __iter__(self):
                     continue
                 self.iterations += 1
                 self._iterations_this_epoch += 1
-                # NOTE: `rnn.pack_padded_sequence` requires that a minibatch be sorted by
-                # decreasing order, which requires reversing relative to typical sort keys
-                minibatch.reverse()
+                if self.sort_within_batch:
+                    # NOTE: `rnn.pack_padded_sequence` requires that a minibatch
+                    # be sorted by decreasing order, which requires reversing
+                    # relative to typical sort keys
+                    if self.sort:
+                        minibatch.reverse()
+                    else:
+                        minibatch.sort(key=self.sort_key, reverse=True)
                 yield Batch(minibatch, self.dataset, self.device,
                             self.train)
             if not self.repeat:


Test patch: