internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pytorch/text

Issue:
ReversibleField performs in-place transpose operation
I am using the recently added `ReversibleField` field.
Converting padded indices into original words was always a cumbersome chore, and the newly added class greatly reduces this burden.
However, I found a strange behavior of the class; when `batch_first` is `True`, it performs in-place transpose operation, which I think not intuitive.
Is it an intended side effect?
The following is the code for reproducing this behavior.
```
TEXT = data.ReversibleField()
LABEL = data.Field(sequential=False)
train, dev, test = datasets.SNLI.splits(TEXT, LABEL)
TEXT.build_vocab(train, max_size=1000) 
LABEL.build_vocab(train) 
train_iter, dev_iter, test_iter = data.BucketIterator.splits((train, dev, test), batch_size=4)
b = next(iter(train_iter))
b.premise.size()  # torch.Size([7, 4])
TEXT.reverse(b.premise.data)
b.premise.size()  # torch.Size([4, 7])
```

And it is also problematic in that calling `TEXT.reverse(b.premise.data)` repeatedly does not give consistent results.
For example, when I call it for the first time, it works nicely:
```
['A girl jumps off of a diving board into a pool.',
 'The mountain biker is UNK the hill on a UNK trail.',
 'A man is standing next to a large black UNK statue.',
 'A young gentleman is pointing to a sign on a building.']
```
However if I call it again, since `b.premise.data` is transposed, it emits wrong results.
```
['A The A A',
 'girl mountain man young',
 'jumps biker is gentleman',
 'off is standing is',
 'of UNK next pointing',
 'a the to to',
 'diving hill a a',
 'board on large sign',
 'into a black on',
 'a UNK UNK a',
 'pool trail statue building',
 '....']
```


Discussion:
Good catch! You're right, that should definitely be out-of-place.

Resolving patch:
diff --git a/torchtext/data/field.py b/torchtext/data/field.py
--- a/torchtext/data/field.py
+++ b/torchtext/data/field.py
@@ -331,7 +331,7 @@ def reverse(self, batch):
                 print("Please install revtok.")
                 raise
         if not self.batch_first:
-            batch.t_()
+            batch = batch.t()
         with torch.cuda.device_of(batch):
             batch = batch.tolist()
         batch = [[self.vocab.itos[ind] for ind in ex] for ex in batch]  # denumericalize


Test patch:
Repository: pytorch/text

Issue:
len of iterator incorrect for dynamic batching
The `__len__` method of `Iterator` (defined [here](https://github.com/pytorch/text/blob/master/torchtext/data/iterator.py#L157)) returns a wrong result for dynamic batching (i.e. if [batch_size_fn](https://github.com/pytorch/text/blob/master/torchtext/data/iterator.py#L50) is not None).

For example if we set `batch_size_fn` to
```
lambda x, n, b: b + len(x.text)
```
there might be more or fewer batches in the dataset than the `__len__` outputs. 


Discussion:

Resolving patch:
diff --git a/torchtext/data/iterator.py b/torchtext/data/iterator.py
--- a/torchtext/data/iterator.py
+++ b/torchtext/data/iterator.py
@@ -72,7 +72,7 @@ class Iterator(object):
     """
 
     def __init__(self, dataset, batch_size, sort_key=None, device=None,
-                 batch_size_fn=lambda new, count, sofar: count, train=True,
+                 batch_size_fn=None, train=True,
                  repeat=None, shuffle=None, sort=None,
                  sort_within_batch=None):
         self.batch_size, self.train, self.dataset = batch_size, train, dataset
@@ -155,6 +155,8 @@ def epoch(self):
         return self.iterations / len(self)
 
     def __len__(self):
+        if self.batch_size_fn is not None:
+            raise NotImplementedError
         return math.ceil(len(self.dataset) / self.batch_size)
 
     def __iter__(self):
@@ -266,8 +268,11 @@ def create_batches(self):
                                 random_shuffler=self.random_shuffler)
 
 
-def batch(data, batch_size, batch_size_fn=lambda new, count, sofar: count):
+def batch(data, batch_size, batch_size_fn=None):
     """Yield elements from data in chunks of batch_size."""
+    if batch_size_fn is None:
+        def batch_size_fn(new, count, sofar):
+            return count
     minibatch, size_so_far = [], 0
     for ex in data:
         minibatch.append(ex)


Test patch:
Repository: pytorch/text

Issue:
Side effect in Vocab __init__
The constructor of Vocab accumulates input data in its `specials` argument/variable and pollute the input argument `counter`.  The constructed object is also wrong because of this side effect.  Please find reproducible example below:
```
>>> c1 = Counter([1])
>>> v1 = Vocab(c1)
>>> print(c1)
Counter({1: 1, u'<pad>': 0})
>>> print(v1.stoi)
defaultdict(<function _default_unk_index at 0x10b4aa758>, {1: 1, u'<pad>': 0})

>>> c2 = Counter([2])
>>> print(c2)
Counter({2: 1})
>>> v2 = Vocab(c2)
>>> print(c2)
Counter({2: 1, 1: 0, u'<pad>': 0}) # c2 is changed after passing as argument
>>> print(v2.stoi)
defaultdict(<function _default_unk_index at 0x10b4aa758>, {1: 1, u'<pad>': 0, 2: 2}) # resulting vocabulary is wrong
```


Discussion:

Resolving patch:
diff --git a/torchtext/vocab.py b/torchtext/vocab.py
--- a/torchtext/vocab.py
+++ b/torchtext/vocab.py
@@ -45,7 +45,8 @@ def __init__(self, counter, max_size=None, min_freq=1, specials=['<pad>'],
                 or custom pretrained vectors (see Vocab.load_vectors);
                 or a list of aforementioned vectors
         """
-        self.freqs = counter.copy()
+        self.freqs = counter
+        counter = counter.copy()
         min_freq = max(min_freq, 1)
         counter.update(specials)
 


Test patch:
Repository: pytorch/text

Issue:
text/test/translation.py fails for custom paths
`text/test/translation.py` currently fails on the last section:

```python
train, val = datasets.TranslationDataset.splits(
    path='.data/multi30k/', train='train',
    validation='val', exts=('.de', '.en'),
    fields=(DE, EN))
```

because `splits` expects TranslationDataset.name to be defined, but it isn't.
Possible fix: add `name = ''` to `TranslationDataset`


Discussion:

Resolving patch:
diff --git a/torchtext/datasets/translation.py b/torchtext/datasets/translation.py
--- a/torchtext/datasets/translation.py
+++ b/torchtext/datasets/translation.py
@@ -40,12 +40,13 @@ def __init__(self, path, exts, fields, **kwargs):
         super(TranslationDataset, self).__init__(examples, fields, **kwargs)
 
     @classmethod
-    def splits(cls, exts, fields, root='.data',
+    def splits(cls, exts, fields, path=None, root='.data',
                train='train', validation='val', test='test', **kwargs):
         """Create dataset objects for splits of a TranslationDataset.
 
         Arguments:
-
+            path (str): Common prefix of the splits' file paths, or None to use
+                the result of cls.download(root).
             root: Root dataset storage directory. Default is '.data'.
             exts: A tuple containing the extension to path for each language.
             fields: A tuple containing the fields that will be used for data
@@ -56,7 +57,8 @@ def splits(cls, exts, fields, root='.data',
             Remaining keyword arguments: Passed to the splits method of
                 Dataset.
         """
-        path = cls.download(root)
+        if path is None:
+            path = cls.download(root)
 
         train_data = None if train is None else cls(
             os.path.join(path, train), exts, fields, **kwargs)


Test patch:
Repository: pytorch/text

Issue:
A batch object created by fromvars does not have "fields" attribute
When making a batch object, the value of the `fields` attribute is set in its `__init__` method.
However, when created with `fromvars` class method, `fields` attribute is not set since the method first creates an empty object and then add information.
It should be modified to be analogous with the one created by `__init__` method.
It can be simply done by adding the following after https://github.com/pytorch/text/blob/master/torchtext/data/batch.py#L36:
```
batch.fields = dataset.fields.keys()
```

This kind of object creation is found when using BPTT iterator. Without `fields` attribute, printing a batch object is not possible due to https://github.com/pytorch/text/blob/master/torchtext/data/batch.py#L49.


Discussion:

Resolving patch:
diff --git a/torchtext/data/batch.py b/torchtext/data/batch.py
--- a/torchtext/data/batch.py
+++ b/torchtext/data/batch.py
@@ -34,6 +34,7 @@ def fromvars(cls, dataset, batch_size, train=True, **kwargs):
         batch.batch_size = batch_size
         batch.dataset = dataset
         batch.train = train
+        batch.fields = dataset.fields.keys()
         for k, v in kwargs.items():
             setattr(batch, k, v)
         return batch


Test patch:
Repository: pytorch/text

Issue:
Use getattr rather than __dict__ in Batch (adds support for __slots__ in Example subclasses)
This is a proposal to change [one line of code](https://github.com/pytorch/text/blob/c839a7934930819be7e240ea972e4d600966afdc/torchtext/data/batch.py#L27) in Batch.py

I suggest `[x.__dict__[name] for x in data]` should become `[getattr(x, name) for x in data]`

A major advantage to doing this is compatibility with `__slots__`. A class that is going to be instantiated for every data point is an ideal use-case for `__slots__`, which reduces per-instance memory overhead. It makes sense for specific projects to subclass Example using `__slots__` with the known fields of the project. If you do, the instances will have empty `__dicts__` but the slots can be accessed via `getattr`.

I don't _think_ this change would break anything...


Discussion:

Resolving patch:
diff --git a/torchtext/data/batch.py b/torchtext/data/batch.py
--- a/torchtext/data/batch.py
+++ b/torchtext/data/batch.py
@@ -24,7 +24,7 @@ def __init__(self, data=None, dataset=None, device=None, train=True):
 
             for (name, field) in dataset.fields.items():
                 if field is not None:
-                    batch = [x.__dict__[name] for x in data]
+                    batch = [getattr(x, name) for x in data]
                     setattr(self, name, field.process(batch, device=device, train=train))
 
     @classmethod


Test patch:
Repository: pytorch/text

Issue:
dataset.sort_key not retained after dataset.split()
Hi I was trying out the new `split()` functionality to split a test set into test and validation set. When the `BucketIterator` is being set up from a newly split dataset the sorting fails because `dataset.sort_key` is None. Looking at the `split()`  function I see that a new instance of type `Dataset` is created in which `sort_key` is None by default:
```
def split():
...
if not stratified:
            train_data, test_data, val_data = rationed_split(self.examples, train_ratio,
                                                             test_ratio, val_ratio, rnd)
            return tuple(Dataset(d, self.fields)
                         for d in (train_data, val_data, test_data) if d)
```

I guess one way would be to explicitly copy the `sort_key` into the new instance. A more generic way could be to make a copy of the current instance and replace only the `examples` attribute but I'm not sure if that is really needed.

Thanks!


Discussion:
Correct, thanks for spotting this. I'll issue a fix soon.

Resolving patch:
diff --git a/torchtext/data/dataset.py b/torchtext/data/dataset.py
--- a/torchtext/data/dataset.py
+++ b/torchtext/data/dataset.py
@@ -108,8 +108,6 @@ def split(self, split_ratio=0.7, stratified=False, strata_field='label',
         if not stratified:
             train_data, test_data, val_data = rationed_split(self.examples, train_ratio,
                                                              test_ratio, val_ratio, rnd)
-            return tuple(Dataset(d, self.fields)
-                         for d in (train_data, val_data, test_data) if d)
         else:
             if strata_field not in self.fields:
                 raise ValueError("Invalid field name for strata_field {}"
@@ -125,8 +123,14 @@ def split(self, split_ratio=0.7, stratified=False, strata_field='label',
                 test_data += group_test
                 val_data += group_val
 
-            return tuple(Dataset(d, self.fields)
-                         for d in (train_data, val_data, test_data) if d)
+        splits = tuple(Dataset(d, self.fields)
+                       for d in (train_data, val_data, test_data) if d)
+
+        # In case the parent sort key isn't none
+        if self.sort_key:
+            for subset in splits:
+                subset.sort_key = self.sort_key
+        return splits
 
     def __getitem__(self, i):
         return self.examples[i]


Test patch:
Repository: pytorch/text

Issue:
TypeError in Python 2.7
https://github.com/pytorch/text/blob/a2795e5731d1b7c0298a1b5087bb8142e1c39d0b/torchtext/datasets/imdb.py#L32

In python 2.7, it will report that `TypeError: 'encoding' is an invalid keyword argument for this function`.

I replace `open` with `io.open` to fix it.


Discussion:

Resolving patch:
diff --git a/torchtext/datasets/imdb.py b/torchtext/datasets/imdb.py
--- a/torchtext/datasets/imdb.py
+++ b/torchtext/datasets/imdb.py
@@ -1,5 +1,6 @@
 import os
 import glob
+import io
 
 from .. import data
 
@@ -29,7 +30,7 @@ def __init__(self, path, text_field, label_field, **kwargs):
 
         for label in ['pos', 'neg']:
             for fname in glob.iglob(os.path.join(path, label, '*.txt')):
-                with open(fname, 'r', encoding="utf-8") as f:
+                with io.open(fname, 'r', encoding="utf-8") as f:
                     text = f.readline()
                 examples.append(data.Example.fromlist([text, label], fields))
 


Test patch:
Repository: pytorch/text

Issue:
Automatically create cache directories for pretrained embeddings
When you try to load your own pretrained embeddings - using eg. ```torchtext.vocab.Vectors``` - if the ```.vector_cache``` directory/the directory in the args doesn't exist, torch's save routine throws an exception. Whilst in theory it is possible to just create the directory yourself, this is annoying on servers with issues with cuda and relative paths. It seems to be pretty simple to just check whether the cache directory exists or not, though, and create it if it doesn't.


Discussion:

Resolving patch:
diff --git a/torchtext/vocab.py b/torchtext/vocab.py
--- a/torchtext/vocab.py
+++ b/torchtext/vocab.py
@@ -327,6 +327,8 @@ def cache(self, name, cache, url=None):
             self.vectors = torch.Tensor(vectors).view(-1, dim)
             self.dim = dim
             logger.info('Saving vectors to {}'.format(path_pt))
+            if not os.path.exists(cache):
+                os.makedirs(cache)
             torch.save((self.itos, self.stoi, self.vectors, self.dim), path_pt)
         else:
             logger.info('Loading vectors from {}'.format(path_pt))


Test patch:
Repository: pytorch/text

Issue:
wordvec of unk token
https://github.com/pytorch/text/blob/master/torchtext/vocab.py#L242

When the word is in the dictionary, the returned is a one-dim vector, while unk vector is a two-dim vector.

I assume this is not expected, right?


Discussion:
@jekbradbury What would be preferred out of those two? I could make a PR for this.
Probably the one-dim vector. Does nothing else in the codebase currently rely on this `__getitem__` method?
@jekbradbury I've been searching through the codebase, but I wasn't able to find anything. It seems that after the vectors are finally loaded, it's up to user's model to deal with it. I'll see whether something breaks when I change this.

Resolving patch:
diff --git a/torchtext/vocab.py b/torchtext/vocab.py
--- a/torchtext/vocab.py
+++ b/torchtext/vocab.py
@@ -239,7 +239,7 @@ def __getitem__(self, token):
         if token in self.stoi:
             return self.vectors[self.stoi[token]]
         else:
-            return self.unk_init(torch.Tensor(1, self.dim))
+            return self.unk_init(torch.Tensor(self.dim))
 
     def cache(self, name, cache, url=None):
         if os.path.isfile(name):


Test patch:
diff --git a/test/data/test_field.py b/test/data/test_field.py
--- a/test/data/test_field.py
+++ b/test/data/test_field.py
@@ -762,7 +762,7 @@ def test_build_vocab(self):
                    [['o', 'n', 'e'], ['l', 'a', 's', 't'], ['s', 'e', 'n', 't']]]
 
         field.build_vocab(sources, vectors='glove.6B.50d',
-                          unk_init=init.xavier_normal,
+                          unk_init=init.normal_,
                           vectors_cache=".vector_cache")
Repository: pytorch/text

Issue:
Translation splits error when not downloading dataset first
Thanks @AngusMonroe for finding this! The problem is that the absence of dataset is not addressed when creating splits. Minimal example:

```

from torchtext.datasets import Multi30k
from torchtext.data import Field
EN = Field()
DE = Field()
ds = Multi30k.splits(('.de','.en'),[('de',DE),('en',EN)],'data/multi30k')
```



Discussion:
I am currently busy and can't work on it (possibly for a few days) so if anyone wants to submit PR, go for it. Otherwise, I will take a look at it as soon as possible.

Resolving patch:
diff --git a/torchtext/datasets/translation.py b/torchtext/datasets/translation.py
--- a/torchtext/datasets/translation.py
+++ b/torchtext/datasets/translation.py
@@ -97,7 +97,9 @@ def splits(cls, exts, fields, root='.data',
             Remaining keyword arguments: Passed to the splits method of
                 Dataset.
         """
-        path = os.path.join('data', cls.name)
+        expected_folder = os.path.join(root, cls.name)
+        path = expected_folder if os.path.exists(expected_folder) else None
+
         return super(Multi30k, cls).splits(
             exts, fields, path, root, train, validation, test, **kwargs)
 
@@ -206,6 +208,8 @@ def splits(cls, exts, fields, root='.data',
             Remaining keyword arguments: Passed to the splits method of
                 Dataset.
         """
-        path = os.path.join('data', cls.name)
+        expected_folder = os.path.join(root, cls.name)
+        path = expected_folder if os.path.exists(expected_folder) else None
+
         return super(WMT14, cls).splits(
             exts, fields, path, root, train, validation, test, **kwargs)


Test patch:
diff --git a/test/translation.py b/test/translation.py
--- a/test/translation.py
+++ b/test/translation.py
@@ -78,7 +78,7 @@ def tokenize_en(text):
 
 train, val = datasets.TranslationDataset.splits(
     path='.data/multi30k/', train='train',
-    validation='val', exts=('.de', '.en'),
+    validation='val', test=None, exts=('.de', '.en'),
     fields=(DE, EN))
 
 print(train.fields)
Repository: pytorch/text

Issue:
SubwordVocab's stoi and itos change if it is called more than once
If `SubwordVocab` is called more than once , `SubwordVocab.stoi` and `SubwordVocab.itos` will be strange except for the first one.

Example code:

```pythonn
from torchtext.vocab import SubwordVocab
from collections import Counter
counter = Counter("torch torch test".split())

v1 = SubwordVocab(counter, max_size=10)
print(v1.itos, v1.stoi)
v2 = SubwordVocab(counter, max_size=10)
print(v2.itos, v2.stoi)
```

Output:

```python
# v1
# stoi and itos are correct
enumerating ngrams: 100%|██████████| 2/2 [00:00<00:00, 4064.25it/s]
building subword vocab: 100%|██████████| 3/3 [00:00<00:00, 12409.18it/s]For faster subwords, please install Julia 0.6, pyjulia, and Revtok.jl. Falling back to Python implementation...

['<pad>', 't', 'c', 'e', 'h', 'o', 'r', 's', 'torch', 'test', 'est'] defaultdict(<function _default_unk_index at 0x1203dd158>,
{'<pad>': 0, 't': 1, 'c': 2, 'e': 3, 'h': 4, 'o': 5, 'r': 6, 's': 7, 'torch': 8, 'test': 9, 'est': 10})

# v2
# stoi and itos are incorrect. several indeces are skipped.
enumerating ngrams: 100%|██████████| 2/2 [00:00<00:00, 4064.25it/s]
building subword vocab: 100%|██████████| 3/3 [00:00<00:00, 12409.18it/s]For faster subwords, please install Julia 0.6, pyjulia, and Revtok.jl. Falling back to Python implementation...

['<pad>', 't', 'c', 'e', 'h', 'o', 'r', 's', 'torch', 'test', 'est', 't', 'c', 'e', 'h', 'o', 'r', 's', 'torch', 'test', 'est'] defaultdict(<function _default_unk_index at 0x1203dd158>,
{'<pad>': 0, 't': 11, 'c': 12, 'e': 13, 'h': 14, 'o': 15, 'r': 16, 's': 17, 'torch': 18, 'test': 19, 'est': 20})
```

This is because `specials` is updated inside `SubwordVocab__init__` due to call by reference.



Discussion:

Resolving patch:
diff --git a/torchtext/vocab.py b/torchtext/vocab.py
--- a/torchtext/vocab.py
+++ b/torchtext/vocab.py
@@ -199,7 +199,7 @@ def __init__(self, counter, max_size=None, specials=['<pad>'],
 
         self.stoi = defaultdict(_default_unk_index)
         self.stoi.update({tok: i for i, tok in enumerate(specials)})
-        self.itos = specials
+        self.itos = specials.copy()
 
         self.segment = revtok.SubwordSegmenter(counter, max_size)
 
@@ -210,6 +210,8 @@ def __init__(self, counter, max_size=None, specials=['<pad>'],
                       key=lambda tup: (len(tup[0]) != 1, -tup[1], tup[0]))
 
         for tok, _ in toks:
+            if len(self.itos) == max_size:
+                break
             self.itos.append(tok)
             self.stoi[tok] = len(self.itos) - 1
 


Test patch:
Repository: pytorch/text

Issue:
Setting `is_target` to True by default in `LabelField`
Hi, 

With the addition of `is_target` in #288, it might be a good idea for `LabelField` to come with `is_target` as True by default.


Discussion:
Yeah, this makes sense. Will be added (if you wish to setup a PR, it would be appreciated).

Resolving patch:
diff --git a/torchtext/data/field.py b/torchtext/data/field.py
--- a/torchtext/data/field.py
+++ b/torchtext/data/field.py
@@ -692,8 +692,10 @@ class LabelField(Field):
     """
 
     def __init__(self, **kwargs):
-        # whichever value is set for sequential and unk_token will be overwritten
+        # whichever value is set for sequential, unk_token, and is_target
+        # will be overwritten
         kwargs['sequential'] = False
         kwargs['unk_token'] = None
+        kwargs['is_target'] = True
 
         super(LabelField, self).__init__(**kwargs)
diff --git a/torchtext/data/iterator.py b/torchtext/data/iterator.py
--- a/torchtext/data/iterator.py
+++ b/torchtext/data/iterator.py
@@ -32,8 +32,7 @@ class Iterator(object):
         repeat: Whether to repeat the iterator for multiple epochs. Default: False.
         shuffle: Whether to shuffle examples between epochs.
         sort: Whether to sort examples according to self.sort_key.
-            Note that repeat, shuffle, and sort default to train, train, and
-            (not train).
+            Note that shuffle and sort default to train and (not train).
         sort_within_batch: Whether to sort (in descending order according to
             self.sort_key) within each batch. If None, defaults to self.sort.
             If self.sort is True and this is False, the batch is left in the
@@ -192,8 +191,7 @@ class BPTTIterator(Iterator):
         repeat: Whether to repeat the iterator for multiple epochs. Default: False.
         shuffle: Whether to shuffle examples between epochs.
         sort: Whether to sort examples according to self.sort_key.
-            Note that repeat, shuffle, and sort default to train, train, and
-            (not train).
+            Note that shuffle and sort default to train and (not train).
         device (str or torch.device): A string or instance of `torch.device`
             specifying which device the Variables are going to be created on.
             If left as default, the tensors will be created on cpu. Default: None.


Test patch: