Repository: pytorch/text

Issue:
max_size vocab is not consistent. 
**Context:**
Num field includes the numbers 0 - 9. I set `max_size=10`. Then I print the vocab that was built:

```
    num_field.build_vocab(train, max_size=10)
    print(num_field.vocab.itos)
    # ['<unk>', '<pad>', '
', '
', u'1', u'2']
    print(len(num_field.vocab.itos))
    # 6
```

Then I checked the `words` created from tokenization:
```
print(words)
# [(u'1', 11308), (u'2', 11270), (u'9', 11058), (u'0', 11020), (u'5', 10952), (u'4', 10942), (u'6', 10914), (u'8', 10820), (u'3', 10766), (u'7', 10706), ('
', 0), ('<pad>', 0), ('
', 0), ('<unk>', 0)]
```

Looks like the vocab built includes only 6 tokens yet the max_size is 10 while there are 14 possible tokens.

**Problem:**
If the number of tokens is larger than `max_size`, `build_vocab` does not fill up the vocabulary up till `max_size`.

**Possible Solution:**
Update https://github.com/pytorch/text/blob/master/torchtext/vocab.py#L129 to not subtract `len(self.itos)` from `max_size`.



Discussion:
i'm a bit confused by your code example --- when you say "`num_field` includes the number 1-9", do you mean that the train set contains 1-9? Could you give a full snippet to reproduce?
@nelson-liu Yup! 

So the training data, when tokenized includes the tokens 0 - 9.

The goal is to train a RNN to reverse a string of numbers `"7 1 5 5 2 5 5 0"` => `"0 5 5 2 5 5 1 7"`.

Example `train.txt`.
Format: `source sequence + \t + target sequence`
Tokenized by spaces (`"5 0"` => `["5", "0"]`)
```
5 0	0 5
4 4 0	0 4 4
2	2
5 2 7 0	0 7 2 5
5 6	6 5
7 1 5 5 2 5 5 0	0 5 5 2 5 5 1 7
9 5 2 1 8 0 4 5	5 4 0 8 1 2 5 9
6	6
3 4 0 6 1 5 4 2	2 4 5 1 6 0 4 3
5 8 0 3	3 0 8 5
2 7 1	1 7 2
7 7 8 8 8 6 2	2 6 8 8 8 7 7
1 7	7 1
1 8 5 1	1 5 8 1
0 4 2 3	3 2 4 0
2 4 9 1 8	8 1 9 4 2
4 9 6 6 6 5 5 0	0 5 5 6 6 6 9 4
6 0 2	2 0 6
1 0 6 4	4 6 0 1
0 9 4 2 7 1 5	5 1 7 2 4 9 0
8 2 3 6 9 2 4 6 8 1	1 8 6 4 2 9 6 3 2 8
2 9 7 4 3	3 4 7 9 2
4 3 6 2 8 4 0 2 7 3	3 7 2 0 4 8 2 6 3 4
6 3	3 6
0 0 0 4 9	9 4 0 0 0
2 2 0 1 4 6	6 4 1 0 2 2
9 3	3 9
8 4 9 3 4 6 8 1 3 1	1 3 1 8 6 4 3 9 4 8
1 2 2 5 7 5 8 8 8 1	1 8 8 8 5 7 5 2 2 1
0 7 0 8 4 6 8 1	1 8 6 4 8 0 7 0
9 7 9	9 7 9
3 3 0 6 4 7 0	0 7 4 6 0 3 3
1 0	0 1
1 6 1 8 8 3 9 8	8 9 3 8 8 1 6 1
5 6 0 9 3 7 6 1 4 8	8 4 1 6 7 3 9 0 6 5
0 5	5 0
4 4 3 7 6 8 0	0 8 6 7 3 4 4
5 0 8 9	9 8 0 5
9 9 5 9 6 0 2 8 4 5	5 4 8 2 0 6 9 5 9 9
8 1 2 4 5 5 3 9	9 3 5 5 4 2 1 8
5 0 9 7 1 1 9 5	5 9 1 1 7 9 0 5
9 2 4 1	1 4 2 9
8 0 0 0 3 1 5 4 6 4	4 6 4 5 1 3 0 0 0 8
4	4
5 1 7 1 2 4 8	8 4 2 1 7 1 5
1 4 6 1 2 2	2 2 1 6 4 1
5 0 9 0 4 1 0	0 1 4 0 9 0 5
5 8 7	7 8 5
4 8 1 5 5	5 5 1 8 4
3	3
7 8 6 7 0 1 6	6 1 0 7 6 8 7
8 3	3 8
9 1 6 5 0 9 6 8 9 5	5 9 8 6 9 0 5 6 1 9
3 8 9 7 3 3 7 5	5 7 3 3 7 9 8 3
5 9 2 6 4	4 6 2 9 5
6 0 4	4 0 6
1 2 3 1 2 5 0 0 2 7	7 2 0 0 5 2 1 3 2 1
3 4 5 2 0 7 5 9 1 4	4 1 9 5 7 0 2 5 4 3
```
Definitely a mistake! Thanks for noticing; this is the kind of breaking bug fix change we need to get in as soon as possible.
I guess this means I should ask for input on the decision whose buggy implementation is at issue here: should `max_vocab=10` include special tokens in that 10 or not? As you can see, I was leaning towards not including them in the 10, but I don't feel strongly either way.
> As you can see, I was leaning towards not including them in the 10 

+1
@jekbradbury Consider renaming `max_vocab` to `max_tokens`? Then define `tokens` to be distinct from `special_tokens` or `specials`.
@jekbradbury @nelson-liu Do you think we should add a label "contributions welcome" to these issues. 
> Do you think we should add a label "contributions welcome" to these issues.

sure, i think it's a bit implicit but could be worth labeling as well. I feel like anything is "contributions welcome", but there aren't any formal contribution guidelines or anything which might make it hard for first-time contributors to OSS in general. For the reference, I'm also just a user who cares about making the library better, and i've found @jekbradbury to be quite receptive to PRs and discussion. (so thanks for that!)

> Consider renaming `max_vocab` to `max_tokens`? Then define `tokens` to be distinct from `special_tokens` or `specials`.

This could work as well, but it might be a bit complex. IMO, the set of specials is small enough that it isn't generally a big deal to me whether my embedding matrix/other stuff is of size `|tokens|` or `|tokens + specials|` (in your definitions of the terms). Better to just pick one and run with it i think.

Additionally, when i think of the semantics of the word `token`, I'm under the impression that it includes specials (since eos, bos, and padding are all tokens as well). `max_vocab` seems clearer to me in that regard, but this is largely just personal preference.

Resolving patch:
diff --git a/torchtext/vocab.py b/torchtext/vocab.py
--- a/torchtext/vocab.py
+++ b/torchtext/vocab.py
@@ -126,7 +126,7 @@ def __init__(self, counter, max_size=None, min_freq=1, wv_dir=os.getcwd(),
         self.itos = ['<unk>'] + specials
 
         counter.subtract({tok: counter[tok] for tok in ['<unk>'] + specials})
-        max_size = None if max_size is None else max_size - len(self.itos)
+        max_size = None if max_size is None else max_size + len(self.itos)
 
         # sort by frequency, then alphabetically
         words = sorted(counter.items(), key=lambda tup: tup[0])


Test patch: