Repository: pytorch/text

Issue:
text_classification dataset default ngrams argument does not match comments
## 🐛 Bug

`_setup_datasets` for the `text_classification` datasets has the `ngrams` argument default to 2, [see](https://github.com/pytorch/text/blob/master/torchtext/datasets/text_classification.py#L116)

However the comments for each of the `text_classification` datasets says that the default `ngrams` argument is 1, [see](https://github.com/pytorch/text/blob/master/torchtext/datasets/text_classification.py#L159).

So the default in `_setup_datasets` should be changed.

I would submit a pull request but it looks like this can be incorporated as part of adding the IMDB dataset: https://github.com/pytorch/text/pull/651





Discussion:
@bentrevett Please let us know if you have any feedbacks for the new datasets in `text_classification` and `language_modeling`. I will do a cherry-up for the bug in doc.

@cpuhrsch 
@zhangguanheng66 I think they look good so far - but do they not lose functionality offered by `data.Field` and `build_vocab`?

A lot of the functionality offered by the `Field` can be incorporated into the `tokenizer` argument, e.g. `init_token`, `eos_token`, `fix_length`, `lower`, but some cannot. How can I get the same functionality as `include_lengths` (necessary for packed padded sequences) for these? How can I change the default pad or unk tokens?

Is this all supposed to be done in the `collate_fn` of the `DataLoader` now?

It also creates a vocabulary but does not have a way to pass arguments such as `max_size` and `min_freq` to it - both of which I think are important. I don't think this can be done in the `collate_fn`.
> @zhangguanheng66 I think they look good so far - but do they not lose functionality offered by `data.Field` and `build_vocab`?
> 
> A lot of the functionality offered by the `Field` can be incorporated into the `tokenizer` argument, e.g. `init_token`, `eos_token`, `fix_length`, `lower`, but some cannot. How can I get the same functionality as `include_lengths` (necessary for packed padded sequences) for these? How can I change the default pad or unk tokens?
> 
> Is this all supposed to be done in the `collate_fn` of the `DataLoader` now?
> 
> It also creates a vocabulary but does not have a way to pass arguments such as `max_size` and `min_freq` to it - both of which I think are important. I don't think this can be done in the `collate_fn`.

Hi @bentrevett  You are right and we are still at the early stage to incorporate those functions and we need to figure out a better way to have these functionals. For example, you could write a padding function in `collate_fn`. 

Resolving patch:
diff --git a/torchtext/datasets/text_classification.py b/torchtext/datasets/text_classification.py
--- a/torchtext/datasets/text_classification.py
+++ b/torchtext/datasets/text_classification.py
@@ -113,7 +113,7 @@ def get_vocab(self):
         return self._vocab
 
 
-def _setup_datasets(dataset_name, root='.data', ngrams=2, vocab=None, include_unk=False):
+def _setup_datasets(dataset_name, root='.data', ngrams=1, vocab=None, include_unk=False):
     dataset_tar = download_from_url(URLS[dataset_name], root=root)
     extracted_files = extract_archive(dataset_tar)
 


Test patch: