Repository: pytorch/text

Issue:
Using a field representing real numbers with the iterator
I am trying to learn a regressor on text data and I use torchtext in all my other tasks but I see a problem in using it for this use case.

I define the field for targets as follows:
```
TARGETS = data.Field(
            sequential=False, tensor_type=torch.DoubleTensor, batch_first=True)
self.fields = [('targets', TARGETS), ('text', TEXT)]
self.train, self.val, self.test = data.TabularDataset.splits(
            path=self.path,
            train=self.train_suffix,
            validation=self.val_suffix,
            test=self.test_suffix,
            format=formatting,
            fields=self.fields)
TEXT.build_vocab(self.train)
```

I have a file that contains tab separate <values>\t<text>

When I make iterators out of it, 
```
train_iter, val_iter, test_iter = data.Iterator.splits(
                (self.train, self.val, self.test),
                batch_sizes=(self.batch_size, self.test_batch_size,
                             self.test_batch_size),
                sort_key=lambda x: len(x.text),
                shuffle=True)
print(next(iter(train_iter)))
```
it gives me an error when getting the next batch:

> AttributeError: 'Field' object has no attribute 'vocab'

I know this is because I didn't run .build_vocab for the TARGETS field. But why do I really need to do this? What if I just want to get real numbers and compute losses on them? 

Any workaround is appreciated. If I am doing something wrong, please let me know too.


Discussion:
Found use_vocab argument 😞 
Even after setting use_vocab=False. I get 

> RuntimeError: already counted a million dimensions in a given sequence. Most likely your items are also sequences and there's no way to infer how many dimension should the tensor have

It is the same error that one gets when you try to do torch.DoubleTensor('1.2'). Is there something I am doing wrong?
Thanks for the issue.

Torchtext needs to convert the string number to an `int` or `float` somewhere down the line and it currently doesn't do this. A quick fix would be to manually add a pipeline to the `postprocessing` argument that converts everything in the `TARGETS` field to int. With a slightly modified version of your code:

Edit: just noticed that your example uses doubles. changed my code accordingly

(tab separated file)
```
$ cat test.txt
1.1   test string
1.2   test string2
1.3   test string3
```

The following works on my machine in the meantime while we patch this:

```python
In [1]: import torch

In [2]: from torchtext import data

In [3]: TEXT = data.Field(batch_first=True)

In [4]: TARGETS = data.Field(sequential=False, tensor_type=torch.DoubleTensor, batch_first=True, use_vocab=False, postprocessing=data.Pipeline(lambda x: float(x)))

In [5]: fields = [('targets', TARGETS), ('text', TEXT)]

In [6]: dataset = data.TabularDataset(path="test.txt", format="tsv", fields=fields)

In [7]: TEXT.build_vocab(dataset)

In [8]: train_iter = data.Iterator(dataset, batch_size=1, sort_key=lambda x: len(x.text), shuffle=True)

In [9]: batch = next(iter(train_iter))

In [10]: batch.targets
Out[10]: 
Variable containing:
 1.3000
[torch.cuda.DoubleTensor of size 1 (GPU 0)]
```

Hope that helps.
Thanks for the solution @nelson-liu 
could you leave this open for now --- there is a bug behind this that would be nice to track (the fact that we do not actually convert values with `use_vocab=False` to numbers). Thanks!
Sure, I agree. 
Yeah, I was originally imagining that values would be provided as Python numerical types -- but that isn't really consistent with the nature of the library as loading mostly text values. Certainly if it sees strings it should convert them!

Resolving patch:
diff --git a/torchtext/data/field.py b/torchtext/data/field.py
--- a/torchtext/data/field.py
+++ b/torchtext/data/field.py
@@ -57,6 +57,28 @@ class Field(object):
         pad_token: The string token used as padding. Default: "<pad>".
     """
 
+    # Dictionary mapping PyTorch tensor types to the appropriate Python
+    # numeric type.
+    tensor_types = {
+        torch.FloatTensor: float,
+        torch.cuda.FloatTensor: float,
+        torch.DoubleTensor: float,
+        torch.cuda.DoubleTensor: float,
+        torch.HalfTensor: float,
+        torch.cuda.HalfTensor: float,
+
+        torch.ByteTensor: int,
+        torch.cuda.ByteTensor: int,
+        torch.CharTensor: int,
+        torch.cuda.CharTensor: int,
+        torch.ShortTensor: int,
+        torch.cuda.ShortTensor: int,
+        torch.IntTensor: int,
+        torch.cuda.IntTensor: int,
+        torch.LongTensor: int,
+        torch.cuda.LongTensor: int
+    }
+
     def __init__(
             self, sequential=True, use_vocab=True, init_token=None,
             eos_token=None, fix_length=None, tensor_type=torch.LongTensor,
@@ -103,7 +125,8 @@ def pad(self, minibatch):
         the longest example in the batch. Prepends self.init_token and appends
         self.eos_token if those attributes are not None. Returns a tuple of the
         padded list and a list containing lengths of each example if
-        `self.include_lengths` is `True`, else just returns the padded list.
+        `self.include_lengths` is `True` and `self.sequential` is `True`, else just
+        returns the padded list. If `self.sequential` is `False`, no padding is applied.
         """
         minibatch = list(minibatch)
         if not self.sequential:
@@ -162,16 +185,25 @@ def numericalize(self, arr, device=None, train=True):
         included in the return value.
 
         Arguments:
-            arr: List of tokenized and padded examples, or tuple of a padded
-                list and a list of lengths if self.include_lengths is True.
-            device: Device to create the Variable's Tensor on. Use -1 for
-                CPU and None for the currently active GPU device. Default:
-                None.
-            train: Whether the batch is for a training set. If False, the
-                Variable will be created with volatile=True. Default: True.
+            arr (List[List[str]], or tuple of (List[List[str]], List[int])):
+                List of tokenized and padded examples, or tuple of List of
+                tokenized and padded examples and List of lengths of each
+                example if self.include_lengths is True.
+            device (-1 or None): Device to create the Variable's Tensor on.
+                Use -1 for CPU and None for the currently active GPU device.
+                Default: None.
+            train (boolean): Whether the batch is for a training set.
+                If False, the Variable will be created with volatile=True.
+                Default: True.
         """
+        if self.include_lengths and not isinstance(arr, tuple):
+            raise ValueError("Field has include_lengths set to True, but "
+                             "input data is not a tuple of "
+                             "(data batch, batch lengths).")
         if isinstance(arr, tuple):
             arr, lengths = arr
+            lengths = torch.LongTensor(lengths)
+
         if self.use_vocab:
             if self.sequential:
                 arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
@@ -180,11 +212,23 @@ def numericalize(self, arr, device=None, train=True):
 
             if self.postprocessing is not None:
                 arr = self.postprocessing(arr, self.vocab, train)
-        elif self.postprocessing is not None:
-            arr = self.postprocessing(arr, train)
+        else:
+            if self.tensor_type not in self.tensor_types:
+                raise ValueError(
+                    "Specified Field tensor_type {} can not be used with "
+                    "use_vocab=False because we do not know how to numericalize it. "
+                    "Please raise an issue at "
+                    "https://github.com/pytorch/text/issues".format(self.tensor_type))
+            numericalization_func = self.tensor_types[self.tensor_type]
+            # It doesn't make sense to explictly coerce to a numeric type if
+            # the data is sequential, since it's unclear how to coerce padding tokens
+            # to a numeric type.
+            if not self.sequential:
+                arr = [numericalization_func(x) for x in arr]
+            if self.postprocessing is not None:
+                arr = self.postprocessing(arr, train)
+
         arr = self.tensor_type(arr)
-        if self.include_lengths:
-            lengths = torch.LongTensor(lengths)
         if self.sequential and not self.batch_first:
             arr.t_()
         if device == -1:


Test patch:
diff --git a/test/common/torchtext_test_case.py b/test/common/torchtext_test_case.py
--- a/test/common/torchtext_test_case.py
+++ b/test/common/torchtext_test_case.py
@@ -20,6 +20,8 @@ def setUp(self):
             os.path.dirname(os.path.realpath(__file__)), os.pardir, os.pardir)))
         self.test_dir = tempfile.mkdtemp()
         self.test_ppid_dataset_path = os.path.join(self.test_dir, "test_ppid_dataset")
+        self.test_numerical_features_dataset_path = os.path.join(
+            self.test_dir, "test_numerical_features_dataset")
 
     def tearDown(self):
         try:
@@ -54,3 +56,46 @@ def write_test_ppid_dataset(self, data_format="csv"):
                                     example["question2"], example["label"]])))
                 else:
                     raise ValueError("Invalid format {}".format(data_format))
+
+    def write_test_numerical_features_dataset(self):
+        with open(self.test_numerical_features_dataset_path,
+                  "w") as test_numerical_features_dataset_file:
+            test_numerical_features_dataset_file.write("0.1\t1\tteststring1\n")
+            test_numerical_features_dataset_file.write("0.5\t12\tteststring2\n")
+            test_numerical_features_dataset_file.write("0.2\t0\tteststring3\n")
+            test_numerical_features_dataset_file.write("0.4\t12\tteststring4\n")
+            test_numerical_features_dataset_file.write("0.9\t9\tteststring5\n")
+
+
+def verify_numericalized_example(field, test_example_data,
+                                 test_example_numericalized,
+                                 test_example_lengths=None,
+                                 batch_first=False, train=True):
+    """
+    Function to verify that numericalized example is correct
+    with respect to the Field's Vocab.
+    """
+    if isinstance(test_example_numericalized, tuple):
+        test_example_numericalized, lengths = test_example_numericalized
+        assert test_example_lengths == lengths.tolist()
+    if batch_first:
+        test_example_numericalized.data.t_()
+    # Transpose numericalized example so we can compare over batches
+    for example_idx, numericalized_single_example in enumerate(
+            test_example_numericalized.t()):
+        assert len(test_example_data[example_idx]) == len(numericalized_single_example)
+        assert numericalized_single_example.volatile is not train
+        for token_idx, numericalized_token in enumerate(
+                numericalized_single_example):
+            # Convert from Variable to int
+            numericalized_token = numericalized_token.data[0]
+            test_example_token = test_example_data[example_idx][token_idx]
+            # Check if the numericalized example is correct, taking into
+            # account unknown tokens.
+            if field.vocab.stoi[test_example_token] != 0:
+                # token is in-vocabulary
+                assert (field.vocab.itos[numericalized_token] ==
+                        test_example_token)
+            else:
+                # token is OOV and <unk> always has an index of 0
+                assert numericalized_token == 0
diff --git a/test/data/test_field.py b/test/data/test_field.py
--- a/test/data/test_field.py
+++ b/test/data/test_field.py
@@ -1,6 +1,12 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+from collections import Counter
+
+from numpy.testing import assert_allclose
+import torch
 import torchtext.data as data
 
-from ..common.torchtext_test_case import TorchtextTestCase
+from ..common.torchtext_test_case import TorchtextTestCase, verify_numericalized_example
 
 
 class TestField(TorchtextTestCase):
@@ -27,7 +33,7 @@ def test_preprocess(self):
         # Non-regression test that we do not try to decode unicode strings to unicode
         field_not_sequential = data.Field(sequential=False, lower=True,
                                           preprocessing=preprocess_pipeline)
-        assert field_not_sequential.preprocess(u"Test string.") == "test string.!"
+        assert field_not_sequential.preprocess("ᑌᑎIᑕOᗪᕮ_Tᕮ᙭T") == "ᑌᑎiᑕoᗪᕮ_tᕮ᙭t!"
 
     def test_pad(self):
         # Default case.
@@ -92,3 +98,272 @@ def test_pad(self):
         field = data.Field(init_token="<bos>", eos_token="<eos>",
                            sequential=False, include_lengths=True)
         assert field.pad(minibatch) == minibatch
+
+    def test_build_vocab(self):
+        # Set up fields
+        question_field = data.Field(sequential=True)
+        label_field = data.Field(sequential=False)
+
+        # Write TSV dataset and construct a Dataset
+        self.write_test_ppid_dataset(data_format="tsv")
+        tsv_fields = [("id", None), ("q1", question_field),
+                      ("q2", question_field), ("label", label_field)]
+        tsv_dataset = data.TabularDataset(
+            path=self.test_ppid_dataset_path, format="tsv",
+            fields=tsv_fields)
+
+        # Write JSON dataset and construct a Dataset
+        self.write_test_ppid_dataset(data_format="json")
+        json_fields = {"question1": ("q1", question_field),
+                       "question2": ("q2", question_field),
+                       "label": ("label", label_field)}
+        json_dataset = data.TabularDataset(
+            path=self.test_ppid_dataset_path, format="json",
+            fields=json_fields)
+
+        # Test build_vocab default
+        question_field.build_vocab(tsv_dataset, json_dataset)
+        assert question_field.vocab.freqs == Counter(
+            {'When': 4, 'do': 4, 'you': 4, 'use': 4, 'instead': 4,
+             'of': 4, 'was': 4, 'Lincoln': 4, 'born?': 4, 'シ': 2,
+             'し?': 2, 'Where': 2, 'What': 2, 'is': 2, '2+2': 2,
+             '"&"': 2, '"and"?': 2, 'Which': 2, 'location': 2,
+             'Abraham': 2, '2+2=?': 2})
+        expected_stoi = {'<unk>': 0, '<pad>': 1, 'Lincoln': 2, 'When': 3,
+                         'born?': 4, 'do': 5, 'instead': 6, 'of': 7,
+                         'use': 8, 'was': 9, 'you': 10, '"&"': 11,
+                         '"and"?': 12, '2+2': 13, '2+2=?': 14, 'Abraham': 15,
+                         'What': 16, 'Where': 17, 'Which': 18, 'is': 19,
+                         'location': 20, 'し?': 21, 'シ': 22}
+        assert dict(question_field.vocab.stoi) == expected_stoi
+        # Turn the stoi dictionary into an itos list
+        expected_itos = [x[0] for x in sorted(expected_stoi.items(),
+                                              key=lambda tup: tup[1])]
+        assert question_field.vocab.itos == expected_itos
+
+        label_field.build_vocab(tsv_dataset, json_dataset)
+        assert label_field.vocab.freqs == Counter({'1': 4, '0': 2})
+        expected_stoi = {'1': 1, '0': 2, '<unk>': 0}
+        assert dict(label_field.vocab.stoi) == expected_stoi
+        # Turn the stoi dictionary into an itos list
+        expected_itos = [x[0] for x in sorted(expected_stoi.items(),
+                                              key=lambda tup: tup[1])]
+        assert label_field.vocab.itos == expected_itos
+
+        # Test build_vocab default
+        question_field.build_vocab(tsv_dataset, json_dataset)
+        assert question_field.vocab.freqs == Counter(
+            {'When': 4, 'do': 4, 'you': 4, 'use': 4, 'instead': 4,
+             'of': 4, 'was': 4, 'Lincoln': 4, 'born?': 4, 'シ': 2,
+             'し?': 2, 'Where': 2, 'What': 2, 'is': 2, '2+2': 2,
+             '"&"': 2, '"and"?': 2, 'Which': 2, 'location': 2,
+             'Abraham': 2, '2+2=?': 2})
+        expected_stoi = {'<unk>': 0, '<pad>': 1, 'Lincoln': 2, 'When': 3,
+                         'born?': 4, 'do': 5, 'instead': 6, 'of': 7,
+                         'use': 8, 'was': 9, 'you': 10, '"&"': 11,
+                         '"and"?': 12, '2+2': 13, '2+2=?': 14, 'Abraham': 15,
+                         'What': 16, 'Where': 17, 'Which': 18, 'is': 19,
+                         'location': 20, 'し?': 21, 'シ': 22}
+        assert dict(question_field.vocab.stoi) == expected_stoi
+        # Turn the stoi dictionary into an itos list
+        expected_itos = [x[0] for x in sorted(expected_stoi.items(),
+                                              key=lambda tup: tup[1])]
+        assert question_field.vocab.itos == expected_itos
+
+        label_field.build_vocab(tsv_dataset, json_dataset)
+        assert label_field.vocab.freqs == Counter({'1': 4, '0': 2})
+        expected_stoi = {'1': 1, '0': 2, '<unk>': 0}
+        assert dict(label_field.vocab.stoi) == expected_stoi
+        # Turn the stoi dictionary into an itos list
+        expected_itos = [x[0] for x in sorted(expected_stoi.items(),
+                                              key=lambda tup: tup[1])]
+        assert label_field.vocab.itos == expected_itos
+
+        # Test build_vocab with extra kwargs passed to Vocab
+        question_field.build_vocab(tsv_dataset, json_dataset, max_size=8,
+                                   min_freq=3)
+        assert question_field.vocab.freqs == Counter(
+            {'When': 4, 'do': 4, 'you': 4, 'use': 4, 'instead': 4,
+             'of': 4, 'was': 4, 'Lincoln': 4, 'born?': 4, 'シ': 2,
+             'し?': 2, 'Where': 2, 'What': 2, 'is': 2, '2+2': 2,
+             '"&"': 2, '"and"?': 2, 'Which': 2, 'location': 2,
+             'Abraham': 2, '2+2=?': 2})
+        expected_stoi = {'<unk>': 0, '<pad>': 1, 'Lincoln': 2, 'When': 3,
+                         'born?': 4, 'do': 5, 'instead': 6, 'of': 7,
+                         'use': 8, 'was': 9}
+        assert dict(question_field.vocab.stoi) == expected_stoi
+        # Turn the stoi dictionary into an itos list
+        expected_itos = [x[0] for x in sorted(expected_stoi.items(),
+                                              key=lambda tup: tup[1])]
+        assert question_field.vocab.itos == expected_itos
+
+    def test_numericalize_basic(self):
+        self.write_test_ppid_dataset(data_format="tsv")
+        question_field = data.Field(sequential=True)
+        tsv_fields = [("id", None), ("q1", question_field),
+                      ("q2", question_field), ("label", None)]
+        tsv_dataset = data.TabularDataset(
+            path=self.test_ppid_dataset_path, format="tsv",
+            fields=tsv_fields)
+        question_field.build_vocab(tsv_dataset)
+
+        test_example_data = [["When", "do", "you", "use", "シ",
+                              "instead", "of", "し?"],
+                             ["What", "is", "2+2", "<pad>", "<pad>",
+                              "<pad>", "<pad>", "<pad>"],
+                             ["Here", "is", "a", "sentence", "with",
+                              "some", "oovs", "<pad>"]]
+
+        # Test default
+        default_numericalized = question_field.numericalize(
+            test_example_data, device=-1)
+        verify_numericalized_example(question_field, test_example_data,
+                                     default_numericalized)
+        # Test with train=False
+        volatile_numericalized = question_field.numericalize(
+            test_example_data, device=-1, train=False)
+        verify_numericalized_example(question_field, test_example_data,
+                                     volatile_numericalized, train=False)
+
+    def test_numericalize_include_lengths(self):
+        self.write_test_ppid_dataset(data_format="tsv")
+        question_field = data.Field(sequential=True, include_lengths=True)
+        tsv_fields = [("id", None), ("q1", question_field),
+                      ("q2", question_field), ("label", None)]
+        tsv_dataset = data.TabularDataset(
+            path=self.test_ppid_dataset_path, format="tsv",
+            fields=tsv_fields)
+        question_field.build_vocab(tsv_dataset)
+
+        test_example_data = [["When", "do", "you", "use", "シ",
+                              "instead", "of", "し?"],
+                             ["What", "is", "2+2", "<pad>", "<pad>",
+                              "<pad>", "<pad>", "<pad>"],
+                             ["Here", "is", "a", "sentence", "with",
+                              "some", "oovs", "<pad>"]]
+        test_example_lengths = [8, 3, 7]
+
+        # Test with include_lengths
+        include_lengths_numericalized = question_field.numericalize(
+            (test_example_data, test_example_lengths), device=-1)
+        verify_numericalized_example(question_field,
+                                     test_example_data,
+                                     include_lengths_numericalized,
+                                     test_example_lengths)
+
+    def test_numericalize_batch_first(self):
+        self.write_test_ppid_dataset(data_format="tsv")
+        question_field = data.Field(sequential=True, batch_first=True)
+        tsv_fields = [("id", None), ("q1", question_field),
+                      ("q2", question_field), ("label", None)]
+        tsv_dataset = data.TabularDataset(
+            path=self.test_ppid_dataset_path, format="tsv",
+            fields=tsv_fields)
+        question_field.build_vocab(tsv_dataset)
+
+        test_example_data = [["When", "do", "you", "use", "シ",
+                              "instead", "of", "し?"],
+                             ["What", "is", "2+2", "<pad>", "<pad>",
+                              "<pad>", "<pad>", "<pad>"],
+                             ["Here", "is", "a", "sentence", "with",
+                              "some", "oovs", "<pad>"]]
+
+        # Test with batch_first
+        include_lengths_numericalized = question_field.numericalize(
+            test_example_data, device=-1)
+        verify_numericalized_example(question_field,
+                                     test_example_data,
+                                     include_lengths_numericalized,
+                                     batch_first=True)
+
+    def test_numericalize_postprocessing(self):
+        self.write_test_ppid_dataset(data_format="tsv")
+
+        def reverse_postprocess(arr, vocab, train):
+            return [list(reversed(sentence)) for sentence in arr]
+
+        question_field = data.Field(sequential=True,
+                                    postprocessing=reverse_postprocess)
+        tsv_fields = [("id", None), ("q1", question_field),
+                      ("q2", question_field), ("label", None)]
+
+        tsv_dataset = data.TabularDataset(
+            path=self.test_ppid_dataset_path, format="tsv",
+            fields=tsv_fields)
+        question_field.build_vocab(tsv_dataset)
+
+        test_example_data = [["When", "do", "you", "use", "シ",
+                              "instead", "of", "し?"],
+                             ["What", "is", "2+2", "<pad>", "<pad>",
+                              "<pad>", "<pad>", "<pad>"],
+                             ["Here", "is", "a", "sentence", "with",
+                              "some", "oovs", "<pad>"]]
+        reversed_test_example_data = [list(reversed(sentence)) for sentence in
+                                      test_example_data]
+
+        postprocessed_numericalized = question_field.numericalize(
+            (test_example_data), device=-1)
+        verify_numericalized_example(question_field,
+                                     reversed_test_example_data,
+                                     postprocessed_numericalized)
+
+    def test_numerical_features_no_vocab(self):
+        self.write_test_numerical_features_dataset()
+        # Test basic usage
+        int_field = data.Field(sequential=False, use_vocab=False)
+        float_field = data.Field(sequential=False, use_vocab=False,
+                                 tensor_type=torch.FloatTensor)
+        tsv_fields = [("int", int_field), ("float", float_field), ("string", None)]
+        tsv_dataset = data.TabularDataset(
+            path=self.test_numerical_features_dataset_path, format="tsv",
+            fields=tsv_fields)
+        int_field.build_vocab(tsv_dataset)
+        float_field.build_vocab(tsv_dataset)
+        test_int_data = ["1", "0", "1", "3", "19"]
+        test_float_data = ["1.1", "0.1", "3.91", "0.2", "10.2"]
+
+        numericalized_int = int_field.numericalize(test_int_data, device=-1)
+        assert_allclose(numericalized_int.data.numpy(), [1, 0, 1, 3, 19])
+        numericalized_float = float_field.numericalize(test_float_data, device=-1)
+        assert_allclose(numericalized_float.data.numpy(), [1.1, 0.1, 3.91, 0.2, 10.2])
+
+        # Test with postprocessing applied
+        int_field = data.Field(sequential=False, use_vocab=False,
+                               postprocessing=lambda arr, _: [x + 1 for x in arr])
+        float_field = data.Field(sequential=False, use_vocab=False,
+                                 tensor_type=torch.FloatTensor,
+                                 postprocessing=lambda arr, _: [x * 0.5 for x in arr])
+        tsv_fields = [("int", int_field), ("float", float_field), ("string", None)]
+        tsv_dataset = data.TabularDataset(
+            path=self.test_numerical_features_dataset_path, format="tsv",
+            fields=tsv_fields)
+        int_field.build_vocab(tsv_dataset)
+        float_field.build_vocab(tsv_dataset)
+        test_int_data = ["1", "0", "1", "3", "19"]
+        test_float_data = ["1.1", "0.1", "3.91", "0.2", "10.2"]
+
+        numericalized_int = int_field.numericalize(test_int_data, device=-1)
+        assert_allclose(numericalized_int.data.numpy(), [2, 1, 2, 4, 20])
+        numericalized_float = float_field.numericalize(test_float_data, device=-1)
+        assert_allclose(numericalized_float.data.numpy(), [0.55, 0.05, 1.955, 0.1, 5.1])
+
+    def test_errors(self):
+        # Test that passing a non-tuple (of data and length) to numericalize
+        # with Field.include_lengths = True raises an error.
+        with self.assertRaises(ValueError):
+            self.write_test_ppid_dataset(data_format="tsv")
+            question_field = data.Field(sequential=True, include_lengths=True)
+            tsv_fields = [("id", None), ("q1", question_field),
+                          ("q2", question_field), ("label", None)]
+            tsv_dataset = data.TabularDataset(
+                path=self.test_ppid_dataset_path, format="tsv",
+                fields=tsv_fields)
+            question_field.build_vocab(tsv_dataset)
+            test_example_data = [["When", "do", "you", "use", "シ",
+                                  "instead", "of", "し?"],
+                                 ["What", "is", "2+2", "<pad>", "<pad>",
+                                  "<pad>", "<pad>", "<pad>"],
+                                 ["Here", "is", "a", "sentence", "with",
+                                  "some", "oovs", "<pad>"]]
+            question_field.numericalize(
+                test_example_data, device=-1)