UnicodeDecodeError when running test iterator

Steboss89: <p>Hello everybody,</p>
<p>I am currently running  a Seq2Seq model, using the dataset <code>Multi30k</code> from <code>torchtext</code>. In particular, I structured the code in order to use <code>torch.utils.data.DataLoader</code> as follows:</p>
<pre><code class="lang-auto"># Define tokenizer
de_tokenizer = get_tokenizer('spacy', language='de_core_news_sm')
en_tokenizer = get_tokenizer('spacy', language='en_core_web_sm')
# Define special tokens
specials = ['&lt;unk&gt;', '&lt;pad&gt;', '&lt;bos&gt;', '&lt;eos&gt;']
# Load the Multi30k dataset
train_data, valid_data, test_data = Multi30k()
de_vocab = build_vocab_from_iterator(yield_tokens(train_data, de_tokenizer),
                                     specials=specials,
                                     min_freq=2)
en_vocab = build_vocab_from_iterator(yield_tokens(train_data, en_tokenizer),
                                     specials=specials,
                                     min_freq=2)
# Add the &lt;unk&gt; token to the vocabulary
de_vocab.set_default_index(de_vocab['&lt;unk&gt;'])
en_vocab.set_default_index(en_vocab['&lt;unk&gt;'])
# Get the index of the &lt;pad&gt; token
pad_idx = de_vocab['&lt;pad&gt;']


# Define the device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
train_iterator = DataLoader(train_data,
                            batch_size=BATCH_SIZE,
                            collate_fn=collate_fn)
valid_iterator = DataLoader(valid_data,
                            batch_size=BATCH_SIZE,
                            collate_fn=collate_fn)
test_iterator = DataLoader(test_data,
                           batch_size=BATCH_SIZE,
                           collate_fn=collate_fn)

hidden_size = 256
# Create model and define optimizer
encoder = Encoder(len(de_vocab), hidden_size).to(device)
decoder = Decoder(hidden_size, len(en_vocab)).to(device)
model = Seq2Seq(encoder, decoder, device)
optimizer = optim.Adam(model.parameters(), lr=0.001)


for epoch in range(EPOCHS):
    for batch in train_iterator:
        input_seq, target_seq = batch
        input_seq = input_seq.to(device)
        target_seq = target_seq.to(device)
        output_seq = model(input_seq, target_seq)

        # Compute loss and perform backpropagation
        loss = F.cross_entropy(output_seq.view(-1, len(en_vocab)), target_seq.view(-1))
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # Print some information during training
        print(f"Epoch: {epoch + 1}, Loss: {loss.item():.4f}")

    # Validation loop
    model.eval()
    with torch.no_grad():
        total_loss = 0
        num_items = 0
        for batch in valid_iterator:
            input_seq, target_seq = batch
            output_seq = model(input_seq, target_seq)
            val_loss = F.cross_entropy(output_seq.view(-1, len(en_vocab)), target_seq.view(-1))
            total_loss += val_loss.item()
            num_items += len(batch)

        average_val_loss = total_loss / num_items
        print(f"Epoch: {epoch + 1}, Validation Loss: {average_val_loss:.4f}")

# Testing loop
model.eval()
with torch.no_grad():
    total_test_loss = 0
    test_num_items = 0
    for batch in test_iterator:
        input_seq = batch.src
        target_seq = batch.trg
        print(input_seq)
        print(target_seq)
        output_seq = model(input_seq, target_seq)
        test_loss = F.cross_entropy(output_seq.view(-1, len(en_vocab)), target_seq.view(-1))
        total_test_loss += test_loss.item()
        test_num_items += len(batch)

    average_test_loss = total_test_loss / test_num_items
    print(f"Test Loss: {average_test_loss:.4f}")
</code></pre>
<p>Everything runs perfectly at training and validation time. However, when I am running the the test step I have tan error at <code>for batch in test_iterator</code>:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "mymodel.py", line 202, in &lt;module&gt;
    for batch in test_iterator:
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 674, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 154, in __next__
    return self._get_next()
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 142, in _get_next
    result = next(self.iterator)
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 226, in wrap_next
    result = next_func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/datapipes/datapipe.py", line 381, in __next__
    return next(self._datapipe_iter)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 183, in wrap_generator
    response = gen.send(None)
               ^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/sharding.py", line 75, in __iter__
    for i, item in enumerate(self.source_datapipe):
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 183, in wrap_generator
    response = gen.send(None)
               ^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/combinatorics.py", line 124, in __iter__
    yield from self.datapipe
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 183, in wrap_generator
    response = gen.send(None)
               ^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/combining.py", line 624, in __iter__
    yield from zip(*iterators)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 183, in wrap_generator
    response = gen.send(None)
               ^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torchdata/datapipes/iter/util/plain_text_reader.py", line 138, in __iter__
    yield from self._helper.return_path(stream, path=path)  # type: ignore[misc]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torchdata/datapipes/iter/util/plain_text_reader.py", line 69, in return_path
    yield from stream
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torchdata/datapipes/iter/util/plain_text_reader.py", line 62, in decode
    yield from stream
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torchdata/datapipes/iter/util/plain_text_reader.py", line 54, in strip_newline
    for line in stream:
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torchdata/datapipes/iter/util/plain_text_reader.py", line 45, in skip_lines
    yield from file
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torch/utils/data/datapipes/utils/common.py", line 369, in __iter__
    yield from self.file_obj
  File "&lt;frozen codecs&gt;", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 37: invalid start byte
</code></pre>
<p>I am using <code>torch</code> version 2.1.1 and <code>torchtext</code> version 0.16.1.<br>
I tried to understand where the error is from, applyign some modifications and <code>try...except</code> to my helper functions, but no luck yet. These are the function I am using for yielding tokens and collate:</p>
<pre><code class="lang-auto">def yield_tokens(data_iter: Iterable, tokenizer) -&gt; List[str]:
    for _, text in data_iter:
        try:
            yield tokenizer(text)
        except UnicodeDecodeError:
            yield ""  # or some other default value


def collate_fn(batch):
    de_batch, en_batch = [], []
    for (de_item, en_item) in batch:
        de_batch.append(torch.tensor([de_vocab[token] for token in de_tokenizer(de_item)], dtype=torch.long))
        en_batch.append(torch.tensor([en_vocab[token] for token in en_tokenizer(en_item)], dtype=torch.long))
    de_batch = pad_sequence(de_batch, padding_value=pad_idx)
    en_batch = pad_sequence(en_batch, padding_value=pad_idx)
    return de_batch, en_batch
</code></pre>
<p>Thanks very much for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ChristLBUPT: <p>It seems that your dataset contains bytes that UTF-8 cannot read.<br>
Are you able to inspect the detailed file contents?</p>
<pre><code class="lang-auto"> File "/opt/homebrew/Caskroom/miniconda/base/envs/pytorch/lib/python3.11/site-packages/torchdata/datapipes/iter/util/plain_text_reader.py", line 138, in __iter__
    yield from self._helper.return_path(stream, path=path)
</code></pre>
<p>You can navigate to the above source file and set a <code>breakpoint()</code> before that statement and check the file path</p>

Dmitriy_Subbotin: <p>It seems ‘test’ dataset from Multi30k was corrupted.</p>
<p>Multi30k uses the following URLs to retrive the data:</p>
<pre><code class="lang-auto"># TODO: Update URL to original once the server is back up (see https://github.com/pytorch/text/issues/1756)
URL = {
    "train": r"https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/training.tar.gz",
    "valid": r"https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/validation.tar.gz",
    "test": r"https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/mmt16_task1_test.tar.gz",
}
</code></pre>
<p>Now, if we look at the test dataset <em>mmt16_task1_test.tar.gz</em>, in addition to the expected <em><a href="http://test.de" rel="noopener nofollow ugc">test.de</a></em>, <em>test.en</em>, and <em><a href="http://test.fr" rel="noopener nofollow ugc">test.fr</a></em> the dataset strangely contains extra 3 files <em>._test.de</em>, <em>._test.en</em>, and <em>._test.fr</em> files:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/5/156b93e37cb3f014b4f8f4ac7fb30939219f15d4.png" alt="image" data-base62-sha1="33uvGlGEMLZPpqTcpPnNvl66MSM" width="589" height="193"></p>
<p>The extra <em>._test.de</em>, <em>._test.en</em>, and <em>._test.fr</em> files seem to be Apple metadata, however Multi30k code picks them up instead of the desired files because the <em>_filter_fn</em> function just searches for the filename <em>in</em> the file:</p>
<pre><code class="lang-auto">def _filter_fn(split, language_pair, i, x):
    return f"{_PREFIX[split]}.{language_pair[i]}" in x[0]
</code></pre>
<p>There are 2 simple fixes:</p>
<p>Fix option 1) Find <em>mmt16_task1_test.tar.gz</em> in pytorch cache folder (by default ~/.data/datasets/Multi30k) and manually unzip the correct files.</p>
<p>Fix option 2) Patch <em>multi30k._filter_fn</em> function to find correct file:</p>
<pre><code class="lang-auto">def _filter_fn(split, language_pair, i, x):
    return f"/{torchtext.datasets.multi30k._PREFIX[split]}.{language_pair[i]}" in x[0]
torchtext.datasets.multi30k._filter_fn = _filter_fn
</code></pre>
<p>Someone might want to raise this to the data owner to fix the original file.</p>

Dmitriy_Subbotin: <p>I’ve found out there is already a relevant discussion here:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/text/issues/1756">
  <header class="source">

      <a href="https://github.com/pytorch/text/issues/1756" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/text</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/text/issues/1756" target="_blank" rel="noopener nofollow ugc">Multi30K dataset link is broken</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-06-01" data-time="21:26:36" data-timezone="UTC">09:26PM - 01 Jun 22 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/xuzhao9" target="_blank" rel="noopener nofollow ugc">
          <img alt="xuzhao9" src="https://avatars.githubusercontent.com/u/502017?v=4" class="onebox-avatar-inline" width="20" height="20">
          xuzhao9
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">The link to Multi30K dataset at `http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">/training.tar.gz` is broken: https://github.com/pytorch/text/blob/73bf4fa8cedc12d910ab76190e446bd2e47a8325/torchtext/datasets/multi30k.py#L16</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>