Build_vocab_from_iterator does not work in notebook

AfonsoSalgadoSousa: <p>Hi. I was trying to run <a href="https://pytorch.org/tutorials/beginner/translation_transformer.html" rel="noopener nofollow ugc">this notebook</a>, but the following line times out:</p>
<pre><code class="lang-auto">vocab_transform[ln] = build_vocab_from_iterator(yield_tokens(train_iter, ln),
                                                    min_freq=1,
                                                    specials=special_symbols,
                                                    special_first=True)
</code></pre>
<p>Specifically, it raises a TimeoutError: [Errno 110] Connection timed out, and the last line of the trace is:<br>
Exception: Could not get the file at <a href="http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/training.tar.gz" rel="noopener nofollow ugc">http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/training.tar.gz</a>. [RequestException] None.</p>
<p>What can be done to circumvent this issue? Thanks in advance for any help you can provide.</p>

dongqi_cai: <p>Met the same question today.<br>
I try to directly access the file link. But it fails either, saying " The requested URL could not be retrieved".<br>
I guess the database server is down or their network is broken.</p>
<p>Hope it can recover soon… Or are there any alternatives?</p>

JensHeinrich: <p>Already tried <a href="http://archive.org" rel="noopener nofollow ugc">archive.org</a>, but sadly the files are unavailable there too</p>

Alex_Ambrioso: <p>I am having exactly the same problem.   I am new to NLP so I am not sure what to do.   Perhaps somebody knows another source of English/German (or another language) word pairs that we can use instead.  I will check back later.</p>

Alex_Ambrioso: <p>The maintainer of this GitHub file: <a href="https://github.com/PetrochukM/PyTorch-NLP/blob/master/torchnlp/datasets/multi30k.py" class="inline-onebox" rel="noopener nofollow ugc">PyTorch-NLP/multi30k.py at master · PetrochukM/PyTorch-NLP · GitHub</a></p>
<p>claims the following:</p>
<pre><code>Status:
    Host ``www.quest.dcs.shef.ac.uk`` forgot to update their SSL
    certificate; therefore, this dataset does not download securely.
References:
    * http://www.statmt.org/wmt16/multimodal-task.html
    * http://shannon.cs.illinois.edu/DenotationGraph/
</code></pre>
<p>He seems to have constructed a workaround program but I have not managed to get it to work.</p>

JensHeinrich: <p>the page is not delivering any content though</p>

ttssp: <p>Same problem… Tried to email the site owner but no feedbacks…</p>

pete1313: <p>I have those files but don’t know where to put them to make them available for everyone.</p>

ttssp: <p>is it possible to put them in the dropbox or google drive and share them using a public link?</p>

elliottd: <p>First author of the Multi30K dataset here <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=12" title=":wave:" class="emoji" alt=":wave:" loading="lazy" width="20" height="20">.</p>
<p>I didn’t know these were being used in a PyTorch tutorial so we are working on hosting these files elsewhere. Alternatively, if someone understands how the files are being used by <code>torchtext.datasets.Multi30K</code>, would one solution be to re-route the data loading to the <a href="https://github.com/multi30k/dataset" rel="noopener nofollow ugc">Multi30K Github repository</a>?</p>

ttssp: <p>Im a beginner but I found the source code of torchtext.datasets.multi30k <a href="https://pytorch.org/text/stable/_modules/torchtext/datasets/multi30k.html#Multi30k" rel="noopener nofollow ugc">here</a>. One may change the URLs and MD5s to make it work ~</p>

pete1313: <p>that is preciously what I was thinking but I do not own that repo so couldn’t do it.</p>

Alex_Ambrioso: <p>Please note that I have been working on the following code:</p>
<p><a href="http://nlp.seas.harvard.edu/annotated-transformer/" class="onebox" target="_blank" rel="noopener nofollow ugc">http://nlp.seas.harvard.edu/annotated-transformer/</a></p>
<p>This code uses the same Multi30K database.  I was able to get the code to work by using another data file.   The basic idea is that the training, validation, and test sets are all lists of tuples.    The tuples consist of sentence pairs in each language.    This insight is nice since it makes it easy to create any language pairing you would like.   Here is my implementation in Colab along with lots of notes:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/131hohvAKRqzHg4K3_68UGL4oi4SGOB45?usp=sharing">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20220621-060047-RC00_456220956" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/131hohvAKRqzHg4K3_68UGL4oi4SGOB45?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/131hohvAKRqzHg4K3_68UGL4oi4SGOB45?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Hope this helps.   Any comments are welcome.</p>

yanielc: <p>The slightly different way the dataset is downloaded <a href="https://torchtutorialstaging.z5.web.core.windows.net/beginner/translation_transformer.html" rel="noopener nofollow ugc">here</a> is working right now.</p>

Alex_Ambrioso: <p>Thank you for this link, Yaniel.    This is a very nice, compact, and up-to-date implementation of a transformer using Pytorch!</p>
<p>-Alex</p>

deng_wang: <p>I think I found the train, val , test file url , you can change the multi30k URL to update the origal invalid url:</p>
<pre><code class="lang-auto">from torchtext.datasets import multi30k, Multi30k

multi30k.URL["train"] = "https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/training.tar.gz"
multi30k.URL["valid"] = "https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/validation.tar.gz"
multi30k.URL["test"] = "https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/mmt16_task1_test.tar.gz"
</code></pre>
<p>if you do not want to update the URL of multi30k, you can just download the file from url above and put the tar.gz file to the torch cache directory. for my machine, the directory is :/root/.cache/torch/text/datasets/Multi30k , copy the tar.gz file into directory ,and run the code. pytorch will uncompress the file , get <a href="http://train.de/" rel="noopener nofollow ugc">train.de</a> train.en files</p>

Alberto_Ferreira_De: <p>I have changed the following lines in your code:</p>
<pre><code class="lang-auto">    print("Building German Vocabulary ...")
    train, val, test = datasets.Multi30k(language_pair=("de", "en"))
</code></pre>
<p>to</p>
<pre><code class="lang-auto">    print("Building German Vocabulary ...")
    from torchtext.datasets import multi30k
    multi30k.URL["train"] = "https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/training.tar.gz"
    multi30k.URL["valid"] = "https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/validation.tar.gz"
    multi30k.URL["test"] = "https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/mmt16_task1_test.tar.gz"

    multi30k.MD5["train"] = "20140d013d05dd9a72dfde46478663ba05737ce983f478f960c1123c6671be5e"
    multi30k.MD5["valid"] = "a7aa20e9ebd5ba5adce7909498b94410996040857154dab029851af3a866da8c"
    multi30k.MD5["test"] = "6d1ca1dba99e2c5dd54cae1226ff11c2551e6ce63527ebb072a1f70f72a5cd36"

    train, val, test = datasets.Multi30k(language_pair=("de", "en"))
</code></pre>
<p>and it worked.</p>