Torchtext dataset: don't load the whole file into memory

username: ecolss
cooked: <p>Hi guys, I’m super new to <code>torchtext</code> lib and currently learning.<br>
I’ve read some tutorials of <a href="https://torchtext.readthedocs.io/en/latest/data.html#tabulardataset" rel="noopener nofollow ugc"><code>torchtext.data</code></a>, but still not sure what should I do for some use-cases.</p>
<p>For example, an ordinary task is to train a <code>word2vec</code> model in NLP, and I’m not sure what’s the correct and most efficient way to prepare/load the training data for this model training task?</p>
<p>I tried the most naive approch, i.e. write an adhoc python script to pre-process the corpus into a tsv format, for example like this,</p>
<pre><code class="lang-auto">anarchism	originated	1
anarchism	as	1
anarchism	a	1
anarchism	term	1
anarchism	of	1
anarchism	race	0
anarchism	one	0
anarchism	from	0
anarchism	details	0
anarchism	hereditary	0
</code></pre>
<p>the 1st column is target word, 2nd is context word, 3nd is label (0 means negative sampling).<br>
Then, tried to load this preprocessed corpus via <code>torchtext.data.TabularDataset</code>.</p>
<p>But this cannot work, since it took so much memory and time, and never finished.<br>
Please provide some comments/answers, what is the right way to do this?</p>

username: Abhilash_Srivastava
cooked: <p>Two things:</p>
<ol>
<li>If you plan to use <code>torchtext.data.TabularDataset</code>, your data should be able to fit in the memory. Only then can torchtext read and create batches of data.</li>
<li>If the above is not possible, you can use lazy loading to create your custom batches (use <code>yield</code> like used in Python generators).</li>
</ol>

username: ecolss
cooked: <p>Thanks <a class="mention" href="/u/abhilash_srivastava">@Abhilash_Srivastava</a>!<br>
I think it’s the second situation for me at the moment, but the follow-up question is, if I use lazy loading, I’m afraid that some <code>torchtext</code> functionalities cannot be used, such as <code>Field.build_vocab</code>, right?<br>
If so, I don’t need to use torchtext at the first place then.</p>

username: BramVanroy
cooked: <p>You can have a look at the “datasets” library, which is highly optimized.</p>

username: Abhilash_Srivastava
cooked: <p>That’s right. With the second option, <code>torchtext</code> is out of the picture. Hence, you’ll need to use create your vocab, along with <code>stoi</code> and <code>itos</code> dictionaries.</p>

username: ecolss
cooked: <p>Sorry, you mean which one? Could you please drop a link?<br>
I thought you were talking about the <code>torchvision.datasets</code>, but doesn’t feel this is correct here.</p>

username: BramVanroy
cooked: <p>Ths one. It uses the Apache Arrow format which allows incredible speed with a low memory footprint.</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/huggingface/datasets" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/172;"><img src="https://repository-images.githubusercontent.com/250213286/617cf300-f385-11ea-8b5c-cb17766db38e" class="thumbnail" width="690" height="172"></div>

<h3><a href="https://github.com/huggingface/datasets" target="_blank" rel="noopener nofollow ugc">huggingface/datasets</a></h3>

<p>🤗 Fast, efficient, open-access datasets and evaluation metrics for Natural Language Processing and more in PyTorch, TensorFlow, NumPy and Pandas - huggingface/datasets</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: ecolss
cooked: <p>Thanks, surely will try it!</p>