How do I use torchtext with contextual word embeddings?

username: dm23
cooked: <p>Hi, I am trying to use torchtext while using pre-trained elmo as word embeddings. My attempt is as follows :</p>
<p>I have a csv, where a column contains the strings of the training data.</p>
<p>The pretrained elmo provides me a simple function elmo() which when called with a sentence as input, automatically tokenizes it and returns a list of torch Tensors, where number of elements in list is same as number of words in sentence.</p>
<p>So, I wrote a simple tokenizer which is :<br>
def myTokenizer(sent):<br>
return elmo(sent)</p>
<p>I defined the field for the column as :<br>
elmo_field = Field( sequential=True, use_vocab=False, batch_first=True, pad_token = torch.zeros(DIMS,dtype=torch.float64),  dtype=torch.float64, tokenize=myTokenizer, fix_length=100<br>
)</p>
<p>But when I iterate over this using BucketIterator, I get this error :</p>
<pre><code>for series,_ in valid_loader:
</code></pre>
<p>File “xxx/lib/python3.7/site-packages/torchtext/data/iterator.py”, line 156, in <strong>iter</strong><br>
yield Batch(minibatch, self.dataset, self.device)<br>
File “xxx/lib/python3.7/site-packages/torchtext/data/batch.py”, line 34, in <strong>init</strong><br>
setattr(self, name, field.process(batch, device=device))<br>
File “xxx/lib/python3.7/site-packages/torchtext/data/field.py”, line 237, in process<br>
tensor = self.numericalize(padded, device=device)<br>
File “xxx/lib/python3.7/site-packages/torchtext/data/field.py”, line 359, in numericalize<br>
var = torch.tensor(arr, dtype=self.dtype, device=device)<br>
ValueError: only one element tensors can be converted to Python scalars</p>
<p>I have no clue what might be going wrong, I tried writing custom field where in numericalize function, I simply return the input argument, that gives issues in later parts of the training code.</p>
<p>Any help or pointers would be really appreciated, spent a lot of time but couldn’t figure anything out.</p>
<p>If any more information is required, please let me know.<br>
Versions -<br>
torch                    1.4.0<br>
torchtext                0.6.0</p>

username: ptrblck
cooked: <p>I’m not sure where your code is exactly failing, but the error message is raised if you try to convert a tensor with more than a single element to a Python scalar value:</p>
<pre><code class="lang-python">torch.randn(2).item()
&gt; ValueError: only one element tensors can be converted to Python scalars

torch.randn(1).item() # works
</code></pre>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>

username: dm23
cooked: <p>That example helps. I will try looking in that direction.</p>
<p>A related question, if I iterate over the data multiple times (like when I have multiple epochs), will the myTokenizer and the processing functions run every time I iterate over the whole data, or only at the first iteration? Because, if all the tokenizing and processing is performed every time, then wouldn’t it be better to first store all of the processed data in a matrix and then perform multiple iterations using it?</p>

username: ptrblck
cooked: <p>I don’t know where the tokenizing is performed, but the <code>__getitem__</code> method of your <code>Dataset</code> will be called to load and process each sample.<br>
If the tokenizing (and other processing steps) are performed there, they will be executed in each iteration for all epochs, so you might want to preprocess the data beforehand.</p>