Converting a tensor to a Python number for ONNX convertation

PlaeryinBol: <p>I’m trying to convert lstm-decoder with beam search to onnx, but I got some TracerWarnings. Is it possible to operate with int-number item from 1-size input tensor in my future code like this?</p>
<pre><code class="lang-python">def forward(self, img_features, beam_size):
    '''
    Arguments:
        img_features (tensor): Extracted features from the encoder module. (batch_size, feature_pixels = 7x7, encoder_dim = 2048)
        beam_size (tensor): Number of top candidates to consider for beam search. (int, = 3 or =6 or =9)
    '''
    item = beam_size.item()
    #some code...
    output = torch.tensor([item])
    return output
</code></pre>
<p>Now I have this warnings and constant output of Onnx-model:</p>
<pre><code class="lang-auto">TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  item = beam_size.item()
</code></pre>
<pre><code class="lang-auto">TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  output = torch.tensor([item])
</code></pre>
<p>I tried using <span class="mention">@torch.jit.script</span> but it didn’t help too</p>

ptrblck: <p>Could you post your real use case, i.e. how the Python integer is used?<br>
Based on your current code snippet you should be able to just use the tensor instead of converting it to an integer, but I assume it’s just an example and not your real use case.</p>

PlaeryinBol: <p>Ok, it’s just a bit of my problems with Onnx-convertation of lstm beam-search, and converting a tensor to a Python number is the first of them. When I export this for Onnx, I got the constant output from the model with incorrect tracing. I found, that I should script it with torch.script, but those examples not easy to understand <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>
<pre><code class="lang-auto">    def forward(self, img_features, beam_size):
        """
        Function to generate the caption for the corresponding encoded image using beam search to provide the most optimal caption 
        combination. 
        
        Arguments:
            img_features (tensor): Extracted features from the encoder module. (batch_size, feature_pixels = 7x7, encoder_dim = 2048)
            beam_size (tensor): Number of top candidates to consider for beam search. (int = 3 or 6 or 9)
            
        Output:
            sentence (tensor): ordered list of words of the final optimal caption (list)

        """
        beam_size = beam_size.item()#!Converting a tensor to a Python number
        
        prev_words = torch.zeros(beam_size, 1).long()
        
        sentences = prev_words
        
        top_preds = torch.zeros(beam_size, 1)
        
        completed_sentences = []
        completed_sentences_preds = []
        
        step = 1
        h, c = self.get_init_lstm_state(img_features)

        while True:
            embedding = self.embedding(prev_words).squeeze(1)
            context = self.attention(img_features, h)[0]
            gate = self.sigmoid(self.f_beta(h))
            gated_context = gate * context

            lstm_input = torch.cat((embedding, gated_context), dim=1)
            h, c = self.lstm(lstm_input, (h, c))
            output = self.deep_output(h)
            output = top_preds.expand_as(output) + output

            if step == 1:
                top_preds, top_words = output[0].topk(beam_size, 0, True, True)
            else:
                top_preds, top_words = output.view(-1).topk(beam_size, 0, True, True)
            prev_word_idxs = top_words / output.size(1)
            next_word_idxs = top_words % output.size(1)

            sentences = torch.cat((sentences[prev_word_idxs], next_word_idxs.unsqueeze(1)), dim=1)

            incomplete = [idx for idx, next_word in enumerate(next_word_idxs) if next_word != 1]#!Converting a tensor to a Python boolean
            complete = list(set(range(len(next_word_idxs))) - set(incomplete))#!Converting a tensor to a Python index

            if len(complete) &gt; 0:
                completed_sentences.extend(sentences[complete].tolist())#!Converting a tensor to a Python list
                completed_sentences_preds.extend(top_preds[complete])#!Converting a tensor to a Python index
            beam_size -= len(complete)

            if beam_size == 0:
                break
            sentences = sentences[incomplete]
            h = h[prev_word_idxs[incomplete]]
            c = c[prev_word_idxs[incomplete]]
            img_features = img_features[prev_word_idxs[incomplete]]
            top_preds = top_preds[incomplete].unsqueeze(1)
            prev_words = next_word_idxs[incomplete].unsqueeze(1)

            if step &gt; 50:
                break
            step += 1

        idx = completed_sentences_preds.index(max(completed_sentences_preds))#!Converting a tensor to a Python boolean
        sentence = completed_sentences[idx]

        sentence = torch.tensor(sentence)#!torch.tensor results are registered as constants
        return sentence
</code></pre>

ptrblck: <p>Is scripting working for you now or are you still facing some issues?<br>
I’m not an expert in using ONNX, but could you try to use tensors instead of Python literals via e.g.:</p>
<pre><code class="lang-python">beam_size = torch.tensor([5])
prev_words = torch.zeros(beam_size, 1).long()
</code></pre>

PlaeryinBol: <p>Yes, your case is working, but in this line, for example:</p>
<pre><code class="lang-auto">top_preds, top_words = output[0].topk(beam_size, 0, True, True)
</code></pre>
<p>‘topk()’ first argument must be int, not Tensor, so I still need this option (converatation tensor to a single number).<br>
I try to script it like this:</p>
<pre><code class="lang-auto">        from torch import Tensor
        @torch.jit.script
        def get_item(x: Tensor):
            item = x.item()
            return item
        item = get_item(beam_size)
</code></pre>
<p>but got this error:</p>
<pre><code class="lang-auto">RuntimeError: get_item() Expected a value of type 'Tensor' for argument 'x' but instead found type 'int'.
Position: 0
Value: 1
Declaration: get_item(Tensor x) -&gt; (Scalar)
Cast error details: Unable to cast Python instance of type &lt;class 'int'&gt; to C++ type 'at::Tensor'
</code></pre>
<p>What do you think about possible solution for this problems?</p>

ptrblck: <p>The last error message points towards an integer input, while a tensor is expected.<br>
Did you accidentally pass <code>x.item()</code> to <code>get_item()</code>?</p>

PlaeryinBol: <p>Sorry, I’m forgot to save changes in module <img src="https://discuss.pytorch.org/images/emoji/apple/smile_cat.png?v=9" title=":smile_cat:" class="emoji" alt=":smile_cat:"></p>
<p>Now, when I try to export to Onnnx this model with above <code>get_item(beam_size)</code> scripted function, I got<br>
<code>RuntimeError: Tracer cannot set value trace for type Int. Supported types are tensor, tensor list, and tuple of tensors.</code></p>
<p>Is it problem from Onnx-part, or am I doing something wrong with torch.script and it leads to errors?</p>

ptrblck: <p>I’m not sure, but the standalone PyTorch script seems to work:</p>
<pre><code class="lang-python">from torch import Tensor
@torch.jit.script
def get_item(x: Tensor):
    item = x.item()
    return item

beam_size = torch.randn(1)
item = get_item(beam_size)
print(get_item.graph)
&gt; graph(%x.1 : Tensor):
  %item.1 : Scalar = aten::item(%x.1) # &lt;ipython-input-109-1655baf12d25&gt;:4:11
  return (%item.1)
print(item)
&gt; 0.18689797818660736
</code></pre>

PlaeryinBol: <p>Yes, this is work in practice, but not in onnx-export(</p>
<p>Anyway, thanks for your help, you are best <img src="https://discuss.pytorch.org/images/emoji/apple/kissing_heart.png?v=9" title=":kissing_heart:" class="emoji" alt=":kissing_heart:"></p>