Batch as input to module->forward({inputs}).toTensor()

username: dambo
cooked: <p>Hello,<br>
I am trying to work in <strong>batches</strong> while reading frames using CV2 from a video like so. I first read 8 frames, batch them into <code>std::vector&lt;torch::jit::IValue&gt;</code> and then run them through <code>module-&gt;forward({inputs}).toTensor();</code>. However, if the batch size is larger than 1, it crashes. I guess there is some reshaping involved to add the batch dimension I am just not sure how to do it in C++.</p>
<pre><code class="lang-auto">    int batch_size =1;
    int b_counter=0;

    std::vector&lt;torch::jit::IValue&gt; inputs; //Accumulates the batches

    for (num_frames = 0; num_frames &lt; nb_frames; num_frames++) {
        video_reader &gt;&gt; frame;
        cv::cvtColor(frame, frame, CV_BGR2RGB);
        frame.convertTo(frame, CV_32FC3, 1.0f / 255.0f);
        auto input_tensor = torch::from_blob(frame.data, {1, frame_h, frame_w, kCHANNELS});
        
        input_tensor = input_tensor.permute({0, 3, 1, 2});
        inputs.emplace_back(input_tensor.to(torch::kCUDA)); // Add the frame to the vector
        //Ok we now have enough inputs
        if (batch_size == inputs.size()) {
            b_counter++;
            std::cout &lt;&lt;"Batch:" &lt;&lt; b_counter&lt;&lt;std::endl;
//            std::cout &lt;&lt;"inputs.size() loop:" &lt;&lt; inputs.size()&lt;&lt;std::endl;
            torch::Tensor out_tensor = module-&gt;forward({inputs}).toTensor(); // This line fails
</code></pre>
<p>Thanks,</p>

username: Fighting-JJ
cooked: <p>I have tried jit.trace the model again with batchsize m, then</p>
<p><code>module-&gt;forward({inputs})</code> the same batchsize data, but it still failed.</p>
<p>I do not want use <code>for(;;;)</code> too.</p>

username: mhubii
cooked: <p>you could just create a tensor of size BxCxHxW, instead of pushing multiple tensors of size 1xCxHxW to your  <code>std::vector&lt;torch::jit::IValue&gt;</code>. For example</p>
<pre><code class="lang-auto">auto module = torch::jit::load("../model.pt");

// Thats how to forward a whole batch.
std::vector&lt;torch::jit::IValue&gt; inputs;
inputs.push_back(torch::ones({10, 3, 224, 224})); // BxCxHxW
module-&gt;forward(inputs);

// You only need to push back multiple inputs if, say your forward takes two arguments, then
std::vector&lt;torch::jit::IValue&gt; inputs;
inputs.push_back(torch::ones({1, 3, 224, 224})); // 1xCxHxW
inputs.push_back(torch::ones({1, 3, 224, 224})); // 1xCxHxW
module-&gt;forward(inputs);
</code></pre>

username: Fighting-JJ
cooked: <p>I just tried your code:</p>
<p><code>std::vector&lt;torch::jit::IValue&gt; inputs;</code><br>
<code>inputs.push_back(torch::ones({1, 3, 224, 224}));</code><br>
<code>inputs.push_back(torch::ones({1, 3, 224, 224}));</code><br>
<code>module-&gt;forward(inputs);</code></p>
<p>but it failed.</p>
<p>the code down works:</p>
<pre><code class="lang-auto">`std::vector&lt;at::Tensor&gt; inputs_vec;`
`inputs_vec.push_back(torch::ones({1, 3, 224, 224})); // 1xCxHxW`
`inputs_vec.push_back(torch::ones({1, 3, 224, 224})); // 1xCxHxW`
`at::Tensor input_ = torch::cat(inputs_vec);`
`std::vector&lt;torch::jit::IValue&gt; inputs;`
`inputs.push_back(input_ )`
`module-&gt;forward(inputs);`
</code></pre>
<p>That is all. Thanks again. sorry about the code format.</p>

username: mhubii
cooked: <p>yes, as explained, the code only works if your forward function expects 2 arguments. This is not the case here. It was just an example. Please take some time and read the annotations and it will become clear</p>

username: dambo
cooked: <p>Thanks. However, how come my loop does work in the same way as your:</p>
<p><code>inputs.push_back(torch::ones({10, 3, 224, 224}));</code><br>
?</p>

username: mhubii
cooked: <p>I dont understand the question, can you explain it further?</p>

username: jy_mo
cooked: <p>May I ask if this is batch=2 forward ?</p>