Newbie. C++. 2 classes classifier - how to make training faster

Tomasz_S: <p>Hi Guys,<br>
I have data sets consisting of about 100,000 of training examples. Every example has roughly 100 features (normalized between -2 to 2) and 2 labels (1, 0 - good, 0, 1 - bad) which looks something like this:</p>
<p>112 inputs | 2 outputs<br>
-0.58572,-0.0248404,…,-0.926154,1.14224,-0.731815  | 0,1<br>
-1.05784,-0.71582,…,0.989832,1.99402,-1.60195 | 0,1<br>
1.92291,1.79471,…,-0.339433,0.572841,-0.306174 |1,0<br>
…</p>
<p>This data sets are generated by c++ code and then passed to custom handcrafted NeuralNet. I was asked to rewrite old NN part using libtorch. I used the following NN definition (which more or less follows the old NN structure):</p>
<pre><code class="lang-auto">class TorchNet : public torch::nn::Module {
    torch::nn::Linear input{ nullptr };
    torch::nn::Linear hidden1{ nullptr };
    torch::nn::Linear hidden2{ nullptr };
    torch::nn::Linear hidden3{ nullptr };
    torch::nn::Linear hidden4{ nullptr };
    torch::nn::Linear hidden5{ nullptr };
    torch::nn::Linear output{ nullptr };
    int input_size = 0;

public:
    TorchNet(int inputs_count);
    torch::Tensor forward(torch::Tensor x);

    // dataVec - vector of pairs of (input values, targets)
    void train_step(std::vector&lt;std::pair&lt;torch::Tensor, torch::Tensor&gt;&gt; dataVec, torch::optim::Optimizer&amp; optimizer);
    // ...
};

TorchNet::TorchNet(int inputs_count) 
{
    input_size = inputs_count;
    input = register_module("input", torch::nn::Linear(inputs_count, 30));
    hidden1 = register_module("hidden1", torch::nn::Linear(30, 20));
    hidden2 = register_module("hidden2", torch::nn::Linear(20, 15));
    hidden3 = register_module("hidden3", torch::nn::Linear(15, 10));
    hidden4 = register_module("hidden4", torch::nn::Linear(10, 5));
    hidden5 = register_module("hidden5", torch::nn::Linear(5, 3));
    output = register_module("output", torch::nn::Linear(3, 2));
}

torch::Tensor TorchNet::forward(torch::Tensor x) {
    x = torch::tanh(input-&gt;forward( x ) );
    x = torch::dropout(x, /*p=*/0.2, /*train=*/is_training());
    x = torch::tanh(hidden1-&gt;forward(x));
    x = torch::tanh(hidden2-&gt;forward(x));
    x = torch::tanh(hidden3-&gt;forward(x));
    x = torch::tanh(hidden4-&gt;forward(x));
    x = torch::tanh(hidden5-&gt;forward(x));
    x = output-&gt;forward(x);
    return torch::log_softmax(x, 0);
}

// dataVec - vector of pairs of (input values, targets)
void TorchNet::train_step(std::vector&lt;std::pair&lt;torch::Tensor, torch::Tensor&gt;&gt; dataVec, torch::optim::Optimizer&amp; optimizer)
{
    train();
    size_t batch_idx = 0;
    double count = 0;
    double max = dataVec.size();
    double pct = 0;

    for (auto&amp; sample : dataVec) {
        auto data = sample.first, targets = sample.second.to(at::kLong);
        optimizer.zero_grad();
        auto output = forward(data);
        // data.sizes() = [112], output.sizes() = [2], targets.sizes() = [2]
        auto loss = torch::nll_loss(output, targets);
        AT_ASSERT(!std::isnan(loss.item&lt;float&gt;()));
        loss.backward();
        optimizer.step();
        if (count / max &gt;= pct) 
        {
            std::cout &lt;&lt; "after " &lt;&lt; (100.0 * count / max) &lt;&lt; "% loss = " &lt;&lt; loss.item&lt;double&gt;() &lt;&lt; std::endl;
            pct += 0.2;
        }
        ++count;
    }
}

</code></pre>
<p>to prepare a training data I do the following:</p>
<pre><code class="lang-auto">// data - pair of 2 vectors - first: vector of 112 input values, second: vector of 2 labels 
double TorchNetTest::train(std::vector&lt;std::pair&lt;std::vector&lt;double&gt;, std::vector&lt;double&gt;&gt;&gt; data, ...)
{
...
   auto input_size = data[0].first.size();
   model = std::make_unique&lt; TorchNet&gt;(input_size);
  
    std::vector&lt;std::pair&lt;torch::Tensor, torch::Tensor&gt;&gt; train_vec;
    for (auto data_label : data)
    {
        auto inputs = data_label.first; // a vector of 112 doubles
        auto labels = data_label.second; // a vector of 2 doubles

        auto sample = std::pair&lt;torch::Tensor, torch::Tensor&gt;();
        sample.first = torch::tensor(c10::ArrayRef(inputs));
        sample.second = torch::tensor(c10::ArrayRef(labels));

        train_vec.push_back(sample);        
    }

   torch::optim::SGD optimizer(model-&gt;parameters(), 
       torch::optim::SGDOptions(0.01).momentum(0.5));
...
   model-&gt;train_step(train_vec, optimizer);
...
}
</code></pre>
<p>It works (model can be trained with proper accuracy) but its very slow comparing to old NN. I am using a CPU version of libtorch (old NN also works on CPU only) and I know it could be much faster on GPU but I would like to make it at least usable with CPU only. As I’m a newbie I’m sure my implementation is very far from optimal.<br>
Do you have any suggestions on how to make it more performant?<br>
For example:<br>
Can I rearrange the data to make training step work faster?<br>
Is it possible to perform training with multiple threads?<br>
Is it possible to modify forword() / nll_loss() so it accepts multiple training examples instead of just one?</p>
<p>Thank you in advance.</p>

Matej_Kompanek: <p>Your model and loss should accept batches without changes (forwarding tensor of size [N x 112] ). For example - stacking of inputs:</p>
<pre><code class="lang-auto">std::vector&lt;double&gt; data1(112);
std::vector&lt;double&gt; data2(112);
std::vector&lt;double&gt; data3(112);

auto input1 = torch::tensor(c10::ArrayRef(data1));
auto input2 = torch::tensor(c10::ArrayRef(data2));
auto input3 = torch::tensor(c10::ArrayRef(data3));

auto input = torch::stack({ input1, input2, input3 });

auto model = std::make_unique&lt;TorchNet&gt;(112);

auto output = model-&gt;forward(input); // size [3, 2]
</code></pre>
<p>Instead of creating batches in your code, you should probably create a custom dataset with stack transform which will do it for you → <a href="https://github.com/pytorch/examples/blob/master/cpp/custom-dataset/custom-dataset.cpp" rel="noopener nofollow ugc">example</a></p>

Tomasz_S: <p>Hey Matej,</p>
<p>Thank you for your help. Your first suggestion (stacking) worked for forwarding but it failed for loss. I assumed I should do the same stacking for targets as you showed it for inputs.</p>
<pre><code class="lang-auto">        auto data_vec = std::vector&lt;torch::Tensor&gt;();
        auto target_vec = std::vector&lt;torch::Tensor&gt;();

        for (auto&amp; data : samples)
        {
            data_vec.push_back(data.first);         // input - torch::tensor(c10::ArrayRef(data_x)); &lt;-[112]
            target_vec.push_back(data.second); // target - torch::tensor(c10::ArrayRef(target_x)); &lt;- [2]
        }

        auto data = torch::stack(data_vec);
        auto targets = torch::stack(target_vec);

        
        auto output = forward(data);   // &lt;- forward accepts stacked inputs
        try {
            auto loss = torch::nll_loss(output, targets); &lt;- exception thrown: 0D or 1D target tensor expected, multi-target not supported

            optimizer.zero_grad();
            loss.backward();
            optimizer.step();
        }
        catch (const std::exception&amp; e) {
            std::cout &lt;&lt; "Exception occured\n" &lt;&lt; e.what() &lt;&lt; std::endl;
            exit(-1);
        }
</code></pre>
<p>torch::nll_loss(output, targets) throws “0D or 1D target tensor expected, multi-target not supported”. Should I stack inputs belonging to the same class and pass a single target for such batch?</p>
<p>I will also try you second suggestion (custom dataset) and share the results.</p>

Matej_Kompanek: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html" rel="noopener nofollow ugc">NLLLoss</a> from docs: The target that this loss expects should be a class index in the range [0,C−1] where C = number of classes (instead of one hot vectors). You can use argmax on the stacked tensor (preferably just change the way you load the targets).<br>
Example:</p>
<pre><code class="lang-auto">std::vector&lt;double&gt; data1(112);
std::vector&lt;double&gt; data2(112);
std::vector&lt;double&gt; label1 = { 0, 1 };
std::vector&lt;double&gt; label2 = { 1, 0 };

auto inputs = torch::stack({ torch::tensor(data1), torch::tensor(data2) });
auto labels = torch::stack({ torch::tensor(label1), torch::tensor(label2) });

auto model = std::make_unique&lt;TorchNet&gt;(112);

auto output = model-&gt;forward(inputs);

labels = torch::argmax(labels, 1);
		
auto loss = torch::nll_loss(output, labels);
</code></pre>

Tomasz_S: <p>Thank you very much for your help. It worked. It also showed how much I have yet to learn <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>