Forward processing speed is slow increasingly during iterative run [libtorch. torchscript]

AppleTree: <p>Processing speed is slow increasingly, perhaps GPU Memory problem.<br>
Except first process time, early processing time is pretty fast.<br>
However, suddenly processing time is slow after 13th run.<br>
After 30 iteration, program was sleep 3 sec to show that processing time is recovery during 12 iteration.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82cfe5348aaaab83a14047e4d9ff2c81a8ea82b8.png" data-download-href="https://discuss.pytorch.org/uploads/default/82cfe5348aaaab83a14047e4d9ff2c81a8ea82b8" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82cfe5348aaaab83a14047e4d9ff2c81a8ea82b8.png" alt="image" data-base62-sha1="iFdu3Mx2u0q4wjfqEypaTqDh9Og" width="383" height="500" data-dominant-color="070707"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">708×924 7.68 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>how can resolve this problem?</p>
<p><strong>Enviroment.</strong><br>
GPU : gtx1060 6gb<br>
Image size : 800x800x1</p>
<p><strong>Code.</strong></p>
<pre><code>torch::jit::script::Module module= torch::jit::load(NetworkPath, torch::kCUDA);
torch::NoGradGuard no_grad;
torch::Tensor outputArr[100];
std::cout &lt;&lt; "Start Iteration run" &lt;&lt; std::endl;
for (int i = 0; i &lt; 100; i++)	{
	if (i != 0 &amp;&amp; i % 30 == 0)	{
		std::cout &lt;&lt; "------ each 30 iteration forced Sleep 3sec -----" &lt;&lt; std::endl;
		Sleep(3000);
	}
	t = clock();
	outputArr[i] = module.forward({ aa_cu }).toTensor();
	std::cout &lt;&lt; "[Time] Iter : " &lt;&lt; i+1 &lt;&lt; " // Time(ms) : " &lt;&lt; clock() - t &lt;&lt; std::endl;
}
</code></pre>

ptrblck: <p>CUDA operations are asynchronous, so you would have to synchronize the timer before starting and stopping the timer.</p>

AppleTree: <p>hm… I thought that only the CUDA operations are synchronize in forward function.<br>
as result, whole processing time is also same.<br>
Does it have some way to synchronize forward function ?</p>
<p>Result (Check whole Process time)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/9/49276627fdf587db8702a930fea466963bf158b1.png" alt="image" data-base62-sha1="ar9gzW3QXGjtI9B0cFUmbXDPWU1" width="472" height="150"></p>
<p>Code</p>
<pre><code>t = clock();
torch::Tensor outputArr[5];
std::cout &lt;&lt; "Start Iteration run" &lt;&lt; std::endl;
for (int i = 0; i &lt; 100; i++)	{
	outputArr[i % 5] = module.forward({ aa_cu }).toTensor();
}
std::cout &lt;&lt; "[Time] 100 Iter : " &lt;&lt; clock() - t &lt;&lt; std::endl;
</code></pre>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">CUDAStream stream = getCurrentCUDAStream();
AT_CUDA_CHECK(cudaStreamSynchronize(stream));
</code></pre>

AppleTree: <p>as your suggestion, i used “cudaStreamSynchronize” function, however nothing to channge.<br>
Even I don’t know if i correctly used the function <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"><br>
i think i didn’t get cuda Stream from “getCurrent CUDA Stream” function.<br>
Because stream variable look like empty as 2nd screenshot.</p>
<p>Should I make the stream to input data for using cudaStreamSynchronize?</p>
<p>Result<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9df91e50280034412e028089ac1b7806a2cf7e7.png" alt="image" data-base62-sha1="xmWfj9oKpNKSkdbyI9LPrQrsSPB" width="368" height="152"></p>
<p>Code<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/09d347f2e0810b7e4d31ebd9b4ff52d4230722ee.png" data-download-href="https://discuss.pytorch.org/uploads/default/09d347f2e0810b7e4d31ebd9b4ff52d4230722ee" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/09d347f2e0810b7e4d31ebd9b4ff52d4230722ee.png" alt="image" data-base62-sha1="1oUXOFM0TBOAWTKYEi9K1f7yqrY" width="690" height="121" data-dominant-color="2B2C2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">737×130 3.87 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Kiki_Rizki_Arpiandi: <p><a class="mention" href="/u/tom">@tom</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@albanD</a>  I am sorry to mention you guys.Please help <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=9" title=":sob:" class="emoji" alt=":sob:"> it happen to me as well. I Syncronized the gpu and the problem still persist.</p>
<p>I use : gtx1050m, cuda10.1, libtorch 1.6</p>

Kiki_Rizki_Arpiandi: <p><a class="mention" href="/u/appletree">@AppleTree</a> did u resolve the problem in the end?</p>

ptrblck: <p>Could you post a minimal, executable code snippet, which shows the slowdown with proper synchronizations, please?</p>

Kiki_Rizki_Arpiandi: <p>Thank you very much for your response here is the minimal working code:</p>
<pre><code class="lang-auto">
#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;
#include &lt;cuda_runtime_api.h&gt;

int main() {
    torch::jit::script::Module script_module = torch::jit::load("detector.pt");
    std::vector &lt;torch::jit::IValue&gt; inputs;
    auto img_tensor = torch::zeros({1,3,640,640}).cuda();
    inputs.emplace_back(img_tensor);
    for(int i=0;i&lt;10000;i++){
        clock_t begin = clock();
        torch::NoGradGuard no_grad_guard;
        cudaDeviceSynchronize();
        auto output = script_module.forward(inputs);
        cudaDeviceSynchronize();
        clock_t end = clock();
        double elapsed_time = double(end - begin) / 1000;
        if(i%100==0) {
            std::cout &lt;&lt; "Iteration : "&lt;&lt;i&lt;&lt;" Elapsed time " &lt;&lt; elapsed_time &lt;&lt; std::endl;
        }
    }
    return 0;
}
</code></pre>
<p>Here is the output</p>
<blockquote>
<p>Iteration : 0 Elapsed time 586.484<br>
Iteration : 100 Elapsed time 29.035<br>
Iteration : 200 Elapsed time 28.963<br>
Iteration : 300 Elapsed time 29.021<br>
Iteration : 400 Elapsed time 29.029<br>
Iteration : 500 Elapsed time 29.407<br>
Iteration : 600 Elapsed time 112.363<br>
Iteration : 700 Elapsed time 141.069<br>
Iteration : 800 Elapsed time 34.88<br>
Iteration : 900 Elapsed time 104.44<br>
Iteration : 1000 Elapsed time 116.31<br>
Iteration : 1100 Elapsed time 100.833</p>
</blockquote>
<p><strong>Environtment</strong> :  NVIDIA-SMI 440.100      Driver Version: 440.100      CUDA Version: 10.1, Ubuntu 18.04, GTX1050M, libtorch 1.6</p>

albanD: <p>Hi,</p>
<p>The runtime seems to be actually be fluctuating quite a lot.<br>
Are there other processes that use the CPU or GPU on the same machine?</p>

Kiki_Rizki_Arpiandi: <p>Thanks for your response <a class="mention" href="/u/alband">@albanD</a>, I m sure that there is no background process or another heavy lifting process that use CPU or GPU on the same machine, I can also provide the traced model if you need here is the link <a href="https://drive.google.com/file/d/1jPVGtMcbqvl3r9oGuqBlbgyKsDLY5RSv/view?usp=sharing" rel="noopener nofollow ugc">https://drive.google.com/file/d/1jPVGtMcbqvl3r9oGuqBlbgyKsDLY5RSv/view?usp=sharing</a></p>

Kiki_Rizki_Arpiandi: <p>I tested the executable for memory leak as well, using valgrind. And there is no leak</p>

Kiki_Rizki_Arpiandi: <p>Please any update on this?</p>

ptrblck: <p>I tested your code and get stable results, even though I would recommend to accumulate the time for a few hundred iterations and calculate the mean once.<br>
Anyway, for <code>resnet101</code>, this is the result for libtorch 1.6.0:</p>
<pre><code class="lang-python">Iteration : 0 Elapsed time 1837.04
Iteration : 100 Elapsed time 22.734
Iteration : 200 Elapsed time 22.597
Iteration : 300 Elapsed time 22.264
Iteration : 400 Elapsed time 22.724
Iteration : 500 Elapsed time 22.885
Iteration : 600 Elapsed time 22.647
Iteration : 700 Elapsed time 22.73
Iteration : 800 Elapsed time 22.692
Iteration : 900 Elapsed time 22.541
Iteration : 1000 Elapsed time 22.549
Iteration : 1100 Elapsed time 22.788
Iteration : 1200 Elapsed time 22.694
Iteration : 1300 Elapsed time 22.637
Iteration : 1400 Elapsed time 22.744
Iteration : 1500 Elapsed time 22.663
Iteration : 1600 Elapsed time 22.66
Iteration : 1700 Elapsed time 22.785
Iteration : 1800 Elapsed time 22.681
Iteration : 1900 Elapsed time 22.786
Iteration : 2000 Elapsed time 22.668
Iteration : 2100 Elapsed time 22.697
Iteration : 2200 Elapsed time 22.749
...
</code></pre>
<p>Note that the first iteration takes longer, since you are already starting the timer before calling the synchronize op.</p>

Kiki_Rizki_Arpiandi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Kindly thank you for your response. What is the possible reasons behind the fluctuation, beside there are another processes on the background?<br>
Is possible that the deep learning model cause such fluctuation?</p>

ptrblck: <aside class="quote no-group" data-username="Kiki_Rizki_Arpiandi" data-post="15" data-topic="71075">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kiki_rizki_arpiandi/48/10715_2.png" class="avatar"> Kiki_Rizki_Arpiandi:</div>
<blockquote>
<p>Is possible that the deep learning model cause such fluctuation?</p>
</blockquote>
</aside>
<p>Could be the case, e.g. if the model uses different paths.<br>
If you can post the model definition, I could rerun the script and check the timings on my system.</p>

Kiki_Rizki_Arpiandi: <p>Here is the model :</p>
<pre><code class="lang-auto">from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import math

import torch
import torch.nn as nn
import torch.utils.model_zoo as model_zoo

BN_MOMENTUM = 0.01
model_urls = {
    'mobilenet_v2': 'https://download.pytorch.org/models/mobilenet_v2-b0353104.pth',
}


def conv_bn(inp, oup, stride):
    conv_3x3 = nn.Sequential(
        nn.Conv2d(inp, oup, 3, stride, 1, bias=False),
        nn.BatchNorm2d(oup),
        nn.ReLU(inplace=True)
    )
    for m in conv_3x3.modules():
        if isinstance(m, nn.Conv2d):
            nn.init.kaiming_normal_(m.weight, mode='fan_out')
            if m.bias is not None:
                nn.init.zeros_(m.bias)
        elif isinstance(m, nn.BatchNorm2d):
            nn.init.ones_(m.weight)
            nn.init.zeros_(m.bias)
    return conv_3x3


def conv_1x1_bn(inp, oup):
    conv1x1 = nn.Sequential(
        nn.Conv2d(inp, oup, 1, 1, 0, bias=False),
        nn.BatchNorm2d(oup),
        nn.ReLU(inplace=True))
    for m in conv1x1.modules():
        if isinstance(m, nn.Conv2d):
            nn.init.kaiming_normal_(m.weight, mode='fan_out')
            if m.bias is not None:
                nn.init.zeros_(m.bias)
        elif isinstance(m, nn.BatchNorm2d):
            nn.init.ones_(m.weight)
            nn.init.zeros_(m.bias)
    return conv1x1


def deconv_bn_relu(in_channels, out_channels, kernel_size, padding, output_padding, bias):
    deconv = nn.Sequential(
        nn.ConvTranspose2d(
            in_channels=in_channels,
            out_channels=out_channels,
            kernel_size=kernel_size,
            stride=2,
            padding=padding,
            output_padding=output_padding,
            bias=bias),
        nn.BatchNorm2d(out_channels, momentum=BN_MOMENTUM),
        nn.ReLU(inplace=True)
    )
    for m in deconv.modules():
        if isinstance(m, nn.ConvTranspose2d):
            fill_up_weights(m)
            if m.bias is not None:
                nn.init.zeros_(m.bias)
        elif isinstance(m, nn.BatchNorm2d):
            nn.init.ones_(m.weight)
            nn.init.zeros_(m.bias)
    return deconv


def fill_fc_weights(layers):
    for m in layers.modules():
        if isinstance(m, nn.Conv2d):
            if m.bias is not None:
                nn.init.constant_(m.bias, 0)


def fill_up_weights(up):
    w = up.weight.data
    f = math.ceil(w.size(2) / 2)
    c = (2 * f - 1 - f % 2) / (2. * f)
    for i in range(w.size(2)):
        for j in range(w.size(3)):
            w[0, 0, i, j] = \
                (1 - math.fabs(i / f - c)) * (1 - math.fabs(j / f - c))
    for c in range(1, w.size(0)):
        w[c, 0, :, :] = w[0, 0, :, :]


class InvertedResidual(nn.Module):
    def __init__(self, inp, oup, stride, expand_ratio):
        super(InvertedResidual, self).__init__()
        self.stride = stride
        assert stride in [1, 2]

        hidden_dim = round(inp * expand_ratio)
        self.use_res_connect = self.stride == 1 and inp == oup

        if expand_ratio == 1:
            self.conv = nn.Sequential(
                # dw
                nn.Conv2d(hidden_dim, hidden_dim, 3, stride, 1, groups=hidden_dim, bias=False),
                nn.BatchNorm2d(hidden_dim),
                nn.ReLU(inplace=True),
                # pw-linear
                nn.Conv2d(hidden_dim, oup, 1, 1, 0, bias=False),
                nn.BatchNorm2d(oup),
            )
        else:
            self.conv = nn.Sequential(
                # pw
                nn.Conv2d(inp, hidden_dim, 1, 1, 0, bias=False),
                nn.BatchNorm2d(hidden_dim),
                nn.ReLU(inplace=True),
                # dw
                nn.Conv2d(hidden_dim, hidden_dim, 3, stride, 1, groups=hidden_dim, bias=False),
                nn.BatchNorm2d(hidden_dim),
                nn.ReLU(inplace=True),
                # pw-linear
                nn.Conv2d(hidden_dim, oup, 1, 1, 0, bias=False),
                nn.BatchNorm2d(oup),
            )

    def forward(self, x):
        if self.use_res_connect:
            return x + self.conv(x)
        else:
            return self.conv(x)


class MobileNetv2Det(nn.Module):
    def __init__(self, heads, head_conv, width_mult=1., is_train=True):
        super(MobileNetv2Det, self).__init__()
        self.inplanes = 32
        self.last_channel = 64  # backbone
        self.deconv_with_bias = False
        self.is_train = is_train
        self.heads = heads

        block = InvertedResidual
        interverted_residual_setting = [
            # t, c, n, s
            [1, 16, 1, 1],
            [6, 24, 2, 2],
            [6, 32, 3, 2],
            [6, 64, 4, 2],
            [6, 96, 3, 1],
            [6, 160, 3, 2],
            [6, 320, 1, 1],
        ]

        # build backbone
        # building first layer
        # assert input_size % 32 == 0
        input_channel = int(self.inplanes * width_mult)
        self.last_channel = int(self.last_channel * width_mult) if width_mult &gt; 1.0 else self.last_channel
        self.features = [conv_bn(3, input_channel, 2)]
        # building inverted residual blocks
        for t, c, n, s in interverted_residual_setting:
            output_channel = int(c * width_mult)
            for i in range(n):
                if i == 0:
                    self.features.append(block(input_channel, output_channel, s, expand_ratio=t))
                else:
                    self.features.append(block(input_channel, output_channel, 1, expand_ratio=t))
                input_channel = output_channel

        # make it nn.Sequential
        self.features = nn.Sequential(*self.features)

        # building last several layers
        self.backbone_lastlayer = conv_1x1_bn(input_channel, self.last_channel)

        self.ups = []
        for i in range(3):
            up = deconv_bn_relu(self.last_channel, self.last_channel, 2, 0, 0, self.deconv_with_bias)
            self.ups.append(up)
        self.ups = nn.Sequential(*self.ups)

        self.conv_dim_matchs = []
        self.conv_dim_matchs.append(conv_1x1_bn(96, self.last_channel))
        self.conv_dim_matchs.append(conv_1x1_bn(32, self.last_channel))
        self.conv_dim_matchs.append(conv_1x1_bn(24, self.last_channel))
        self.conv_dim_matchs = nn.Sequential(*self.conv_dim_matchs)

        self.last_context_conv = conv_bn(self.last_channel, self.last_channel, 1)

        for head in sorted(self.heads):
            num_output = self.heads[head]


            if head_conv &gt; 0:
                fc = nn.Sequential(
                    nn.Conv2d(self.last_channel, head_conv,
                              kernel_size=3, padding=1, bias=True),
                    nn.ReLU(inplace=True),
                    nn.Conv2d(head_conv, num_output,
                              kernel_size=1, stride=1, padding=0))
                if 'hm' in head:
                    fc[-1].bias.data.fill_(-2.19)
                else:
                    fill_fc_weights(fc)
            else:
                fc = nn.Conv2d(
                    in_channels=self.last_channel,
                    out_channels=num_output,
                    kernel_size=1,
                    stride=1,
                    padding=0
                )
                if 'hm' in head:
                    fc.bias.data.fill_(-2.19)
                else:
                    fill_fc_weights(fc)
            self.__setattr__(head, fc)

    def init_weights(self, pretrained=True):
        if pretrained:
            for head in self.heads:
                final_layer = self.__getattr__(head)
                for i, m in enumerate(final_layer.modules()):
                    if isinstance(m, nn.Conv2d):
                        nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
                        if m.weight.shape[0] == self.heads[head]:
                            if 'hm' in head:
                                nn.init.constant_(m.bias, -2.19)
                            else:
                                nn.init.normal_(m.weight, std=0.001)
                                nn.init.constant_(m.bias, 0)
            url = model_urls['mobilenet_v2']
            pretrained_state_dict = model_zoo.load_url(url)
            print('=&gt; loading pretrained model {}'.format(url))
            self.features.load_state_dict(pretrained_state_dict, strict=False)
        else:
            for m in self.modules():
                if isinstance(m, nn.Conv2d):
                    nn.init.kaiming_normal_(m.weight, mode='fan_out')
                    if m.bias is not None:
                        nn.init.zeros_(m.bias)
                elif isinstance(m, nn.BatchNorm2d):
                    nn.init.ones_(m.weight)
                    nn.init.zeros_(m.bias)
                elif isinstance(m, nn.Linear):
                    nn.init.normal_(m.weight, 0, 0.01)
                    nn.init.zeros_(m.bias)
                elif isinstance(m, nn.ConvTranspose2d):
                    fill_up_weights(m)
                    if m.bias is not None:
                        nn.init.zeros_(m.bias)

    def forward(self, x):
        xs = []

        for n in range(0, 4):
            x = self.features[n](x)
        xs.append(x)

        for n in range(4, 7):
            x = self.features[n](x)
        xs.append(x)

        for n in range(7, 14):
            x = self.features[n](x)
        xs.append(x)

        for n in range(14, 18):
            x = self.features[n](x)

        x = self.backbone_lastlayer(x)

        for i in range(3):
            x = self.ups[i](x)
            x = x + self.conv_dim_matchs[i](xs[3 - i - 1])

        x = self.last_context_conv(x)
        if self.is_train == True:
            ret = {}
            for head in self.heads:
                ret[head] = self.__getattr__(head)(x)
            return [ret]
        else:
            ret = []
            for head in self.heads:
                ret.append(self.__getattr__(head)(x))
            return torch.cat(ret, 1)


def get_mv2relu_net(num_layers, heads, head_conv, is_train):
    model = MobileNetv2Det(heads, head_conv=head_conv, width_mult=1.0, is_train=is_train)
    model.init_weights()
    return model


def create_model(heads={'hm': 1, 'wh': 2, 'reg': 2}, head_conv=64, is_train=True):
    model = get_mv2relu_net(num_layers=0, heads=heads, head_conv=head_conv, is_train=is_train)
    return model


def demo():
    model = create_model(is_train=False).cuda()
    model.init_weights()

    model = model.eval()
    model.is_train = False
    model.eval()

    random_sample = torch.rand(2, 3, 640, 640).cuda()
    out = model(random_sample)


    traced_module = torch.jit.trace(model, random_sample)
    traced_module.save("traced_model.pt")


if __name__ == '__main__':
    demo()

</code></pre>

ptrblck: <p>I cannot trace the model and get:</p>
<pre><code class="lang-python">RuntimeError: Only tensors, lists, tuples of tensors, or dictionary of tensors can be output from traced functions
</code></pre>
<p>and scripting it yields:</p>
<pre><code class="lang-python">RuntimeError: 
Expected integer literal for index:
  File "&lt;ipython-input-194-4d44670bb81d&gt;", line 254
        
        for n in range(0, 4):
            x = self.features[n](x)
                ~~~~~~~~~~~~~~~~ &lt;--- HERE
        xs.append(x)
</code></pre>

Kiki_Rizki_Arpiandi: <p>I am sorry, my bad  <a class="mention" href="/u/ptrblck">@ptrblck</a>. I have edited the code it should works now</p>

ptrblck: <p>Thanks, the model timing is also table for this model:</p>
<pre><code class="lang-python">Iteration : 0 Elapsed time 352.787
Iteration : 100 Elapsed time 6.162
Iteration : 200 Elapsed time 6.17
Iteration : 300 Elapsed time 6.19
Iteration : 400 Elapsed time 6.197
Iteration : 500 Elapsed time 6.204
Iteration : 600 Elapsed time 6.169
Iteration : 700 Elapsed time 6.193
Iteration : 800 Elapsed time 6.168
Iteration : 900 Elapsed time 6.128
Iteration : 1000 Elapsed time 6.209
Iteration : 1100 Elapsed time 6.183
Iteration : 1200 Elapsed time 6.17
Iteration : 1300 Elapsed time 6.18
Iteration : 1400 Elapsed time 6.181
Iteration : 1500 Elapsed time 6.177
Iteration : 1600 Elapsed time 6.175
Iteration : 1700 Elapsed time 6.186
Iteration : 1800 Elapsed time 6.185
Iteration : 1900 Elapsed time 6.18
Iteration : 2000 Elapsed time 6.136
Iteration : 2100 Elapsed time 6.192
Iteration : 2200 Elapsed time 6.158
Iteration : 2300 Elapsed time 6.15
Iteration : 2400 Elapsed time 6.14
Iteration : 2500 Elapsed time 6.166
Iteration : 2600 Elapsed time 6.169
Iteration : 2700 Elapsed time 6.178
Iteration : 2800 Elapsed time 6.177
Iteration : 2900 Elapsed time 6.173
Iteration : 3000 Elapsed time 6.166
</code></pre>
<p>(besides the previously mentioned wrongly timed first iteration).</p>