Forward method slows down at some point if it is executed repeatedly

nvkz: <p>We are currently working with pytorch 1.9.1.</p>
<p>We built pytorch with <code>LITE_INTERPRETER</code> and converted our original fully quantized model that contains only conv and relu to NNAPI supported format.<br>
Test inference code that is shown below worked successfully on Android. But <code>forward</code> method slows down at some point.</p>
<p>As shown below, in first 37 iterations, each execution takes about 78 - 80 msec. After that, however, it takes about 90 - 102 msec.</p>
<p>What is the cause of this feature? We found the similar issue <a href="https://discuss.pytorch.org/t/forward-function-slows-down-after-several-iterations/71336">here</a>. Do these issues come from the same cause? I’d like to know the cause and solution of this issue.</p>
<p>Thanks in advance!</p>
<pre><code class="lang-auto">void testLoadAndForward(
        JNIEnv *env,
        jstring jModelPath)
{
    c10::InferenceMode guard;
    const char* modelPath = env-&gt;GetStringUTFChars(jModelPath, 0);

    torch::jit::mobile::Module module = torch::jit::_load_for_mobile(modelPath);

    auto get_method = module.find_method("get_all_bundled_inputs");
    auto all_inputs = (*get_method)({}).toList();
    auto inputs = all_inputs.get(0).toTuple()-&gt;elements();

    c10::IValue t_out;
    int iter = 100;
    for (int i = 0; i &lt; iter; ++i) {
        std::chrono::system_clock::time_point start = std::chrono::system_clock::now();
        t_out = module.forward(inputs);
        std::chrono::system_clock::time_point end = std::chrono::system_clock::now();
        auto duration = std::chrono::duration_cast&lt;std::chrono::nanoseconds&gt;(end - start).count() / 1000000.0;
        LOGD("%d duration: %lf msec", i, duration);
    }
}
</code></pre>
<p>Output is:</p>
<pre><code class="lang-auto">D/NATIVE_CODE: 0 duration: 9284.693000 msec
D/NATIVE_CODE: 1 duration: 79.760000 msec
D/NATIVE_CODE: 2 duration: 78.123000 msec
D/NATIVE_CODE: 3 duration: 78.468000 msec
D/NATIVE_CODE: 4 duration: 77.882000 msec
D/NATIVE_CODE: 5 duration: 78.987000 msec
D/NATIVE_CODE: 6 duration: 77.897000 msec
D/NATIVE_CODE: 7 duration: 78.845000 msec
D/NATIVE_CODE: 8 duration: 78.278000 msec
D/NATIVE_CODE: 9 duration: 77.879000 msec
D/NATIVE_CODE: 10 duration: 78.108000 msec
D/NATIVE_CODE: 11 duration: 78.109000 msec
D/NATIVE_CODE: 12 duration: 78.286000 msec
D/NATIVE_CODE: 13 duration: 78.376000 msec
D/NATIVE_CODE: 14 duration: 78.368000 msec
D/NATIVE_CODE: 15 duration: 78.295000 msec
D/NATIVE_CODE: 16 duration: 78.213000 msec
D/NATIVE_CODE: 17 duration: 78.345000 msec
D/NATIVE_CODE: 18 duration: 77.938000 msec
D/NATIVE_CODE: 19 duration: 78.280000 msec
D/NATIVE_CODE: 20 duration: 78.160000 msec
D/NATIVE_CODE: 21 duration: 78.574000 msec
D/NATIVE_CODE: 22 duration: 78.711000 msec
D/NATIVE_CODE: 23 duration: 78.490000 msec
D/NATIVE_CODE: 24 duration: 78.499000 msec
D/NATIVE_CODE: 25 duration: 78.521000 msec
D/NATIVE_CODE: 26 duration: 78.173000 msec
D/NATIVE_CODE: 27 duration: 78.130000 msec
D/NATIVE_CODE: 28 duration: 77.988000 msec
D/NATIVE_CODE: 29 duration: 78.555000 msec
D/NATIVE_CODE: 30 duration: 78.568000 msec
D/NATIVE_CODE: 31 duration: 78.566000 msec
D/NATIVE_CODE: 32 duration: 80.458000 msec
D/NATIVE_CODE: 33 duration: 79.139000 msec
D/NATIVE_CODE: 34 duration: 78.517000 msec
D/NATIVE_CODE: 35 duration: 78.168000 msec
D/NATIVE_CODE: 36 duration: 78.810000 msec
D/NATIVE_CODE: 37 duration: 78.541000 msec
D/NATIVE_CODE: 38 duration: 87.629000 msec
D/NATIVE_CODE: 39 duration: 89.959000 msec
D/NATIVE_CODE: 40 duration: 93.983000 msec
D/NATIVE_CODE: 41 duration: 97.401000 msec
D/NATIVE_CODE: 42 duration: 96.597000 msec
D/NATIVE_CODE: 43 duration: 90.576000 msec
D/NATIVE_CODE: 44 duration: 92.451000 msec
D/NATIVE_CODE: 45 duration: 97.267000 msec
D/NATIVE_CODE: 46 duration: 97.906000 msec
D/NATIVE_CODE: 47 duration: 97.167000 msec
D/NATIVE_CODE: 48 duration: 94.146000 msec
D/NATIVE_CODE: 49 duration: 94.663000 msec
D/NATIVE_CODE: 50 duration: 93.778000 msec
D/NATIVE_CODE: 51 duration: 99.721000 msec
D/NATIVE_CODE: 52 duration: 97.753000 msec
D/NATIVE_CODE: 53 duration: 97.071000 msec
D/NATIVE_CODE: 54 duration: 97.035000 msec
D/NATIVE_CODE: 55 duration: 101.194000 msec
D/NATIVE_CODE: 56 duration: 102.563000 msec
D/NATIVE_CODE: 57 duration: 99.331000 msec
D/NATIVE_CODE: 58 duration: 98.304000 msec
D/NATIVE_CODE: 59 duration: 91.064000 msec
D/NATIVE_CODE: 60 duration: 96.322000 msec
D/NATIVE_CODE: 61 duration: 94.546000 msec
D/NATIVE_CODE: 62 duration: 93.846000 msec
D/NATIVE_CODE: 63 duration: 98.800000 msec
D/NATIVE_CODE: 64 duration: 96.586000 msec
D/NATIVE_CODE: 65 duration: 99.268000 msec
D/NATIVE_CODE: 66 duration: 96.090000 msec
D/NATIVE_CODE: 67 duration: 96.336000 msec
D/NATIVE_CODE: 68 duration: 99.795000 msec
D/NATIVE_CODE: 69 duration: 100.935000 msec
D/NATIVE_CODE: 70 duration: 99.763000 msec
D/NATIVE_CODE: 71 duration: 98.782000 msec
D/NATIVE_CODE: 72 duration: 101.088000 msec
D/NATIVE_CODE: 73 duration: 101.943000 msec
D/NATIVE_CODE: 74 duration: 99.099000 msec
D/NATIVE_CODE: 75 duration: 99.384000 msec
D/NATIVE_CODE: 76 duration: 91.306000 msec
D/NATIVE_CODE: 77 duration: 93.050000 msec
D/NATIVE_CODE: 78 duration: 92.522000 msec
D/NATIVE_CODE: 79 duration: 91.041000 msec
D/NATIVE_CODE: 80 duration: 99.242000 msec
D/NATIVE_CODE: 81 duration: 99.460000 msec
D/NATIVE_CODE: 82 duration: 99.413000 msec
D/NATIVE_CODE: 83 duration: 99.119000 msec
D/NATIVE_CODE: 84 duration: 102.010000 msec
D/NATIVE_CODE: 85 duration: 98.925000 msec
D/NATIVE_CODE: 86 duration: 99.047000 msec
D/NATIVE_CODE: 87 duration: 95.927000 msec
D/NATIVE_CODE: 88 duration: 95.569000 msec
D/NATIVE_CODE: 89 duration: 94.219000 msec
D/NATIVE_CODE: 90 duration: 96.446000 msec
D/NATIVE_CODE: 91 duration: 96.937000 msec
D/NATIVE_CODE: 92 duration: 95.171000 msec
D/NATIVE_CODE: 93 duration: 90.502000 msec
D/NATIVE_CODE: 94 duration: 97.297000 msec
D/NATIVE_CODE: 95 duration: 100.063000 msec
D/NATIVE_CODE: 96 duration: 97.878000 msec
D/NATIVE_CODE: 97 duration: 97.199000 msec
D/NATIVE_CODE: 98 duration: 99.090000 msec
D/NATIVE_CODE: 99 duration: 92.561000 msec
</code></pre>