memory benchmarking: should the cudnn kernels loading be included
Author: stas00
Pull request: no
Labels: name: wontfix

Bringing this discussion from slack to a dedicated issue so it's easier to track and discuss:

Here is the gist of the discussion so far:

I said:

Currently `benchmark._measure_memory()` doesn't take into account cudnn kernel loading, so even if you measure the most trivial function on pytorch you end up with ~0.6 (Titan X) to 1GB (Tesla T4) reported.

Usually, I deal with this issue by first doing: `torch.ones((1, 1)).cuda()` and then doing any measuring.

For example here is a rough prototype on a new test I'm working to do regressions on mem/speed for basic functionality:
https://colab.research.google.com/drive/1n6J3tc8FT4ER1vBCTAtU4__U5Px2mxwI?usp=sharing
All it does is measuring memory consumption and speed for init, fwd and fwd+bwd - hope it's easy to read.

As you can see I had to first measure a baseline with the cheat I mentioned above, and then subtract it from all the subsequent memory measurements.

----

@patrickvonplaten suggested that it is that way so that:

a) The number includes all the required memory to run a model.
b) better comparison with tensorflow

---

fwd+init is fwd+init+cudnn load. In order to measure fwd+init, you need to load cudnn kernels first. You can do multiple concurrent inits and cudnn overheard will be happening only once.

I think the API should provide for flexible approach, by allowing:

1. show me the bottom line for any operation - i.e. how much memory was consumed when `func` is run
2. show me the delta, subtracting the cudnn overhead which happens once

Obviously, the caller can ask the benchmarking function to do all those measurements, including the manual measurement of cudnn overhead, and then do the accounting herself. But perhaps it could have an optional argument that would perform something like torch.ones((1, 1)).cuda() and substract that.

Well, perhaps the simplest approach is to just have a way to query how much memory cudnn loading takes and then leave the rest of it as it is, so the caller will do all the accounting. And perhaps if such accounting is repetitive than it can be abstracted into another layer.

It might be easier to show in code, so a new shortcut is added to measure the loading of cudnn:
```
mem, _ = benchmark._measure_memory_cudnn_load()
mem_load = mem.bytes - mem_load
mem, _ = benchmark._measure_memory(func)
mem_diff = mem.bytes - mem_load
```
Now this can be wrapped into:
```
mem, _ = benchmark._measure_memory_delta(func)
mem_diff = mem.bytes - mem_load
```
does this make sense?

---

So this is where we are at, thoughts?

Comments:
Thinking more about it. Perhaps it's a question of benchmarking the whole vs. parts. 

For example, if we want to memory profile a specific function, measuring the whole (that is including everything else that came before the function) could give us wrong results, because perhaps some parts of the whole got bigger, while others smaller, and the total outcome is unpredictable.

Therefore, if a purpose of a memory consumption regression test is so see whether the whole thing (say finetuning) is still in the ballpark of certain baseline, then measuring the whole is good enough. If however we want to make sure that each major component  (init/fwd/fwd+bwd/inference) remains lean, then we should measure and track the specific deltas.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Still an open issue

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.