opened
xilenteyex
username_0
### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow)**: Yes
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 16.04
-   **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue
    happens on a mobile device**: N/A
-   **TensorFlow installed from (source or binary)**: Source
-   **TensorFlow version (use command below)**: r1.14
-   **Python version**: 2.7
-   **Bazel version (if compiling from source)**:
-   **GCC/Compiler version (if compiling from source)**: 5.4
-   **CUDA/cuDNN version**: 10.1/7.1
-   **GPU model and memory**: NVIDIA Corporation GV100GL [Tesla V100 SXM2 16GB] (rev a1)
-   **Exact command to reproduce**: N/A

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with:

```bash
python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
```

### Describe the problem
I am running a Tramsformer Model in which If I add some control dependencies (which are not required for its design, but performance can be improved if I can add those dependencies), I get OOM error. I wanted to log the memory Usage for each TF operation (including CUDA context memory and Memory Allocator Reservations in addition to the raw bytes needed to input and output tensors.)

I looked at profiling using Timeline as suggested in this [StackOverFlow post](https://stackoverflow.com/questions/36123740/is-there-a-way-of-determining-how-much-gpu-memory-is-in-use-by-tensorflow), but this is just logging raw bytes needed by the tensors, not the additional memory reserved or is marked free but is not garbage collected yet.

In addition, I also came across [tf.Profiler](https://github.com/tensorflow/tensorflow/tree/r1.14/tensorflow/core/profiler), but it has this [comment](https://github.com/tensorflow/tensorflow/blob/00fad90125b18b80fe054de1055770cfb8fe4ba3/tensorflow/core/profiler/tfprof_log.proto#L128) saying that numbers are not reliable, which otherwise might have sufficed the requirement. Is there a better way to log all the memory used by each TF operation?
Logging Memory Usage For each TensorFlow Operation
issue
created
saikumarchalla
username_1
@username_0  Please provide the complete standalone code and error log as well to replicate the issue at our end.

Also,could you please update the Tensorflow version to 2.3 since TF 1.14 is not supported actively. Thanks!
comment
created
xilenteyex
username_0
@username_1 thanks for looking into this issue. I am working on creating a standalone example for you to reproduce the issue. Will share ASAP. 

Actually, I am more interested in knowing if there is a way to log per op memory utilization in a GPU (including workspace memory, CUDA context and Allocator Reservations [in addition to memory required by live tensors]). Is this feature available in TensorFlow or will be made available in near future ? Using Timeline or TF-Profiler, I can only get the memory requirements for Tensors, not the additional overheads. If you can point me to logger which will be helpful in this regard, that would be great and maybe sufficient to resolve this GitHub issue.

Also, as far as the version of TF is concerned, it's a research project which I am working for a long time now and we are comfortable with TF 1.14. It will be hard to upgrade. to TF 2.3 promptly, but we will be moving to TF 2.3 later, but it will be hard to switch quickly. Hope you understand.
comment
created
prb12
username_2
I do not work on TensorFlow any more.
comment
created
rmothukuru
username_3
@username_0,
I'm not sure if [this comment](https://github.com/tensorflow/tensorflow/blob/00fad90125b18b80fe054de1055770cfb8fe4ba3/tensorflow/core/profiler/tfprof_log.proto#L128) exactly means that the Statistics shown in the `Tensorboard Profiler` is unreliable. But the Fact is that we can understand which **`Tensorflow Op`** is consuming more `Memory` (compared to other OPs) and it will provide Recommendations to reduce the bottlenecks. 

Please refer [Profiler Guide](https://www.tensorflow.org/guide/profiler) and the Example Tutorial. Thanks!
comment
created
rmothukuru
username_3
@username_0,
Can you please respond to the above comment. Thanks!
comment
closed
rmothukuru
username_3
issue
created
rmothukuru
username_3
Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!
comment