Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

Code in Golang

Used GO Tensorflow

MobileNet model

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Raspberry Pi (1.4GHz, 1024 RAM, 4 Cores)

TensorFlow version (use command below):

v1.12.0

Describe the current behavior

I have written a Face Classification code which uses MobileNet (depth 80) model. But When I ran the model after 8-10 min, tensorflow is throwing OOM allocation error.

Describe the expected behavior

Because raspberry pi has limited memory. But as the model is very small it has to run perfectly.

Code to reproduce the issue

Run MobileNet model classification in Golang Env

Input tensor shape [5,224,224,3], tried other batch sizes as well 1 and 3 but no luck.

Other info / logs

2019-02-20 10:52:26.403200: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at fused_batch_norm_op.cc:573 : Resource exhausted: OOM when allocating tensor with shape[5,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
2019-02-20 10:52:26.478760: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at pad_op.cc:137 : Resource exhausted: OOM when allocating tensor with shape[5,113,113,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
E0220 10:52:26.483588 3282 classifier.go:203] Failed to classify face: OOM when allocating tensor with shape[5,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
[[{{node mobilenet_1.00_224/conv_pw_1_bn/FusedBatchNorm}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc

Issue 2:
@poxvoculi @penpornk @nhasabni @reedwm @shashishekhar @claynerobison @andydavis1

Hi guys, I have checked some previous issues on bad_allocation and checked all the discussions and I could not reach to a conclusion how to resolve my bad_allocation issue. Below is my details for my issue with log. If I could get some insights, it will be very helpful.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

Written custom code for face image classification. Model accepts 224x224x3 tensor. The model has 4,253,864 parameters and model depth is 88.

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Raspberry Pi, CPU: 1.4Ghz, 980 Mb RAM.

Programming Language

Golang

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy)

No

TensorFlow installed from (source or binary):

Build from source

TensorFlow version (use command below):

v1.12.0

Python version:

NO

Bazel version (if compiling from source):

v0.15.2

GCC/Compiler version (if compiling from source):

6.3

CUDA/cuDNN version:

NO

GPU model and memory:

NO

Describe the current behavior

I am using GoLang enviroment to load Tensorflow model and run. My Tensorflow model is MobileNet.

It does face classification. I faced issue in memory allocation.

Describe the expected behavior

Model loading is fine and runs perfectly for 15-20 min and after that, it throws bad_allocation error and exits.

Code to reproduce the issue

Generate libtensorflow.so and libtensorflow_framework.so artifact for raspberry pi.

Load tensorflow model into Golang Env.

While model is running and after 15-20 min this happens.

Other info / logs

2019-02-06 10:41:44.057443: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

E0206 10:41:44.128595 2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

2019-02-06 10:41:44.313010: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

E0206 10:41:44.314077 2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

2019-02-06 10:41:44.409438: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

E0206 10:41:44.411047 2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

2019-02-06 10:41:44.504644: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

E0206 10:41:44.505833 2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

2019-02-06 10:41:44.601160: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

E0206 10:41:44.602079 2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

I0206 10:41:48.258835 2719 camera_input.go:69] Captured Image, Len Queue 0

I0206 10:41:48.898151 2719 runner.go:85] Detection time 638.594898ms

2019-02-06 10:41:48.997640: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

E0206 10:41:48.999425 2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

2019-02-06 10:41:49.100121: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

E0206 10:41:49.101226 2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

2019-02-06 10:41:49.197890: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops.cc:470 : Resource exhausted: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

E0206 10:41:49.199211 2719 classifier.go:148] Failed to classify face: OOM when allocating tensor with shape[1,112,112,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

[[{{node mobilenet_1.00_224/conv_pw_1/convolution}}]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

terminate called after throwing an instance of 'std::bad_alloc'

what(): std::bad_alloc

Aborted