Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): nvidia-docker tensor:19.02-py3 (Ubuntu 16.04)

TensorFlow installed from (source or binary): Source

TensorFlow version (use command below): v1.14.0-rc1-22-gaf24dc91b5 1.14.0

Python version: Python 3.5.2

Bazel version (if compiling from source): NA

GCC/Compiler version (if compiling from source): NA

CUDA/cuDNN version: 10.0.130 / 7.4.2

GPU model and memory: Tesla V100 32GB

Describe the current behavior

I am trying to optimize (decrease) the inference time and model size of my Tiny Yolov3 model. I currently have it as a frozen graph. When I run the timing and evaluation script from here, fps actually decreases as the optimizations are tried, and file sizes stay constant.

Describe the expected behavior

I expect file sizes and inference time to decrease.

Code to reproduce the issue

I used this nvidia-docker image.

This script with a small preprocessing change:

def resize_image(image, target_height, target_width):
image = image.resize((target_width, target_height), Image.BICUBIC).convert('RGB')
return np.array(image)

def preprocess_image(file_name, output_height=416, output_width=416,
num_channels=3):
return resize_image(Image.open(file_name), output_height, output_width).astype('float32')

I also removed the printing of predictions, as the script is built for a classifier rather than a detector.

Other info / logs

Command executed (image file is attached; model is here):

python timed.py --frozen_graph=tiny-yolov3_frozen.pb \
--image_file=11075842.jpg \
--native --fp32 --fp16 --int8 \
--output_dir=/workspace \
--input_node=inputs --output_node=output_boxes

==========================
network: native_tiny-yolov3_frozen.pb, batchsize 256, steps 100
fps median: 993.1, mean: 959.8, uncertainty: 8.9, jitter: 7.7
latency median: 0.25778, mean: 0.26986, 99th_p: 0.43523, 99th_uncertainty: 0.05783

==========================
network: tftrt_fp32_tiny-yolov3_frozen.pb, batchsize 256, steps 100
fps median: 912.7, mean: 896.3, uncertainty: 8.2, jitter: 4.6
latency median: 0.28048, mean: 0.29338, 99th_p: 0.42830, 99th_uncertainty: 0.43877

==========================
network: tftrt_fp16_tiny-yolov3_frozen.pb, batchsize 256, steps 100
fps median: 823.5, mean: 831.9, uncertainty: 8.4, jitter: 96.2
latency median: 0.31087, mean: 0.31129, 99th_p: 0.43070, 99th_uncertainty: 0.07565

==========================
network: tftrt_int8_tiny-yolov3_frozen.pb, batchsize 256, steps 100
fps median: 803.3, mean: 820.6, uncertainty: 9.9, jitter: 19.9
latency median: 0.31867, mean: 0.31814, 99th_p: 0.60985, 99th_uncertainty: 0.01011

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04, nvcr.io/nvidia/tensorrt:19.02-py3

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: NA

TensorFlow installed from (source or binary): binary

TensorFlow version (use command below): v1.12.1-7024-g24b3e6cf73 1.15.0-dev20190725

Python version: 3.5.2

Bazel version (if compiling from source): NA

GCC/Compiler version (if compiling from source): NA

CUDA/cuDNN version: 10.0.130 / 7.4.2 as per the above linked container

GPU model and memory: Tesla V100 32GB

Relevant output from pip freeze:

tf-estimator-nightly==1.14.0.dev2019072201
tf-nightly-gpu==1.15.0.dev20190725

Describe the current behavior

Inference Speed (frames per second)

Model
Native
FP32
FP16
INT8

tiny-yolo
348
333
402
415

big-yolo
125
140
243
208

1. Why is there a slowdown for tiny native->FP32? (@pooyadavoodi same as here)

2. Why is this slowdown not consistent for big?

3. Why is there a slowdown for big FP16->INT8?

Model Size (megabytes)

Model
Native
FP32
FP16
INT8

tiny-yolo
35
67
44
51

big-yolo
238
439
288
332

I understand that there is currently an issue where new graph weights are saved twice to the .pb file (#30717, #30789). Once the weights in the table are adjusted for this double weight saving, the resulting sizes for fp32 and fp16 seem reasonable.

1. Why is there an increase in size for fp16->int8?

Describe the expected behavior

I am trying to quantize two different YOLO models (one tiny, one normal) with TensorRT. The goals of this quantization are:

speed up inference

decrease model size

As quantization and conversion proceeds from native->fp32->fp16->int8, I expect inference time to decrease (FPS to increase), and model size to decrease.

Code to reproduce the issue

I am using this script and a few helper functions from here. The two exact scripts that I use are do.py and utilities.py. Here are the tiny model and the big model.

My command for running the experiments:

python do.py \
--frozen_graph=big-yolov3_frozen.pb \ # or tiny-yolov3_frozen.pb
--native --fp32 --fp16 --int8 \
--batch_size=32 \ # or 128 for tiny
--output_dir=/workspace \
--input_node=inputs --output_node=output_boxes

Other info / logs

I ran a couple of experiments just to make sure that the results were consistent.

==========================
network: native_tiny-yolov3_frozen.pb, batchsize 128, steps 100
fps median: 350.2, mean: 348.0, uncertainty: 1.4, jitter: 5.1
latency median: 0.36551, mean: 0.36846, 99th_p: 0.42946, 99th_uncertainty: 0.01453

==========================
network: tftrt_fp32_tiny-yolov3_frozen.pb, batchsize 128, steps 100
fps median: 340.9, mean: 333.4, uncertainty: 1.5, jitter: 4.4
latency median: 0.37546, mean: 0.38470, 99th_p: 0.47469, 99th_uncertainty: 0.06110

==========================
network: tftrt_fp16_tiny-yolov3_frozen.pb, batchsize 128, steps 100
fps median: 403.3, mean: 402.3, uncertainty: 0.6, jitter: 3.6
latency median: 0.31740, mean: 0.31824, 99th_p: 0.34266, 99th_uncertainty: 0.00263

==========================
network: tftrt_int8_tiny-yolov3_frozen.pb, batchsize 128, steps 100
fps median: 417.7, mean: 414.9, uncertainty: 1.1, jitter: 4.4
latency median: 0.30641, mean: 0.30873, 99th_p: 0.35451, 99th_uncertainty: 0.01511

==========================
network: native_big-yolov3_frozen.pb, batchsize 32, steps 100
fps median: 125.2, mean: 124.7, uncertainty: 0.3, jitter: 1.4
latency median: 0.25553, mean: 0.25677, 99th_p: 0.28257, 99th_uncertainty: 0.00308

==========================
network: tftrt_fp32_big-yolov3_frozen.pb, batchsize 32, steps 100
fps median: 140.3, mean: 140.2, uncertainty: 0.4, jitter: 1.9
latency median: 0.22802, mean: 0.22839, 99th_p: 0.25419, 99th_uncertainty: 0.00890

==========================
network: tftrt_fp16_big-yolov3_frozen.pb, batchsize 32, steps 100
fps median: 237.6, mean: 242.5, uncertainty: 1.4, jitter: 5.4
latency median: 0.13469, mean: 0.13245, 99th_p: 0.17733, 99th_uncertainty: 0.04387

==========================
network: tftrt_int8_big-yolov3_frozen.pb, batchsize 32, steps 100
fps median: 210.1, mean: 207.5, uncertainty: 1.5, jitter: 2.7
latency median: 0.15231, mean: 0.15657, 99th_p: 0.16928, 99th_uncertainty: 0.16613

==========================
network: native_tiny-yolov3_frozen.pb, batchsize 128, steps 100
fps median: 357.4, mean: 354.3, uncertainty: 1.7, jitter: 14.5
latency median: 0.35814, mean: 0.36215, 99th_p: 0.44575, 99th_uncertainty: 0.00629

==========================
network: tftrt_fp32_tiny-yolov3_frozen.pb, batchsize 128, steps 100
fps median: 324.9, mean: 319.4, uncertainty: 1.4, jitter: 2.5
latency median: 0.39401, mean: 0.40173, 99th_p: 0.49218, 99th_uncertainty: 0.06484

==========================
network: tftrt_fp16_tiny-yolov3_frozen.pb, batchsize 128, steps 100
fps median: 376.1, mean: 372.8, uncertainty: 1.1, jitter: 1.7
latency median: 0.34036, mean: 0.34363, 99th_p: 0.38601, 99th_uncertainty: 0.02051

==========================
network: tftrt_int8_tiny-yolov3_frozen.pb, batchsize 128, steps 100
fps median: 392.1, mean: 391.3, uncertainty: 0.5, jitter: 1.8
latency median: 0.32645, mean: 0.32717, 99th_p: 0.33765, 99th_uncertainty: 0.01967

==========================
network: native_big-yolov3_frozen.pb, batchsize 32, steps 100
fps median: 124.3, mean: 124.0, uncertainty: 0.4, jitter: 1.7
latency median: 0.25737, mean: 0.25842, 99th_p: 0.31292, 99th_uncertainty: 0.00355

==========================
network: tftrt_fp32_big-yolov3_frozen.pb, batchsize 32, steps 100
fps median: 141.0, mean: 140.7, uncertainty: 0.3, jitter: 0.3
latency median: 0.22690, mean: 0.22761, 99th_p: 0.24239, 99th_uncertainty: 0.00581

==========================
network: tftrt_fp16_big-yolov3_frozen.pb, batchsize 32, steps 100
fps median: 247.4, mean: 245.9, uncertainty: 1.0, jitter: 4.4
latency median: 0.12934, mean: 0.13044, 99th_p: 0.16018, 99th_uncertainty: 0.02011

==========================
network: tftrt_int8_big-yolov3_frozen.pb, batchsize 32, steps 100
fps median: 206.0, mean: 204.5, uncertainty: 1.4, jitter: 1.0
latency median: 0.15536, mean: 0.15885, 99th_p: 0.16454, 99th_uncertainty: 0.17423