Issue 1:
I was trying to add profiler to the existing tensorflow object detection API, the following part of code is added to

tensorflow/tensorflow/contrib/slim/python/slim/learning.py

Line 489
in
dfa0f87

if run_metadata is not None:
logging.info('Added profiling by User')
ProfileOptionBuilder = profiler.ProfileOptionBuilder
opts = ProfileOptionBuilder(ProfileOptionBuilder.time_and_memory()).with_step(np_global_step).with_timeline_output(train_step_kwargs['logdir']+'/profile.json').build()
logging.info(train_step_kwargs['logdir']+'/profile.json')
profiler.profile(
ops.get_default_graph(),
run_meta=run_metadata,
cmd='graph',
options=opts)

but when I was trying to run the training for resnet_101_coco with no other changes, the training progress with no error, but the profiler output incompatible shapes:

Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 300, 453) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 300, 453) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 300, 453) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/pool1/MaxPool incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/shortcut/Conv2D incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/shortcut/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_1/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_2/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/shortcut/MaxPool incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 150, 227) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/Pad incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 152, 229) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 64) and (2, 64, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block1/unit_3/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 256) and (2, 256, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/shortcut/Conv2D incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/shortcut/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_1/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_2/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv1/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv1/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv2/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv2/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv2/Relu incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv3/Conv2D incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/conv3/BatchNorm/FusedBatchNorm incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/add incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_3/bottleneck_v1/Relu incompatible shapes: Shapes (2, ?, ?, 512) and (2, 512, 75, 114) are not compatible.
Node clone_3/FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/block2/unit_4/bottleneck_v1/conv1/Conv2D incompatible shapes: Shapes (2, ?, ?, 128) and (2, 128, 75, 114) are not compatible.
Node

Issue 2:
Hi! I was using the official minist summaries example which uses run_metadata to track model runtime statistics:

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/mnist_with_summaries.py

I tried to add simple profiler python API as suggest by

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/profiler/g3doc/python_api.md

So, my code is:

https://github.com/MrWanter/mnist-proflie/blob/c0f5b59403fbf3258e8236fa7c983750623354e6/mnist-profile.py#L175-L184

but running the model gives

Added profiling by user
Parsing Inputs...
Incomplete shape.
Incomplete shape.
Options -step=199 not found.
Available steps: 0
Adding run metadata for 199

but tensorboard shows the memory consumption of nodes, which suggests runtime statistics are captured:

what is incomplete shape, the input shape has None for batch size, but since it is runtime statistics, I thought it would work, but it does not, can someone help?

As in #19962, it is hard to know how to deal with profiler problem since documentation is not enough and codes are hard to read as it is implemented with C++ and heavily wrapped

UPDATA:

Th problem is caused by input shape of None for batch size, change it to fixed shape solve the problem, but there would inevitably be a large amount of None in tensors since we have to deal with variable sized inputs, in those cases, it should be reasonable for the profiler to capture runtime statistics, so how can we do it? where am I wrong?

UPDATE

Found a quote on the official profiler doc:

It must have known "shape" information for RegisterStatistics('flops') to calculate the statistics. It is suggested to pass in -run_meta_path if shape is only known during runtime. tfprof can fill in the missing shape with the runtime shape information from RunMetadata. Hence, it is suggested to use -account_displayed_op_only option so that you know the statistics are only for the operations printed out.

seems profiler can fill in the missing shape with runtime shape information, but how to pass in - run_meta_path, I did not find anything relevant in the profiler options:

self._options = {'max_depth': 100,
'min_bytes': 0,
'min_micros': 0,
'min_params': 0,
'min_float_ops': 0,
'min_occurrence': 0,
'order_by': 'name',
'account_type_regexes': ['.*'],
'start_name_regexes': ['.*'],
'trim_name_regexes': [],
'show_name_regexes': ['.*'],
'hide_name_regexes': [],
'account_displayed_op_only': False,
'select': ['micros'],
'step': -1,
'output': 'stdout'}