Error when executing resnet-training

bucky: <p>./bin/resnet-training -backend=CPU -resnet50=resnet50 file.txt</p>
<p>contents of file.txt:<br>
tests/images/imagenet/cat_285.png,285<br>
tests/images/imagenet/dog_207.png,207<br>
tests/images/imagenet/zebra_340.png,340</p>
<p>here is the error:</p>
<p>Loading resnet50 model.<br>
WARNING: Logging before InitGoogleLogging() is written to STDERR<br>
I0316 18:32:00.168864 11616 Graph.cpp:5880] Layout requirements checking is enabled<br>
Preparing for training.<br>
Differentiating graph.<br>
Compiling backend.<br>
I0316 18:32:00.199743 11616 Partitioner.cpp:482] The model is too small for applying partition.<br>
Model size : 103058848<br>
Backend Name : CPU<br>
Device memory: 2000000000<br>
I0316 18:32:00.702248 11616 Partitioner.cpp:88] The number of partitions is : 2<br>
I0316 18:32:00.702265 11616 PartitionerUtils.cpp:549] 	 Partition 0:<br>
Name :	resnet50<br>
BackendKind :	CPU<br>
context count :	1<br>
total Memory :	103062848<br>
input size:	103042720<br>
input count :	268<br>
input only from peers count :	267<br>
output size:	4000<br>
constant size:	16128<br>
I0316 18:32:00.702288 11616 PartitionerUtils.cpp:570] 		 LogicalDeviceIDs :	0<br>
I0316 18:32:00.702291 11616 PartitionerUtils.cpp:549] 	 Partition 1:<br>
Name :	resnet50_grad<br>
BackendKind :	CPU<br>
context count :	1<br>
total Memory :	409959712<br>
input size:	205270848<br>
input count :	429<br>
input only from peers count :	0<br>
output size:	204672736<br>
constant size:	16128<br>
I0316 18:32:00.702301 11616 PartitionerUtils.cpp:570] 		 LogicalDeviceIDs :	0<br>
Training - epoch <span class="hashtag">#0</span> from total 32<br>
Segmentation fault (core dumped)</p>
<p>when i tried with the opencl backend:</p>
<p>Loading resnet50 model.<br>
WARNING: Logging before InitGoogleLogging() is written to STDERR<br>
I0316 21:45:21.926693 31914 Graph.cpp:5880] Layout requirements checking is enabled<br>
Preparing for training.<br>
Differentiating graph.<br>
Compiling backend.<br>
I0316 21:45:21.972465 31914 Partitioner.cpp:482] The model is too small for applying partition.<br>
Model size : 103058848<br>
Backend Name : OpenCL<br>
Device memory: 16943087616<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: AvgPoolGrad<br>
name : gpu_0_pool5__1_grad<br>
Input : float&lt;1 x 7 x 7 x 2048&gt;<br>
OriginalOutputForResult : float&lt;1 x 1 x 1 x 2048&gt;<br>
GradOfOriginalOutputNamedResult : float&lt;1 x 1 x 1 x 2048&gt;<br>
Kernels : [7, 7]<br>
Strides : [1, 1]<br>
Pads : [0, 0, 0, 0]<br>
Layout : 0<br>
CountIncludePads : 1<br>
users : 1<br>
GradOfInputNamedInput : float&lt;1 x 7 x 7 x 2048&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res5_2_branch2c__6<br>
Input : bool&lt;1 x 7 x 7 x 2048&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 2048 x 7 x 7&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res5_2_branch2b__5<br>
Input : bool&lt;1 x 7 x 7 x 512&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 512 x 7 x 7&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res5_2_branch2a__5<br>
Input : bool&lt;1 x 7 x 7 x 512&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 512 x 7 x 7&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res5_1_branch2c__6<br>
Input : bool&lt;1 x 7 x 7 x 2048&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 2048 x 7 x 7&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res5_1_branch2b__5<br>
Input : bool&lt;1 x 7 x 7 x 512&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 512 x 7 x 7&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res5_1_branch2a__5<br>
Input : bool&lt;1 x 7 x 7 x 512&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 512 x 7 x 7&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res5_0_branch2c__6<br>
Input : bool&lt;1 x 7 x 7 x 2048&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 2048 x 7 x 7&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res5_0_branch2b__5<br>
Input : bool&lt;1 x 7 x 7 x 512&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 512 x 7 x 7&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res5_0_branch2a__5<br>
Input : bool&lt;1 x 14 x 14 x 512&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 512 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_5_branch2c__6<br>
Input : bool&lt;1 x 14 x 14 x 1024&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 1024 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_5_branch2b__5<br>
Input : bool&lt;1 x 14 x 14 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_5_branch2a__5<br>
Input : bool&lt;1 x 14 x 14 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_4_branch2c__6<br>
Input : bool&lt;1 x 14 x 14 x 1024&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 1024 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_4_branch2b__5<br>
Input : bool&lt;1 x 14 x 14 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_4_branch2a__5<br>
Input : bool&lt;1 x 14 x 14 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_3_branch2c__6<br>
Input : bool&lt;1 x 14 x 14 x 1024&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 1024 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_3_branch2b__5<br>
Input : bool&lt;1 x 14 x 14 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_3_branch2a__5<br>
Input : bool&lt;1 x 14 x 14 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_2_branch2c__6<br>
Input : bool&lt;1 x 14 x 14 x 1024&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 1024 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_2_branch2b__5<br>
Input : bool&lt;1 x 14 x 14 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_2_branch2a__5<br>
Input : bool&lt;1 x 14 x 14 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_1_branch2c__6<br>
Input : bool&lt;1 x 14 x 14 x 1024&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 1024 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_1_branch2b__5<br>
Input : bool&lt;1 x 14 x 14 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_1_branch2a__5<br>
Input : bool&lt;1 x 14 x 14 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_0_branch2c__6<br>
Input : bool&lt;1 x 14 x 14 x 1024&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 1024 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_0_branch2b__5<br>
Input : bool&lt;1 x 14 x 14 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 14 x 14&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res4_0_branch2a__5<br>
Input : bool&lt;1 x 28 x 28 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 28 x 28&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res3_3_branch2c__6<br>
Input : bool&lt;1 x 28 x 28 x 512&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 512 x 28 x 28&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res3_3_branch2b__5<br>
Input : bool&lt;1 x 28 x 28 x 128&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 128 x 28 x 28&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res3_3_branch2a__5<br>
Input : bool&lt;1 x 28 x 28 x 128&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 128 x 28 x 28&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res3_2_branch2c__6<br>
Input : bool&lt;1 x 28 x 28 x 512&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 512 x 28 x 28&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res3_2_branch2b__5<br>
Input : bool&lt;1 x 28 x 28 x 128&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 128 x 28 x 28&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res3_2_branch2a__5<br>
Input : bool&lt;1 x 28 x 28 x 128&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 128 x 28 x 28&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res3_1_branch2c__6<br>
Input : bool&lt;1 x 28 x 28 x 512&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 512 x 28 x 28&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res3_1_branch2b__5<br>
Input : bool&lt;1 x 28 x 28 x 128&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 128 x 28 x 28&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res3_1_branch2a__5<br>
Input : bool&lt;1 x 28 x 28 x 128&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 128 x 28 x 28&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res3_0_branch2c__6<br>
Input : bool&lt;1 x 28 x 28 x 512&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 512 x 28 x 28&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res3_0_branch2b__5<br>
Input : bool&lt;1 x 28 x 28 x 128&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 128 x 28 x 28&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res3_0_branch2a__5<br>
Input : bool&lt;1 x 56 x 56 x 128&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 128 x 56 x 56&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res2_2_branch2c__6<br>
Input : bool&lt;1 x 56 x 56 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 56 x 56&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res2_2_branch2b__5<br>
Input : bool&lt;1 x 56 x 56 x 64&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 64 x 56 x 56&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res2_2_branch2a__5<br>
Input : bool&lt;1 x 56 x 56 x 64&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 64 x 56 x 56&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res2_1_branch2c__6<br>
Input : bool&lt;1 x 56 x 56 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 56 x 56&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res2_1_branch2b__5<br>
Input : bool&lt;1 x 56 x 56 x 64&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 64 x 56 x 56&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res2_1_branch2a__5<br>
Input : bool&lt;1 x 56 x 56 x 64&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 64 x 56 x 56&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res2_0_branch2c__6<br>
Input : bool&lt;1 x 56 x 56 x 256&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 256 x 56 x 56&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res2_0_branch2b__5<br>
Input : bool&lt;1 x 56 x 56 x 64&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 64 x 56 x 56&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_res2_0_branch2a__5<br>
Input : bool&lt;1 x 56 x 56 x 64&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 64 x 56 x 56&gt;<br>
Unsupported node found while compiling Function resnet50_grad for backend OpenCL: Transpose<br>
name : gpu_0_conv1__5<br>
Input : bool&lt;1 x 112 x 112 x 64&gt;<br>
Shuffle : [0, 3, 1, 2]<br>
Layout : NCHW<br>
users : 1<br>
Result : bool&lt;1 x 64 x 112 x 112&gt;<br>
F0316 21:45:22.983700 31914 Error.cpp:121] exitOnError(Error) got an unexpected ErrorValue:<br>
Error code: COMPILE_UNSUPPORTED_NODE_AFTER_OPTIMIZE<br>
Error message: Unsupported node(s) found after optimizing Function resnet50_grad for backend OpenCL</p>

jfix: <p>So, a few things:</p>
<ul>
<li>Running on CPU and seg faulting clearly is bad. I haven’t tried reproducing this yet, but curious if you have tried running in dbg mode to see if an assert is hit, or if you have a stack trace.</li>
<li>On OpenCL, it seems AvgPoolGrad isn’t supported for the OpenCL backend. So that would need to be added to use the OpenCL backend.</li>
<li>Also on OpenCL it looks like there are a lot of transposes of booleans. That would also need to be added (should be very simple since there is already <code>ElemKind::Int8QTy</code> support).</li>
</ul>
<p>In general Glow is more focused on inference than training as of late, so I imagine there isn’t nearly as much coverage on the training side of things. Apologies in advance!</p>