Issue 1:
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Ubuntu 16.04 (ppc64le)

TensorFlow installed from (source or binary):

Installed from source (v1.0.1)

TensorFlow version (use command below):

('v1.0.1-0-ge895d5c-dirty', '1.0.1')

Bazel version (if compiling from source):

0.4.4-2017-05-26 (@80a07b5)

CUDA/cuDNN version:

CUDA = 8.0 and cuDNN = 5.1

GPU model and memory:

GPU 0: Tesla P100-SXM2-16GB

GPU 1: Tesla P100-SXM2-16GB

Exact command to reproduce:

bazel test --config=opt --config=cuda //tensorflow/core/kernels:sparse_matmul_op_test_gpu

Describe the problem

This is regarding failure of test case SparseMatmulOpTest.BroadcastPacketTest in tensorflow/core/kernels/sparse_matmul_op_test.cc file.While executing this test case on ppc64le, it was observed that following line returns unexpected results:

https://github.com/tensorflow/tensorflow/blob/v1.0.1/tensorflow/core/kernels/sparse_matmul_op_test.cc#L255

internal::pstoreu(data2, internal::pbroadcast_first<Packet>(
internal::ploadu<Packet>(data1)));

Here we are getting expected result on x86 for data2 array = [0.170094 0.170094 0.170094 0.170094], however on ppc64le getting incorrect result i.e. [ 0.170094 0.14922 -0.0823886 0.026985]

I have done some investigation around this - using print/cout statement I tried to understand the code flow on ppc64le as well as on X86 platform.

Here I found that for internal::pbroadcast_first<Packet> line ,on both the platform executed different functions, see below-

On x86 executed EIGEN_STRONG_INLINE Packet4f pbroadcast_first<Packet4f>(const Packet4f& a) function(https://github.com/tensorflow/tensorflow/blob/v1.0.1/tensorflow/core/kernels/sparse_matmul_op.h#L197)

And on ppc64le executed some different function i.e. EIGEN_DEVICE_FUNC inline Packet pbroadcast_first(const Packet& a) (https://github.com/tensorflow/tensorflow/blob/v1.0.1/tensorflow/core/kernels/sparse_matmul_op.h#L92)

That is why we are getting incorrect result on ppc64le for data2 array and test fails.

I have done some debugging on this but couldn't find the reason - why control is going to different functions on both the platform for internal::pbroadcast_first<Packet> ?

If could get any suggestions/pointers to why this is happening that would be great!

(I am new to tensorflow code but interested/would-like-to debug & help)

Thanks!

Source code / logs

$ bazel test --config=opt --config=cuda //tensorflow/core/kernels:sparse_matmul_op_test_gpu

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
Running main() from test_main.cc
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from SparseMatmulOpTest
[ RUN ] SparseMatmulOpTest.BroadcastPacketTest
[0.170094 0.170094 0.170094 0.170094] != [ 0.170094 0.14922 -0.0823886 0.026985], differences: [ 0 -0.0208738 -0.252482 -0.143109]
tensorflow/core/kernels/sparse_matmul_op_test.cc:257: Failure
Value of: areApprox(ref, data2, PacketSize)
Actual: false
Expected: true
[ FAILED ] SparseMatmulOpTest.BroadcastPacketTest (0 ms)
[ RUN ] SparseMatmulOpTest.InterleavePacketTest
[ OK ] SparseMatmulOpTest.InterleavePacketTest (0 ms)
[ RUN ] SparseMatmulOpTest.Bfloat16ExpandTest
[ OK ] SparseMatmulOpTest.Bfloat16ExpandTest (0 ms)
[ RUN ] SparseMatmulOpTest.Bfloat16LoadTest
[ OK ] SparseMatmulOpTest.Bfloat16LoadTest (0 ms)
[----------] 4 tests from SparseMatmulOpTest (0 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (0 ms total)
[ PASSED ] 3 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SparseMatmulOpTest.BroadcastPacketTest

1 FAILED TEST

Issue 2:
###Environment info:

Operating System: Ubuntu 16.04 (ppc64le)

###Installed version of CUDA and cuDNN: cuda & OpenCL disabled and all other features enabled

###Build and test INFO

Built TF (version 1.0.1) successfully on Ubuntu 16.04 and RHEL 7.3.

Now I am trying to run the bazel tests - 12 tests are failing out of 1044. Please find the attachment for log details.

Thanks!

C-C++Test Failure analysis for TF.xlsx