Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary): not sure

TensorFlow version (use command below): v1.12.0-0-ga6d8ffa

Python version: 3.6.6

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version: CUDA 10.0

GPU model and memory: GeForce GTX 1080 Ti, 11178MiB

Describe the current behavior

When computing a batched matmul with 32-bit precision and batch size >= 2 ** 16, the first 2 ** 16 - 1 batch elements of the result are correct, and the remaining elements are arbitrary - often zero, but not always. When the sess.run call is made multiple times, the result is usually the same or similar. The bug only seems to occur when using a variable or placeholder.

Describe the expected behavior

Every batch element of the result should be correct.

Code to reproduce the issue

import numpy as np
import tensorflow as tf

s = (100000, 1, 1)
p = tf.placeholder(shape=s, dtype=tf.float32)
x = tf.matmul(tf.ones(s), p)

with tf.Session() as sess:
r = sess.run(x, feed_dict={p: np.ones(s, dtype=np.float32)})
print(r[2**16 - 5:2**16 + 5, 0, 0])

Typical output:

[1. 1. 1. 1. 0. 0. 0. 0. 0. 0.]

Other info / logs

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary): binary

TensorFlow version (use command below): v1.14.0-rc1-22-gaf24dc91b5 1.14.0

Python version: 3.6.7

GCC/Compiler version (if compiling from source): 8.2.0

CUDA/cuDNN version: cuda-10.0

GPU model and memory:

Describe the current behavior

tf.matmul on tensorflow-gpu gave wrong results. Here is a simplified version of the code.

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

r = [[ 1.0, 0.0], [0.0, 1.0]]

x, y = np.meshgrid(list(range(400)), list(range(400)))
coords = np.stack([x,y],-1).reshape((400,400,2,1))
coords = tf.convert_to_tensor(coords,dtype=tf.float32)

r1 = tf.constant(r)

newCoords = tf.matmul(r1, coords)

sess = tf.Session()
ret = sess.run(newCoords,feed_dict={r1:r})

plt.matshow(ret[:,:,0,0])
plt.show()

When I ran it on my tensorflow-gpu, here is the result:

Looks like it stops computing halfway through and gave the rest 0 as a result.

Describe the expected behavior

Here is the result with CPU:

Below is my old post. Initially, I thought the problem was related to TFRecord, but seems like this problem occurs without even using tfrecord too. For completeness, I keep the old example code with tfrecord.

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

def parser(serialized_example):
fs = tf.io.parse_single_example(
serialized_example,
features={ "r": tf.FixedLenFeature([4], tf.float32) })
fs["r"] = tf.reshape(fs["r"], [2, 2])
return fs

r = [[ 1.0, 0.0],[0.0, 1.0]]

with tf.io.TFRecordWriter("cc.test") as tfrecord_writer:
feature = {"r": tf.train.Feature(float_list=tf.train.FloatList(value=np.array(r).flatten() ))}
example = tf.train.Example(features=tf.train.Features(feature=feature))
tfrecord_writer.write(example.SerializeToString())
dataset = tf.data.TFRecordDataset(["cc.test"])
dataset = dataset.map(parser).repeat().make_one_shot_iterator()
features = dataset.get_next()

x, y = tf.meshgrid(list(range(400)), list(range(400)))
coords = tf.stack([x, y], -1) #(h,w,2)
coords = tf.expand_dims(tf.cast(coords,tf.float32),-1) #(h,w,2,1)

r1 = features["r"]
r2 = tf.constant(r)

newCoords = tf.matmul(r1, coords)

sess = tf.Session()
ret = sess.run(newCoords[:,:,0,0])
plt.matshow(ret)
plt.show()

The code will create "cc.test" file to save the variable r and load it as r1. Then matmul r1 with some big varibles.