My Experience Installing and Debugging Tensorflow for Cuda 10.2
By: Michael Deleo
Tags: Bug Fixes, Guides And Tutorials, Cuda, Cudnn, TensorFlow

Building tensorflow from source

The next part of my philosophy is to accept that brute force won’t work, and I need to get my hands dirty…we’ll build tensorflow from source. TODO Link: On the tensorflow site, it is stated that you can build 1.14 from source for Cuda 10.2.

My caveat here is that this process is long, and it may not even work for you. It didn’t work for me. I would recommend using the docker container *TODO LINK*. But if you must build it, then make sure you have Cudnn installed (of some version), and discoverable in your installed directory for Cuda.

Cudnn

Download the runtime and developer for Ubuntu 18.04

If these instructions are outdated, then you can refer to NVIDIA’s guide for installing Cudnn.

NOTE: These packages must be added in this order, otherwise you will get errors.



sudo dpkg -i

sudo apt-get update sudo dpkg -i libcudnn8_8.0.5.39-1+cuda10.2_amd64.deb sudo dpkg -i libcudnn8-dev_8.0.5.39-1+cuda10.2_amd64.deb sudo apt-get update

Then verify the install

Copy the cuDNN samples to a writable path.

cp -r /usr/src/cudnn_samples_v8/ $HOME

Go to the writable path.

cd $HOME/cudnn_samples_v8/mnistCUDNN

Compile the mnistCUDNN sample.

make clean && make

Run the mnistCUDNN sample.

./mnistCUDNN

If cuDNN is properly installed and running on your Linux system, you will see a message similar to the following:

Test passed!

If this is not working, I would go over what you did to install Cudnn. Then I would check out Nvidia’s guide.

Tensorflow Configure.py Issue

If you are like me, you will get this error on configuring Tensorflow

Traceback (most recent call last):

File "./configure.py", line 1554, in <module>

main()

File "./configure.py", line 1437, in main

if validate_cuda_config(environ_cp):

File "./configure.py", line 1318, in validate_cuda_config

config = dict(

ValueError: dictionary update sequence element #9 has length 1; 2 is required

What does this mean?? Well, it means that a dictionary is being created with a key and not a value. This will similarly throw that error

>>> dict("hello") Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: dictionary update sequence element #0 has length 1; 2 is required

But still, why does a perfectly written package throw this error? Well tensorflow has no error catching there, and this issue has been passed around on their issues for a while. Here is the github issues link. It means that tensorflow cannot find your cudnn version or some other parameter.

Your input parameters to the configure may look like this

cublas_include_dir: /usr/local/cuda/include

cublas_library_dir: /usr/local/cuda/lib64

cuda_binary_dir: /usr/local/cuda/bin

cuda_include_dir: /usr/local/cuda/include

cuda_library_dir: /usr/local/cuda/lib64

cuda_toolkit_path: /usr/local/cuda

cuda_version: 10.2

cudnn_include_dir: /usr/local/cuda/include

cudnn_library_dir: /usr/local/cuda/lib64

cudnn_version:

cupti_include_dir: /usr/local/cuda/include

cupti_library_dir: /usr/local/cuda/lib64

nvvm_library_dir: /usr/local/cuda/nvvm/libdevice

If the issue is still not fixed, you will need to change the configure.py file as such. Lines 1399, 1400:

config = dict(tuple(line.decode('ascii').rstrip().split(': ')) for line in proc.stdout)

Should be changed to:

config = dict([tuple(line.decode('ascii').rstrip().split(': ')) for line in proc.stdout])

Notice the square brackets. This is giving the dict() constructor a list of tuples, instead of just the tuples, which threw errors on my machine and was fine when corrected with this fix.

You need to link the cudnn.h file in /usr/include/cudnn.h as well as all of the other cudnn files in /usr/include/ to /usr/local/cuda/include/. If this doesn’t work, then go to /tensorflow/python third_party/gpus/find_cuda_config.py line 344

header_path, header_version = _find_header(base_paths, "cudnn.h",

You can change “cudnn.h” to “cudnn_version.h”, or edit your cudnn.h file to have the macro definition of your version. This will allow tensorflow to see what cudnn version you have and the configuration will proceed.

From there proceed to follow the build tutorial.

Takeaways from doing a Build

Don’t do it. I spent hours getting it to work and could not, and I would not recommend it unless you are good in that sort of work.