Native build procedure of Tensorflow v2.0.0 for RaspberryPi (Accelerated Tensorflow Lite PythonAPI enabled)

**Tensorflow-bin**　[![GitHub stars](https://img.shields.io/github/stars/PINTO0309/Tensorflow-bin.svg?style=social&label=Stars)](https://github.com/PINTO0309/Tensorflow-bin)
<a href="https://github.com/PINTO0309/Tensorflow-bin"><img src="https://github-link-card.s3.ap-northeast-1.amazonaws.com/PINTO0309/Tensorflow-bin.png" width="460px"></a>
**Bazel_bin**　[![GitHub stars](https://img.shields.io/github/stars/PINTO0309/Bazel_bin.svg?style=social&label=Stars)](https://github.com/PINTO0309/Bazel_bin)
<a href="https://github.com/PINTO0309/Bazel_bin"><img src="https://github-link-card.s3.ap-northeast-1.amazonaws.com/PINTO0309/Bazel_bin.png" width="460px"></a>

# １．Introduction
Leave the procedure to build Tensorflow v2.0.0 for RaspberryPi3/4. The pre-built installer can be downloaded from the Github link above.

# ２．Environment

- RaspberryPi3 / 4 (Python3.7)
- Raspbian Buster / Debian Buster
- **[openjdk-8-jdk](https://qiita.com/PINTO/items/612718c0ce4f1def6c6e)**
- **[Bazel 0.26.1](https://github.com/PINTO0309/Bazel_bin.git)**
- **[Tensorflow v2.0.0 (glibc2.28, armhf)](https://github.com/tensorflow/tensorflow/tree/v2.0.0)**

# ３．Procedure
## ３−１．Install openjdk-8-jdk
Openjdk-8-jdk cannot be installed from apt repository of Raspbian Buster and Debian Buster. Download and install the offline installer (wheel) file from my Google Drive.

```console:For_armhf
$ curl -sc /tmp/cookie "https://drive.google.com/uc?export=download&id=1LQUSal55R6fmawZS9zZuk6-5ZFOdUqRK" > /dev/null
$ CODE="$(awk '/_warning_/ {print $NF}' /tmp/cookie)"
$ curl -Lb /tmp/cookie "https://drive.google.com/uc?export=download&confirm=${CODE}&id=1LQUSal55R6fmawZS9zZuk6-5ZFOdUqRK" -o adoptopenjdk-8-hotspot_8u222-b10-2_armhf.deb

$ sudo apt-get install -y ./adoptopenjdk-8-hotspot_8u222-b10-2_armhf.deb
```
For how to procure openjdk-8-jdk for Raspbian Buster / Debian Buster, please refer to my article below. **[[Stable] Install openjdk-8-jdk safely in Raspbian Buster (Debian 10) environment](https://qiita.com/PINTO/items/612718c0ce4f1def6c6e)** 

## ３−２．Preparation for Tensorflow build

```console:
$ sudo nano /etc/dphys-swapfile
CONF_SWAPFILE=2048

$ sudo systemctl stop dphys-swapfile
$ sudo systemctl start dphys-swapfile

$ wget https://github.com/PINTO0309/Tensorflow-bin/raw/master/zram.sh
$ chmod 755 zram.sh
$ sudo mv zram.sh /etc/init.d/
$ sudo update-rc.d zram.sh defaults
$ sudo reboot

$ sudo apt-get install -y libhdf5-dev libc-ares-dev libeigen3-dev libatlas-base-dev libopenblas-dev
$ sudo pip3 install keras_applications==1.0.8 --no-deps
$ sudo pip3 install keras_preprocessing==1.1.0 --no-deps
$ sudo pip3 install h5py==2.9.0
$ sudo apt-get install -y openmpi-bin libopenmpi-dev
$ sudo -H pip3 install -U --user six numpy wheel mock

$ cd ~
$ git clone https://github.com/PINTO0309/Bazel_bin.git
$ cd Bazel_bin
$ ./0.26.1/Raspbian_Debian_Buster_armhf/openjdk-8-jdk/install.sh

$ cd ~
$ git clone -b v2.0.0 https://github.com/tensorflow/tensorflow.git
$ cd tensorflow
$ git checkout -b v2.0.0
```

Some program files have been modified to tune Tensorflow Lite acceleration and fix bugs in the official Tensorflow repository.

```python:tensorflow/lite/python/interpreter.py
# Add the following two lines to the last line
  def set_num_threads(self, i):
    return self._interpreter.SetNumThreads(i)
```

```C++:tensorflow/lite/python/interpreter_wrapper/interpreter_wrapper.cc
// Corrected the vicinity of the last line as follows
PyObject* InterpreterWrapper::ResetVariableTensors() {
  TFLITE_PY_ENSURE_VALID_INTERPRETER();
  TFLITE_PY_CHECK(interpreter_->ResetVariableTensors());
  Py_RETURN_NONE;
}

PyObject* InterpreterWrapper::SetNumThreads(int i) {
  interpreter_->SetNumThreads(i);
  Py_RETURN_NONE;
}

}  // namespace interpreter_wrapper
}  // namespace tflite
```

```C++:tensorflow/lite/python/interpreter_wrapper/interpreter_wrapper.h
  // should be the interpreter object providing the memory.
  PyObject* tensor(PyObject* base_object, int i);

  PyObject* SetNumThreads(int i);

 private:
  // Helper function to construct an `InterpreterWrapper` object.
  // It only returns InterpreterWrapper if it can construct an `Interpreter`.
```

```python:tensorflow/lite/tools/make/Makefile
BUILD_WITH_NNAPI=false
```

```C++:tensorflow/lite/experimental/ruy/pack_arm.cc_Line1292
"mov r0, 0\n"
  ↓
"mov r0, #0\n"
```

## ３−３．Build Tensorflow v2.0.0

```console:configure
$ sudo ./configure 
Extracting Bazel installation...
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
You have bazel 0.26.1- (@non-git) installed.
Please specify the location of python. [Default is /usr/bin/python]: /usr/bin/python3

Found possible Python library paths:
  /usr/local/lib
  /usr/lib/python3/dist-packages
  /home/pi/inference_engine_vpu_arm/python/python3.7
  /usr/local/lib/python3.7/dist-packages
Please input the desired Python library path to use.  Default is [/usr/local/lib]
/usr/local/lib/python3.7/dist-packages
Do you wish to build TensorFlow with XLA JIT support? [Y/n]: n
No XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: n
No OpenCL SYCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with ROCm support? [y/N]: n
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: n
No CUDA support will be enabled for TensorFlow.

Do you wish to download a fresh release of clang? (Experimental) [y/N]: n
Clang will not be downloaded.

Do you wish to build TensorFlow with MPI support? [y/N]: n
No MPI support will be enabled for TensorFlow.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native -Wno-sign-compare]: 

Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: n
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
	--config=mkl         	# Build with MKL support.
	--config=monolithic  	# Config for mostly static monolithic build.
	--config=gdr         	# Build with GDR support.
	--config=verbs       	# Build with libverbs support.
	--config=ngraph      	# Build with Intel nGraph support.
	--config=numa        	# Build with NUMA support.
	--config=dynamic_kernels	# (Experimental) Build kernels into separate shared objects.
	--config=v2             # Build Tensorflow 2.x instead of 1.x
Preconfigured Bazel build configs to DISABLE default on features:
	--config=noaws       	# Disable AWS S3 filesystem support.
	--config=nogcp       	# Disable GCP support.
	--config=nohdfs      	# Disable HDFS support.
	--config=noignite    	# Disable Apache Ignite support.
	--config=nokafka     	# Disable Apache Kafka support.
	--config=nonccl      	# Disable NVIDIA NCCL support.
Configuration finished
```

### （１） For RaspberryPi3

```console:build
$ sudo bazel --host_jvm_args=-Xmx512m build \
--config=opt \
--config=noaws \
--config=nohdfs \
--config=noignite \
--config=nokafka \
--config=nonccl \
--config=v2 \
--local_resources=1024.0,0.5,0.5 \
--copt=-mfpu=neon-vfpv4 \
--copt=-ftree-vectorize \
--copt=-funsafe-math-optimizations \
--copt=-ftree-loop-vectorize \
--copt=-fomit-frame-pointer \
--copt=-DRASPBERRY_PI \
--host_copt=-DRASPBERRY_PI \
//tensorflow/tools/pip_package:build_pip_package
```

### （２） For RaspberryPi4

```console:build
$ sudo bazel --host_jvm_args=-Xmx512m build \
--config=opt \
--config=noaws \
--config=nohdfs \
--config=noignite \
--config=nokafka \
--config=nonccl \
--config=v2 \
--local_resources=4096.0,3.0,1.0 \
--copt=-mfpu=neon-vfpv4 \
--copt=-ftree-vectorize \
--copt=-funsafe-math-optimizations \
--copt=-ftree-loop-vectorize \
--copt=-fomit-frame-pointer \
--copt=-DRASPBERRY_PI \
--host_copt=-DRASPBERRY_PI \
//tensorflow/tools/pip_package:build_pip_package
```

## ３−４．Generate Wheel file

```console:Generate_Wheel_file
$ su --preserve-environment
# ./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
# exit
$ sudo cp /tmp/tensorflow_pkg/tensorflow-2.0.0-cp37-cp37m-linux_arm7l.whl ~
```

## ３−５．Install Tensorflow v2.0.0 including accelerated Tensorflow Lite

```console:Install_Tensorflow_v2.0.0_including_accelerated_TensorflowLite
$ cd ~
$ sudo pip3 uninstall tensorflow
$ sudo -H pip3 install tensorflow-2.0.0-cp37-cp37m-linux_armv7l.whl 
```

# ４．Reference articles

1. **[[Stable] Install openjdk-8-jdk safely in Raspbian Buster (Debian 10) environment](https://qiita.com/PINTO/items/612718c0ce4f1def6c6e)**
2. **[Ultra-fast build of Tensorflow with Bazel Remote Caching [Docker version]](https://qiita.com/PINTO/items/49ce77a6ee03b404c156)**