How to build MediaPipe hand tracking (iOS) as an Xcode project

# Agenda
1. Preparation (MediaPipe)
2. Build from terminal
3. Preparation (Tulsi)
4. Build as an Xcode project

# Demo

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Hand Tracking on iOS📱<a href="https://twitter.com/hashtag/MediaPipe?src=hash&amp;ref_src=twsrc%5Etfw">#MediaPipe</a> <a href="https://t.co/mgHPfUIHwK">pic.twitter.com/mgHPfUIHwK</a></p>&mdash; MIWA Tetsushi (@WWWPONTE) <a href="https://twitter.com/WWWPONTE/status/1293183995086172171?ref_src=twsrc%5Etfw">August 11, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

# DEV
- MacBook Pro: Catalina 10.15.4
    - Xcode: 11.6
- iPhone SE (2nd generation): iOS 13.5.1

# Preparation (MediaPipe)
### 1. Install Homebrew
- Copy the command at [https://brew.sh](https://brew.sh/)
- Paste to your terminal and execute
- The version of Homebrew I used this time was 2.4.7.

```zsh

$ brew -v
# Homebrew 2.4.7
```

### 2. Install Command Line Tools
- Go to [https://developer.apple.com/download/more](https://developer.apple.com/download/more/) and download Command Line Tools for Xcode XX ※You need to sign in Apple Developer Program.
- Double-click `.dmg`
- Execute the following command

```
$ sudo xcodebuild -license
```

### 3. Check the Python version
Python is pre-installed on mac by default. Depending on the Python version, the build may not pass, so it is necessary to check the Python version. I confirmed that the build passed with `Python 3.7.5`, so I recommend building with this version of Python. At that time, there is a version manager called `pyenv` that can switch the Python version, so I will explain how to use it.

- Clone the pyenv repository

```
$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv
```
- Add to path `.zsh_profile`

`zsh` 

```zsh
$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zsh_profile
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zsh_profile
$ echo 'eval "$(pyenv init -)"' >> ~/.zsh_profile
```

- Install `Python 3.7.5`

```
$ pyenv install 3.7.5
```

- Rehash `shim`

```
$ pyenv rehash
```

- Specifying the Python version

The global command sets the global Python version. This can be overridden with other commands, but is useful for ensuring you use a particular Python version by default. If you wanted to use `3.7.5` by default, then you could run this:

```
$ pyenv global 3.7.5
```

The local command is often used to set an application-specific Python version. You could use it to set the version to `3.7.5`:

```
$ pyenv local 3.7.5
```

### 4. Install six library
Install `six` library for absorbing the difference between Python 2 and Python 3.

```
$ pip install –user future six
```

### 5. Clone the MediaPipe repository

```
$ git clone https://github.com/google/mediapipe.git
```

### 6. Install Bazel

- Insatll `Bazel`

```
$ brew install bazel
```

- The version of bazel I used this time was 3.3.0.

```zsh

$ bazel --version
# bazel 3.3.0
```

### 7. Install OpenCV and FFmpeg

```
$ brew install opencv@3
```

### 8. Install numpy

```
$ pip install numpy
```

### 9. Check installation with Hello World

- Execute `Hello World desktop example`

```zsh

$ export GLOG_logtostderr=1
$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
    mediapipe/examples/desktop/hello_world:hello_world

# After building bazel (it takes a few minutes), it is OK if "Hello World!" is displayed 10 times as shown below.
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
```

# Build from terminal

### 1. Prepare Provisioning Profile

In order to execute the iOS app on your device, a file for identifying the iOS device or app called the `Provisioning Profile` is required. Apple Developer Program subscribers can create and download from [https://developer.apple.com/jp](https://developer.apple.com/jp/). Rename the downloaded file to `provisioning_profile.mobileprovision` and place it in `mediapipe/mediapipe/`.

### 2. Change Bundle Identifier

Next, fix `BUILD` at `mediapipe/mediapipe/examples/ios/handtrackinggpu/`. Change `bundle_id` to the same one which you set at `Provisioning Profile `.

```zsh:mediapipe/examples/ios/handtrackinggpu/BUILD：36

bundle_id = BUNDLE_ID_PREFIX + ".HandTrackingGpu",
　　　　　　　　　↓
bundle_id = "(Bundle Identifier)",
```

### 3. Build

Move `mediapipe/` and execute the following command.

```zsh

$ bazel build -c opt –config=ios_arm64 mediapipe/example/ios/handtrackinggpu:HandTrackingGpuApp
```

An IPA file is generated at the following directory.

```zsh

bazel-bin/mediapipe/examples/ios/handtrackinggpu/
```

# Preparation (Tulsi)

### 1. Clone the Tulsi repository

```
$ git clone https://github.com/bazelbuild/tulsi.git
```

### 2. Apply the patch 

```
$ cd tulsi
$ git fetch origin pull/99/head:xcodefix
$ git checkout xcodefix
```

### 3. Execute the build script

```
sh build_and_run.sh
```

Occurred an error!! 

```zsh

ERROR: /Users/miwa/tulsi/BUILD:62:18: Linking of rule '//:tulsi.__internal__.apple_binary' failed (Exit 1) wrapped_clang failed: error executing command external/local_config_cc/wrapped_clang -Xlinker -objc_abi_version -Xlinker 2 -fobjc-link-runtime -ObjC -arch x86_64 -filelist ... (remaining 26 argument(s) skipped)
```

After checking [this](https://github.com/bazelbuild/tulsi/issues/102), I could execute the build script!

```zsh:tulsi/WORKSPACE：6

tag = "0.17.2",
　　　　　　　　　↓
tag = "0.18.0",
```

### 4. Open MediaPipe.tulsiproj and generate the Xcode project
- Launch Tulsi.app and open `Mediapipe.tulsiproj` at `mediapipe/mediapipe/`.

<img width="614" alt="Tulsi_Start.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/557107/7c9e7bee-19ed-b202-0f9a-b66f0631f557.png">

- Push `Generate` button in `Configs` tab

<img width="661" alt="Tulsi_Generate.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/557107/68576854-fd14-9619-795d-76dc6bd1204f.png">


# Build as an Xcode project
Now connect your iPhone to your mac. Then open the generated Xcode project and start the build. After a while, the build will be completed and it will be installed on your iPhone.

<img width="1440" alt="Xcode_handtrackinggpu.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/557107/c742e374-ec90-8ede-6db7-1c86c69d6607.png">

Set to use the rear camera.

```zsh:mediapipe/examples/ios/handtrackinggpu/ViewController.mm：107

_cameraSource.cameraPosition = AVCaptureDevicePositionFront;
　　　　　　　　　↓
_cameraSource.cameraPosition = AVCaptureDevicePositionBack;
```

Displaying reversed image, so I changed the value of `Mirrored`.

```zsh:mediapipe/examples/ios/handtrackinggpu/ViewController.mm：111

_cameraSource.videoMirrored = YES;
　　　　　　　　　↓
_cameraSource.videoMirrored = NO;
```

# Summary

I succeed in implementing high-precision real-time hand tracking on iOS.

In the future, I will develop learning iOS applications for the visually impaired with this technology.