PythonでRealSenseD400シリーズ入門（Windows10、Python3.6）

# はじめに
[3D Sensor Advent Calendar 2019](https://qiita.com/advent-calendar/2019/3d-sensor)の3日目の記事です。
今日は、PythonでRealSenseD400シリーズに入門しましょう。（というか、備忘録）

センサーについてはこちらがよくまとまっていますのでご参考まで。
[【記事】RealSense D415とD435の比較と選び方](https://www.tegara.com/news/2018/07/realsense-d415-d435.html)
[【記事】RealSense D415とD435 および D435iとT265の比較と選び方](https://www.tegara.com/news/2019/04/realsense-compare.html)

# 開発環境
・Windows 10
・Anaconda 2019.10
・Python 3.6.9
・opencv-python 4.1.1.26
・opencv-contrib-python 4.1.1.26
・pyrealsense2 2.30.0.1184
・open3d-python 0.7.0.0
・RealSenseD435（D415／D435i）

# RealSense SDK のインストール
Intel® RealSense™ SDK 2.0 (build 2.29.0) の [Intel.RealSense.SDK.exe](https://github.com/IntelRealSense/librealsense/releases/download/v2.29.0/Intel.RealSense.SDK.exe) を実行し、インストールしてください。

# Anaconda のインストール

1．[Anaconda3-2019.10-Windows-x86_64.exe](https://repo.anaconda.com/archive/Anaconda3-2019.10-Windows-x86_64.exe) を実行し、インストールする
2．Anaconda Promptを開き、仮想環境を作成する

~~~
(base) C:\Users\xxx> conda create -n py36 python=3.6 anaconda
~~~

3．仮想環境を立ち上げる

~~~
(base) C:\Users\xxx> conda activate py36
(py36) C:\Users\xxx> 
~~~

# 各ライブラリのインストール

~~~
(py36) C:\Users\xxx> pip install open3d-python
(py36) C:\Users\xxx> pip install pyrealsense2
(py36) C:\Users\xxx> pip install opencv-python
(py36) C:\Users\xxx> pip install opencv-contrib-python
~~~

※アップデートする場合、***pip install***の後に***-U***オプションをつける

# サンプルコードのダウンロード
・OpenCV4.1.1 のサンプルは[こちら](https://github.com/opencv/opencv/archive/4.1.1.zip)
・OpenCV Contrib 4.1.1 のサンプルは[こちら](https://github.com/opencv/opencv_contrib/archive/4.1.1.zip)
・RealSense のサンプルは[こちら](https://github.com/IntelRealSense/librealsense/archive/v2.29.0.zip)
・Open3Dのサンプルは[こちら](https://github.com/intel-isl/Open3D/archive/v0.7.0.zip)

# RealSenseのサンプルコードをやってみる

ダウンロードした *librealsense-2.29.0\wrappers\python\examples* へ移動

~~~
(py36) C:\Users\xxx> cd librealsense-2.29.0\wrappers\python\examples
~~~


## align-depth2color.py

RGBD画像の取得、表示するプログラム

~~~
(py36) C:\Users\xxx> python align-depth2color.py
Depth Scale is:  0.0010000000474974513
~~~

<blockquote class="twitter-tweet"><p lang="und" dir="ltr"><a href="https://t.co/Mjp7ZeY6yr">https://t.co/Mjp7ZeY6yr</a> <a href="https://twitter.com/hashtag/RealSense?src=hash&amp;ref_src=twsrc%5Etfw">#RealSense</a> <a href="https://twitter.com/hashtag/D400Series?src=hash&amp;ref_src=twsrc%5Etfw">#D400Series</a> <a href="https://twitter.com/hashtag/Python?src=hash&amp;ref_src=twsrc%5Etfw">#Python</a> <a href="https://t.co/YbHnHQjv6D">pic.twitter.com/YbHnHQjv6D</a></p>&mdash; 藤本賢志(ガチ本)@MAリーグ2019決勝 (@sotongshi) <a href="https://twitter.com/sotongshi/status/1201721205289390080?ref_src=twsrc%5Etfw">December 3, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

qキーで閉じる

## export_ply_example.py
1フレーム読み取り、3D点群を.plyとして保存するプログラム

~~~
(py36) C:\Users\xxx>python export_ply_example.py
Saving to 1.ply...
Done
~~~

examplesフォルダ以下に1.plyが生成されているので、[MeshLab2016.12](http://www.meshlab.net/#download)等で開いて確認してみましょう。

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">export_ply_example.py <a href="https://twitter.com/hashtag/RealSense?src=hash&amp;ref_src=twsrc%5Etfw">#RealSense</a> <a href="https://twitter.com/hashtag/D400Series?src=hash&amp;ref_src=twsrc%5Etfw">#D400Series</a> <a href="https://twitter.com/hashtag/Python?src=hash&amp;ref_src=twsrc%5Etfw">#Python</a> <a href="https://t.co/Yww18O36gr">pic.twitter.com/Yww18O36gr</a></p>&mdash; 藤本賢志(ガチ本)@MAリーグ2019決勝 (@sotongshi) <a href="https://twitter.com/sotongshi/status/1201724715405963265?ref_src=twsrc%5Etfw">December 3, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

## frame_queue_example.py
slower processing functionがどうとか

~~~
(py36) C:\Users\xxx> python frame_queue_example.py
Slow callback
1
2
3
・・・
Slower callback + queue
1
2
3
・・・
138
139
140
Slower callback + keeping queue
__init__(): incompatible constructor arguments. The following argument types are supported:
    1. pyrealsense2.pyrealsense2.frame_queue(arg0: int)
    2. pyrealsense2.pyrealsense2.frame_queue()

Invoked with: 50; kwargs: keep_frames=True
~~~

## opencv_pointcloud_viewer.py
OpenCVで3D点群を表示するプログラム

~~~
(py36) C:\Users\xxx> python opencv_pointcloud_viewer.py
~~~

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">opencv_pointcloud_viewer.py <a href="https://twitter.com/hashtag/RealSense?src=hash&amp;ref_src=twsrc%5Etfw">#RealSense</a> <a href="https://twitter.com/hashtag/D400Series?src=hash&amp;ref_src=twsrc%5Etfw">#D400Series</a> <a href="https://twitter.com/hashtag/Python?src=hash&amp;ref_src=twsrc%5Etfw">#Python</a> <a href="https://t.co/ldRRXeeIZm">pic.twitter.com/ldRRXeeIZm</a></p>&mdash; 藤本賢志(ガチ本)@MAリーグ2019決勝 (@sotongshi) <a href="https://twitter.com/sotongshi/status/1201726581908987905?ref_src=twsrc%5Etfw">December 3, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

qキーで閉じる

## opencv_viewer_example.py
OpenCVでDepth画像、Color画像を表示するプログラム

~~~
(py36) C:\Users\xxx> python opencv_viewer_example.py
~~~

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">opencv_viewer_example.py <a href="https://twitter.com/hashtag/RealSense?src=hash&amp;ref_src=twsrc%5Etfw">#RealSense</a> <a href="https://twitter.com/hashtag/D400Series?src=hash&amp;ref_src=twsrc%5Etfw">#D400Series</a> <a href="https://twitter.com/hashtag/Python?src=hash&amp;ref_src=twsrc%5Etfw">#Python</a> <a href="https://t.co/7kH4F3Gu3L">pic.twitter.com/7kH4F3Gu3L</a></p>&mdash; 藤本賢志(ガチ本)@MAリーグ2019決勝 (@sotongshi) <a href="https://twitter.com/sotongshi/status/1201728212075597824?ref_src=twsrc%5Etfw">December 3, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

コマンドプロンプトからCtrl+Cで閉じる

cv2.waitKey(1)のところを

~~~python
if cv2.waitKey(1) & 0xFF == ord('q'):
    break
~~~
に書き換えると、qキーで閉じれる

## pybackend_example_1_general.py

~~~
(py36) C:\Users\xxx> python pybackend_example_1_general.py
Traceback (most recent call last):
  File "pybackend_example_1_general.py", line 9, in <module>
    import pybackend2 as rs
ModuleNotFoundError: No module named 'pybackend2'
~~~

pip install できなそうだからやめとく
[ModuleNotFoundError: No module named 'pybackend2' #3204](https://github.com/IntelRealSense/librealsense/issues/3204)


## pyglet_pointcloud_viewer.py
pygletを用いて3D点群を表示するプログラム

~~~
(py36) C:\Users\xxx> pip install pyglet
(py36) C:\Users\xxx> pyglet_pointcloud_viewer.py
Traceback (most recent call last):
  File "pyglet_pointcloud_viewer.py", line 151, in <module>
    fps_display = pyglet.clock.ClockDisplay()
AttributeError: module 'pyglet.clock' has no attribute 'ClockDisplay'
~~~

Python2っぽい

## python-rs400-advanced-mode-example.py
D400シリーズのパラメータを取得するプログラム

~~~
(py36) C:\Users\xxx> python python-rs400-advanced-mode-example.py
Found device that supports advanced mode: Intel RealSense D435
Advanced mode is enabled
Depth Control:
 minusDecrement: 10, deepSeaMedianThreshold: 500, scoreThreshA: 1, scoreThreshB: 2047, textureDifferenceThreshold: 0, textureCountThreshold: 0, deepSeaSecondPeakThreshold: 325, deepSeaNeighborThreshold: 7, lrAgreeThreshold: 24
RSM:
 rsmBypass: 0, diffThresh: 4, sloRauDiffThresh: 1, removeThresh: 63
RAU Support Vector Control:
 minWest: 1, minEast: 1, minWEsum: 3, minNorth: 1, minSouth: 1, minNSsum: 3, uShrink: 3, vShrink: 1
Color Control:
 disableSADColor: 0, disableRAUColor: 0, disableSLORightColor: 0, disableSLOLeftColor: 0, disableSADNormalize: 0
RAU Thresholds Control:
 rauDiffThresholdRed: 51, rauDiffThresholdGreen: 51, rauDiffThresholdBlue: 51
SLO Color Thresholds Control:
 diffThresholdRed: 72, diffThresholdGreen: 72, diffThresholdBlue: 72
SLO Penalty Control:
 sloK1Penalty: 60, sloK2Penalty: 342, sloK1PenaltyMod1: 105, sloK2PenaltyMod1: 190, sloK1PenaltyMod2: 70, sloK2PenaltyMod2: 130
HDAD:
 lambdaCensus: 26, lambdaAD: 800, ignoreSAD: 0
Color Correction:
 colorCorrection1: 0.461914, colorCorrection2: 0.540039, colorCorrection3: 0.540039, colorCorrection4: 0.208008, colorCorrection5: -0.332031, colorCorrection6: -0.212891, colorCorrection7: -0.212891, colorCorrection8: 0.68457, colorCorrection9: 0.930664, colorCorrection10: -0.553711, colorCorrection11: -0.553711, colorCorrection12: 0.0458984
Depth Table:
 depthUnits: 1000, depthClampMin: 0, depthClampMax: 65536, disparityMode: 0, disparityShift: 0
Auto Exposure Control:
 Mean Intensity Set Point: 1536
Census:
 uDiameter: 9, vDiameter: 9
Depth Control Min Values:
  minusDecrement: 0, deepSeaMedianThreshold: 0, scoreThreshA: 0, scoreThreshB: 0, textureDifferenceThreshold: 0, textureCountThreshold: 0, deepSeaSecondPeakThreshold: 0, deepSeaNeighborThreshold: 0, lrAgreeThreshold: 0
Depth Control Max Values:
  minusDecrement: 255, deepSeaMedianThreshold: 1023, scoreThreshA: 1023, scoreThreshB: 4095, textureDifferenceThreshold: 4095, textureCountThreshold: 1023, deepSeaSecondPeakThreshold: 1023, deepSeaNeighborThreshold: 1023, lrAgreeThreshold: 2047
After Setting new value, Depth Control:
 minusDecrement: 10, deepSeaMedianThreshold: 500, scoreThreshA: 511, scoreThreshB: 2047, textureDifferenceThreshold: 0, textureCountThreshold: 0, deepSeaSecondPeakThreshold: 325, deepSeaNeighborThreshold: 7, lrAgreeThreshold: 24
Controls as JSON:
 {
    "aux-param-autoexposure-setpoint": "1536",
    "aux-param-colorcorrection1": "0.461914",
    "aux-param-colorcorrection10": "-0.553711",
    "aux-param-colorcorrection11": "-0.553711",
    "aux-param-colorcorrection12": "0.0458984",
    "aux-param-colorcorrection2": "0.540039",
    "aux-param-colorcorrection3": "0.540039",
    "aux-param-colorcorrection4": "0.208008",
    "aux-param-colorcorrection5": "-0.332031",
    "aux-param-colorcorrection6": "-0.212891",
    "aux-param-colorcorrection7": "-0.212891",
    "aux-param-colorcorrection8": "0.68457",
    "aux-param-colorcorrection9": "0.930664",
    "aux-param-depthclampmax": "65536",
    "aux-param-depthclampmin": "0",
    "aux-param-disparityshift": "0",
    "controls-autoexposure-auto": "True",
    "controls-autoexposure-manual": "8500",
    "controls-color-autoexposure-auto": "True",
    "controls-color-autoexposure-manual": "156",
    "controls-color-backlight-compensation": "0",
    "controls-color-brightness": "0",
    "controls-color-contrast": "50",
    "controls-color-gain": "64",
    "controls-color-gamma": "300",
    "controls-color-hue": "0",
    "controls-color-power-line-frequency": "3",
    "controls-color-saturation": "64",
    "controls-color-sharpness": "50",
    "controls-color-white-balance-auto": "True",
    "controls-color-white-balance-manual": "4600",
    "controls-depth-gain": "16",
    "controls-laserpower": "150",
    "controls-laserstate": "on",
    "ignoreSAD": "0",
    "param-amplitude-factor": "0",
    "param-autoexposure-setpoint": "1536",
    "param-censusenablereg-udiameter": "9",
    "param-censusenablereg-vdiameter": "9",
    "param-censususize": "9",
    "param-censusvsize": "9",
    "param-depthclampmax": "65536",
    "param-depthclampmin": "0",
    "param-depthunits": "1000",
    "param-disableraucolor": "0",
    "param-disablesadcolor": "0",
    "param-disablesadnormalize": "0",
    "param-disablesloleftcolor": "0",
    "param-disableslorightcolor": "0",
    "param-disparitymode": "0",
    "param-disparityshift": "0",
    "param-lambdaad": "800",
    "param-lambdacensus": "26",
    "param-leftrightthreshold": "24",
    "param-maxscorethreshb": "2047",
    "param-medianthreshold": "500",
    "param-minscorethresha": "511",
    "param-neighborthresh": "7",
    "param-raumine": "1",
    "param-rauminn": "1",
    "param-rauminnssum": "3",
    "param-raumins": "1",
    "param-rauminw": "1",
    "param-rauminwesum": "3",
    "param-regioncolorthresholdb": "0.0499022",
    "param-regioncolorthresholdg": "0.0499022",
    "param-regioncolorthresholdr": "0.0499022",
    "param-regionshrinku": "3",
    "param-regionshrinkv": "1",
    "param-robbinsmonrodecrement": "10",
    "param-robbinsmonroincrement": "10",
    "param-rsmdiffthreshold": "4",
    "param-rsmrauslodiffthreshold": "1",
    "param-rsmremovethreshold": "0.375",
    "param-scanlineedgetaub": "72",
    "param-scanlineedgetaug": "72",
    "param-scanlineedgetaur": "72",
    "param-scanlinep1": "60",
    "param-scanlinep1onediscon": "105",
    "param-scanlinep1twodiscon": "70",
    "param-scanlinep2": "342",
    "param-scanlinep2onediscon": "190",
    "param-scanlinep2twodiscon": "130",
    "param-secondpeakdelta": "325",
    "param-texturecountthresh": "0",
    "param-texturedifferencethresh": "0",
    "param-usersm": "1",
    "param-zunits": "1000"
}
~~~

パラメータがいっぱい

## python-tutorial-1-depth.py

Depthを取得して文字で表現するプログラム

~~~
(py36) C:\Users\xxx> python python-tutorial-1-depth.py
~~~

<blockquote class="twitter-tweet"><p lang="und" dir="ltr"><a href="https://t.co/G93Lf8gYd8">https://t.co/G93Lf8gYd8</a> <a href="https://twitter.com/hashtag/RealSense?src=hash&amp;ref_src=twsrc%5Etfw">#RealSense</a> <a href="https://twitter.com/hashtag/D400Series?src=hash&amp;ref_src=twsrc%5Etfw">#D400Series</a> <a href="https://twitter.com/hashtag/Python?src=hash&amp;ref_src=twsrc%5Etfw">#Python</a> <a href="https://t.co/nPELNS3bUN">pic.twitter.com/nPELNS3bUN</a></p>&mdash; 藤本賢志(ガチ本)@MAリーグ2019決勝 (@sotongshi) <a href="https://twitter.com/sotongshi/status/1201730688715653120?ref_src=twsrc%5Etfw">December 3, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

## read_bag_example.py

ROSでよく使う.bagファイルを読み込み、表示するプログラム

１．Interl RealSense Viewerを用いて、Recordボタンを押し録画。
２．.bagファイルができているので、それを読み込む。

~~~
(py36) C:\Users\xxx> python read_bag_example.py -i 20191203_143057.bag
~~~

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">read_bag_example.py <a href="https://twitter.com/hashtag/RealSense?src=hash&amp;ref_src=twsrc%5Etfw">#RealSense</a> <a href="https://twitter.com/hashtag/D400Series?src=hash&amp;ref_src=twsrc%5Etfw">#D400Series</a> <a href="https://twitter.com/hashtag/Python?src=hash&amp;ref_src=twsrc%5Etfw">#Python</a> <a href="https://t.co/6FinAbjS6d">pic.twitter.com/6FinAbjS6d</a></p>&mdash; 藤本賢志(ガチ本)@MAリーグ2019決勝 (@sotongshi) <a href="https://twitter.com/sotongshi/status/1201736264837259264?ref_src=twsrc%5Etfw">December 3, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

ESCキーで閉じる

# FPS-解像度の対応表

解像度に対する最大FPS

|Resolution|Depth|RGB|
|:--       |:--  |:--|
| 320,  180|　   |60 |
| 320,  240|　   |60 |
| 424,  240|90   |60 |
| 480,  270|90   |　 |
| 640,  360|90   |60 |
| 640,  480|90   |60 |
| 848,  480|90   |60 |
| 960,  540|　   |60 |
|1280,  720|30   |30 |
|1920, 1080|     |30 |

FPSに対する最大解像度

|FPS|Depth|RGB|
|:--|:--      |:--       |
| 6|1280,  720|1920, 1080|
|15|1280,  720|1920, 1080|
|30|1280,  720|1920, 1080|
|60| 848,  480| 960,  540|
|90| 848,  480|          |

# まとめ
RealSenseのサンプルをひととおり動かしてみました。open3dやopencvは今後紹介したいと思います。
明日は @UnaNancyOwen さんによる「みんな大好きOpenNI2でAzure Kinectを動かしてみる」です。お楽しみに！

## Intel RealSense Viewer のフィルター項目一覧
### Stereo Module
- Emitter Enabled
- Enable Auto Exposure Set ROI
- Controls
    - Exposure
    - Gain
    - Laser Power
    - Asic Temperature
    - Error Polling Enabled
    - Projector Temperature
    - Output Trigger Enabled
    - Depth Units
    - Stereo Baseline
    - Inter Cam Sync Mode
    - Emitter On Off
    - Global Time Enabled
- Advanced Controls
    - Depth Control
        - DS Second Peak Threshold
        - DS Neighbor Threshold
        - DS Median Threshold
        - Estimate Median Increment
        - Estimate Median Decrement
        - Score Minimum Threshold 
        - Score Maximum Threshold
        - DS LR Threshold
        - Texture Count Threshold 
        - Texture Difference Threshold
    - Rsm
        - RSM Bypass
        - Disparity Difference Threshold
        - SLO RAU Difference Threshold
        - Remove Threshold
    - Rau Support Vector Control
        - Min West 
        - Min East
        - Min WE Sum
        - Min North
        - Min South
        - Min NS Sum
        - U Shrink
        - V Shrink
    - Color Control
        - Disable SAD Color 
        - Disable RAU Color
        - Disable SLO Right Color
        - Disable SLO Left Color 
        - Disable SAD Normalize
    - Rau Color Thresholds Control
        - Diff Threshold Red
        - Diff Threshold Green
        - Diff Threshold Blue
    - SLO Color Thresholds Control
        - Diff Threshold Red
        - Diff Threshold Green
        - Diff Threshold Blue
    - SLO Penalty Control
        - K1 Penalty
        - K2 Penalty
        - K1 Penalty Mod1
        - K1 Penalty Mod2
        - K2 Penalty Mod1
        - K2 Penalty Mod2
    - HDAD
        - Ignore SAD
        - AD Lambda
        - Census Lambda
    - Color Correction 
        - Color Correction 1
        - Color Correction 2
        - Color Correction 3
        - Color Correction 4
        - Color Correction 5
        - Color Correction 6
        - Color Correction 7
        - Color Correction 8
        - Color Correction 9
        - Color Correction 10 
        - Color Correction 11
        - Color Correction 12
    - Depth Table
        - Depth Units
        - Depth Clamp Min
        - Depth Clamp Max
        - Disparity Mode
        - Disparity Shift
    - AE Control
        - Mean Intensity Set Point
    - Census Enable Reg
        - u-Diameter
        - v-Diameter
    - Disparity Modulation
        - A Factor
- Depth Visualization
    - Visual Preset 
        - Dynamic, Fixed, Near, Far
    - Color Scheme
        - Jet, Classic, White to Black, Black to White, Bio, Cold, Warm
    - Histogram Equalization Enabled
    - Min Distance
    - Max Distance
- Post-Processing
    -  Decimation Filter
        - Filter Magnitude
    - Threshold Filter
        - Min Distance
        - Max Distance
    - Depth to Disparity
    - Spatial Filter
         - Filter Magnitude
         - Filter Smooth Alpha 
         - Filter Delta
         - Holes filling mode
             - Disabled, 2-pixel radius, 4-pixel radius, 8-pixel radius, 16-pixel radius, Unlimited
    - Temporal Filter
        - Filter Smooth Alpha
        - Filter Smooth Delta
        - Persistency mode
            - Disabled, Valid in 8/8, Valid in 2/, Valid in 2/8, Valid in 1/la, Valid in 1/la, Valid in 2/
    - Hole Filling Filter
        - Hole Filling mode
            - Fill from Left, Farest from a, Nearest from 
    - Disparity to Depth

### RGB Camera
- Color
    - YUYV, BGR8, RGBA8, BGRA8, Y16, RGB8, RAW16 
- Enable Auto Exposure
- Controls
    - Backlight Compensation
    - Brightness
    - Contrast
    - Exposure
    - Gain
    - Gamma
    - Hue
    - Saturation
    - Sharpness
    - White Balance
    - Enable Auto White Balance
    - Power Line Frequency
    - Auto Exposure Priority
    - Global Time Enabled
- Post-Processing
    - Decimation Filter
        - Filter Magnitude