360度カメラでDeepStreamSDKのnvdewarperの最新版を試してみる


## 必要なソフトウェア

* DeepStreamSDK 5.1
* https://github.com/NVIDIA-AI-IOT/Deepstream-Dewarper-App の`libnvdgst_dewarper.so`と`nvds_dewarper_meta.h` 

## 必要なハードウェア

|画像|型番|
|:--|:--|
|![elecom3.jpg](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63450/97bd112d-b2e5-e3d4-1baa-401f2687e6c1.jpeg)|OCAM-VRU01BK|
||USB TypeCメス->USB TypeAオス変換ケーブル|

https://www.amazon.co.jp/%E3%82%A8%E3%83%AC%E3%82%B3%E3%83%A0-360%E5%BA%A6%E3%82%AB%E3%83%A1%E3%83%A9-%E3%82%B9%E3%83%9E%E3%83%9B%E7%9B%B4%E6%8C%BF%E3%81%97%E3%82%BF%E3%82%A4%E3%83%97-%E3%82%AA%E3%83%A0%E3%83%8B%E3%82%B7%E3%83%A7%E3%83%83%E3%83%88%E3%83%9F%E3%83%8B-OCAM-VRU01BK/dp/B07BKF66GF/ref=sr_1_3?__mk_ja_JP=%E3%82%AB%E3%82%BF%E3%82%AB%E3%83%8A&dchild=1&keywords=elecom%2B360%E5%BA%A6%E3%82%AB%E3%83%A1%E3%83%A9&qid=1632749702&sr=8-3&th=1

https://www.amazon.co.jp/gp/product/B098LCDSGY/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1


## DeepStreamSDK用のDockerを展開

```
xhost +
```

```
sudo docker run -it --ipc=host --net=host --runtime nvidia  --device /dev/video0:/dev/video0:mwr -e DISPLAY=$DISPLAY -w /opt/nvidia/deepstream/deepstream-5.1  -v /tmp/argus_socket:/tmp/argus_socket -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/deepstream-l4t:5.1-21.02-samples
```

## 必要なパッケージをインストール

```
apt update
apt install v4l-utils vim -y
```

## 動作テスト

```
gst-launch-1.0 v4l2src device="/dev/video0" ! \
'image/jpeg, width=1472, height=736, framerate=30/1' ! \
nvv4l2decoder mjpeg=1 ! \
nvvideoconvert ! \
nvegltransform ! \
nveglglessink sync=0
```

![Screenshot from 2021-09-27 23-25-10.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63450/e72365a8-7ecf-e2a9-f867-e967f0cf59e5.png)


## nvdewarperの最新ライブラリに更新

```
cd /opt/nvidia/deepstream/deepstream-5.1/lib/gst-plugins/
mv libnvdsgst_dewarper.so libnvdsgst_dewarper_backup.so
wget https://github.com/NVIDIA-AI-IOT/Deepstream-Dewarper-App/raw/main/plugin_libraries/jetson_bin/libnvdsgst_dewarper.so
chmod 755 libnvdsgst_dewarper.so
```

```
cd /opt/nvidia/deepstream/deepstream-5.1/sources/includes/
mv nvds_dewarper_meta.h nvds_dewarper_meta_backup.h
wget https://raw.githubusercontent.com/NVIDIA-AI-IOT/Deepstream-Dewarper-App/main/nvds_dewarper_meta.h
```

## nvdewarperで歪み補正

`dewarp_config.txt`

```
[property]
  
[surface0]
# 1=PushBroom, 2=VertRadCyl 3= Perspective_Perspective FISH_PERSPECTIVE=4, FISH_FISH=5, FISH_CYL=6, FISH_EQUIRECT=7,
# FISH_PANINI=8, PERSPECTIVE_EQUIRECT=9, PERSPECTIVE_PANINI=10, EQUIRECT_CYLINDER=11, EQUIRECT_EQUIRECT=12 EQUIRECT_FISHEYE=13,
# EQUIRECT_PANINI=14, EQUIRECT_PERSPECTIVE=15, EQUIRECT_PUSHBROOM=16, EQUIRECT_STEREOGRAPHIC=17, EQUIRECT_VERTCYLINDER=18
projection-type=15
surface-index=0
#dewarped surface parameters
width=600
height=480
top-angle=15
bottom-angle=-15
pitch=0
yaw=0
roll=0
src-x0=380
src-y0=368
focal-length=850

[surface1]
# 1=PushBroom, 2=VertRadCyl 3= Perspective_Perspective FISH_PERSPECTIVE=4, FISH_FISH=5, FISH_CYL=6, FISH_EQUIRECT=7,
# FISH_PANINI=8, PERSPECTIVE_EQUIRECT=9, PERSPECTIVE_PANINI=10, EQUIRECT_CYLINDER=11, EQUIRECT_EQUIRECT=12 EQUIRECT_FISHEYE=13,
# EQUIRECT_PANINI=14, EQUIRECT_PERSPECTIVE=15, EQUIRECT_PUSHBROOM=16, EQUIRECT_STEREOGRAPHIC=17, EQUIRECT_VERTCYLINDER=18
projection-type=15
surface-index=1
#dewarped surface parameters
width=600
height=480
top-angle=15
bottom-angle=-15
pitch=0
yaw=0
roll=0
src-x0=1080
src-y0=368
focal-length=850
```

```
gst-launch-1.0 v4l2src device="/dev/video0" ! 'image/jpeg, width=1472, height=736, framerate=30/1' ! nvv4l2decoder mjpeg=1 ! nvvideoconvert ! nvdewarper config-file=dewarp_config.txt  ! m.sink_0 nvstreammux name=m width=600 height=480 batch-size=2 num-surfaces-per-frame=2 ! nvmultistreamtiler width=600 height=480 ! nvegltransform ! nveglglessink sync=0
```

![Screenshot from 2021-09-27 23-26-35.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63450/f61269d1-90d8-489e-a09f-1027cbe73adf.png)

## 参考

https://forums.developer.nvidia.com/t/360-degree-camera-config/171049

https://rb-station.com/blogs/article/360-degree-camera-jetsonnano-dewarp

https://github.com/NVIDIA-AI-IOT/Deepstream-Dewarper-App