Jetbot, CNNでラインフォローから停止まで

停止線をチェッカー柄にすることで認識性をUP。
単純なCNNでラインフォローから停止までできました。
動画はこちら。
ピタゴラスイッチ感もUP！
https://www.youtube.com/watch?v=ux2GdgBG7oM
![IMG_E3057small.JPG](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/617683/501209db-75bf-c88f-c29a-ea7317793b58.jpeg)




データ採取、学習は以前使ったものを使用。
「Jetbot & PyTorch CNN（４）ライントレースのデータ収集プログラム」
「Jetbot & PyTorch CNN（５）ライントレースの学習プログラム」
学習データは1,000枚ぐらい撮りました。

initialize the PyTorch model（変更なし）

```ruby:live_demo.ipynb
import torch
import torchvision
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from PIL import Image
import numpy as np
```

Define network（今回はモデル１つです）

```ruby:live_demo.ipynb
class miniNet2(nn.Module):
    
    def __init__(self, num_outputs=2):
        super(miniNet2, self).__init__()

        # 特徴抽出用の部品
        self.conv1 = nn.Conv2d(3, 6, 3)
        self.pool1 = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 3)
        self.pool2 = nn.MaxPool2d(2, 2)
        self.conv3 = nn.Conv2d(16, 32, 3)
        self.pool3 = nn.MaxPool2d(2, 2)

        # ニューラルネットワーク用の部品
        self.fc1 = nn.Linear(4160, 1024)
        self.fc2 = nn.Linear(1024, 512)
        self.fc3 = nn.Linear(512, num_outputs)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = self.pool1(x)
        x = F.relu(self.conv2(x))
        x = self.pool2(x)
        x = F.relu(self.conv3(x))
        x = self.pool3(x)
        
        # 行列をベクトルに変換
        x = x.view(-1, 4160 )
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)

        return x
```

Load model（モデルは1つ）

```ruby:live_demo.ipynb
Steer_net = miniNet2()
Steer_net.load_state_dict(torch.load('best_steering_model_xy2.pth'))
```

transfer to the GPU device（モデルは1つ）

```ruby:live_demo.ipynb
device = torch.device('cuda')
Steer_net.to(device)
Steer_net = Steer_net.eval().half()
```

some preprocessing（修正なし）

```ruby:live_demo.ipynb
import torchvision.transforms as transforms
import torch.nn.functional as F
import cv2
import PIL.Image
import numpy as np

mean = torch.Tensor([0.485, 0.456, 0.406]).cuda().half()
std = torch.Tensor([0.229, 0.224, 0.225]).cuda().half()

def preprocess(image):
    image2 = np.copy(image)         # trimmed image
    image2 = image2[168:224, 0:224] # trimmed image
    
    image = PIL.Image.fromarray(image2)
    image = transforms.functional.to_tensor(image).to(device).half()
    image.sub_(mean[:, None, None]).div_(std[:, None, None])
    return image[None, ...]
```

start and display our camera（修正なし）

```ruby:live_demo.ipynb
from IPython.display import display
import ipywidgets
import traitlets
from jetcam.csi_camera import CSICamera    ###
from jetcam.utils import bgr8_to_jpeg      ###
camera = CSICamera(width=224, height=224)  ###
camera.running = True                      ###

image_widget = ipywidgets.Image()

def display_xy2(camera_image):      # trimmed
    image1 = np.copy(camera_image)  # trimmed
    image1 = image1[168:224, 0:224] #trimmedthickness=-1)
    jpeg_image1 = bgr8_to_jpeg(image1)
    return jpeg_image1


camera_link = traitlets.dlink((camera, 'value'), (image_widget, 'value'), transform=display_xy2) # trimmed

display(image_widget)
```

create our robot instance（修正なし）

```ruby:live_demo.ipynb
from jetbot import Robot
robot = Robot()
```

define sliders to control JetBot（修正なし）

```ruby:live_demo.ipynb
speed_gain_slider = ipywidgets.FloatSlider(min=0.0, max=1.0, step=0.01,  value=0.2, description='speed gain')
steering_gain_slider = ipywidgets.FloatSlider(min=0.0, max=1.0, step=0.01, value=0.20, description='steering gain')
steering_dgain_slider = ipywidgets.FloatSlider(min=0.0, max=1.0, step=0.01, value=1, description='steering kd')
steering_bias_slider = ipywidgets.FloatSlider(min=-0.3, max=0.3, step=0.01, value=0.0, description='steering bias')

display(speed_gain_slider, steering_gain_slider, steering_dgain_slider, steering_bias_slider)
display(image_widget)
```

display some sliders（修正なし）

```ruby:live_demo.ipynb
xy0_slider = ipywidgets.FloatSlider(min=-1.0, max=1.0, description='xy0; steering')
xy1_slider = ipywidgets.FloatSlider(min=-4, max=4.0, description='xy1; stop')
steering_slider = ipywidgets.FloatSlider(min=-1.0, max=1.0, description='steering')
speed_slider = ipywidgets.FloatSlider(min=0, max=1.0, description='speed')

display(ipywidgets.HBox([xy0_slider, steering_slider]))
display(xy1_slider)
```

create a function(camera's value changes)（変更あり）

```ruby:live_demo.ipynb
import time
angle = 0.0
angle_last = 0.0

def execute(change):
    global angle, angle_last
    image = change['new']
    
    image1 = preprocess(image)
    
    xy = Steer_net(image1).detach().float().cpu().numpy().flatten()
    x = xy[0]
    y = (0.5 - xy[1]) / 2.0

    xy0_slider.value = x
    xy1_slider.value = y

    speed_slider.value = speed_gain_slider.value

    angle = x
    pid =angle * steering_gain_slider.value

    steering_slider.value = steering_bias_slider.value +     pid
    
    if y > 0.2 :
        robot.forward(0.0)
        
    else:
        robot.left_motor.value = max(min(speed_slider.value + steering_slider.value, 1.0), 0.025)
        robot.right_motor.value = max(min(speed_slider.value - steering_slider.value, 1.0), 0.025)



    time.sleep(0.001)

execute({'new': camera.value})
```

camera for processing（変更なし）

```ruby:live_demo.ipynb
camera.observe(execute, names='value')
```

ビデオのストリーミングを中止しプログラム負荷を減らす。（追加）

```ruby:live_demo.ipynb
camera_link.unlink()  # don't stream to browser (will still run camera)
```

ロボットの停止（変更なし）

```ruby:live_demo.ipynb
import time

camera.unobserve(execute, names='value')

time.sleep(0.1)  # add a small sleep to make sure frames have finished processing

robot.stop()
```

close the camera conneciton（変更なし）

```ruby:live_demo.ipynb
camera.stop()
robot.stop()
```
#よかったら、LGTMお願いします!