「Codama」ボードを使った音声認識によるデバイス制御実験（おまけ：認識結果・カメラ映像のモニタリング）

[ビデオで紹介した今回の実験システム](https://youtu.be/MBEKwfJghEw)の全体構成です。　Part-1～3で、欠けていた部分を掲載します。([Part-1](https://qiita.com/wanic/items/a8cfd5bdda62b301ce12)、[Part-2](https://qiita.com/wanic/items/11963ac57a6e1d412c74)、[Part-3](https://qiita.com/wanic/items/b3a05618b8b5ffc8a98e)）
## ■　全体構成
![codama_21.JPG](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/134874/6e3842fa-1d44-e195-e0ef-0bc9aee02a2b.jpeg)

### ■ 実験システムのファイル構成
拡張性と理解しやすさを考えてファイルを分けています。
データの授受（認識結果・ビデオ映像）はRedis D/Bを介して行うことにより分散処理を可能にしています。
>・static
　　├ css
　　　　└ style.css
　　├ js
　　　　└ app.js
・template
　　└ index.html
・server.py
・Picam.py
・PiVedeoCamera.py
・define.py (Part-3)
・devcont.py (Part-3)
・DevControl.py (Part-3)

### ■ 起動方法
```exec.sh
python3 Picam.py &
python3 DevControl.py &
python3 server.py
```

### ■ Tornade Web Socket Server
認識結果及びビデオ映像をJsonに変換してブラウザに送信しています。

#### ■ Server.py
```python:server.py
# -*- coding: utf-8 -*-
import sys
import os
import json
import time
from struct import *
import random
# Import 3rd-party modules.
from tornado import websocket, web, ioloop
import redis

hostname = '192.168.xxx.xxx'
MAX_FPS = 100

# ============================
# Tornade web socket server
# ============================
class IndexHandler( web.RequestHandler):
    def get( self):
        self.render( 'index.html')

class SocketHandler( websocket.WebSocketHandler):
    def __init__(self, *args, **kwargs):
        super(SocketHandler, self).__init__(*args, **kwargs)
        pool = redis.ConnectionPool( host = hostname, port = 6379, db = 0)
        self._r = redis.StrictRedis( connection_pool = pool)
        self._store = redis.StrictRedis( host='localhost', port='6379')
        self._prec_id = None
        self._prev_image_id = None
        self.result = ""

    def open( self):
        print( "Tornado WebSocket opened")

    def on_message( self, dummy):
        while True:
            time.sleep( 1./MAX_FPS)
            image_id = self._store.get( 'image_id')
            if image_id != self._prev_image_id:
                break
        self._prev_image_id = image_id
        image = self._store.get( 'image').decode( 'utf-8')
        rec_id = self._r.get( "Rec_id")
        if rec_id != self._prec_id:
            # byte型を文字列型に変換
            self.result = self._r.get( "Result").decode( 'utf-8')
            self._prec_id = rec_id

        #　データをDict形式で格納
        tmpDic = { 'Result': self.result, 'Image': image}
        #　jsonデータに変換
        toBrowser = json.dumps( tmpDic)
        #print( "JsonData:", toBrowser, "\n")
        # ブラウザに送信
        self.write_message( toBrowser)

    def on_close(self):
        print( "WebSocket closed")

app = web.Application([
    ( r'/', IndexHandler),
    ( r'/ws', SocketHandler),
    ],
    template_path=os.path.join( os.getcwd(),  "templates"),
    static_path=os.path.join( os.getcwd(),  "static"),
)

if __name__ == '__main__':
    try:
        print( "Tornado Server is running : port 9000")
        app.listen( 9000)
        ioloop.IOLoop.instance().start()

    except KeyboardInterrupt:
        sys.exit(0)
```

#### ■ index.html
```html:index.html
<!doctype html>
<html lang="ja">
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>Voice Device Control Test</title>
    <link type="text/css" rel="stylesheet" href="http://code.jquery.com/ui/1.10.3/themes/cupertino/jquery-ui.min.css" />
    <link rel="stylesheet" href="{{ static_url("css/style.css") }}"/>
    <script type="text/javascript" src="http://code.jquery.com/jquery-1.10.2.min.js"></script>
    <script type="text/javascript" src="http://code.jquery.com/ui/1.10.3/jquery-ui.min.js"></script>
    <script type="text/javascript" src="{{ static_url("js/app.js") }}"></script>
</head>
<body>
<h1>Voice Control Device</h1>
<div class="container">
    <dic id="camera_info">
        <img id="cam" src=""/>
    </div>
    <div id="data_info">
        <dl>
            <dt>Result:</dt>
            <dd id="result">@@@</dd>
        </dl>
	</div>
</div>
</body>
</html>
```
#### ■ app.js
```javascript:app.js
$(function() {
    var nextRequest = "Send me next";
    var result = "";
    var image = "";
    var ws_path = "ws://192.168.xxx.xxx:9000/ws";
    var ws = new WebSocket( ws_path);

    console.log( ws_path);

    $( "#result").text( result);
        ws.onopen = function() {
        ws.send( nextRequest);
        console.log( "Request send to Tornado Server");
    };

    // ラズパイからのデータ受信
    // e は [object MessageEvent]
    ws.onmessage = function( e) {
        // 受信データをデコード（ラズパイでの定義）
        var decJson = JSON.parse( e.data);
        result = decJson.Result;
        image = decJson.Image;

        $( "#cam").attr( 'src', 'data:image/jpg;base64,' + image);
        $( "#result").text( result);
        ws.send( 1);
    };

    ws.onerror = function( e) {
        console.log( e);
    };
});
```
#### ■ style.css
```stylesheet:style.css
@charset "utf-8";
* {
    margin: 0;
    padding: 0;
}
body {
    padding: 20px;
}
.container {
    position: relative;
    margin-top: 0px;
    margin-left: 10px;
}
#camera_info {
    position: absolute;
    top: 0px;
    left: 15px;
}
#data_info {
    position: absolute;
    top: 560px;
    left: 50px;
}
#data_info dl {
}
#data_info dt {
    padding: 3px;
    font-size: 20px;
    font-weight: bold;
    color: blue;
    float: left;
}
#data_info dd {
    margin-left: 100px;
    padding: 3px;
    font-size: 24px;
    font-weight: bold;
    color: red;
}
```
## ■　ラズパイカメラ画像の入力

#### ■ Picam.py
```python:Picam.py
# -*- coding: utf-8 -*-
import sys
import os
import time
import base64
import redis
import numpy
import imutils
import cv2
from PIL import Image
from imutils.video.pivideostream import PiVideoStream

# Create client to the Redis store.
store = redis.StrictRedis( host='localhost', port='6379')

# Create video capture object and socket client.
vs = PiVideoStream().start()

# allow the camera to warmup
time.sleep( 2)

while True:
    frame = vs.read()
    if frame is None:
        time.sleep( 0.5)
        continue

    frame = imutils.resize( frame, width=640)
    # jpegに圧縮
    result, encimg = cv2.imencode( '.jpg', frame)
    # 画像をBase64の文字列に変換
    # prifixは、Javascriptで追加（ 'data:image/jpg;base64,'）
    enc64img = base64.b64encode( encimg)

    store.set( 'image', enc64img)
    image_id = os.urandom( 4)
    store.set( 'image_id', image_id)
```
#### ■ PiVedeoCamera.py
```python:PiVedeoCamera.py
# -*- coding: utf-8 -*-
# import the necessary packages
from picamera.array import PiRGBArray
from picamera import PiCamera
from threading import Thread

class PiVideoStream:
    def __init__(self, resolution=(320, 240), framerate=32):
        # initialize the camera and stream
        self.camera = PiCamera()
        self.camera.resolution = resolution
        self.camera.framerate = framerate
        self.rawCapture = PiRGBArray( self.camera, size=resolution)
        self.stream = self.camera.capture_continuous( self.rawCapture, format="bgr", use_video_port=True)

        # initialize the frame and the variable used to indicate
        # if the thread should be stopped
        self.frame = None
        self.stopped = False

    def start(self):
        # start the thread to read frames from the video stream
        Thread(target=self.update, args=()).start()
        return self

    def update(self):
        # keep looping infinitely until the thread is stopped
        for f in self.stream:
            # grab the frame from the stream and clear the stream in
            # preparation for the next frame
            self.frame = f.array
            self.rawCapture.truncate(0)

            # if the thread indicator variable is set, stop the thread
            # and resource camera resources
            if self.stopped:
                self.stream.close()
                self.rawCapture.close()
                self.camera.close()
                return

    def read(self):
        # return the frame most recently read
        return self.frame

    def stop(self):
        # indicate that the thread should be stopped
        self.stopped = True
```

## 参考資料
Websocket／Redis／Picamera　などで大変お世話になったサイトです。

* http://www.pyimagesearch.com/2015/12/28/increasing-raspberry-pi-fps-with-python-and-opencv/
* https://www.pyimagesearch.com/2016/01/04/unifying-picamera-and-cv2-videocapture-into-a-single-class-with-opencv/
* https://github.com/vmlaker/hello-websocket