Package: rpi-deep-pantilt
Raspberry Pi Deep PanTiltREAD THIS FIRST!A detailed walk-through is available inReal-time Object Tracking with TensorFlow, Raspberry Pi, and Pan-tilt HAT.Build ListRaspberry Pi 4 (4GB recommended)Raspberry Pi Camera V2Pimoroni Pan-tilt KitMicro SD card 16+ GBMicro HDMI Cable12" CSI/DSI ribbon for Raspberry Pi Camera(optional, but highly recommended)Coral Edge TPU USB Accelerator(optional)RGB NeoPixel Stick(optional, makes lighting conditions more consistent)An example of deep object detection and tracking with a Raspberry PiFree software: MIT licenseDocumentation:https://rpi-deep-pantilt.readthedocs.io.Basic SetupBefore you get started, you should have an up-to-date installation of Raspbian 10 (Buster) running on your Raspberry Pi. You'll also need to configure SSH access into your Pi.Install RaspbianConfigure WiFiConfigure SSH AccessInstallationInstall system dependencies$sudoapt-getupdate&&sudoapt-getinstall-y\cmakepython3-devlibjpeg-devlibatlas-base-devraspi-gpiolibhdf5-devpython3-smbusCreate new virtual environment$python3-mvenv.venvActivate virtual environment$source.venv/bin/activateUpgrade setuptools$pipinstall--upgradesetuptoolsInstall TensorFlow 2.2 (community-built wheel)$pipinstallhttps://github.com/leigh-johnson/Tensorflow-bin/releases/download/v2.2.0/tensorflow-2.2.0-cp37-cp37m-linux_armv7l.whlInstall therpi-deep-pantiltpackage.pipinstallrpi-deep-pantiltInstall Coral Edge TPUtflite_runtime(optional)NOTE: This step is only required if you are usingCoral's Edge TPU USB Accelerator. If you would like to run TFLite inferences using CPU only, skip this step.$pipinstallhttps://dl.google.com/coral/python/tflite_runtime-2.1.0.post1-cp37-cp37m-linux_armv7l.whl=======ConfigurationWARNING: Do not skip this section! You will not be able to userpi-deep-pantiltwithout properly configuring your Pi.Enable Pi CameraRunsudo raspi-configand selectInterfacing Optionsfrom the Raspberry Pi Software Configuration Tool’s main menu. Press ENTER.Select the Enable Camera menu option and press ENTER.In the next menu, use the right arrow key to highlight ENABLE and press ENTER.Enable i2c in Device TreeOpen/boot/config.txtand verify the followingdtparamslines are uncommented:dtparam=i2c1=ondtparam=i2c_arm=onExample UsageObject DetectionThedetectcommand will start a PiCamera preview and render detected objects as an overlay. Verify you're able to detect an object before trying to track it.Supports Edge TPU acceleration by passing the--edge-tpuoption.rpi-deep-pantilt detect [OPTIONS] [LABELS]...rpi-deep-pantilt detect --help
Usage: rpi-deep-pantilt detect [OPTIONS] [LABELS]...

  rpi-deep-pantilt detect [OPTIONS] [LABELS]

    LABELS (optional)     One or more labels to detect, for example:     
    $ rpi-deep-pantilt detect person book "wine glass"

    If no labels are specified, model will detect all labels in this list:
    $ rpi-deep-pantilt list-labels

    Detect command will automatically load the appropriate model

    For example, providing "face" as the only label will initalize
    FaceSSD_MobileNet_V2 model $ rpi-deep-pantilt detect face

    Other labels use SSDMobileNetV3 with COCO labels $ rpi-deep-pantilt detect
    person "wine class" orange

Options:
  --loglevel TEXT  Run object detection without pan-tilt controls. Pass
                   --loglevel=DEBUG to inspect FPS.
  --edge-tpu       Accelerate inferences using Coral USB Edge TPU
  --rotation INTEGER  PiCamera rotation. If you followed this guide, a
                      rotation value of 0 is correct.
                      https://medium.com/@grepLeigh/real-time-object-tracking-
                      with-tensorflow-raspberry-pi-and-pan-tilt-
                      hat-2aeaef47e134
  --help           Show this message and exit.Object TrackingThe following will start a PiCamera preview, render detected objects as an overlay, and track an object's movement with Pimoroni pan-tilt HAT.By default, this will track anypersonin the frame. You can track other objects by passing--label <label>. For a list of valid labels, runrpi-deep-pantilt list-labels.rpi-deep-pantilt trackSupports Edge TPU acceleration by passing the--edge-tpuoption.Usage: rpi-deep-pantilt track [OPTIONS] [LABEL]

  rpi-deep-pantilt track [OPTIONS] [LABEL]

  LABEL (required, default: person) Exactly one label to detect, for example:     
  $ rpi-deep-pantilt track person

  Track command will automatically load the appropriate model

  For example, providing "face" will initalize FaceSSD_MobileNet_V2 model
  $ rpi-deep-pantilt track face

  Other labels use SSDMobileNetV3 model with COCO labels 
  $ rpi-deep-pantilt detect orange

Options:
  --loglevel TEXT  Pass --loglevel=DEBUG to inspect FPS and tracking centroid
                   X/Y coordinates
  --edge-tpu       Accelerate inferences using Coral USB Edge TPU
  --rotation INTEGER  PiCamera rotation. If you followed this guide, a
                      rotation value of 0 is correct.
                      https://medium.com/@grepLeigh/real-time-object-tracking-
                      with-tensorflow-raspberry-pi-and-pan-tilt-
                      hat-2aeaef47e134
  --help           Show this message and exit.Valid labels for Object Detection/Trackingrpi-deep-pantilt list-labelsThe following labels are valid tracking targets.['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush']Face Detection (NEW in v1.1.x)The following command will detect human faces.NOTE: Face detection uses a specialized model (FaceSSD_MobileNet_V2), while other labels are detecting using SSDMobileNetV3_COCO. You cannot detect both face and COCO labels at this time.Watch this repo for updates that allow you to re-train these models to support a custom mix of object labels!rpi-deep-pantilt detect face
Usage: cli.py face-detect [OPTIONS]

Options:
  --loglevel TEXT  Run object detection without pan-tilt controls. Pass
                   --loglevel=DEBUG to inspect FPS.
  --edge-tpu       Accelerate inferences using Coral USB Edge TPU
  --help           Show this message and exit.Face Tracking (NEW in v1.1.x)The following command will track a human face.rpi-deep-pantilt track face
Usage: cli.py face-detect [OPTIONS]

Options:
  --loglevel TEXT  Run object detection without pan-tilt controls. Pass
                   --loglevel=DEBUG to inspect FPS.
  --edge-tpu       Accelerate inferences using Coral USB Edge TPU
  --help           Show this message and exit.Model SummaryThe following section describes the models used in this project.Object Detection & TrackingFLOAT32model (ssd_mobilenet_v3_small_coco_2019_08_14)rpi-deep-pantilt detectandrpi-deep-pantilt trackperform inferences using this model. Bounding box and class predictions render at roughly6 FPSon aRaspberry Pi 4.The model is derived fromssd_mobilenet_v3_small_coco_2019_08_14intensorflow/models. I extended the model with an NMS post-processing layer, then converted to a format compatible with TensorFlow 2.x (FlatBuffer).I scripted the conversion steps intools/tflite-postprocess-ops-float.sh.QuantizedUINT8model (ssdlite_mobilenet_edgetpu_coco_quant)If you specify--edge-tpuoption,rpi-deep-pantilt detectandrpi-deep-pantilt trackperform inferences using this model. Rounding box and class predictions render at roughly24+ FPS (real-time) on Raspberry Pi 4.This modelREQUIRESaCoral Edge TPU USB Acceleratorto run.This model is derived fromssdlite_mobilenet_edgetpu_coco_quantintensorflow/models. I reversed the frozen.tflitemodel into a protobuf graph to add an NMS post-processing layer, quantized the model in a.tfliteFlatBuffer format, then converted using Coral'sedgetpu_compilertool.I scripted the conversion steps intools/tflite-postprocess-ops-128-uint8-quant.shandtools/tflite-edgetpu.sh.Face Detection & TrackingI was able to use the same model architechture for FLOAT32 and UINT8 input,facessd_mobilenet_v2_quantized_320x320_open_image_v4_tflite2.This model is derived fromfacessd_mobilenet_v2_quantized_320x320_open_image_v4intensorflow/models.Common Issuesi2c is not enabledIf you run$ rpi-deep-pantilt test pantiltand see a similar error, check your Device Tree configuration.File"/home/pi/projects/rpi-deep-pantilt/.venv/lib/python3.7/site-packages/pantilthat/pantilt.py",line72,insetupself._i2c=SMBus(1)FileNotFoundError:[Errno2]NosuchfileordirectoryOpen/boot/config.txtand ensure the following lines are uncommented:dtparam=i2c1=ondtparam=i2c_arm=onCreditsThe MobileNetV3-SSD model in this package was derived fromTensorFlow's model zoo, withpost-processing ops added.The PID control scheme in this package was inspired byAdrian RosebrocktutorialPan/tilt face tracking with a Raspberry Pi and OpenCVThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History1.0.0 (2019-12-01)First release on PyPI.