Package: azure-percept
This is the source code for the azure-percept package - an unofficial Python library to access the sensors of Azure Percept in Python.IMPORTANT: This is a community-driven open source library without any warranty, expect bugs. If you encounter them, pleaseopen an issueon Github.Connect to your PerceptPlease refer to the official documentation to learn how to connect to the device:https://docs.microsoft.com/en-us/azure/azure-percept/how-to-ssh-into-percept-dkInstallFrom PyPI (recommended)This package is intended to run on an Azure Percept device (or a container hosted on Azure Percept). Make sure you use one of the following Python versions: 3.6, 3.7, 3.8, 3.9, 3.10. Then install the package with:sudo yum install python3-pip
python3 -m venv ~/.venv
source ~/.venv/bin/activate
pip3 install --upgrade pip
pip3 install azure-percept
sudo usermod -aG apdk_accessories,audio $(whoami)After running these commands,log out and log in againso the group membership changes take effect.From sourceMake sure the following is installed on your Percept device or the container you want to use:libalsa, libusb, gcc, binutils, Python headers, setuptools and pip (runsudo yum install -y git alsa-lib-devel libusb-devel gcc glibc-devel kernel-devel kernel-headers binutils python3-devel python3-setuptools python3-pip)pthreads (libpthread should be available on most OS by default, check your library path - for example /usr/lib/ - to be sure)Clone the source code on your Percept devicegit clone https://github.com/christian-vorhemus/azure-percept-py.gitOpen a terminal and cd intoazure-percept-pyRunsudo pip3 install .In case you get an error message like "module_info.ld: No such file or directory", runsudo /usr/lib/rpm/mariner/gen-ld-script.shto create the necessary scripts.Runsudo usermod -aG apdk_accessories,audio $(whoami)Log out and log in againTo uninstall runsudo pip3 uninstall azure-perceptNote that the package includes pre-built libraries that will only run on an aarch64 architecture!Azure Percept Audio sampleThe following sample authenticates the Azure Percept Audio sensor, records audio for 5 seconds and saves the result locally as a WAV file. Create a new fileperceptaudio.pywith the following contentfromazure.iot.perceptimportAudioDeviceimporttimeaudio=AudioDevice()print("Authenticating sensor...")whileTrue:ifaudio.is_ready()isTrue:breakelse:time.sleep(1)print("Authentication successful!")print("Recording...")audio.start_recording("./sample.wav")time.sleep(5)audio.stop_recording()print("Recording stopped")audio.close()Typepython3 perceptaudio.pyto run the script.Azure Percept Vision samplesRun a machine learning model on the VPUThe following sample shows how to run a model on the Azure Vision Myriad VPU. It assumes we have a .onnx model ready for inference. If not, download a model from theONNX Model Zoo, for exampleResNet-18or browse thesample modelswhich are already converted to the .blob format. Create a new fileperceptvision.pywith the following contentfromazure.iot.perceptimportVisionDevice,InferenceResultimporttimeimportnumpyvision=VisionDevice()print("Authenticating sensor...")whileTrue:ifvision.is_ready()isTrue:breakelse:time.sleep(1)print("Authentication successful!")# this will convert a ONNX model to a model file with the same name# and a .blob suffix to the output directory "/path/to"vision.convert_model("/path/to/resnet18-v1-7.onnx",scale_values=[58.395,57.120,57.375],mean_values=[123.675,116.28,103.53],reverse_input_channels=True,output_dir="/path/to")vision.start_inference("/path/to/resnet18-v1-7.blob")res:InferenceResult=vision.get_inference(return_image=True)print(res.inference)print(res.image)vision.stop_inference()vision.close()Typepython3 perceptvision.pyto run the script. Especially the model conversion can take several minutes. The preprocessing of images (if needed) is baked into the model itself and applied by first converting the BGR camera frame to RGB (if specified withreverse_input_channels=True), then subtracting themean_valuesfrom the input and finally dividing all tensor elements per channel byscale_values.vision.start_inference(blob_model_path)will initialize the Azure Percept Vision camera as well as the VPU. To specify the input camera sources, pass theinput_srcargument, for examplevision.start_inference(blob_model_path, input_src=["/dev/video0", "/dev/video2"])whereas/camera1would identify the Percept module camera and/dev/video0,/dev/video2are conventional USB cameras plugged into the Percept DK.  Withvision.get_inference()the prediction results are returned as anInferenceResultobject or as a list ofInferenceResultobjects in case of multiple input sources. The prediction is stored as a numpy array inres.inference.Process local image files on the VPUIt's also possible to use a local image file instead of reading from a camera device. To do so, convert the image into a BGR sequence of bytes and pass them in theinputargument ofget_inference():fromazure.iot.perceptimportVisionDevice,InferenceResultimporttimefromPILimportImageimportnumpyasnpvision=VisionDevice()print("Authenticating sensor...")whileTrue:ifvision.is_ready()isTrue:breakelse:time.sleep(1)print("Authentication successful!")image=Image.open("./<yourfile>.jpg")image_np=np.array(image)image_np=np.moveaxis(image_np,-1,0)r=image_np[0].tobytes()g=image_np[1].tobytes()b=image_np[2].tobytes()img=b+g+rvision.start_inference("<model>.blob")res:InferenceResult=vision.get_inference(input=img,input_shape=(image.height,image.width))print(res.inference)vision.stop_inference()vision.close()Take a picture and save it locallyThe following sample gets an image (as a numpy array) from the Azure Percept Vision device in BGR format with shape (height, width, channels) and saves it as a JPG file (you need Pillow for this sample to work:pip3 install Pillow)fromazure.iot.perceptimportVisionDeviceimporttimeimportnumpyasnpfromPILimportImagevision=VisionDevice()print("Authenticating sensor...")whileTrue:ifvision.is_ready()isTrue:breakelse:time.sleep(1)print("Authentication successful!")img=vision.get_frame()# get a camera frame from the Azure Vision deviceimg=img[...,::-1].copy()# copy the BGR image as RGBpil_img=Image.fromarray(img)# convert the numpy array to a Pillow imagepil_img.save("frame.jpg")vision.close()Record a videoThe following sample records a video for 5 seconds and saves it locally as a MP4 file.fromazure.iot.perceptimportVisionDeviceimporttimevision=VisionDevice()print("Authenticating sensor...")whileTrue:ifvision.is_ready()isTrue:breakelse:time.sleep(1)print("Authentication successful!")print("Recording...")vision.start_recording("./sample.mp4")time.sleep(5)vision.stop_recording()print("Recording stopped")vision.close()TroubleshootingDuring model conversion I get "Cannot create X layer from unsupported opset".This indicates that the model contains a layer that can't be converted to a model definition the VPU can process. For a list of supported layers seehere.Reading audio data fails with "ValueError: Device not found" or "Exception: Azure Ear could not authenticate".Type in "lsusb". You should see a list of several devices, try to find ID "045e:0673". If this device is not present, unplug and plug in your Azure Audio device again and restart the device. Additionally make sure your device has Internet connectivity during the authentication process. It's also possible that the user you run the command with has no rights to access soundcards. You can check this if you install alsa utils (sudo yum install alsa-utils) and then runaplay -l. If you see an output like "no soundcards found", add the user you run the script with (e..g the current usersudo usermod -aG audio $(whoami)) to the audio group, log out and log in and test again.When running the package in a docker container I get "Exception: Azure Eye could not authenticate" or "Failed to find MX booted device. Retrying..." is running in a loop.Type "lsusb" and check if "045e:066f" is present. If not, unplug the vision device and plug it in again. If it's visible, check if "03e7:2485" is in the list. If not, the authentication process is not finished. Check if your device has Internet connectivity. Additionally, make sure the container is using the host's network to receive udev events and, mount the /dev path to the container and run in privileged mode (e.g.,docker run --net=host -v /dev:/dev --privileged <imagename>)When trying to install azure-percept I get "Could not find a version that satisfies the requirement azure-percept".Make sure that you install the package on the Azure Percept DK directly and you use a supported Python version (check withpython3 --version). Also make sure you use a recent version of pip (python3 -m pip install --upgrade pip)LicenseThis library is licensed underApache License Version 2.0and uses binaries and scripts from theOpenVINO toolkitwhich is as well licensed under Apache License Version 2.0.