Live Video Streaming Using Multiple Smartphones With ImageZMQ | by Leon Lok | Towards Data Science

Lately, I’ve been delving into computer vision. Live object detection and object tracking are especially interesting to me. However, I realised that I’ll need my own live video stream where I could test my projects.
The first problem is that I don’t have any external webcams available. I have a laptop which does have a built-in webcam, but setting up a laptop just to use the webcam wouldn’t be very practical due to its size and power consumption. I do, however, have quite a few number of spare smartphones that I don’t use anymore.
The second problem is that I also need something which would allow me to send frames over a network to a remote server where there’s enough processing power to run machine learning algorithms on each video stream. This is an important one. My laptop is nowhere near powerful enough to run things like YOLO and Deep SORT on multiple streams.
Could I use my old smartphones to set up a simple camera network?
At first, I successfully managed to stream from a single device over a network using an IP camera app and OpenCV. However, once I tried to stream from multiple devices, things got messy very quickly. There were also issues with performance since I wasn’t processing the frames asynchronously.
After struggling for a while with OpenCV, I came across an amazing library by Jeff Bass called ImageZMQ for Python. It allows you to create a video streaming network in just a few lines of code! It’s designed to run on Raspberry Pis but I didn’t have any available. Instead, I used my laptop to process and send the frames from my smartphones over a network instead. ImageZMQ also significantly improved my streaming performance.
If you’re looking to get started with computer vision or perhaps want to recycle your old smartphones to build a surveillance system, I’d say this is a great starting point.
As the name suggests, ImageZMQ uses the brokerless asynchronous messaging library called ZeroMQ.
When it comes to live video streaming, we ideally want the lowest latency possible. Asynchronous processing allows us to achieve a much better framerate and latency. This becomes especially true with larger camera networks.
It’s also brokerless. This means less running processes and removes the possibility of a broker bottleneck. It’s particularly important for live video streaming since we want high throughput and low latency performance.
There’s a fantastic article by Adrian Rosebrock which goes in-depth on how to set up a surveillance network with motion detection using Raspberry Pis with Pi Camera modules. This was overkill for me and as I stated earlier, I didn’t have any of these available.
Instead, I used some old smartphones with an IP camera application installed on each one. I streamed the smartphones to my laptop where they’re redirected over a network to a central server. If you do happen to own some Raspberry Pis, then I’d recommend you start with Adrian’s guide instead of mine.
For this to work, we need to set up multiple clients and a server that’ll act as a central processing hub. We also need to install an IP camera application on each smartphone. To be clear, all my smartphones use the Android OS. If you’re also using Android, then there’s plenty of IP cameras on the Google Play store. Personally, I’ve been using IP Webcam which offers a few useful video settings. But I’m sure the other IP camera applications work pretty much the same.
Before continuing, please make sure that the following libraries are installed in your Python environment:
imutils,
opencv-python,
socket,
and of course, imagezmq.
Just in case any issues occur with these libraries, I’m using Python 3.6. I’d recommend using the same version of Python as mine or higher.
Setting up the server to receive incoming frames is extremely simple. It should look familiar if you’ve had experience using OpenCV.
First, we need to import OpenCV and ImageZMQ. Note that when we install opencv-python, we import it as cv2. Then, we need to create an image processing hub using ImageZMQ’s ImageHub that’ll receive and process the incoming frames from each stream which can be done in a single line.
import cv2import imagezmqimage_hub = imagezmq.ImageHub()
ImageHub() has two additional arguments that you can use within it. The first argument is open_port, which is defaulted to accept any incoming TCP traffic at the port 5555. The second argument is REQ_REP, which stands for request-reply and is defaulted to True. This is a type of ZeroMQ pattern where every frame sent by the client will have to wait for a reply from the server before sending the next frame. If we set this to False, then the pattern will be changed to a publish-subscribe one instead.
For our scenario, we want to have a single server that does all the processing with multiple clients sending their video frames to it. A publish-subscribe pattern would require each client to also be a server itself, with the main processing hub subscribing to each of these servers. This requires knowing the address of each stream beforehand so it’s not the most practical for what we want to do. Hence, we keep the default values and leave it as it is.
However, Jeff has explained the benefits of using the publish-subscribe pattern very well here. If you’re interested in learning more about ZeroMQ patterns then I’d recommend giving that a read.
The next step is to create a loop that will take each frame from each stream and display them accordingly. This can be done using a while loop and using OpenCV’s imshow() method, as shown below.
while True:      cam_id, frame = image_hub.recv_image()    cv2.imshow(cam_id, frame)      cv2.waitKey(1)    image_hub.send_reply(b'OK')
We unpackage the incoming stream using recv_image() to get the frames of each video stream along with the names of each camera device.
To display the frames for each camera, we then use cv2.imshow(). The best part is that this method will automagically separate and display the frames for each video stream! We then need cv2.waitKey(1) is then needed to allow each frame to refresh after 1ms.
Lastly, since we’re using the request-reply pattern, we need to send an OK message to the client to let them know the server has received and processed the frame successfully. The client will then know to continue sending frames to the server.
The code for each client is also quite straightforward. The only thing we need to be aware of is that to send frames to a server, we need to configure each client with the correct stream path and server address.
For the clients, we need to import ImageZMQ, socket, and also the VideoStream() method from imutils. Then, we use VideoStream() to capture frames from our camera. We could have used OpenCV’s VideoCapture() instead, but imutil’s VideoStream() combines the picamera module with OpenCV, which is good to have in case we do decide to use Raspberry Pis in the future.
from imutils.video import VideoStreamimport imagezmqimport socket# change this to your stream addresspath = "rtsp://192.168.1.70:8080//h264_ulaw.sdp"cap = VideoStream(path)
The path should be a link to the stream coming from your IP camera application. You can find this easily by opening the application on your smartphone.
Note that the address can change if you’re streaming via router. You need to give the device a static IP address within the router or it’ll break your stream if you try to run the code again.
We then need to create an image sender using ImageZMQ and give our camera a name. We can now start the stream.
# change this to your server addresssender = imagezmq.ImageSender(connect_to='tcp://localhost:5555') cam_id = socket.gethostname()stream = cap.start()
ImageSender() also has the REQ_REP argument, but as discussed earlier, we leave that as the default value of True. We only need to specify the server address to send the frames to. In this case, I’ve used localhost at port 5555 since I’m streaming from the same machine. If for example we’re streaming to a server in the cloud, then we’d replace localhost with the IP address of that machine. This is assuming that the server allows traffic by opening the correct ports, which by default would be port 5555.
We also use socket.gethostname() to get the name of the device we’re streaming from. If we used Raspberry Pis, then this is an easy way of automatically getting the names of each device. If you only have a few devices, then you could manually change cam_id to something simple like a number, for example.
To send the frames, we again need a loop that reads the frames from the video streams and then send them using the ImageSender() from earlier.
while True:    frame = stream.read()    sender.send_image(cam_id, frame)
... and we’re done! Now all we need to do is create a client script for each device and make sure that the path and cam_id variables are configured correctly for each device.
Once each device is ready with its own client script and IP camera running, make sure the stream path and server addresses are correct and then start the server. Only once the server is up and running can the devices start sending the frames. Run each client and you should be able to see each individual live video stream appearing on the server side!