The Streaming and Image processing challenge with Project Immersion Develop
Tags: Image Processing, Object Detection, Opencv Python, Image Recognition, Python

The Team

The Problem

After experiencing the drastic shift from face-to-face education to remote education, I noticed that the only solution offered to all the professions that were forced to move to remote realms in the internet world were video conferencing, screen sharing, and video conferencing. slates. it was not enough, work stress levels increased, direct contact was lost, children lost attention, the world lost interest.

Development Approach

We proposed the development of Project Immersion, to provide the new digital daily life with tools that would allow us to recover the lost affection of a face-to-face world, and even extend that capacity, focused on learning processes, taking as the most extreme example the first educational stages In the formation of every human being and its serious affectation in pandemic modern times, the proposed development time is one month.

It is clear that the new AR / VR technologies have solved many of these problems, but the access barrier due to the necessary hardware (VR GLASS, GPUs, etc.) and its consequent costs for the customer, which limits the solutions to a small Spectrum of the population, having raised this problem, we decided to develop a solution with AR characteristics as a web application that would make it compatible as a puglin of any video calling platform accessible on the market.

My Role

This consists of devising how to use the webcam of the devices by default, and the recognition of objects in the image for through the processing of the image in the backend provided by the client from the browser to assume the procedural load on the server .

We decided to attack the monotony of presentations by means of slides where contact with the speaker is lost. So the idea was to represent the face of the speaker on the slides without interfering with the content of these, without reducing their seriousness and recovering the attention of the public, also allowing the speaker total freedom of expression and adding it as an added value to his presentation, too gives you the possibility to point with a red dot that is detected in the palm of your hands, and with this, when you bring it closer to the lower edges, you can change the slide, so the user is not forced to make his presentation tied to his keyboard.

The Challenges

For the above, the first challenge was to capture the face on the client’s webcam and express it proportionally in the image of the slides, for this it was necessary to agree on a universal format from which the client could be generated from any slide editor, We We decided on the universal PDF, which by means of a python script converts its pages into images and is provided at the image resolution obtained from the camera, so the client will have the entire dimension of the slide to apply the AR effects.

Using Jquery, call the universal API in browsers to access the client’s camera, draw it and portion it in Canvas to send it through the Socket.io protocol to the API rest in the backend, once that problem is solved, I start with the challenges object detection.

I started the development of image processing by calling my camera directly in the development process, executing the entire script process natively on my own machine.

I developed a model to detect the reference points on the user’s face once detected in the camera image, detection executed with the default dlib tools model, for this model I use a convolutional neural network model in Keras using the set of iBUG 300-W data with which 68 points are marked on facial features, but I developed a model that only took into account 56 of those 68 points, although I lose the ability to detect certain facial features, I am forced to reduce the detail to achieve a clearer model.

To use the palm of the hand as a pointer, it was necessary to develop a HOG (Histogram of Oriented Gradients) model with dlib tools and the MOHI dataset and then train using the Support Vector Machine (SVM) algorithm, and thus detect only the hand when you expose your palm.

From the implementation of the detection of these models in each camera frame, a second image is created in which the slide becomes the background, the 56 landmarks of the face send the coordinates to redraw the image in 2 dimensions the face with its expressions on the slide, then with the coordinates of the hand a red dot is superimposed that acts as a pointer and selector when executing the triggers in the lower corners of the image.

Results

Once achieved, I started to refactor the code to adapt it to the image flow that the API receives through the socket.io protocol, if it is the same image that we draw on the canvas before sending it from the html, here I face my first challenge.

The image is a structure given as a three-dimensional matrix composed only of integers, but when traveling through the https protocol this structure is transformed into a list of binary numbers, one for each frame, in the backend the numerical transformation was binary required to the base 64 bit image and then to a string image and then to a numpy array, apply the previously developed image processing algorithm and reverse the format so that it is returned to the client via web protocol.

I did not expect that the frames of the video sent by the camera would be forced to transform the object into so many structures, a situation that adds to the risk of delay and processing load the transformation of the object in the middle of the data flow.

To lower the risk of delay and the processing load, configure the entire workflow to a resolution of 640 x 480 pixels, thus reducing the weight of the stremming, then it was time to test, and indeed the delay was excessive and accumulated, for which it was necessary to set a top streaming speed of 2 fps.

New Challenges to Resolve

The reason why I was forced to affect the quality of the final result so much is because the image transformation work was developed in python and this language is characterized by being slower than others, for which I am migrating the Backend support to Ruby on rails, re-training the models to make them lighter, due to the absence of a person dedicated to devops in the development team, the project used the hosting services of heroku, which reaffirmed its versatility and better the fast debugging, trial and error, however I work to migrate to a hosting service that offers me processing in dedicated gpu how Azure or AWS and thus reduce the time that the processing retains the image in the backend and reduce all the risk of possible dalay that occupies your execution.

I also work on the application of two more main features for which it is necessary to first solve the shortcomings in the video flow.

Look the demo!

http://project-inmersion.herokuapp.com

By Andre Felipe Miranda

Cali, Colombia

https://www.linkedin.com/in/andres-miranda-2451b51a1/

@AelFM

andreselfm@gmail.com