Creating a Virtual Background App... | by Swati Modi | Aug, 2020 | Towards Data Science | Towards Data Science

This is the Part 2 of the MediaPipe Series I am writing.
Previously, we saw how to get started with MediaPipe and use it with your own tflite model. If you haven’t read it yet, check it out here.
We had tried using the portrait segmentation tflite model in the existing segmentation pipeline with the calculators already present in MediaPipe
After getting bored with this Blue Background, I decided to have some fun with it by having some Zoom like Virtual Backgrounds like beautiful stars or some crazy clouds instead :)
For this, I wrote a custom calculator and used it with the existing pipeline.
So today I’ll show how did I go about making this App
Before we get started, I would suggest you to go through this part of the documentation which explains the Flow of a basic calculator.
https://google.github.io/mediapipe/framework_concepts/calculators.html
Now, let’s get started with the code
$ git clone https://github.com/SwatiModi/portrait-segmentation-mediapipe.git
So earlier, the rendering/coloring was done by the RecolorCalculator , it used to take image and mask gpu buffer as input and returned gpu buffer rendered output (rendered using opengl)
Here, for replacing the Background with an Image(jpg/png), I have used OpenCV operations.
NOTE : OpenCV operations are performed on CPU — ImageFrame datatype where as opengl operations are performed on GPU — Image-buffer datatype
portrait_segmentation.pbtxt
We will replace the RecolorCalculator with the BackgroundMaskingCalculator
node {  calculator: "BackgroundMaskingCalculator"  input_stream: "IMAGE_CPU:mask_embedded_input_video_cpu"  input_stream: "MASK_CPU:portrait_mask_cpu"  output_stream: "OUTPUT_VIDEO:output_video_cpu"  }
This calculator takes transformed Image and the Mask (Both ImageFrame Datatype) as input and changes the background with given image.
For converting the Image and Mask GpuBuffer to ImageFrame, I used the GpuBufferToImageFrameCalculator.
In this we will write the logic of processing and creating the background masking effect
background_masking_calculator.cc
place this background_masking_calculator.cc in mediapipe/calculators/image/
a. Extending the Base Calculator
b. GetContract()
Here we are just verifying our inputs and their data types
c. Open()
Prepares the calculator’s per-graph-run state.
d. Process()
Taking the inputs, converting to OpenCV Mat and further processing for desired output
BUILD file in mediapipe/calculators/image
We need to do this so the calculator is available and accessible while compilation and execution
the deps (dependencies) you see here are the imports you are using in your calculator
The name used here is referred in graph BUILD file as well
BUILD file in graphs/portrait_segmentation
Add background_masking_calculator to deps and remove recolor_calculator which we are not using now
Now we are ready to build, we just have to add the asset(image to be masked on the background)
NOTE: This asset reading method is straight forward when trying on Desktop but quite different for the Android Build.
So we will look at the Desktop and Android build one by one
In the background_masking_calculator.cc , read the asset file as
Now, we are ready to build
bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11  mediapipe/examples/desktop/portrait_segmentation:portrait_segmentation_gpu
The build will finish successfully in around 10–15 minutes
INFO: Build completed successfully, 635 total actions
Now, you can run the Pipeline using the following
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/portrait_segmentation/portrait_segmentation_gpu   --calculator_graph_config_file=mediapipe/graphs/portrait_segmentation/portrait_segmentation.pbtxt
the webcam starts and you’ll be able to see the output window
Add the following code to the BUILD file in testdata folder (folder containing the background asset images)
exports_files(    srcs = glob(["**"]),)
Adding the files to android_library along with the tflite model
Modify in android project BUILD file in examples/android/..../portraitsegmentationgpu/
Here, first using the MediaPipe’s PathToResourceAsFile method, we try to look if the asset is available and if available we read it using it’s value as path to file
On a side note, this method was a part of resource_util.h present in mediapipe/util/ folder. So whenever you import a new file into your calculator, make sure to add it to BUILD file mentioned in pt. 4
# BUILD bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/portraitsegmentationgpu# On successfully building the APK, it printsINFO: Elapsed time: 1499.898s, Critical Path: 753.09sINFO: 2002 processes: 1849 linux-sandbox, 1 local, 152 worker.INFO: Build completed successfully, 2140 total actions
This would take around 20–25 minutes when building for the first time because it downloads all the external dependencies for the Build. Next time it uses the cached dependencies, so it builds much faster.
# INSTALLadb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/portraitsegmentationgpu/portraitsegmentationgpu.apk
Now, you are ready to run the APK and test it.
Here we successfully added our own calculator and modified the pipeline and graph according by first testing it on Desktop and then finally on Android with minimal changes.
Feel free to ask any questions in the comments or you can reach me out personally.
You can find more about me on swatimodi.com
https://blog.gofynd.com/mediapipe-with-custom-tflite-model-d3ea0427b3c1https://mediapipe.readthedocs.io/en/latest/framework_concepts.htmlhttps://mediapipe.readthedocs.io/en/latest/calculator.htmlhttps://github.com/google/mediapipe/issuesGrab the code from here — https://github.com/SwatiModi/virtual-background-app
https://blog.gofynd.com/mediapipe-with-custom-tflite-model-d3ea0427b3c1
https://mediapipe.readthedocs.io/en/latest/framework_concepts.html
https://mediapipe.readthedocs.io/en/latest/calculator.html
https://github.com/google/mediapipe/issues
Grab the code from here — https://github.com/SwatiModi/virtual-background-app