KubeROS for Deploying ROS 2 based Robotic Applications
with Kubernetes- Challenges, Concept, Architecture, and Case StudyYongzhou Zhang [1,2]; Gergely Soti [1,2]; Björn Hein[1,2][1] Karlsruhe University of Applied Sciences – Institut for Robotics and Autonomous Systems (IRAS)[2] Karlsruhe Institute of Technology (KIT)

2023/11/23

Acknowledgement

Acknowledgment: parts of this work are conducted as part of the project KI5GRob funded byGerman Federal Ministry of Education and Research (BMBF).

Context
Q1: Are the onboard computing resources sufficient?
Q2: Is robot software becoming increasingly complex?
Q3: How to scale the robotic system based on ROS/ROS2?

Example: Mobile Manipulator for Industry
Q1: Are the onboard computing resources sufficient?
Q2: Is robot software becoming increasingly complex?
Q3: How to scale the robotic system based on ROS/ROS2?
One of previous project: QBIIK - An autonomous comminision system with learning capbility for logistics.

https://www.kit-technology.de/de/blog/picken-und-packen-mit-qbiik
https://www.digitale-technologien.de/DT/Redaktion/DE/Textbausteine/PAiCE_Marginalspalten/PAiCE_Projekt_QBIIK_Infokarte.pdf?__blob=publicationFile&v=1

Example: Mobile Manipulator for Industry
PU/RAMG/UPCeMor

Q1: Are the onboard computing resources sufficient?
Q2: Is robot software becoming increasingly complex?

updyltneuqe
dules, fromynaMated

Q3: How to scale the robotic system based on ROS/ROS2?
Perception:- object detection and segmentation- camera calibration- grasping pose estimation- shelf and placing slot localization

Drivers for sensor & actuator:- camera- trailer actuator- rotation table- lidar- etc.
Forklift truck (AMR):- Localization- Mapping- Navigation- Planning- Controller- etc.

Grasping and Placing:- grasping process control- placing process control
Gripper:- Motor controller with CANOpen- gripper controller

User interface:- task management system- training data management system- graphical user interface (GUI)

Robot arm:
- motion planning- trajectory generation- arm controller- fanuc driver- safety module
Task planning:- finit state machine- interface to WMS

Tele-operation:- interface for human assistant- control interface for VR/ARCPU&GPU

Three Challenges

SensePlanAct

Onboard computing
resources are insufficientHigh software complexity ofthe entire system

Deployment at large scaleComputing resources fromcloud and edge

Containerized softwaremodules

Deployment and orchestration
with Kubernetes

CPU&GPU

Leveraging the Cloud and Edge Computing

Onboard computing
resources are insufficient

Sense

Plan

Act

High software complexity ofthe entire system

Deployment at large scale

ROS 2 with Kubernetes• Cloud OS• Production grade container orchestration• Flexibility and adaptability• Scalability• Ability to manage complex, distributed applications• Etc.
http://download.ros.org/downloads/metrics/metrics-report-2022-07.pdf

Pains of Combining two Complex Systems• Networking and communication• Complex setup and configuration• Access to the hardware• Dynamic resource allocation• Latency• Containerization granularity• Integration with existing systems• Etc.

A Layer Between Two Systems?

KubeROSKubeROS Concept• Abstract the onboard devices, cloud/edge resources as a unified computing infrastructure• Using Kubernetes to orchestrate the containerized ROS 2 software modules• Hide the complex underlying framework• Provide an easy-to-use interface for developers

ROS 2 Modules
cloudedge

Configuration

Containerized ROS 2 Modules

Automated Deployment

Unified Computing Infrastructure

KubeROS High-Level Architecture• Abstract the onboard devices, cloud/edge resources as a unified computing infrastructure• Using Kubernetes to orchestrate the containerized ROS 2 software modules

Software Modules
Drivers

Controllers

Perception

Localization

Manipulation

Navigation

Developers

ROS2 Code

Cluster 1Config

KubeROS
KubeROS APIServerDB

Deployment
Controller

Resource
Manager

Deployment
Scheduler

SoftwareManager

Robot

Robot

Edge ServerEdge VM

RobotStorage

Cluster 2VM

VMCPUGPU

Computing InfrastructureAbstraction between Software and Hardware
Software Modules
DriversControllers

Navigation

Perception

Localization

Manipulation

Fleet (Fleet nodes)

Kubernetes (K8s nodes)

Hardware (Onboard, Edge, Cloud)Create fleet for deployment with FleetManifest

Robot hardware specifications in ClusterInventory

Interface: KubeROS-CLI•

Similar to the ros2cli command tools•

Auto-completion is enabled•

Install via pip$ pip install kuberos-cli

https://github.com/kuberos-io/kuberos-cli

Robot Onboard Only Deployment• Large robot fleet (10+)• 10+ containers for one application• Sufficient onboard computer resources
Containerized
ROS 2 Modules

Robot Cluster

Deployment
Manifest

Robots with Edge/Cloud Computing• Large robot fleet (10+)• 10+ containers for one application• Onboard computer resources are insufficientContainerized
ROS 2 Modules

Robot Cluster

Deployment
Manifest

Robot as Standalone Cluster• Large robot fleet (10+)• Multiple onboard computers• Self-healing even during network connection loss
Deployment
Manifest

Containerized
ROS 2 Modules

Single Robot
ClusterSingle Robot
ClusterSingle Robot
ClusterSingle RobotCluster

Single RobotCluster

Edge Cluster

Case Study
A: Pick-and-Place with Manipulators

B: Navigation of Mobile Robots

Case Study A: Pick and Place with Manipulators
Onboard
Robot Driver& Controllers

Motion Planning

Gripper Driver& Controller

Task Coordinator

Perception andGrasping Analysis

Placing Analysis

Safety Module

A high performance onboard
computer is requiredCase Study A: Pick and Place with Manipulators
OnboardEdge/Cloud

Robot Driver& Controllers

Motion Planning

Gripper Driver& Controller

ManualSetup

Task Coordinator

Perception andGrasping Analysis

Placing Analysis

Safety Module

Case Study A: Pick and Place with Manipulators
Classic Setup-Setup networking
Configure DDS domain ID
Set namespace for multi-robots
Bash-script to start different processes
OnboardEdge/Cloud

Robot Driver& Controllers

Motion Planning

Gripper Driver& Controller

Perception andGrasping Analysis

Task Coordinator

Safety Module

Placing Analysis

Case Study A: Pick and Place with Manipulators
Classic Setup-Setup networking
Configure DDS domain ID
Set namespace for multi-robots
Bash-script to start different processes
OnboardEdge/Cloud

Robot Driver& Controllers

Motion Planning

Gripper Driver& Controller

Edge/Cloud
Motion Planning
Motion Planning

Perception andGrasping Analysis

Task Coordinator

Safety ModuleIt becomes more complex
as the system scale increases.

Placing Analysis

Perception andPerception andGrasping Analysis
Grasping Analysis

Placing Analysis
Placing Analysis

Case Study A: Pick and Place with Manipulators
Classic Setup

Each module is run in an isolated container-

Manageable hardware via ClusterInventory-

Declarative software modules in ApplicationDeployment

Robot Driver& Controllers
Gripper Driver& Controller

Perception and
Grasping Analysis

Task Coordinator

Safety Module

Placing Analysis

Task Coordinator

Safety Module

Motion Planning

Perception andGrasping Analysis

Placing Analysis

Kubernetes Cluster

KubeROS Platform

Motion Planning

Pod

Robot Driver& Controllers
Gripper Driver& Controller

Edge/Cloud

PodOnboard

Pod

Edge/Cloud

PodOnboard

Pod-Pod

Setup networking
Configure DDS domain ID
Set namespace for multi-robots
Bash-script to start different processes

Pod-

With Kubernetes and KubeROS

Case Study A: Pick and Place with Manipulators
With Kubernetes and KubeROS-

Manageable hardware via ClusterInventory-

Declarative software modules in ApplicationDeployment$ kuberos cluster update -f ur-cluster.yaml

Safety Module

Pod

Motion Planning

Perception andGrasping Analysis

Placing Analysis

Kubernetes Cluster

KubeROS Platform

Task Coordinator

Pod

Gripper Driver& Controller

Pod

Robot Driver& Controllers

Pod

Edge/Cloud

PodOnboard

Pod

Describe the infrastructure in the ClusterInventorywith robot specifications

Each module is run in an isolated container

Pod

Infrastructure as a Code (IaaC):-

Case Study A: Pick and Place with Manipulators
With Kubernetes and KubeROS-

Manageable hardware via ClusterInventory-

Declarative software modules in ApplicationDeployment$ kuberos deploy create -f ur-manipulation.yaml

Safety Module

PodMotion Planning

Perception andGrasping Analysis

Placing Analysis

Kubernetes Cluster

KubeROS Platform

Task Coordinator

Pod

Gripper Driver& Controller

Pod

Robot Driver& Controllers

Pod

Edge/Cloud

PodOnboard

Pod

container image, launch parameters,
ROS parameters, deployment requirements

Each module is run in an isolated container

Pod

ROS 2 Modules for the deployment:-

Case Study A: Pick and Place with Manipulators
With Kubernetes and KubeROS-

Manageable hardware via ClusterInventory-

Declarative software modules in ApplicationDeployment

Task Coordinator

Pod

Safety Module

Pod[edge]

Perception andGrasping Analysis

Placing Analysis

Kubernetes Cluster

KubeROS Platform

Gripper Driver& Controller

Motion Planning

Pod

Robot Driver& Controllers

Pod[onboard]

Edge/Cloud

PodOnboard

Podchange the preference

Each module is run in an isolated container

Pod

Offload to the edge:-Case Study A: Pick and Place with Manipulators
With Kubernetes and KubeROS-

Manageable hardware via ClusterInventory-

Declarative software modules in ApplicationDeployment

Safety Module

Perception andGrasping Analysis

Placing Analysis

Kubernetes Cluster

KubeROS Platform

Task Coordinator

Motion Planning

Pod

Pod

Gripper Driver& Controller

Pod

Robot Driver& Controllers

Pod

V0.2.6

Edge/Cloud

PodOnboard

Podupdate the container image tag or address

Each module is run in an isolated container

Pod

Software update:-Case Study A: Pick and Place with Manipulators
With Kubernetes and KubeROS-

Manageable hardware via ClusterInventory-Declarative software modules in ApplicationDeployment

Safety Module

Pod

Placing Analysis

Kubernetes Cluster

KubeROS Platform

Task Coordinator

Perception andGrasping Analysis

Pod

Gripper Driver& Controller

Motion Planning

Pod

Robot Driver& Controllers

Pod

Edge/Cloud

PodOnboard

Podremove the specified targetRobots

Each module is run in an isolated container

Pod

Scale to the entire fleet:-Case Study A: Pick and Place with Manipulators
Classic Setup-Advantages with KubeROS

Setup networking
Configure DDS domain ID
Set namespace for multi-robots
Bash-script to start different processes-

Easy access to the edge/cloud-

Flexibility and adaptability

OnboardEdge/Cloud-

Scalability

Robot Driver& Controllers

Motion Planning-

Reusability of modules-

Simplified launch files-

Reproducible deployment

Gripper Driver& Controller

Perception andGrasping Analysis

Task Coordinator

Safety Module

Placing Analysis

Case Study B: Navigation of Mobile Robots

OnboardEdge/CloudDrivers &Controllers

Navigation(with AMCL)

Task Controller

Navigation(with RTABMap)

Safety Module

Navigation(SLAMToolbox)

Utility Containers for Development
Simulation(Gazebo)Contributors: Frederik Pasch; Florian Mirus

Rviz2

RosbagRecordinghttps://github.com/kuberos-io/kuberos-nav-slamCase Study B: Navigation of Mobile Robots

Drivers

Controllers

Simulation

Navigation(AMCL)

Navigation(RTABMap)

Navigation

Deployment
Manifest

KubeROS-CLI

ClusterInventory

Fleet
Manifest(SLAMToolbox)

Containers

Manifests

KubeROS PlatformTurtleBot 2 (K8s Node)
TurtleBot 1 (K8s Node)

Edge Server (K8s Node)
Edge Server (K8s Node)Controllers

Drivers

TaskController

DaemonContainer

Navigation(Turtlebot 1)

Navigation(Turtlebot 2)

Pod

Pod

Pod

Pod

Pod

Pod

DDS

Kubernetes Container Network Interface (CNI) Plugin
Network InfrastructureManaged Cluster (TurtleBots and Edge Server)

Contributors: Frederik Pasch; Florian Mirus

https://github.com/kuberos-io/kuberos-nav-slamSummary
Benefits with KubeROS:•Simplify access to the edge or the cloud•Improved software flexibility, adaptability, und maintainability•Enable large-scale deployment•

Easy to use (after setup)

Problems:•Kubernetes setup requires experience•

A general hardware interface•

Application decomposition and containerization (granularity)

KubeROS status:•Under active development [Prototype]•Parts of the code available in GitHub: github.com/kuberos-io

KubeROS

Vielen Dank!
ROS 2 Modules
cloudedge

Configuration

Kontakt: Yongzhou Zhang
yongzhou.zhang.d@gmail.com
Acknowledgment: parts of this work are conducted as part of the project KI5GRob funded by German Federal Ministry ofEducation and Research (BMBF).

github.com/kuberos-io