How We Scaled Catalog Ingestion to Hundreds of Thousands of Entities - Damon Kaswell, HP Inc.
Channel: CNCF [Cloud Native Computing Foundation]
Date: 26/10/2022

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io​. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

How We Scaled Catalog Ingestion to Hundreds of Thousands of Entities - Damon Kaswell, HP Inc.

When you need to ingest over 100,000 entities from a data source, you can't just point a Catalog Processor at it and hope for the best. Through a combination of careful planning, properly scaled computing resources, and some clever customization that extends the basic Entity Provider and Catalog Processor functionality, HP Inc. is successfully ingesting hundreds of thousands of separate entities, proving that Backstage is truly ready to scale for the large enterprise. In this presentation, I will:

* Describe the scope of the technical challenge. HP Inc. is using Backstage for many different types of entities from many different sources.
* Do a deep technical dive into how our entity ingestion works, and how we have extended the base Entity Provider mechanism with multiple features that allow us to handle extremely large sets of data.
* Talk about considerations such as entity validation and relations.