Package: cloud-cost-allocation
Cloud cost allocationOverviewThis project provides tools forshared, hierarchical cost allocationbased on user-defined usage metrics.IntroductionMainstreamcloud cost managementtoolsfrom the Cloud Service Providers or 3rd party vendors typicallyrely onthe high level hierarchy of cloud objects (Resource containers, likeProjectsinGoogle Cloud Platform, orSubscriptionsandResource GroupsinAzure) ortags to allocate coststo a given cost center or business unit.
In addition, some tools support one level of cost re-allocation (e.g. based on fixed percentages, overall/specific costs of the consumers). Usually, this does not include the ability to use dynamic, custom defined metrics though.
As a result, the cost allocation process of shared services is often rather:Imprecise, thus unfair, since cost allocation is not done based on the actual consumptionIncomplete, since shared costs, especially if hierarchical, are simply not allocated to their consumers, thus requiring additional tooling on topOpaqueand thus hard to action, since neither the key cost drivers nor their actual consumption (and thus angles for optimization) are visible to the consumersExampleIn this example, classic, usage-based cost allocation would likely stop at the cloud resource level. Thus, only the container service would be visible. With additional instrumentation these costs could be split into the direct costs for each of the services running on it. But the costs of the shared monitoring or history service would not be properly allocated, let alone their contributions to the business services behind that.
This might not be a problem as long as those costs are small and the hierarchy of services using each other is shallow. But in large production environments the key cost drivers for a given business service might thus be hidden making it very difficult to assess the total costs of a given service.
Usually, the only way to resolve this issue is avoid sharing services by duplicating them. While this can work in some cases it often also results in higher overall costs, both for deploying and for managing these services.The goal of the cost allocation model proposed here is to support such complex consumption hierarchies in a usage-based fashion with custom (North Star) Metrics up the business service level.
This does not only enable the more accurate and complete cost allocation required to make proper business decisions and forecasts, but also yields more actionable outputs thanks to the transparency on unit costs, consumption and their evolution over time.
Furthermore, this also allows service owners to easily identify their main cost drivers in the service hierarchy. This in turn helps to ensure that optimization efforts can be focused on the most promising services as well as to simulate and later measure the actual benefits this can bring to all services along the consumption chain.DefinitionsTermDefinitionServiceA technical workload, operated by DevOps, which can either run directly on the cloud, or consume other services, or both.ProductA business product, which relies on one or multiple services. Products are somehow commercial packaging of services under the control of Product Managers.Product DimensionAn optional dimension for the cost report of a product, for example: a product feature, a product flavor, a market or a market segment.Provider ServiceThe role of a service when it is consumed by another other service or product, a.k.a. a shared service.Consumer ServiceThe role of a service when it consumes another service.InstanceA running instance or a deployment of a service. If we think ofServicesasClassesin object-oriented programming, thenInstanceswould be theObjects.DimensionAn optional dimension in the cost allocation, for example an environment like Test or Production, or a component in the architecture of the service.MeterA meter that measures the functional activity or throughput of a service or of a product.Amortized CostThe billing cost incremented with the amortized cost of the commitment-based purchases (reservations, saving plans).On-demand CostThe equivalent on-demand cost, i.e. as if no commitment-based purchase had been made.Cost ItemA cost that is either coming from cloud provider billing or allocated by a provider service.Cost Allocation KeyA numerical value to allocate a share of a cost. Cost share = cost * key / total keys.UML Class DiagramKeep calm and drink coffee.Model principles and considerationsThe model relies solely on cloud resource tags in order to work transparently for all cloud providersNote that policies can be used to propagate tags from cloud provider containers (GCP Projects, Azure Subscriptions or Resource Groups, ...) to individual resources if requiredOn top of that, the model uses cost allocation data provided by the service ownersIf a service owner provides no cost allocation data, the service becomes a leaf in the hierarchy and thus owns all costs allocated along the chainCost allocation data is based on keys: the cost of a service is split and dispatched proportionally to cost allocation keysIdeally, all cloud costs should be allocated to products. This means every service owner should provide cost allocation data to allocate the service cost to consumer services and/or productsServices, products and meters are discovered and linked through tags and through cost allocation dataIt is strongly recommended to use policies or other means to cross-check these names against the relevant product and service catalogsThe pivot of the cost allocation is theInstance: service cost allocation data is per instanceService owners should design carefully service instances, in order to be able to generate cost allocation data for each of themNaming conventions are recommended to reflect the design of service instance, and possibly to provide some information about every service instanceAll other dimensions (e.g. environment) are not structuring for the cost allocation between instances, but they provide commonly used dimensions for cost reportingBoth products and services have their metersProduct meters are defined by product owners (or mangers), while service meters are defined by service ownersA product meter measures the business volume served by the productProduct meters are used to monitor cost efficiency at product level, to perform cloud cost forecast in budget activities, to optimize pricing, and more generally to take data-driven business decisionsA service meter measures the usage of a provider service by consumer services and products, and so it must be reflected in cost allocation keysService meters are used to monitor cost efficiency, to detect drifts, to track the benefit of cloud cost optimizations, and more generally to serve as KPIs for service cloud costsA given product or service can have more than one meterExample 1: container serviceA meter of a container service could be the combination of container CPU request and container Memory requestIn this case the Cost Allocation Keys be computed out of the two metricsExample 2: shared data lakeHere the cost driver could be the amount of data being storedSince this volume is hard to predict and optimize, this could be further broken down into the daily volume stored + the retention timeCost items are taggedCost item tags comes from cloud resource tags and from provider servicesNote thatDimensionscan be seen conceptually as specifictagsthat have been promoted as first-class citizens in the model for cost reportingTheProviderTagSelectorfield in cost allocation data enables a finer grained cost selection and allocation along additional axis (e.g. customer)A cost allocation key can allocate cost both to a service and to a productService cost allocation and product cost allocation actually work in parallelThe testtest2illustrates how these cost allocation flows work togetherService cost allocation basically propagates costs through cost allocation keysThe cost of a service is the sum of the costs of cloud resources it uses directly and of the cost shares of the other services it consumesContrary to this, only the following costs are assigned to products from each serviceThe costs of cloud resources it uses directly andAny costs from consumed services that did not report their costs to productsThis ensures that there is no double counting of costs at product levelAssuming that all cloud resources are properly tagged and that services eventually allocate all their costs to products, then the costs of all cloud resources must be equal to the cost of all productsSimilarly, the costs of a product that exclusively uses a single service would be equal to the total cost of that serviceOn-demand costs and amortized costs are supported in parallel by the modelReporting the two is useful to investigate whether cost variations come from commitment management or something elseReporting the two is also useful for service owners and product managers alike to see the savings made thanks to the commitment-based purchasesCommitment savings = On-demand costs - amortized costsCycles in the cost allocation can be broken by providing aservice precedence listExample: A monitoring systemMrunning on top of a container serviceCis also used by the container service itselfIn this case, part of the costs ofMshould be assigned back toCSince the costs ofCare allocated toM(in part), this creates a loopThis cost allocation cycleC->M->Ccan be broken by specifying a precedence listC,M. In this case,Cis allocated no cost fromMAs another example, seetest5The cost allocation could manage cycles in a more sophisticated way in the futureAs a general rule, the names of all model objects (products, services, instances, tags, ...) are normalized to lower case, in order avoid low-high case mistakesBittersweet ways for a service to allocate its costThe standard way for a service to its allocate cost is to provide dynamic cost allocation data containing cost allocation keys.
Three alternatives exist to this, and they should be used carefully:Use specific configurableconsumer service and product tagsWhile providing an easy way to dynamically allocate costs, it does not offer any means to track usage and thus provides no hint to consumers on what drives their costs (black box)See sectionConsumer service and product tagsfor detailsUse specificcost allocation typein cost allocation dataE.g. to allocate a cost share proportional to the amortized costs of the consumersSee sectionCost allocation types in cost allocation databelow for detailsUse a static cost allocation fileInstead of generating and using accurate consumption data, a static approach (potentially updated at certain intervals) might be sufficient in some cases and in crawl phaseConsumer service and product tagsSpecific configurable tags can be used by a service to allocate its cost to consumer services and to products.
Consumer tags are convenient to get cost allocation keys generated automatically based on tagging.Technically, for everyCostItemhaving consumer tags, a newConsumerCostItemis created with:ProviderServiceandProviderInstance: the ones from theCostItem.ConsumerServiceandConsumerInstance: the ones from the consumer tags.Product: the one from the product tag.ProviderTagSelector: an expression matching the consumer tags.ProviderCostAllocationKey: value 1.0.
Examples are available in the teststest3andtest4.Cost allocation types in cost allocation dataThe default value ofProviderCostAllocationTypein cost allocation data is'Key'.
Three other types of cost allocation are supported:'CloudTagSelector'The consumer service instances are not explicitly specified in the cost allocation dataThey are inferred as the ones consuming the cloud cost items whose tags match theProviderCostAllocationCloudTagSelectorexpressionThe cost allocation keys are set as the amortized costs of the matching cloud cost itemsThis type of cost allocation is convenient to allocate costs to consumer services that run directly on the cloud proportionally to their cloud costsSee example in the testtest3'Cost'The cost is allocated to the consumer services proportionally to their amortized costsTechnically, amortized costs are globally allocated to service instances in a first time, while ignoring this type of cost allocationThe amortized costs resulting from this cost allocation are used in a second pass as cost allocation keys to perform the actual cost allocationAlthough this type of cost allocation is convenient to avoid bothering about cost allocation keys, it can produce results that are hardly predictable and understandable, depending on the complexity of the global cost allocation graphSee example in the testtest3'DefaultProduct'By default, the cost of the provider service is allocated to the specified product(s)This type of cost allocation is convenient to implement centrally a default cost allocation to products, which is overriden by service owners when the latter provide cost allocation dataSee example in the testtest8Examples of'CloudTagSelector'and'Cost'cost allocation types are available in the testtest3.In theory, an even more general type of cost allocation could be considered:'ConsumerTagSelector'Cost would be allocated to service instances whose cost items match theProviderCostAllocationConsumerTagSelectorexpression proportionally to their amortized costsWhile being more generic, this type of cost allocation can be even less predictableThis type of cost allocation has not been required and thus not implemented yetAllocation of further amounts [experimental]After cost allocation, custom amounts, which are calledfurther amounts, can be allocated.
Examples offurther amountsare CO2 and electricity power.
Thefurther amountsto allocate must be first loaded into the cost items, before the allocation is run for them.
Thefurther amountscan be loaded into cloud cost items, or even into consumer cost items: in both cases, the further amounts are allocated along the cost allocation graph.
Custom allocation keys can be used to allocatefurther amounts.ConfigurationPythonconfigparseris used to manage the configuration. Here are the configuration parameters with examples:[General]

# The date format, which is used both in input cost allocation data and output allocated cost data
# The format uses Python datetime syntax
# Do not forget to escape the % character
DateFormat = %%Y-%%m-%%d.

# The name of the default service to use for cost allocation when the service tag is missing
DefaultService = x

# The name of the default product to allocate costs (optional)
DefaultProduct = x

# The dimensions
Dimensions = Component,Environment

# The number of provider meter columns in input cost allocation data and output allocated cost data
NumberOfProviderMeters = 2

# The number of product dimensions in input cost allocation data and output allocated cost data
NumberOfProductDimensions = 2

# The number of product meter columns in input cost allocation data and output allocated cost data
NumberOfProductMeters = 3

[TagKey]

# The tag keys for service and instance
# In case of list, first matching key is used
Service = service,svc
Instance = instance

# The tag keys for the configured dimensions
# In case of list, first matching key is used
Component = component,comp
Environment = environment

# The consumer tag keys for service, instance and the configured dimensions
# In case of list, first matching key is used
ConsumerService = consumer_service
ConsumerInstance = consumer_instance
ConsumerComponent = consumer_component
ConsumerEnvironment = consumer_environment

# List of values to be ignored for the consumer tag
ConsumerServiceIgnoredValue = n/a

# The product tag key
# In case of list, first matching key is used
Product = product

[AzureEaAmortizedCost]

# The service, instance, and dimensions to allocate Azure unused reservation cost
UnusedReservationService = finops
UnusedReservationInstance = unused-commitment
UnusedReservationComponent = reservation

# The service, instance, and dimensions to allocate Azure unused savings plan cost
UnusedSavingsPlanService = finops
UnusedSavingsPlanInstance = unused-commitment
UnusedSavingsPlanComponent = savings-plan

[Cycles]

# The precedence list for breaking cost allocation cycles
# Here, the cycle a -> b -> d ->a will be broken to a -> b -> d (cost allocation d -> a is removed)
ServicePrecedenceList = a,b,c

# The maximum number of cycles to break, as a safe guard to avoid programming loops or
# unexpected long execution time
MaxBreaks = 10

[FurtherAmounts]

# Further amounts to allocate
Amounts = Co2,ElectricityPower

# Allocation keys used for further amounts
AllocationKeys = Co2Key

# The allocation keys to use for further amounts
AmountAllocationKeys = Co2:Co2Key,ElectricityPower:ProviderCostAllocationKeyTestRun the following command in the repository root directory:python -m unittest -v tests.testUsageThe code for a simple command line inmain.pyis a basic example to:Read cloud costsRead cost allocation keysAllocate costsWrite allocated costsThetest.pyfile in thetestsdirectory illustrates how to add custom code and columns in the allocated costs.ContributingBug reportsBugs are both proofs of success and of failure: what do you think?
Before submitting a bug, please check through theGitHub issues,
both open and closed, to confirm that the bug hasn't been reported before.Feature requestsIf you think a feature is missing and could be useful in this module, feel free to raise a feature request through theGitHub issues.
Please keep in mind that the cost allocation model is intended to be as minimal as possible.Code contributionsWhen contributing code, please followthis project-agnostic contribution guide.TODOCost reader for FOCUSSplit and dispatch of the provider meter values when CloudTagSelector is used.Support product dimension in cloud tags