A metric can be split into multiple time series by dimensions. For example, the metric Response latency is monitored for all services owned by the team. The Service category could be used as a dimension to enrich the metric, so we get Response latency split by Service1, Service2, and so on. Each service could be deployed on different machines in multiple data centers, so the metric could be further split by Machine and Data center.
ServiceData centerMachine S1DC1M1 S1DC1M2 S1DC2M3 S1DC2M4 S2DC1M1 S2DC1M2 S2DC2M5 S2DC2M6 ...
Starting from the total Response latency, we can drill down into the metric by Service, Data center and Machine. However, maybe it makes more sense for service owners to use the path Service -> Data center -> Machine, or maybe it makes more sense for infrastructure engineers to use the path Data Center -> Machine -> Service. It all depends on the individual business requirements of your users.
In Metric Advisor, users can specify any path they want to drill down or roll up from one node of the hierarchical topology. More precisely, the hierarchical topology is a directed acyclic graph rather than a tree structure. There's a full hierarchical topology that consists of all potential dimension combinations, like this:
In theory, if the dimension Service has Ls distinct values, dimension Data center has Ldc distinct values, and dimension Machine has Lm distinct values, then there could be (Ls + 1) * (Ldc + 1) * (Lm + 1) dimension combinations in the hierarchical topology.
But usually not all dimension combinations are valid, which can significantly reduce the complexity. Currently if users aggregate the metric themselves, we don't limit the number of dimensions. If you need to use the rollup functionality provided by Metrics Advisor, the number of dimensions shouldn't be more than 6. However, we limit the number of time series expanded by dimensions for a metric to less than 10,000.
The Diagnostic tree tool in the diagnostics page only shows nodes where an anomaly has been detected, rather than the whole topology. This is to help you focus on the current issue. It also may not show all anomalies within the metric, and instead will display the top anomalies based on contribution. In this way, we can quickly find out the impact, scope, and the spread path of the abnormal data. Which significantly reduces the number of anomalies we need to focus on, and helps users to understand and locate their key issues.
For example, when an anomaly occurs on Service = S2 | Data Center = DC2 | Machine = M5, the deviation of the anomaly impacts the parent node Service= S2, which also has detected the anomaly, but the anomaly doesn't affect the entire data center at DC2 and all services on M5. The incident tree would be built as in the below screenshot, the top anomaly is captured on Service = S2, and root cause could be analyzed in two paths which both lead to Service = S2 | Data Center = DC2 | Machine = M5.