Repository: rabbitmq/rabbitmq-server (language: javascript)

Issue:
Introduce a way to tag clusters (and, optionally, nodes)
Per discussion with @SimonUnge but also @stefanmoser @mkuratczyk.

## Problem Definition

Larger users can have thousands of clusters used for all kinds of purposes. Sometimes it may be
necessary to perform certain operations (e.g. upgrades) on a subset of them, e.g. those that are
used by development environments only.

Right now there only so many ways of doing it:

1. Keep this cluster metadata in an external store. This is not always an option
2. Prefix cluster name with `production-*` or something like that. This is fragile and allows for only a couple of tags

## Cluster Tags

One solution can be configuring cluster metadata with a map (a set of pairs) in `rabbitmq.con`:

```ini
cluster_tags.environment = production

cluster_tags.region = us-east
cluster_tags.az = us-east-3
```

This map can be then added to `GET /api/overview` HTTP API responses. In Prometheus responses, this could look something like this (**please read the comments**, with Prometheus it is much more nuanced than it sounds):

```
rabbitmq_identity_cluster_tags{environment = "production", region = "us-east", az = "us-east-3"}
```

This data can be stored in the node's application environment or in a global runtime parameter
(for "last write wins" consistency).

## Node Tags

The same idea can be extended to node tags that will only be stored in the application
environment since this data is local to the node. 

```ini
node_tags.environment = production

node_tags.region = us-east
node_tags.az = us-east-3
```

Which could look approximately like this in Prometheus output (**please read the comments**, with Prometheus it is much more nuanced than it sounds):

```
rabbitmq_identity_node_tags{environment = "production", region = "us-east", az = "us-east-3"}
```

The usefulness of this is less clear.




Discussion / hints:
From a pure Prometheus point of view, I see two potential anti-patterns here:

1.
The suggested new Prometheus metrics `rabbitmq_identity_cluster_tags ` and `rabbitmq_identity_node_tags` are info metrics. According to Prometheus best practices:
> The gauge should have the suffix `_info`

Hence, instead of introducing new Prometheus info metrics, I think it's better to re-use the existing Prometheus metric `rabbitmq_identity_info`.

2.
These labels are meant to be **target labels** rather than **instrumentation labels**. In other words, RabbitMQ itself should probably not emit labels such as `{environment = "production", region = "us-east", az = "us-east-3"}`.
This is explained in "Prometheus: Up & Running":
> Labels come from two sources, instrumentation labels and target labels. When you are
working in PromQL there is no difference between the two, but it’s important to dis‐
tinguish between them in order to get the most benefits from labels.
Instrumentation labels, as the name indicates, come from your instrumentation.
They are about things that are known inside your application or library, such as the
type of HTTP requests it receives, which databases it talks to, and other internal
specifics.
Target labels identify a specific monitoring target; that is, a target that Prometheus
scrapes. A target label relates more to your architecture and may include which appli‐
cation it is, what datacenter it lives in, if it is in a development or production environ‐
ment, which team owns it, and of course, which exact instance of the application it is.
Target labels are attached by Prometheus as part of the process of scraping metrics.
Different Prometheus servers run by different teams may have different views of what
a “team,” “region,” or “service” is, **so an instrumented application should not try to
expose such labels itself**. Accordingly, you will not find any features in client libraries
to add labels across all metrics of a target. Target labels come from service discovery
and relabelling and are discussed further in Chapter 8.
@ansd this feature is not 100% Prometheus-specific, so even if the Prometheus view of the world assumes that these are "deployment labels", RabbitMQ will have to log and expose them e.g. in the HTTP API and CLI tools in order to make them useful.

@SimonUnge will configuring your monitoring and upgrade tooling be possible to comply with the monitoring target concept quoted above?
@michaelklishin yes, my comment above was purely about Prometheus' point of view. Tagging/labelling RabbitMQ nodes with key/value metadata and exposing this metadata via the HTTP API, CLI tools and logs makes sense to me.

Fix patch:
diff --git a/deps/rabbit/priv/schema/rabbit.schema b/deps/rabbit/priv/schema/rabbit.schema
index e930ddbf0fcd..f6ccf7037bb2 100644
--- a/deps/rabbit/priv/schema/rabbit.schema
+++ b/deps/rabbit/priv/schema/rabbit.schema
@@ -2688,6 +2688,20 @@ fun(Conf) ->
     end
 end}.
 
+{mapping, "cluster_tags.$tag", "rabbit.cluster_tags", [
+    {datatype, [binary]}
+]}.
+
+{translation, "rabbit.cluster_tags",
+fun(Conf) ->
+    case cuttlefish:conf_get("cluster_tags", Conf, undefined) of
+        none -> [];
+        _ ->
+            Settings = cuttlefish_variable:filter_by_prefix("cluster_tags", Conf),
+            [ {list_to_binary(K), V} || {[_, K], V} <- Settings]
+    end
+end}.
+
 % ===============================
 % Validators
 % ===============================
diff --git a/deps/rabbit/src/rabbit.erl b/deps/rabbit/src/rabbit.erl
index f6f6fa364278..4a8c0b62d467 100644
--- a/deps/rabbit/src/rabbit.erl
+++ b/deps/rabbit/src/rabbit.erl
@@ -38,7 +38,7 @@
 
 %%---------------------------------------------------------------------------
 %% Boot steps.
--export([maybe_insert_default_data/0, boot_delegate/0, recover/0,
+-export([update_cluster_tags/0, maybe_insert_default_data/0, boot_delegate/0, recover/0,
          pg_local_amqp_session/0,
          pg_local_amqp_connection/0]).
 
@@ -208,6 +208,12 @@
                     {requires,    recovery},
                     {enables,     routing_ready}]}).
 
+
+-rabbit_boot_step({cluster_tags,
+                   [{description, "Set cluster tags"},
+                    {mfa,         {?MODULE, update_cluster_tags, []}},
+                    {requires,    core_initialized}]}).
+
 -rabbit_boot_step({routing_ready,
                    [{description, "message delivery logic ready"},
                     {requires,    [core_initialized, recovery]}]}).
@@ -1138,6 +1144,15 @@ pg_local_amqp_connection() ->
 pg_local_scope(Prefix) ->
     list_to_atom(io_lib:format("~s_~s", [Prefix, node()])).
 
+
+-spec update_cluster_tags() -> 'ok'.
+
+update_cluster_tags() ->
+    Tags = application:get_env(rabbit, cluster_tags, []),
+    ?LOG_DEBUG("Seeding cluster tags from application environment key...",
+                       #{domain => ?RMQLOG_DOMAIN_GLOBAL}),
+    rabbit_runtime_parameters:set_global(cluster_tags, Tags, <<"internal_user">>).
+
 -spec maybe_insert_default_data() -> 'ok'.
 
 maybe_insert_default_data() ->
diff --git a/deps/rabbitmq_cli/lib/rabbitmq/cli/ctl/commands/cluster_status_command.ex b/deps/rabbitmq_cli/lib/rabbitmq/cli/ctl/commands/cluster_status_command.ex
index bc5ca76ca0bc..70bc8f3de5bc 100644
--- a/deps/rabbitmq_cli/lib/rabbitmq/cli/ctl/commands/cluster_status_command.ex
+++ b/deps/rabbitmq_cli/lib/rabbitmq/cli/ctl/commands/cluster_status_command.ex
@@ -33,7 +33,7 @@ defmodule RabbitMQ.CLI.Ctl.Commands.ClusterStatusCommand do
   use RabbitMQ.CLI.Core.RequiresRabbitAppRunning
 
   def run([], %{node: node_name, timeout: timeout} = opts) do
-    status =
+    status0 =
       case :rabbit_misc.rpc_call(node_name, :rabbit_db_cluster, :cli_cluster_status, []) do
         {:badrpc, {:EXIT, {:undef, _}}} ->
           :rabbit_misc.rpc_call(node_name, :rabbit_mnesia, :status, [])
@@ -45,11 +45,13 @@ defmodule RabbitMQ.CLI.Ctl.Commands.ClusterStatusCommand do
           status
       end
 
-    case status do
+    case status0 do
       {:badrpc, _} = err ->
         err
 
-      status ->
+      status0 ->
+        tags = cluster_tags(node_name, timeout)
+        status = status0 ++ [{:cluster_tags, tags}]
         case :rabbit_misc.rpc_call(node_name, :rabbit_nodes, :list_running, []) do
           {:badrpc, _} = err ->
             err
@@ -122,7 +124,6 @@ defmodule RabbitMQ.CLI.Ctl.Commands.ClusterStatusCommand do
 
   def output(result, %{node: node_name}) when is_list(result) do
     m = result_map(result)
-
     total_cores = Enum.reduce(m[:cpu_cores], 0, fn {_, val}, acc -> acc + val end)
 
     cluster_name_section = [
@@ -131,6 +132,15 @@ defmodule RabbitMQ.CLI.Ctl.Commands.ClusterStatusCommand do
       "Total CPU cores available cluster-wide: #{total_cores}"
     ]
 
+    cluster_tag_section =
+      [
+        "\n#{bright("Cluster Tags")}\n"
+      ] ++
+      case m[:cluster_tags] do
+        [] -> ["(none)"]
+        tags -> cluster_tag_lines(tags)
+      end
+
     disk_nodes_section =
       [
         "\n#{bright("Disk Nodes")}\n"
@@ -210,6 +220,7 @@ defmodule RabbitMQ.CLI.Ctl.Commands.ClusterStatusCommand do
 
     lines =
       cluster_name_section ++
+        cluster_tag_section ++
         disk_nodes_section ++
         ram_nodes_section ++
         running_nodes_section ++
@@ -260,6 +271,7 @@ defmodule RabbitMQ.CLI.Ctl.Commands.ClusterStatusCommand do
     #           {rabbit@warp10,[{resource_limit,memory,rabbit@warp10}]}]}]
     %{
       cluster_name: Keyword.get(result, :cluster_name),
+      cluster_tags: result |> Keyword.get(:cluster_tags, []),
       disk_nodes: result |> Keyword.get(:nodes, []) |> Keyword.get(:disc, []),
       ram_nodes: result |> Keyword.get(:nodes, []) |> Keyword.get(:ram, []),
       running_nodes: result |> Keyword.get(:running_nodes, []) |> Enum.map(&to_string/1),
@@ -383,6 +395,18 @@ defmodule RabbitMQ.CLI.Ctl.Commands.ClusterStatusCommand do
     {node, result}
   end
 
+  defp cluster_tags(node, timeout) do
+    case :rabbit_misc.rpc_call(
+          node,
+          :rabbit_runtime_parameters,
+          :value_global,
+          [:cluster_tags],
+          timeout) do
+      :not_found -> []
+      tags -> tags
+    end
+  end
+
   defp node_lines(nodes) do
     Enum.map(nodes, &to_string/1) |> Enum.sort()
   end
@@ -413,4 +437,10 @@ defmodule RabbitMQ.CLI.Ctl.Commands.ClusterStatusCommand do
   defp maintenance_lines(mapping) do
     Enum.map(mapping, fn {node, status} -> "Node: #{node}, status: #{status}" end)
   end
+
+  defp cluster_tag_lines(mapping) do
+    Enum.map(mapping, fn {key, value} ->
+      "#{key}: #{value}"
+    end)
+  end
 end
diff --git a/deps/rabbitmq_management/src/rabbit_mgmt_wm_overview.erl b/deps/rabbitmq_management/src/rabbit_mgmt_wm_overview.erl
index 24ab67ce8f49..817211200f70 100644
--- a/deps/rabbitmq_management/src/rabbit_mgmt_wm_overview.erl
+++ b/deps/rabbitmq_management/src/rabbit_mgmt_wm_overview.erl
@@ -47,6 +47,7 @@ to_json(ReqData, Context = #context{user = User = #user{tags = Tags}}) ->
                  {product_name,              list_to_binary(rabbit:product_name())},
                  {rabbitmq_version,          list_to_binary(rabbit:base_product_version())},
                  {cluster_name,              rabbit_nodes:cluster_name()},
+                 {cluster_tags,              cluster_tags()},
                  {erlang_version,            erlang_version()},
                  {erlang_full_version,       erlang_full_version()},
                  {release_series_support_status, rabbit_release_series:readable_support_status()},
@@ -182,3 +183,10 @@ transform_retention_intervals([{MaxAgeInSeconds, _}|Rest], Acc) ->
                      0
              end,
     transform_retention_intervals(Rest, [AccVal|Acc]).
+
+cluster_tags() ->
+    case rabbit_runtime_parameters:value_global(cluster_tags) of
+        not_found ->
+            [];
+        Tags -> Tags
+    end.


Test patch:
diff --git a/deps/rabbitmq_management/test/rabbit_mgmt_http_SUITE.erl b/deps/rabbitmq_management/test/rabbit_mgmt_http_SUITE.erl
index d517cb4810a8..3fa8a6e697c7 100644
--- a/deps/rabbitmq_management/test/rabbit_mgmt_http_SUITE.erl
+++ b/deps/rabbitmq_management/test/rabbit_mgmt_http_SUITE.erl
@@ -201,7 +201,8 @@ all_tests() -> [
     disabled_qq_replica_opers_test,
     qq_status_test,
     list_deprecated_features_test,
-    list_used_deprecated_features_test
+    list_used_deprecated_features_test,
+    cluster_tags_test
 ].
 
 %% -------------------------------------------------------------------
@@ -282,6 +283,12 @@ init_per_testcase(Testcase = disabled_qq_replica_opers_test, Config) ->
     rabbit_ct_broker_helpers:rpc_all(Config,
       application, set_env, [rabbitmq_management, restrictions, Restrictions]),
     rabbit_ct_helpers:testcase_started(Config, Testcase);
+init_per_testcase(Testcase = cluster_tags_test, Config) ->
+    Tags = [{<<"az">>, <<"us-east-3">>}, {<<"region">>,<<"us-east">>}, {<<"environment">>,<<"production">>}],
+    rpc(
+      Config, rabbit_runtime_parameters, set_global,
+      [cluster_tags, Tags, none]),
+    rabbit_ct_helpers:testcase_started(Config, Testcase);
 init_per_testcase(queues_detailed_test, Config) ->
     IsEnabled = rabbit_ct_broker_helpers:is_feature_flag_enabled(
                   Config, detailed_queues_endpoint),
@@ -348,6 +355,11 @@ end_per_testcase0(disabled_operator_policy_test, Config) ->
 end_per_testcase0(disabled_qq_replica_opers_test, Config) ->
     rpc(Config, application, unset_env, [rabbitmq_management, restrictions]),
     Config;
+end_per_testcase0(cluster_tags_test, Config) ->
+    rpc(
+      Config, rabbit_runtime_parameters, clear_global,
+      [cluster_tags, none]),
+    Config;
 end_per_testcase0(Testcase, Config)
   when Testcase == list_deprecated_features_test;
        Testcase == list_used_deprecated_features_test ->
@@ -3936,6 +3948,14 @@ list_used_deprecated_features_test(Config) ->
     ?assertEqual(list_to_binary(Desc), maps:get(desc, Feature)),
     ?assertEqual(list_to_binary(DocUrl), maps:get(doc_url, Feature)).
 
+cluster_tags_test(Config) ->
+    Overview = http_get(Config, "/overview"),
+    Tags = maps:get(cluster_tags, Overview),
+    ExpectedTags = #{az => <<"us-east-3">>,environment => <<"production">>,
+                     region => <<"us-east">>},
+    ?assertEqual(ExpectedTags, Tags),
+    passed.
+
 %% -------------------------------------------------------------------
 %% Helpers.
 %% -------------------------------------------------------------------