Repository: rabbitmq/rabbitmq-server (language: javascript)

Issue:
When a cluster node is removed, consider deleting all non-mirrored queues hosted on it
[See this rabbitmq-users thread for background](https://groups.google.com/forum/#!topic/rabbitmq-users/rpO3dFzm-yQ). If a node is removed from a cluster, all non-mirrored queues hosted on it should be removed or migrated (depending on their durability: durable queues are not migrated when their hosting node is unavailable but transient queues are).

This is somewhat of an edge case. There can be other edge cases with the proposed solution above.

**Update** from 2020-2021: with the introduction of [maintenance mode](https://www.rabbitmq.com/upgrade.html#maintenance-mode) and planned removal of classic queue mirroring in favor of [quorum queues](https://www.rabbitmq.com/quorum-queues.html) and [streams](https://rabbitmq.com/stream.html),
this becomes both less relevant and smaller in scope/less risky of a change.


Discussion / hints:
+1. And users are most likely to be unaware of which queues are residing on the node being removed, in case any are critical, and the exercise of checking this could be tedious (as queue counts increase). Brainstorming a possibility of queue migration policies, putting onus on users to decide how migration is carried out, and for which queues in particular (durable/transient);
- `migrate-to-none` - simply delete all non-mirrored queues on removed node
- `migrate-to-any` - move matching non-mirrored queues to any other cluster node
- `migrate-to-node` - move matching non-mirrored queues to a specific cluster node
- `migrate-to-min-master` - move matching non-mirrored queues to node with least queue masters

Durability definitely a big factor. Migration would also slow down duration of node removal procedure - but, it would/could be the price to pay, in order to retain any non-mirrored queues.
Will this be implemented soon?
Now that there is a way to transfer leadership of a queue to a different node, it is an option for, say, durable queues.
> Now that there is a way to transfer leadership of a queue to a different node, it is an option for, say, durable queues.

@michaelklishin Which method are you referring to? Thanks
@luddd3 I was referring to an internal API introduced for [maintenance mode](https://www.rabbitmq.com/upgrade.html#maintenance-mode).

You can put a node under maintenance before removing it, the node won't keep or accept any client connections and host any queue leader replicas. So the effect would be comparable to what this issue tries to achieve.
**Update** from 2020-2021: with the introduction of [maintenance mode](https://www.rabbitmq.com/upgrade.html#maintenance-mode) and planned removal of classic queue mirroring in favor of [quorum queues](https://www.rabbitmq.com/quorum-queues.html) and [streams](https://rabbitmq.com/stream.html),
this becomes both less relevant and smaller in scope/less risky of a change.
@michaelklishin 
I tried to reproduce the scenario in rabbitmq 3.8.22. When i removed then node from a cluster,all the queue were deleted. Is this issue has been solved?
Below was my reproduce steps
1.  rabbit@rabbitmqservice-0  and  rabbit@rabbitmqservice-1 was a cluster
2. test,test2,test3,test4 ware four queues created in two nodes
```
test     **durable**    **rabbit@rabbitmqservice-0**
test2   transient  rabbit@rabbitmqservice-0
test3   **durable**    **rabbit@rabbitmqservice-1**
test4   transient  rabbit@rabbitmqservice-1
```
3. stop rabbitmqservice-1
```
[rabbitmq@rabbitmqservice-1 rabbitmq-service]$ rabbitmqctl stop_app
Stopping rabbit application on node rabbit@rabbitmqservice-1 ...
[rabbitmq@rabbitmqservice-1 rabbitmq-service]$
```
test3 queue was deleted,  test4 queue was down

4.   remove rabbitmqservice-1 from cluster
```
[rabbitmq@rabbitmqservice-0 /]$ rabbitmqctl  -n rabbit@rabbitmqservice-0 forget_cluster_node rabbit@rabbitmqservice-1
Removing node rabbit@rabbitmqservice-1 from the cluster
```
5. only test,test2 queues were left.
```
[rabbitmq@rabbitmqservice-0 /]$ rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name    messages
test    0
test2   0
```

@polaris-alioth we can't tell since you haven't shared any details on the properties of those queues. They might have been exclusive, for example. We do not guess in this community.
@michaelklishin 
I created four queues, all were non-mirrored.  **test3 was durable, test4 was transient**
test     durable    on rabbit@rabbitmqservice-0
test2   transient  on  rabbit@rabbitmqservice-0
test3   durable    on  rabbit@rabbitmqservice-1
test4   transient  on  rabbit@rabbitmqservice-1
This issue has been fixed a long time ago. However, turns out it now affects RabbitMQ 4.0 with Khepri enabled. Therefore, let's fix this in Khepri before closing this issue.

Fix patch:
diff --git a/deps/rabbit/src/rabbit_amqqueue.erl b/deps/rabbit/src/rabbit_amqqueue.erl
index f38b24d0b8a6..86edf40847c4 100644
--- a/deps/rabbit/src/rabbit_amqqueue.erl
+++ b/deps/rabbit/src/rabbit_amqqueue.erl
@@ -1815,6 +1815,7 @@ internal_delete(Queue, ActingUser, Reason) ->
 %% TODO this is used by `rabbit_mnesia:remove_node_if_mnesia_running`
 %% Does it make any sense once mnesia is not used/removed?
 forget_all_durable(Node) ->
+    rabbit_log:info("Will remove all classic queues from node ~ts. The node is likely being removed from the cluster.", [Node]),
     UpdateFun = fun(Q) ->
                         forget_node_for_queue(Q)
                 end,
diff --git a/deps/rabbit/src/rabbit_khepri.erl b/deps/rabbit/src/rabbit_khepri.erl
index f6a84a6afcac..b763e3137bd4 100644
--- a/deps/rabbit/src/rabbit_khepri.erl
+++ b/deps/rabbit/src/rabbit_khepri.erl
@@ -538,6 +538,7 @@ remove_reachable_member(NodeToRemove) ->
             NodeToRemove, khepri_cluster, reset, [?RA_CLUSTER_NAME]),
     case Ret of
         ok ->
+            rabbit_amqqueue:forget_all_durable(NodeToRemove),
             ?LOG_DEBUG(
                "Node ~s removed from Khepri cluster \"~s\"",
                [NodeToRemove, ?RA_CLUSTER_NAME],
@@ -559,6 +560,7 @@ remove_down_member(NodeToRemove) ->
     Ret = ra:remove_member(ServerRef, ServerId, Timeout),
     case Ret of
         {ok, _, _} ->
+            rabbit_amqqueue:forget_all_durable(NodeToRemove),
             ?LOG_DEBUG(
                "Node ~s removed from Khepri cluster \"~s\"",
                [NodeToRemove, ?RA_CLUSTER_NAME],


Test patch:
diff --git a/deps/rabbit/test/cli_forget_cluster_node_SUITE.erl b/deps/rabbit/test/cli_forget_cluster_node_SUITE.erl
index b088cf68daff..8ceb2825ea30 100644
--- a/deps/rabbit/test/cli_forget_cluster_node_SUITE.erl
+++ b/deps/rabbit/test/cli_forget_cluster_node_SUITE.erl
@@ -37,7 +37,8 @@ groups() ->
                            forget_cluster_node_with_all_last_streams,
                            forget_cluster_node_with_quorum_queues_and_streams,
                            forget_cluster_node_with_one_last_quorum_member_and_streams,
-                           forget_cluster_node_with_one_last_stream_and_quorum_queues
+                           forget_cluster_node_with_one_last_stream_and_quorum_queues,
+                           forget_cluster_node_with_one_classic_queue
                           ]}
     ].
 
@@ -354,6 +355,30 @@ forget_cluster_node_with_one_last_stream_and_quorum_queues(Config) ->
     ?awaitMatch(Members when length(Members) == 2, get_quorum_members(Rabbit, QQ1), 30000),
     ?awaitMatch(Members when length(Members) == 2, get_quorum_members(Rabbit, QQ2), 30000).
 
+forget_cluster_node_with_one_classic_queue(Config) ->
+    [Rabbit, Hare, Bunny] = rabbit_ct_broker_helpers:get_node_configs(Config, nodename),
+
+    assert_clustered([Rabbit, Hare, Bunny]),
+
+    Ch = rabbit_ct_client_helpers:open_channel(Config, Bunny),
+    CQ1 = <<"classic-queue-1">>,
+    declare(Ch, CQ1, [{<<"x-queue-type">>, longstr, <<"classic">>}]),
+
+    ?awaitMatch([_], rabbit_ct_broker_helpers:rabbitmqctl_list(
+                       Config, Rabbit,
+                       ["list_queues", "name", "--no-table-headers"]),
+                30000),
+
+    ?assertEqual(ok, rabbit_control_helper:command(stop_app, Bunny)),
+    ?assertEqual(ok, forget_cluster_node(Rabbit, Bunny)),
+
+    assert_cluster_status({[Rabbit, Hare], [Rabbit, Hare], [Rabbit, Hare]},
+                          [Rabbit, Hare]),
+    ?awaitMatch([], rabbit_ct_broker_helpers:rabbitmqctl_list(
+                      Config, Rabbit,
+                      ["list_queues", "name", "--no-table-headers"]),
+                30000).
+
 forget_cluster_node(Node, Removee) ->
     rabbit_control_helper:command(forget_cluster_node, Node, [atom_to_list(Removee)],
                                   []).