Repository: MassTransit/MassTransit

Issue #397: Deadlock publishing a message after start/stop the rabbitmq service
Created: 2015-11-19T17:36:28Z
The issue
I'm having a problem with the latest version of masstransit while publishing to the queue and starting/stopping the RabbitMQ service.
So, if I start the bus and, before I send any message to the queue, stop the rabbitmq service and try to publish the first message it will fail (of course, the service is down). Then, I will start the service again and try to publish again and I will get a deadlock.
I created a unit testing to reproduce this behavior, but I think you will need to run visual studio as administrator to be able to start and stop the service.
Unit test
[Test]
public async Task Should_not_block_when_rabbitmqservices_goes_down()
{
    var rabbitMqService = new ServiceController("RabbitMQ");
    if(rabbitMqService.Status != ServiceControllerStatus.Running)
    {
        rabbitMqService.Start();
    }

    var busControl = Bus.Factory.CreateUsingRabbitMq(x =>
    {
        x.Host(new Uri("rabbitmq://localhost/"), h =>
        {
            h.Username("guest");
            h.Password("guest");
        });
    });

    busControl.Start();
    rabbitMqService.Stop();

    int index = 0;
    while (index < 3)
    {
        Debug.WriteLine("Go for the {0} round...", index + 1);
        try
        {
            //await busControl.Publish<object>(new { Id = 1, Name = "Abc" });
            Debug.WriteLine("Publishing");
            TaskUtil.Await(() => busControl.Publish<object>(new { Id = 1, Name = "Abc" }));
        }
        catch (Exception)
        {
            Debug.WriteLine("Error, starting the rabbitmq again...");
            rabbitMqService.Start();
        }

        index++;
    }

    Assert.IsTrue(true);
}
await busControl.Publish<object>(new { Id = 1, Name = "Abc" });

This code is commented but I did try with it also with the same result.
Log
Go for the 1 round...
2015-11-19 17:33:13,560 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Pipeline.RabbitMqBasicConsumer - Consumer Model Shutdown (amq.ctag-Q4Lafwjtvmyv8hKmYAsc1g), Concurrent Peak: 0, 320-CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-11-19 17:33:13,561 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Pipeline.RabbitMqConsumerFilter - Consumer amq.ctag-Q4Lafwjtvmyv8hKmYAsc1g: 0 received, 0 concurrent
2015-11-19 17:33:13,564 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache - Connecting: guest@localhost:5672/
2015-11-19 17:33:15,570 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache - The connection usage threw an exception
RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:5672
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
   at RabbitMQ.Client.Impl.SocketFrameHandler.Connect(TcpClient socket, AmqpTcpEndpoint endpoint, Int32 timeout)
   --- End of inner exception stack trace ---
   at RabbitMQ.Client.Impl.SocketFrameHandler.Connect(TcpClient socket, AmqpTcpEndpoint endpoint, Int32 timeout)
   at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func`2 socketFactory, Int32 timeout)
   at RabbitMQ.Client.Framing.Impl.ProtocolBase.CreateFrameHandler(AmqpTcpEndpoint endpoint, Func`2 socketFactory, Int32 timeout)
   at RabbitMQ.Client.ConnectionFactory.CreateFrameHandler()
   at RabbitMQ.Client.ConnectionFactory.CreateConnection()
   --- End of inner exception stack trace ---
   at RabbitMQ.Client.ConnectionFactory.CreateConnection()
   at MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache.<SendUsingNewConnection>d__13.MoveNext() in C:\Users\bruno.costa\Documents\GitHub\MassTransit\src\MassTransit.RabbitMqTransport\Integration\RabbitMqConnectionCache.cs:line 94
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache.ConnectionScope.<Attach>d__5.MoveNext() in C:\Users\bruno.costa\Documents\GitHub\MassTransit\src\MassTransit.RabbitMqTransport\Integration\RabbitMqConnectionCache.cs:line 213
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache.<SendUsingExistingConnection>d__14.MoveNext() in C:\Users\bruno.costa\Documents\GitHub\MassTransit\src\MassTransit.RabbitMqTransport\Integration\RabbitMqConnectionCache.cs:line 159
Error, starting the rabbitmq again...
Go for the 2 round...
2015-11-19 17:33:15,582 [AMQP Connection amqp://localhost:5672] ERROR MassTransit.RabbitMqTransport.RabbitMqReceiveTransport - RabbitMQ connection failed: Connect failed: guest@localhost:5672/
2015-11-19 17:33:15,583 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache - Connecting: guest@localhost:5672/
2015-11-19 17:33:17,583 [AMQP Connection amqp://localhost:5672] ERROR MassTransit.RabbitMqTransport.RabbitMqReceiveTransport - RabbitMQ connection failed: Connect failed: guest@localhost:5672/
2015-11-19 17:33:17,583 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache - Connecting: guest@localhost:5672/
2015-11-19 17:33:18,086 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache - Connected: guest@localhost:5672/ (address: [::1]:5672, local: [::1]:58007
2015-11-19 17:33:18,086 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache - Using new connection: guest@localhost:5672/
2015-11-19 17:33:18,086 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Pipeline.RabbitMqBasicConsumer - Consumer Model Shutdown (amq.ctag-Q4Lafwjtvmyv8hKmYAsc1g), Concurrent Peak: 0, 320-CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-11-19 17:33:18,088 [9] DEBUG MassTransit.RabbitMqTransport.Pipeline.PrepareReceiveQueueFilter - Queue: bus-FFPTN006-TE.ProcessHost.Managed-bmeyyygqwgsx4jn6bdjxnbhync (auto-delete)
2015-11-19 17:33:18,090 [10] DEBUG MassTransit.RabbitMqTransport.Pipeline.RabbitMqBasicConsumer - ConsumerOk: rabbitmq://localhost:5672/bus-FFPTN006-TE.ProcessHost.Managed-bmeyyygqwgsx4jn6bdjxnbhync?durable=false&autodelete=true&prefetch=16 - amq.ctag-CeyDSRWhPfFWwxMaQv8fiQ

Duplicate issue #3502: Request/Response flow has no timeout during connectivity to Broker is lost
Created: 2022-07-08T12:10:28Z
Is this a bug report?
Yes
Can you also reproduce the problem with the latest version?
Yes, 8.0.3, but also 7.3.1
Environment

Dotnet version: .NET 6
Operating system: both Windows/Linux
IDE (Rider, VS Code, Visual Studio): Visual Studio

Steps to Reproduce

Start bus when broker (RabbitMQ) is available, make sure it's connected: _busControl.StartAsync()
Trigger Request/Response communication when broker is not available. I've used vnet to disable/enable access:

        var requestTimeout = TimeSpan.FromSeconds(10);
        var requestTimeoutSource = new CancellationTokenSource();
        requestTimeoutSource.CancelAfter(requestTimeout);

	var destinationAddress = GetServiceAddress(serviceName);
	var client = _busControl.CreateRequestClient<TRequest>(destinationAddress, requestTimeout);
					                    
        var response = await client.GetResponse<TResponse>(request, requestTimeoutSource.Token, requestTimeout);

Expected Behavior
GetResponse triggers exception as RequestTimeout property passed or/and CancellationToken passed, stops execution.
It works in that way when broker is available, but the request takes too long to execute.
Actual Behavior
Provided options don't stop the execution, a deadlock occurs. In case you make the broker available again - GetResponse just returns a result.
I assume that the client tries to reconnect to the channel infinitely and this process doesn't use the cancellation token passed to the method. The same issue is described here: https://stackoverflow.com/questions/61069577/masstransit-requestclient-timeout-handling.

Closing comment (2022-07-08T20:41:48Z): I'm receiving a loop of a bit different error, found it very similar to: #397
RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable
 ---> System.AggregateException: One or more errors occurred. (Connection failed)
 ---> RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed
 ---> System.Net.Sockets.SocketException (11001): No such host is known.
   at System.Net.NameResolutionPal.ProcessResult(SocketError errorCode, GetAddrInfoExContext* context)
   at System.Net.NameResolutionPal.GetAddressInfoExCallback(Int32 error, Int32 bytes, NativeOverlapped* overlapped)
--- End of stack trace from previous location ---
   at RabbitMQ.Client.Impl.TcpClientAdapter.ConnectAsync(String host, Int32 port)
   at RabbitMQ.Client.Impl.TaskExtensions.TimeoutAfter(Task task, TimeSpan timeout)
   at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient socket, AmqpTcpEndpoint endpoint, TimeSpan timeout)
   --- End of inner exception stack trace ---
   at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient socket, AmqpTcpEndpoint endpoint, TimeSpan timeout)
   at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingAddressFamily(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan timeout, AddressFamily family)
   at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingIPv4(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan timeout)
   at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan connectionTimeout, TimeSpan readTimeout, TimeSpan writeTimeout)
   at RabbitMQ.Client.Framing.Impl.IProtocolExtensions.CreateFrameHandler(IProtocol protocol, AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan connectionTimeout, TimeSpan readTimeout, TimeSpan writeTimeout)
   at RabbitMQ.Client.ConnectionFactory.CreateFrameHandler(AmqpTcpEndpoint endpoint)
   at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func`2 selector)
   --- End of inner exception stack trace ---
   at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func`2 selector)
   at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
   --- End of inner exception stack trace ---
   at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
   at RabbitMQ.Client.ConnectionFactory.CreateConnection(IList`1 hostnames, String clientProvidedName)
   at MassTransit.RabbitMqTransport.ConnectionContextFactory.CreateConnection(ISupervisor supervisor) in /_/src/Transports/MassTransit.RabbitMqTransport/RabbitMqTransport/ConnectionContextFactory.cs:line 86
The thread 0x53cc has exited with code 0 (0x0).
MassTransit: Warning: Connection Failed: rabbitmq://************