Repository: localstack/localstack

Issue:
Bring back SQS queue "dynamic" format
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Enhancement description

Before 3.x, SQS would use the hostname and port of the incoming request to determine the URL to return, so the URL would work both if it was queried from the host or from inside the docker (compose) network. This was a critical and incredibly useful feature for us. While we have tried to not depend on it by providing environment variables that the users of our tool should use, some teams depend on third party libraries that query the URL and won't have the capacity to replace them. 

Additionally, we have the requirement of running with published random ports and there are use cases that get the URL both from the host and the docker network. While we will be looking into workarounds and try to find a way to still upgrade (since the [AWS SDK compatibility fix](https://github.com/localstack/localstack/issues/8267) is also critical) I don't think it's possible to have something that works in both use cases (host and network).

Honestly, I think it was awesome that LocalStack did this and I would have kept it as the default, so I hope you consider and decide bringing it back, maybe as a `SQS_ENDPOINT_STRATEGY` option.

### 🧑‍💻 Implementation

_No response_

### Anything else?

https://localstack-community.slack.com/archives/CMAFN2KSP/p1700665707589479


Discussion:
Affected by the same issue, I have tests breaking if I use Localstack 3.x because they run from outside the docker network.
hey @xtaixe @jospint, sorry for not getting back to you sooner.

with 3.0, we released a major overhaul of our DNS system where `localhost.localstack.cloud` should now always resolve to the localstack container, regardless of whether a request comes from the host or from within the container network. the need for "dynamic" queue URLs based on incoming requests was really a symptom of this missing DNS system. rather than bringing back the dynamic behavior, we'd prefer to understand why the default behavior isn't working for you.
@simonrw gave a comprehensive answer in the thread you posted about how to set up localstack in a way that it should work with. could you help us understand what's blocking you from moving to the standard endpoint strategy?
@thrau Hi there, thanks for replying!

Given this code
```
      sqs.sendMessage(
        new SendMessageRequest()
        .withQueueUrl(sqs.getQueueUrl(queueName).getQueueUrl)
        .withMessageBody(event.asJson.toString())
        )
```
Where `sqs` is an `AmazonSQS` client (AWS SDK v1)

Prior to Localstack 3.0 and AWS 1.12.584, the method `GetQueueUrlResult getQueueUrl(String queueName)`  from `com.amazonaws.services.sqs.AmazonSQS` would return this result (example)
```
http://127.0.0.1:32793/000000000000/queue_name
```
And I could call that exact address returned by the method, from outside the docker network, to send some messages. All good ✅ 

Upgrading to Localstack 3.0.2 and a later version of AWS v1 SDK, the result of `getQueueUrl` changes to this.
```
http://sqs.us-east-1.localhost:4566/000000000000/queue_name
```

Which can't be any more called from outside the docker network. I will get a `Unable to execute HTTP request: sqs.us-east-1.localhost` response.

I found the network migration guide [here](https://discuss.localstack.cloud/t/networking-migration-guide-for-localstack-3-0/588) but I'm not sure that any of the mentioned tips apply to my case. Note that the address returned is not `localhost.localstack.cloud` as you mentioned.

Is there a way that when calling `getQueueUrl(String queueName)`, I can get an URL that I can successfully invoke from outside the Docker network like it used to be?

EDIT: I got into the Localstack Community Slack and I can see that the linked post addresses the same issue and that this issue has been created after the response from @simonrw 

> Context: previous to v3.0.0, SQS (and only SQS) would use the hostname and port of the incoming request, which may be something we want to look into, but we removed it in order to standardize on `LOCALSTACK_HOST`

If no fix is done for this, I will work on running the test from inside the Docker network instead.
Update: I thought that setting `LOCALSTACK_HOST` wouldn't change things, since the default URL in the documentation is `localhost.localstack.cloud` and the one I was getting back from `getQueueUrl(String queueName)` was `sqs.us-east-1.localhost`

Well, I tried setting `LOCALSTACK_HOST` to `127.0.0.1` and it worked :) Both in a local M1 machine and on CI on an Intel machine.

cc/ @thrau @simonrw @xtaixe 
@jospint thanks for all the details. Could you share the config you are using to start LocalStack?
@thrau Using localstack Java library

```
new LocalStackContainer(DockerImageName.parse("localstack/localstack:3.0.2"))
   .withServices(Service.DYNAMODB, Service.SQS)
   .withEnv("LOCALSTACK_HOST", "127.0.0.1")
```
@thrau sorry for the late response, I was on leave.

Hope it was clear that we need random published ports and for returned URLs to work both from the host and the docker (compose) network. Given that:

1. Is there any configuration that would work from the host with a random port assigned by Docker (Compose)?
2. Although a secondary concern we'd rather not be forced to rely on DNS, since in cases where not needed our app would work offline (we build our own LocalStack image to avoid on demand packages downloads and installation and rate limiting errors that can cause tests and build failures).
3. Additionally, AFAIK, the DNS approach requires modifying other containers right? We could look into it if there's a solution for point 1, although is likely that we might find some problems since in our case/tool most containers are provided by plugins and LocalStack is supposed to be hidden and might mean changing the plugins API in weird ways or leaking internal details...

Let me reiterate that for our team this was one of those features that "just works" and you love... I don't even see why would you need the `SQS_ENDPOINT_STRATEGY` feature instead of just listing what different URL formats support (multi-region, etc.)

Resolving patch:
diff --git a/localstack/services/sqs/models.py b/localstack/services/sqs/models.py
--- a/localstack/services/sqs/models.py
+++ b/localstack/services/sqs/models.py
@@ -346,6 +346,12 @@ def url(self, context: RequestContext) -> str:
         elif config.SQS_ENDPOINT_STRATEGY == "path":
             # https?://localhost:4566/queue/us-east-1/00000000000/my-queue (us-east-1)
             host_url = f"{scheme}://{host_definition.host_and_port()}/queue/{self.region}"
+        elif config.SQS_ENDPOINT_STRATEGY == "dynamic":
+            # undocumented strategy to help unblock folks who depended on the feature previously,
+            # especially when running localstack in the container on a different port than the one exposed
+            # by docker (let's say :<random-port>->:4566/tcp.). however, this should be handled in a more
+            # fundamental way in localstack.
+            host_url = f"{context.request.host_url.rstrip('/')}/queue/{self.region}"
         else:
             host_url = f"{scheme}://{host_definition.host_and_port()}"
 


Test patch: