Repository: facebookresearch/Mephisto

Issue:
Unhandled exception in socket server rejected WebSocket connection: HTTP 404
## Issue Description

When I try to deploy my task with heroku + mturk sandbox it keeps thowring `Unhandled exception in socket server rejected WebSocket connection: HTTP 404`

Already turn on `debug` flag but can't see much information

## Expected Behavior

I worked fine a week or two ago. This is very simple task so I would expect it should be fine this time\

## Current Behavior

Keep throwing WebSocket error 404 when `sending alive` after `heroku Scaling dynos` step

## Steps to Reproduce

<!-- If applicable, provide step-by-step instructions to reproduce the issue. -->

1. python3 deploy.py

## Screenshots

```
bash-5.1# python3 ../deploy.py 
/usr/lib/python3.9/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(
[06/18/23 14:44:35] INFO     Prometheus server launched at process 113                                                                                                                                                                                     metrics.py:185

up to date, audited 783 packages in 12s

148 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

> mephisto-task-compiler@1.0.1 dev
> webpack --mode development


🌼 daisyUI components 2.52.0  https://daisyui.com
  ✔︎ Including:  base, components, 29 themes, utilities
  ❤︎ Support daisyUI:  https://opencollective.com/daisyui 
  
asset build/bundle.js 1.4 MiB [compared for emit] (name: main)
runtime modules 1.07 KiB 6 modules
modules by path ./ 1.19 MiB
  modules by path ./node_modules/ 1.13 MiB 12 modules
  modules by path ./src/ 63.2 KiB
    modules by path ./src/css/*.css 55.4 KiB 2 modules
    ./src/main.js 230 bytes [built] [code generated]
    ./src/app.jsx 4.9 KiB [built] [code generated]
    ./src/components/core_components.jsx 2.7 KiB [built] [code generated]
modules by path ../../packages/ 156 KiB
  ../../packages/mephisto-task/build/bundle.js 134 KiB [built] [code generated]
  ../../packages/annotated/annotator-tracker/build/bundle.js 22.1 KiB [built] [code generated]
webpack 5.87.0 compiled successfully in 34243 ms
Locating heroku...
Getting heroku
/usr/local/bin/heroku
 ›   Warning: Our terms of service have changed: https://dashboard.heroku.com/terms-of-service
 ›   Warning: HEROKU_API_KEY is set
[06/18/23 14:46:00] INFO     Creating a task run under task name: s3-voting                                                                                                                                                                               operator.py:262
[06/18/23 14:46:01] DEBUG    type of assignment data: <class 'list'>                                                                                                                                                                                  task_launcher.py:86
Building server files...

up to date, audited 84 packages in 10s

7 packages are looking for funding
  run `npm fund` for details

3 high severity vulnerabilities

Some issues need review, and may require choosing
a different dependency.

Run `npm audit` for details.
Locating heroku...
/usr/local/bin/heroku
 ›   Warning: heroku update available from 8.1.3 to 8.1.8.
 ›   Warning: HEROKU_API_KEY is set
 ›   Warning: heroku update available from 8.1.3 to 8.1.8.
Heroku: Starting server Jay Huynh...
 ›   Warning: heroku update available from 8.1.3 to 8.1.8.
Creating root-s3-voting-22-4a8629019e21... done
 ›   Warning: heroku update available from 8.1.3 to 8.1.8.
Enabling http-session-affinity for root-s3-voting-22-4a8629019e21... done
 ›   Warning: heroku update available from 8.1.3 to 8.1.8.
Scaling dynos... done, now running web at 1:Basic
[06/18/23 14:47:55] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:47:56] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:47:57] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:47:58] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:47:59] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:00] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:00] ERROR    Unhandled exception in socket server rejected WebSocket connection: HTTP 404, InvalidStatusCode(404, Headers([('Content-Length', '563'), ('Cache-Control', 'no-cache, no-store'), ('Content-Type', 'text/html;      websocket_channel.py:165
                             charset=utf-8'), ('Date', '2023-06-18 14:45:09.991494749 +0000 UTC'), ('Server', 'heroku-router')]))                                                                                                                                        
                             Traceback (most recent call last):                                                                                                                                                                                                          
                               File "/mephisto/mephisto/abstractions/architects/channels/websocket_channel.py", line 132, in run_socket                                                                                                                                  
                                 async with websockets.connect(                                                                                                                                                                                                          
                               File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 642, in __aenter__                                                                                                                                              
                                 return await self                                                                                                                                                                                                                       
                               File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 659, in __await_impl_timeout__                                                                                                                                  
                                 return await asyncio.wait_for(self.__await_impl__(), self.open_timeout)                                                                                                                                                                 
                               File "/usr/lib/python3.9/asyncio/tasks.py", line 479, in wait_for                                                                                                                                                                         
                                 return fut.result()                                                                                                                                                                                                                     
                               File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 666, in __await_impl__                                                                                                                                          
                                 await protocol.handshake(                                                                                                                                                                                                               
                               File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 332, in handshake                                                                                                                                               
                                 raise InvalidStatusCode(status_code, response_headers)                                                                                                                                                                                  
                             websockets.exceptions.InvalidStatusCode: server rejected WebSocket connection: HTTP 404                                                                                                                                                     
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:01] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:02] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:03] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:04] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
                    INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
[06/18/23 14:48:05] INFO     Sending alive                                                                                                                                                                                                       client_io_handler.py:243
Exception in thread socket-thread-wss://root-s3-voting-22-4a8629019e21.herokuapp.com/:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/mephisto/mephisto/abstractions/architects/channels/websocket_channel.py", line 185, in async_socket_wrap
    event_loop.run_until_complete(close_websocket())
  File "/usr/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/mephisto/mephisto/abstractions/architects/channels/websocket_channel.py", line 183, in close_websocket
    await self._socket_task
  File "/mephisto/mephisto/abstractions/architects/channels/websocket_channel.py", line 168, in run_socket
    raise e
  File "/mephisto/mephisto/abstractions/architects/channels/websocket_channel.py", line 132, in run_socket
    async with websockets.connect(
  File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 642, in __aenter__
    return await self
  File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 659, in __await_impl_timeout__
    return await asyncio.wait_for(self.__await_impl__(), self.open_timeout)
  File "/usr/lib/python3.9/asyncio/tasks.py", line 479, in wait_for
    return fut.result()
  File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 666, in __await_impl__
    await protocol.handshake(
  File "/usr/lib/python3.9/site-packages/websockets/legacy/client.py", line 332, in handshake
    raise InvalidStatusCode(status_code, response_headers)
websockets.exceptions.InvalidStatusCode: server rejected WebSocket connection: HTTP 404
[06/18/23 14:48:05] ERROR    Encountered error while launching run, shutting down                                                                                                                                                                         operator.py:302
                             Traceback (most recent call last):                                                                                                                                                                                                          
                               File "/mephisto/mephisto/operations/operator.py", line 300, in launch_task_run_or_die                                                                                                                                                     
                                 live_run.client_io.launch_channels()                                                                                                                                                                                                    
                               File "/mephisto/mephisto/operations/client_io_handler.py", line 224, in launch_channels                                                                                                                                                   
                                 self._register_channel(channel)                                                                                                                                                                                                         
                               File "/mephisto/mephisto/operations/client_io_handler.py", line 200, in _register_channel                                                                                                                                                 
                                 raise ConnectionRefusedError(  # noqa F821 we only support py3                                                                                                                                                                          
                             ConnectionRefusedError: Was not able to establish a connection with the server, please try to run again. If that fails,please launch with mephisto.log_level=debug and watch for clear errors. If this doesn't help, feel                   
                             free to open an issue with your debug logs on the Mephisto github.                                                                                                                                                                          
Heroku: Deleting server: root-s3-voting-22-4a8629019e21
 ›   Warning: heroku update available from 8.1.3 to 8.1.8.
Destroying root-s3-voting-22-4a8629019e21 (including all add-ons)... done
[06/18/23 14:48:17] ERROR    Ran into error while launching run:                                                                                                                                                                                          operator.py:522
                             Traceback (most recent call last):                                                                                                                                                                                                          
                               File "/mephisto/mephisto/operations/operator.py", line 518, in launch_task_run                                                                                                                                                            
                                 return self.launch_task_run_or_die(                                                                                                                                                                                                     
                               File "/mephisto/mephisto/operations/operator.py", line 312, in launch_task_run_or_die                                                                                                                                                     
                                 raise e                                                                                                                                                                                                                                 
                               File "/mephisto/mephisto/operations/operator.py", line 300, in launch_task_run_or_die                                                                                                                                                     
                                 live_run.client_io.launch_channels()                                                                                                                                                                                                    
                               File "/mephisto/mephisto/operations/client_io_handler.py", line 224, in launch_channels                                                                                                                                                   
                                 self._register_channel(channel)                                                                                                                                                                                                         
                               File "/mephisto/mephisto/operations/client_io_handler.py", line 200, in _register_channel                                                                                                                                                 
                                 raise ConnectionRefusedError(  # noqa F821 we only support py3                                                                                                                                                                          
                             ConnectionRefusedError: Was not able to establish a connection with the server, please try to run again. If that fails,please launch with mephisto.log_level=debug and watch for clear errors. If this doesn't help, feel                   
                             free to open an issue with your debug logs on the Mephisto github.                                                                                                                                                                          
                    INFO     operator shutting down                                                                                                                                                                                                       operator.py:413
[06/18/23 14:48:27] DEBUG    Prometheus server appears to be running at 9090                                                                                                                                                                               metrics.py:164
                    INFO     Prometheus server shut down   
```

## Environment

- Operating System: Docker container - Alpine
- Project Version/Commit: [https://github.com/d-lab/mephisto/tree/deployment/app](url)

## Additional Information

<!-- Add any additional information or context that might be helpful in resolving the issue. -->

## Possible Solution (Optional)

- Maybe Heroku update there API then we might need to update our backend





Discussion:
Hi @JackUrb . Hope you are doing well.

Our team have been doing very great with Mephisto for a while but we just encounter this error that blocking us from deploying the task.

Hope you might have some ideas how to fix this issue as I have spent a few days on this but no luck at all
I also tested with freshly latest version of mephisto. Still throwing error 404
Hi @jayhuynh, this is certainly an issue! We haven't used Heroku extensively any time recently, so it's certainly possible something has changed. One thing you may do - if you could try to open the heroku logs _on heroku_ when the server launches. You may want to add an `input` statement near the `"Encountered error while launching run, shutting down"` throw to ensure the heroku server stays up.

As Heroku's [free model was removed](https://help.heroku.com/RSBRUH58/removal-of-heroku-free-product-plans-faq), we've been looking to onboard alternatives in  #1010 (fly.io) and also AWS App Runner. 
Figured out the issue - heroku has started to add random hashes to the ends of the app url, so we need to now query for those. Will put up a fix shortly.
Thank you so much Jack. I'm excited to see your change 😄 

Resolving patch:
diff --git a/mephisto/abstractions/architects/heroku_architect.py b/mephisto/abstractions/architects/heroku_architect.py
--- a/mephisto/abstractions/architects/heroku_architect.py
+++ b/mephisto/abstractions/architects/heroku_architect.py
@@ -117,6 +117,7 @@ def __init__(
         self.__heroku_app_name: Optional[str] = args.architect.get(
             "heroku_app_name", None
         )
+        self.__heroku_app_url: Optional[str] = None
         self.__heroku_executable_path: Optional[str] = None
         self.__heroku_user_identifier: Optional[str] = None
 
@@ -124,8 +125,7 @@ def __init__(
 
     def _get_socket_urls(self) -> List[str]:
         """Returns the path to the heroku app socket"""
-        heroku_app_name = self.__get_app_name()
-        return ["wss://{}.herokuapp.com/".format(heroku_app_name)]
+        return ["wss://{}".format(self.__heroku_app_url)]
 
     def get_channels(
         self,
@@ -444,7 +444,11 @@ def __setup_heroku_server(self) -> str:
 
         time.sleep(HEROKU_WAIT_TIME)
 
-        return "https://{}.herokuapp.com".format(heroku_app_name)
+        domains = subprocess.check_output(
+            shlex.split(f"{heroku_executable_path} domains -a {heroku_app_name}")
+        ).decode()
+        self.__heroku_app_url = domains.split("\n")[1]
+        return "https://{}".format(self.__heroku_app_url)
 
     def __delete_heroku_server(self):
         """


Test patch: