Repository: facebookresearch/Mephisto

Issue:
[Clarifications] Testing in a local environment
Dear All,
I've started working with ParlAI and Mephisto some days ago in order to define an interface for data collection. In particular, I've extend the parlai-chat-demo to meet my task-specific requirements.

I was wondering how to test the interface with users in a local network or by remotely connecting to my server. Thus, I've looked up for existing threads about the same topic: [#409](https://github.com/facebookresearch/Mephisto/issues/409) and [#303 ](https://github.com/facebookresearch/Mephisto/issues/303) as the main examples.

Since I'm no expert in this kind of stuff, I'm not completely sure what are the current networking setups that I can consider at the moment.  Therefore, I kindly ask you for a gentle summary/tutorial concerning the particular objective of testing locally/remotely with own server.

To summarize, I would like to know if there's an answer to these main questions:
Q: Is it possible to easily test the interface with other people in a local environment (same network)?
Q: How to do so?

Down below I report a list of additional questions with related comments:

1. My server has a static IP, can I test it locally on the same network? Connecting from the same network to the server brings up a 'disconnected' status in the interface.

2. Can I use the mentioned nginx reverse proxy to access to my interface? What are the steps? I've tried setting it up and then access to the server. I'm able to access to the interface but the connection status is still 'disconnected'.

3. Do I need a specific mephisto configuration for local testing? Up until now I've worked with `"mephisto/architect"="local"` and `"mephisto/provider"="mock"`.

Thanks in advance and apologies if this issue sounds just as a duplicate of some past issues.
Unfortunately, I'm not sure I've done everything right until now.



Discussion:
Hi @federicoruggeri, sorry for the delay here. To answer your questions, testing locally _should_ be possible, and shouldn't require usage of a reverse proxy. If you have a static IP on the network, you can specify the IP itself as the `mephisto.architect.hostname` argument, which updates the `LocalArchitect` to be listening to more than just requests for `localhost`. This should resolve your issue, but let me know if you're still stuck after this.
Hi @JackUrb!
Thank you very much for your reply.
I've tried what you suggested as follows:

`python run.py mephisto.architect.hostname="**static IP**" `

But, unfortunately, I still get 'disconnected' status bar with the parlai_chat_task_demo app.

By the way, what is the corresponding python version for setting the suggested property? I'm referring to the `defaults` variable in the `run.py` script (in the parlai_chat_task_demo, the script is called `parlai_test_script.py`).

I've tried the following ones with no success:

`defaults = [   
    {"mephisto/blueprint": BLUEPRINT_TYPE},
    {"mephisto/architect: "local"},
    {"mephisto/provider": "mock"},
    {"mephisto/hostname": "**static IP**"},
    {"conf": "my_conf"}
]`

and

`defaults = [   
    {"mephisto/blueprint": BLUEPRINT_TYPE},
    {"mephisto/architect: "local"},
    {"mephisto/provider": "mock"},
    {"mephisto/architect/hostname": "**static IP**"},
    {"conf": "my_conf"}
]`

Thanks in advance for your courtesy!

----

Anyway, I've managed to run the custom web app with heroku. I've initially wrongly assumed that I needed to do something more, but the url I was using turned out to be wrong. Stupidly, I was just clicking on the heroku web app base url, while I should have added  `/?worker_id=X&assignment_id=Y` as well.
> I still get 'disconnected' status bar with the parlai_chat_task_demo app.

That's somewhat surprising - anything appear in your javascript console logs, or in the terminal logs, when you try to do this?

> what is the corresponding python version for setting the suggested property?

At the moment we're suggesting using `.yaml` files for this, at least until Hydra 1.1 comes out. Ideally we'd like to do away with using a python defaults list where possible, as it adds to the boilerplate of run files. If you'd like to learn more about how to do it anyways though you can visit the Hydra guides - I expect it would be `... {"mephisto.architect.hostname": "<IP>"} ...` . Hydra uses `/` notation for setting configuration types, and `.` notation for values.

> Stupidly, I was just clicking on the heroku web app base url, while I should have added /?worker_id=X&assignment_id=Y as well.

This is really something we should document, it's one of the most common mistakes new users run into when trying out local development at this point.
Here's the javascript console output (it keeps repeating this output in the console):

![Screenshot 2021-03-23 101905](https://user-images.githubusercontent.com/18099124/112123695-c7162b80-8bc1-11eb-8b89-0b48217e544d.png)

Looking at the terminal output (script output), it doesn't detect any connection: no particular output is produced when connecting to the link.

---

I'm working directly with the repo version (downloaded 6th March 2021). Is it possible that I'm missing some recent update?

Ah okay I think I see the issue, the server may not be configured to handle wss over another IP, and our JS check for if the socket should be served securely is simply a check for localhost. We should be inheriting the security of the regular url.

Resolving patch:
diff --git a/examples/simple_static_task/static_run_with_onboarding.py b/examples/simple_static_task/static_run_with_onboarding.py
--- a/examples/simple_static_task/static_run_with_onboarding.py
+++ b/examples/simple_static_task/static_run_with_onboarding.py
@@ -23,12 +23,7 @@
 TASK_DIRECTORY = os.path.dirname(os.path.abspath(__file__))
 CORRECT_ANSWER = "apple"
 
-defaults = [
-    {"mephisto/blueprint": BLUEPRINT_TYPE},
-    {"mephisto/architect": "local"},
-    {"mephisto/provider": "mock"},
-    {"conf": "onboarding_example"},
-]
+defaults = ["_self_", {"conf": "onboarding_example"}]
 
 from mephisto.operations.hydra_config import RunScriptConfig, register_script_config
 
diff --git a/mephisto/abstractions/blueprints/abstract/static_task/static_agent_state.py b/mephisto/abstractions/blueprints/abstract/static_task/static_agent_state.py
--- a/mephisto/abstractions/blueprints/abstract/static_task/static_agent_state.py
+++ b/mephisto/abstractions/blueprints/abstract/static_task/static_agent_state.py
@@ -100,18 +100,13 @@ def update_data(self, live_data: Dict[str, Any]) -> None:
     def update_submit(self, submission_data: Dict[str, Any]) -> None:
         """Move the submitted output to the local dict"""
         outputs: Dict[str, Any]
-
-        if submission_data.get("onboarding_data") is not None:
-            outputs = submission_data["onboarding_data"]
-        else:
-            outputs = submission_data["task_data"]
+        output_files = submission_data.get("files")
+        if output_files is not None:
+            submission_data["files"] = [f["filename"] for f in submission_data["files"]]
+        self.state["outputs"] = submission_data
         times_dict = self.state["times"]
         assert isinstance(times_dict, dict)
         times_dict["task_end"] = time.time()
-        if submission_data.get("files") != None:
-            logger.info(f"Got files: {str(submission_data['files'])[:500]}")
-            outputs["files"] = [f["filename"] for f in submission_data["files"]]
-        self.state["outputs"] = outputs
         self.save_data()
 
     def get_task_start(self) -> Optional[float]:


Test patch: