Repository: facebookresearch/Mephisto

Issue:
Probably getting null Worker ID in the frontend
Hi,

Thank you for the great crowdsourcing platform. We are using Mephisto for a chat-style data collection.

When the task was live on AMT, many of the workers reported that they saw the following message:

`Sorry, you have already worked on the maximum number of these tasks available to you, or are no longer eligible to work on this task.`

However, we did not set the property `maximum_units_per_worker` in our config.
We did not face any errors in `localhost` or AMT Sandbox.

From the [code](https://github.com/facebookresearch/Mephisto/blob/c571bf9c2e6395553e71ce922c14a14e363ad7d5/packages/mephisto-task/src/index.js#L109), it looks like this message arises when the `workerId` received is `null`.
Is there any other scenario, apart from exceeding `maximum_units_per_worker`, that can result in a null `workerId`?



Discussion:
Hi @vaibhavad - this is curious, what were your parameters on the number of concurrent tasks? And what was the rate of occurrence? Are the chats with a model, or with other people? 

One thought I have is Mephisto also puts up this messaging if it cannot find an available task for someone to work on - if a worker accepts and then returns a `Unit`, and another accepts the same one before the return is properly processed, they may see this message as no units are "available" from Mephisto's perspective.

Mephisto does log these occurrences at the `debug` level though, so you can set the command line parameter `mephisto.log_level=debug` to see the reason for a worker not receiving a task after connecting.
The setup is a 1 turn chat between two people, built on top of `parlai_chat_task_demo` example. The model is not involved.

Both `allowed_concurrent` and `maximum_units_per_worker` are not set, hence they default to 0 (which is infinite).

I think the issue is that sometimes the unit is unavailable but HIT is still available on AMT. I ran a small example on Sandbox (with `mephisto.log_level=debug`). Here are steps to replicate it:
1. Set `num_conversations` to 2 (and `agent_count` per world as 2) and launch on Sandbox.
2. Accept from 2 different worker accounts. Don't submit, but return the HIT.
3. Repeat step 2
4. Repeat step 2, you should see the message now - `Sorry, you have already worked on the maximum number of these tasks available to you, or are no longer eligible to work on this task.`

The logs are attached - [mephisto_log.txt](https://github.com/facebookresearch/Mephisto/files/6728682/mephisto_log.txt)
It confirms that no more units were available for the workers. I have added additional agent status logging. If you need access to the code, let me know, I'll add you to the private repo of our data collection.

It seems that if a worker returns a HIT, the HIT is now available for other workers, but the Unit on the Mephisto side is "consumed". Eventually, it means that if some workers return, we'll end up with non-zero available HITs but zero available Units.
Is there a way to keep available Units and available HITs synchronized?

A quick fix that we are using right now is that we are launching with 'num_conversations' more than we actually require. The risk in this is that if the return rate is low, we might end up with more data than we actually need, and end up paying more than the original estimated budget.

This is definitely unintended behavior - if a worker returns a HIT from a live task requiring more than one concurrent worker, we should be expiring the corresponding `Unit` on MTurk as well as locally. Thanks for the report and repro steps, I'll be looking into it further later this week.

Resolving patch:
diff --git a/mephisto/data_model/unit.py b/mephisto/data_model/unit.py
--- a/mephisto/data_model/unit.py
+++ b/mephisto/data_model/unit.py
@@ -258,6 +258,11 @@ def get_status(self) -> str:
         from mephisto.abstractions.blueprint import AgentState
 
         db_status = self.db_status
+
+        # Expiration is a terminal state, and shouldn't be changed
+        if db_status == AssignmentState.EXPIRED:
+            return db_status
+
         computed_status = AssignmentState.LAUNCHED
 
         agent = self.get_assigned_agent()


Test patch: