opened
kushalchawla
username_0
Hey,

MTurkers are facing one issue: 

When the maximum number of conversations is reached, ParlAI expires everyone else who was in their onboarding survey in the middle. Can I do anything about this? I know I can just pay them for their time, but is there anything that can be done in ParlAI to control this? Any parameters that I can set?

Requesting for a very early reply since I am in the middle of my collection.

@username_1?

Thanks,
Kushal.
HITs being expired in the middle
issue
created
kushalchawla
username_0
Hey. I can also not find a direct way to pay them as well. Since they did not complete the task, there is no assignment ID attached to that Turker -> Means I cannot roll out a bonus for their time. Is there a way to pay these turkers, who could not submit the task but you want to pay them for their time?
comment
created
stephenroller
username_1
Not sure. @username_2  is the expert. I believe he's back from vacation Friday.
comment
created
JackUrb
username_2
@username_0 unfortunately this is a fairly major downside to the ParlAI-MTurk design which was patched out in Mephisto. There's no way to prevent the expirations of in-progress onboarding short of stopping the entire shutdown process, which would require a number of code changes.

There is also no really good way to bonus workers who have never worked on a task for you (we would tend to ask workers for a task they've worked for us on in the past and bonusing there, but that worked b/c we had a lot of tasks running and frequently workers had another task to give us). 

What you _can_ do is use the `mturk_utils.py` file to grant a specific worker a qualification unique to them, and then either using the regular MTurk interface or using the hit creation tools in `mturk_utils.py` you can launch a single HIT that only workers with that qualification can do (title it "<task timeout> compensation task" or something), and then the worker can submit that.
comment
created
kushalchawla
username_0
Hey @username_2. Hope you had a great vacation. Thanks for replying. 

Got it. I will probably just have to live with it for now. I hope this is rare. -> I also feel that the number of parallel connections that I open up might have something to do with it. At least mitigate this effect (default is 30, I am using 100 currently after taking advice from @username_1). Is this true? Do you think I should increase/decrease this number?

The last method is the one which Amazon suggests as well: https://blog.mturk.com/paying-for-non-submitted-hits-245c6c3323bb

Since I am running my experiment in batches, I will request these Turkers to participate in a future batch and if not possible, will use the above method.
comment
created
JackUrb
username_2
Yeah the more connections you have running, the more of a chance a bunch of people will still be working on things when everything shuts down.

Reflecting in the other direction, if you're alright with making the other mistake (and not collecting all of the data you're setting out for in one run), you can override all of the `HIT_MULT_SCALE` values to be 1 [here](https://github.com/facebookresearch/ParlAI/blob/fff92128d69e05aec7d8ec3efb6e85939c0ef33f/parlai/mturk/core/mturk_manager.py#L41). On the downside here, the task may never actually reach the quantity you were trying to launch, in which case it could hang forever waiting on the last results.
comment
created
kushalchawla
username_0
Hey.

What exactly do you mean by "if you're alright with making the other mistake"?

On HIT_MULT_SCALE: Not exactly sure how this works but I will try reducing this if the issue happens a lot.

Thanks.
comment
created
JackUrb
username_2
Setting all of the `HIT_MULT_SCALE`'s to 1 will mean that if workers disconnect or the tasks aren't properly submitted, ParlAI-MTurk will run out of tasks to give to workers, and you may have a situation where the run never ends (rather than ending abruptly for some workers). In the absolute worst case, only one task may remain and there will be nobody to pair with that task endlessly.
comment
created
kushalchawla
username_0
Hey. Oh got it. Let's probably keep this as a last resort. I will just create a dummy compensation task as well.

Thanks a lot, @username_2. Closing this for now.
comment
closed
kushalchawla
username_0
issue