Repository: facebookresearch/ParlAI

Issue:
MTurk: this HIT is expired, please return and take a new one if you'd want to work on this task.
Hi thanks for the great platform, 

I am actually testing the  ```convai2_model_eval``` script by running ```python3 run.py --only-masters True --sandbox```, and the result is the following: 
```
(pytorch4) ➜  convai2_model_eval git:(master) ✗ python3 run.py --only-masters True --sandbox
[ optional arguments: ]
[  ag_shutdown_time: 120 ]
[  auto_approve_delay: 86400 ]
[  datapath: /home/andrea/mama-persona/ParlAI/data ]
[  max_persona_time: 300 ]
[  max_resp_time: 240 ]
[  max_turns: 10 ]
[  only_masters: True ]
[  persona_type: both ]
[  range_turn: 5,6 ]
[  revised: False ]
[ ParlAI Model Arguments: ]
[  dict_class: None ]
[  init_model: None ]
[  model: projects.personachat.kvmemnn.kvmemnn:Kvmemnn ]
[  model_file: /home/andrea/mama-persona/ParlAI/data/models/convai2/kvmemnn/model ]
[ Mechanical Turk: ]
[  allowed_conversations: 0 ]
[  block_qualification: None ]
[  count_complete: False ]
[  disconnect_qualification: None ]
[  hard_block: False ]
[  heroku_team: None ]
[  hobby: False ]
[  is_debug: False ]
[  is_sandbox: True ]
[  local: False ]
[  log_level: 20 ]
[  max_connections: 30 ]
[  max_hits_per_worker: 0 ]
[  max_time: 0 ]
[  max_time_qual: None ]
[  min_messages: 0 ]
[  mturk_log_path: /home/andrea/mama-persona/ParlAI/logs/mturk ]
[  num_conversations: 1 ]
[  reward: 0.05 ]
[  task: None ]
[  unique_qual_name: None ]
[  unique_worker: False ]
[  verbose: False ]
[ warning: overriding opt['no_cuda'] to True (previously: None )]
[ warning: overriding opt['interactive_mode'] to True (previously: False )]
[ warning: overriding opt['tensorboard_log'] to False (previously: None )]
[ creating KvmemnnAgent ]
Dictionary: loading dictionary from /home/andrea/mama-persona/ParlAI/data/models/convai2/kvmemnn/model.dict
[ num words =  19153 ]
Loading existing model params from /home/andrea/mama-persona/ParlAI/data/models/convai2/kvmemnn/model
[loading candidates: /home/andrea/mama-persona/ParlAI/data/models/convai2/kvmemnn/model.candspair*]
[caching..]
=init done=
/home/andrea/anaconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
  warnings.warn(warning.format(ret))
[ Interactive mode ]
=== Actual bot opt === :
 [task] : convai2:SelfRevisedTeacher
[download_path] : /private/home/jase/src/ParlAI/downloads
[datatype] : train
[image_mode] : raw
[numthreads] : 40
[batchsize] : 1
[include_labels] : True
[datapath] : /private/home/jase/src/ParlAI/data
[model] : projects.personachat.kvmemnn.kvmemnn:KvmemnnAgent
[model_file] : /home/andrea/mama-persona/ParlAI/data/models/convai2/kvmemnn/model
[dict_class] : parlai.core.dict:DictionaryAgent
[dict_file] : /home/andrea/mama-persona/ParlAI/data/models/convai2/kvmemnn/model.dict
[dict_initpath] : None
[dict_language] : english
[dict_max_ngram_size] : -1
[dict_minfreq] : 0
[dict_maxtokens] : -1
[dict_nulltoken] : __NULL__
[dict_starttoken] : __START__
[dict_endtoken] : __END__
[dict_unktoken] : __UNK__
[dict_maxexs] : -1
[dict_tokenizer] : split
[dict_lower] : False
[hops] : 1
[twohop_range] : 100
[lins] : 0
[embeddingsize] : 2000
[embeddingnorm] : 10
[share_embeddings] : True
[learningrate] : 0.1
[margin] : 0.1
[optimizer] : sgd
[truncate] : -1
[neg_samples] : 10
[parrot_neg] : 0
[take_next_utt] : False
[tfidf] : False
[cache_size] : 1000
[history_length] : 100
[history_replies] : label
[interactive_mode] : True
[evaltask] : convai2:SelfOriginalTeacher
[display_examples] : False
[num_epochs] : -1
[max_train_time] : 28800.0
[log_every_n_secs] : 60.0
[validation_every_n_secs] : 900.0
[save_every_n_secs] : -1
[save_after_valid] : False
[validation_max_exs] : 100000
[validation_patience] : -1
[validation_metric] : accuracy
[validation_metric_mode] : max
[validation_cutoff] : 1.0
[dict_build_first] : True
[load_from_checkpoint] : False
[parlai_home] : /private/home/jase/src/ParlAI
[override] : {'task': 'convai2:SelfRevisedTeacher', 'model': 'projects.personachat.kvmemnn.kvmemnn:KvmemnnAgent', 'model_file': '/checkpoint/jase/20180328/kvmemnn_sweep10/persona-self_rephraseTrn-True_rephraseTst-False_lr-0.1_esz-2000_margin-0.1_tfidf-False_shareEmb-True_hops1_lins0/model', 'evaltask': 'convai2:SelfOriginalTeacher', 'validation_metric': 'accuracy'}
[no_cuda] : True
[tensorboard_log] : False
[init_model] : None
[mturk_log_path] : /home/andrea/mama-persona/ParlAI/logs/mturk
[num_conversations] : 1
[unique_worker] : False
[max_hits_per_worker] : 0
[unique_qual_name] : None
[reward] : 0.05
[is_sandbox] : True
[is_debug] : False
[verbose] : False
[hard_block] : False
[log_level] : 20
[disconnect_qualification] : None
[block_qualification] : None
[count_complete] : False
[allowed_conversations] : 0
[max_connections] : 30
[min_messages] : 0
[local] : False
[hobby] : False
[max_time] : 0
[max_time_qual] : None
[heroku_team] : None
[max_turns] : 10
[max_resp_time] : 240
[max_persona_time] : 300
[ag_shutdown_time] : 120
[persona_type] : both
[revised] : False
[range_turn] : 5,6
[auto_approve_delay] : 86400
[only_masters] : True
[starttime] : Mar20_17-27
[loss] : cosine

You are going to allow workers from Amazon Mechanical Turk to be an agent in ParlAI.
During this process, Internet connection is required, and you should turn off your computer's auto-sleep feature.
Enough HITs will be launched over time up to a max of 1.5 times the amount requested until the desired number of conversations start.
Please press Enter to continue...

Querying the parlai website for possible notices...
Setting up MTurk server...
Heroku: Collecting files...
Heroku: Starting server...
Creating andrea-775d1b89-convai2self-4b... done
Enabling http-session-affinity for andrea-775d1b89-convai2self-4b... done
Scaling dynos... done, now running web at 1:Free
MTurk server setup done.

Botocore couldn't subscribe to HIT events, perhaps you tried to register to localhost?
Creating HITs...
You should be calling `ready_to_accept_workers` before `create_hits` to ensure that the socket is connected beforehits are added. This will be enforced in future versions.
Link to HIT: https://workersandbox.mturk.com/mturk/preview?groupId=37R8JAFGQ098RQHW35YUR0RY0VCJDO

Waiting for Turkers to respond... (Please don't close your laptop or put your computer into sleep or standby mode.)

Local: Setting up WebSocket...
```

The output looks actually correct, I just two doubts:
- Is it normal the warning?  ```Botocore couldn't subscribe to HIT events, perhaps you tried to register to localhost? ```.
- And, ```You should be calling `ready_to_accept_workers` before `create_hits` to ensure that the socket is connected beforehits are added. This will be enforced in future versions.```. For this I moved the call of```create_hits``` after ```ready_to_accept_workers``` and it actually disappeared. But I am not sure 100% if I should actually do that :)


Anyway, I go the HIT link and I accept the HIT for the worker. However the system gets disconnected almost immidiatelly and the interface says: ``` This HIT is expired, please return and take a new one if you'd want to work on this task.```.  Like: 

![Screenshot 2019-03-20 at 17 36 52](https://user-images.githubusercontent.com/6934703/54674154-c8c23a00-4b36-11e9-9645-087a09fcd9a7.png)

I am actually living outside the US so as suggested in the issue https://github.com/facebookresearch/ParlAI/issues/1184 I am using the following Qualifications

```
MASTER_QUALIF_SDBOX = {
              'QualificationTypeId': '00000000000000000071', 
              'Comparator': 'EqualTo', 
           'LocaleValues': [{'Country': 'IT'}]
}
```


Any suggestion on how to solve this problem? Is it related with the ``` Botocore``` issue? 


Thanks a lot in advance 

Best 

Andrea



Discussion:
Hi @andreamad8! Thanks for reaching out - the two warnings you're experiencing are regular behavior at the moment:
For `Botocore couldn't subscribe to HIT events, perhaps you tried to register to localhost?` - I need to either fix the root cause or update the messaging, AMT slightly changed their messaging protocols and I have to add a retry rather than failing fast. The platform works almost exactly as well without the Botocore HIT events, but workers will sometimes experience rare delays of up to 30 seconds when trying so submit tasks. 
For `You should be calling ready_to_accept_workers before create_hits to ensure that the socket is connected before hits are added. This will be enforced in future versions.` - The `convai2_model_eval` run script should be updated to reflect the new ordering. It was created before I made some changes to the platform that are trying to enforce the noted ordering, as the reverse can result in degenerate behavior in some cases (workers being able to access HITs before the backend server is up to support it, for instance).

As far as the bug you're currently having, it appears the HITs are going up before the socket is properly launched (`Local: Setting up WebSocket...` is the last line of your output). This means that while the frontend is able to hit the server, it isn't going through to the local machine running ParlAI. This is actually exactly the symptom of the degenerate case for which you should be swapping `ready_to_accept_workers` and `create_hits`. Can you share what happens if you run with the swapped ordering?
Hi @JackUrb, thanks for your quick response.

I moved ```create_hits``` after ```ready_to_accept_workers ```. So the code becomes:

```
    try:
        mturk_manager.start_new_run()
        agent_qualifications = []
        if opt['only_masters']:
            if opt['is_sandbox']:
                agent_qualifications.append(MASTER_QUALIF_SDBOX)
            else:
                agent_qualifications.append(MASTER_QUALIF)
        #mturk_manager.create_hits(qualifications=agent_qualifications) ## commented this line

        if not opt['is_sandbox']:
            # ADD SOFT-BLOCKED WORKERS HERE
            # NOTE: blocking qual *must be* specified
            blocked_worker_list = []
            for w in blocked_worker_list:
                print('Soft Blocking {}\n'.format(w))
                mturk_manager.soft_block_worker(w)
                time.sleep(0.1)  # do the sleep to prevent amazon query drop

        def run_onboard(worker):
            worker.persona_generator = persona_generator
            world = PersonaProfileWorld(opt, worker)
            world.parley()
            world.shutdown()
        mturk_manager.set_onboard_function(onboard_function=run_onboard)
        mturk_manager.ready_to_accept_workers()
        mturk_manager.create_hits(qualifications=agent_qualifications)  ## added here
```

then I run again ```python3 run.py --only-masters True --sandbox```, and the result is the following:

```
You are going to allow workers from Amazon Mechanical Turk to be an agent in ParlAI.
During this process, Internet connection is required, and you should turn off your computer's auto-sleep feature.
Enough HITs will be launched over time up to a max of 1.5 times the amount requested until the desired number of conversations start.
Please press Enter to continue...

Querying the parlai website for possible notices...
Setting up MTurk server...
Heroku: Collecting files...
Heroku: Starting server...
 ›   Warning: token will expire 03/19/2020
 ›   Use heroku authorizations:create to generate a long-term token
Creating andrea-791c9040-convai2self-4b... done
Enabling http-session-affinity for andrea-791c9040-convai2self-4b... done
Scaling dynos... done, now running web at 1:Free
MTurk server setup done.

Botocore couldn't subscribe to HIT events, perhaps you tried to register to localhost?
Local: Setting up WebSocket...
WebSocket set up!
Creating HITs...
Link to HIT: https://workersandbox.mturk.com/mturk/preview?groupId=3EYXOXDEN7MBURMSVNNRISA1ODMZJ3

Waiting for Turkers to respond... (Please don't close your laptop or put your computer into sleep or standby mode.)
```

The warning disappears, then I went to the HIT link but still I have the same problem:

![Screenshot 2019-03-21 at 09 44 07](https://user-images.githubusercontent.com/6934703/54729440-70cf1600-4bbe-11e9-8dc0-82d45a6be906.png)

It looks like the server gets disconnected when it is searching for the 'partner' to talk with (which in this case is the KVMM). I have also tried to run the ```run.py``` function in personachat folder, same problem. 

Now I am just wild guessing, maybe could be a problem with the QualificationTypeId, since I used a different one?  

Thanks a lot for your help

Best

Andrea

Hi again - The fact that you can accept the HIT implies there's nothing amiss with the qualifications. I suspect that the world is crashing silently upon creation. How long does the hit show connected before it switches to disconnected? I still don't have enough to go on, perhaps try `--verbose --debug` and we can see what's going on. (It's going to be really verbose, but with that I should be able to find out the root cause).
Hi, 

after few seconds, and the is a little promt that says: "wait for the next person to speak". Like:

<img width="1440" alt="Screenshot 2019-03-21 at 13 51 01" src="https://user-images.githubusercontent.com/6934703/54735310-70477700-4be0-11e9-9de9-c80eba7a7a97.png">

I have run it with ```--verbose --debug``` this is the log:
```
You are going to allow workers from Amazon Mechanical Turk to be an agent in ParlAI.
During this process, Internet connection is required, and you should turn off your computer's auto-sleep feature.
Enough HITs will be launched over time up to a max of 1.5 times the amount requested until the desired number of conversations start.
Please press Enter to continue...

Querying the parlai website for possible notices...
Setting up MTurk server...
Heroku: Collecting files...
Heroku: Starting server...
 ›   Warning: token will expire 03/19/2020
 ›   Use heroku authorizations:create to generate a long-term token
Creating andrea-e58ce006-convai2self-4b... done
Enabling http-session-affinity for andrea-e58ce006-convai2self-4b... done
Scaling dynos... done, now running web at 1:Free
https://andrea-e58ce006-convai2self-4b.herokuapp.com
MTurk server setup done.

Botocore couldn't subscribe to HIT events, perhaps you tried to register to localhost?
Local: Setting up WebSocket...
Socket open: (<websocket._app.WebSocketApp object at 0x7fd85db757b8>,)
WebSocket set up!
Creating HITs...
Creating 2 hits...
https://andrea-e58ce006-convai2self-4b.herokuapp.com/chat_index?task_group_id=convai2:self_1553147054
Link to HIT: https://workersandbox.mturk.com/mturk/preview?groupId=3KBOIXB475YU906GJP0KQPFS196Q5E

Waiting for Turkers to respond... (Please don't close your laptop or put your computer into sleep or standby mode.)

On new message: (<websocket._app.WebSocketApp object at 0x7fd85db757b8>, '{"type":"new packet","content":{"id":"5fd47107-ad74-4f67-baf9-43e0dbe828e5","type":"alive","sender_id":"A226B246YXJDNS","assignment_id":"3UWN2HHPUZCS9YHAB3KWJAX5QGONSY","conversation_id":null,"receiver_id":"[World_convai2:self_1553147054]","data":{"hit_id":"372AGES0I4BJYTX78I2Y2IORZ7ZRX3","assignment_id":"3UWN2HHPUZCS9YHAB3KWJAX5QGONSY","worker_id":"A226B246YXJDNS","conversation_id":null},"status":"init","require_ack":true,"blocking":true}}')
on_agent_alive: Packet <{'id': '5fd47107-ad74-4f67-baf9-43e0dbe828e5', 'type': 'alive', 'sender_id': 'A226B246YXJDNS', 'receiver_id': '[World_convai2:self_1553147054]', 'assignment_id': '3UWN2HHPUZCS9YHAB3KWJAX5QGONSY', 'conversation_id': None, 'data': {'hit_id': '372AGES0I4BJYTX78I2Y2IORZ7ZRX3', 'assignment_id': '3UWN2HHPUZCS9YHAB3KWJAX5QGONSY', 'worker_id': 'A226B246YXJDNS', 'conversation_id': None}}>
Put packet (A226B246YXJDNS_fa6af0d7-230d-4adc-898b-d5c6fbf226cb) in queue (A226B246YXJDNS_3UWN2HHPUZCS9YHAB3KWJAX5QGONSY)
Send packet: Packet <{'id': 'A226B246YXJDNS_fa6af0d7-230d-4adc-898b-d5c6fbf226cb', 'type': 'message', 'sender_id': '[World_convai2:self_1553147054]', 'receiver_id': 'A226B246YXJDNS', 'assignment_id': '3UWN2HHPUZCS9YHAB3KWJAX5QGONSY', 'conversation_id': None, 'data': {'text': 'COMMAND_CHANGE_CONVERSATION', 'conversation_id': 'o_c617cc83-1da1-469a-b0d3-76fc5b4f210a', 'agent_id': 'onboarding', 'type': 'COMMAND'}}>
On new message: (<websocket._app.WebSocketApp object at 0x7fd85db757b8>, '{"type":"new packet","content":{"id":"5fd47107-ad74-4f67-baf9-43e0dbe828e5","type":"alive","sender_id":"A226B246YXJDNS","assignment_id":"3UWN2HHPUZCS9YHAB3KWJAX5QGONSY","conversation_id":null,"receiver_id":"[World_convai2:self_1553147054]","data":{"hit_id":"372AGES0I4BJYTX78I2Y2IORZ7ZRX3","assignment_id":"3UWN2HHPUZCS9YHAB3KWJAX5QGONSY","worker_id":"A226B246YXJDNS","conversation_id":null},"status":"sent","require_ack":true,"blocking":true}}')
on_agent_alive: Packet <{'id': '5fd47107-ad74-4f67-baf9-43e0dbe828e5', 'type': 'alive', 'sender_id': 'A226B246YXJDNS', 'receiver_id': '[World_convai2:self_1553147054]', 'assignment_id': '3UWN2HHPUZCS9YHAB3KWJAX5QGONSY', 'conversation_id': None, 'data': {'hit_id': '372AGES0I4BJYTX78I2Y2IORZ7ZRX3', 'assignment_id': '3UWN2HHPUZCS9YHAB3KWJAX5QGONSY', 'worker_id': 'A226B246YXJDNS', 'conversation_id': None}}>
Channel (A226B246YXJDNS_3UWN2HHPUZCS9YHAB3KWJAX5QGONSY) already open
Agent (A226B246YXJDNS)_(3UWN2HHPUZCS9YHAB3KWJAX5QGONSY) reconnected to o_c617cc83-1da1-469a-b0d3-76fc5b4f210a with status none
Put packet (A226B246YXJDNS_6375e253-fb76-42cb-af54-dbc60350f310) in queue (A226B246YXJDNS_3UWN2HHPUZCS9YHAB3KWJAX5QGONSY)
On new ack: (<websocket._app.WebSocketApp object at 0x7fd85db757b8>, '{"type":"new packet","content":{"id":"A226B246YXJDNS_fa6af0d7-230d-4adc-898b-d5c6fbf226cb","sender_id":"A226B246YXJDNS","receiver_id":"[World_convai2:self_1553147054]","assignment_id":"3UWN2HHPUZCS9YHAB3KWJAX5QGONSY","conversation_id":null,"type":"ack","data":null}}')
Send packet: Packet <{'id': 'A226B246YXJDNS_6375e253-fb76-42cb-af54-dbc60350f310', 'type': 'message', 'sender_id': '[World_convai2:self_1553147054]', 'receiver_id': 'A226B246YXJDNS', 'assignment_id': '3UWN2HHPUZCS9YHAB3KWJAX5QGONSY', 'conversation_id': None, 'data': {'text': 'COMMAND_EXPIRE_HIT', 'inactive_text': "This HIT is expired, please return and take a new one if you'd want to work on this task.", 'type': 'COMMAND'}}>
On new message: (<websocket._app.WebSocketApp object at 0x7fd85db757b8>, '{"type":"new packet","content":{"id":"9b1f3dae-b725-4507-b594-3ca29ee263b2","type":"alive","sender_id":"A226B246YXJDNS","assignment_id":"3UWN2HHPUZCS9YHAB3KWJAX5QGONSY","conversation_id":"o_c617cc83-1da1-469a-b0d3-76fc5b4f210a","receiver_id":"[World_convai2:self_1553147054]","data":{"hit_id":"372AGES0I4BJYTX78I2Y2IORZ7ZRX3","assignment_id":"3UWN2HHPUZCS9YHAB3KWJAX5QGONSY","worker_id":"A226B246YXJDNS","conversation_id":"o_c617cc83-1da1-469a-b0d3-76fc5b4f210a"},"status":"init","require_ack":true,"blocking":true}}')
on_agent_alive: Packet <{'id': '9b1f3dae-b725-4507-b594-3ca29ee263b2', 'type': 'alive', 'sender_id': 'A226B246YXJDNS', 'receiver_id': '[World_convai2:self_1553147054]', 'assignment_id': '3UWN2HHPUZCS9YHAB3KWJAX5QGONSY', 'conversation_id': 'o_c617cc83-1da1-469a-b0d3-76fc5b4f210a', 'data': {'hit_id': '372AGES0I4BJYTX78I2Y2IORZ7ZRX3', 'assignment_id': '3UWN2HHPUZCS9YHAB3KWJAX5QGONSY', 'worker_id': 'A226B246YXJDNS', 'conversation_id': 'o_c617cc83-1da1-469a-b0d3-76fc5b4f210a'}}>
Channel (A226B246YXJDNS_3UWN2HHPUZCS9YHAB3KWJAX5QGONSY) already open
Agent (A226B246YXJDNS)_(3UWN2HHPUZCS9YHAB3KWJAX5QGONSY) reconnected to o_c617cc83-1da1-469a-b0d3-76fc5b4f210a with status expired
Put packet (A226B246YXJDNS_b56f66ae-3f75-4173-b8f1-ad975fb2fb06) in queue (A226B246YXJDNS_3UWN2HHPUZCS9YHAB3KWJAX5QGONSY)
On new message: (<websocket._app.WebSocketApp object at 0x7fd85db757b8>, '{"type":"new packet","content":{"id":"9b1f3dae-b725-4507-b594-3ca29ee263b2","type":"alive","sender_id":"A226B246YXJDNS","assignment_id":"3UWN2HHPUZCS9YHAB3KWJAX5QGONSY","conversation_id":"o_c617cc83-1da1-469a-b0d3-76fc5b4f210a","receiver_id":"[World_convai2:self_1553147054]","data":{"hit_id":"372AGES0I4BJYTX78I2Y2IORZ7ZRX3","assignment_id":"3UWN2HHPUZCS9YHAB3KWJAX5QGONSY","worker_id":"A226B246YXJDNS","conversation_id":"o_c617cc83-1da1-469a-b0d3-76fc5b4f210a"},"status":"sent","require_ack":true,"blocking":true}}')
on_agent_alive: Packet <{'id': '9b1f3dae-b725-4507-b594-3ca29ee263b2', 'type': 'alive', 'sender_id': 'A226B246YXJDNS', 'receiver_id': '[World_convai2:self_1553147054]', 'assignment_id': '3UWN2HHPUZCS9YHAB3KWJAX5QGONSY', 'conversation_id': 'o_c617cc83-1da1-469a-b0d3-76fc5b4f210a', 'data': {'hit_id': '372AGES0I4BJYTX78I2Y2IORZ7ZRX3', 'assignment_id': '3UWN2HHPUZCS9YHAB3KWJAX5QGONSY', 'worker_id': 'A226B246YXJDNS', 'conversation_id': 'o_c617cc83-1da1-469a-b0d3-76fc5b4f210a'}}>
Channel (A226B246YXJDNS_3UWN2HHPUZCS9YHAB3KWJAX5QGONSY) already open
Agent (A226B246YXJDNS)_(3UWN2HHPUZCS9YHAB3KWJAX5QGONSY) reconnected to o_c617cc83-1da1-469a-b0d3-76fc5b4f210a with status expired
Put packet (A226B246YXJDNS_5480eaf4-665b-4b80-93fa-df89810dfc36) in queue (A226B246YXJDNS_3UWN2HHPUZCS9YHAB3KWJAX5QGONSY)
Send packet: Packet <{'id': 'A226B246YXJDNS_6375e253-fb76-42cb-af54-dbc60350f310', 'type': 'message', 'sender_id': '[World_convai2:self_1553147054]', 'receiver_id': 'A226B246YXJDNS', 'assignment_id': '3UWN2HHPUZCS9YHAB3KWJAX5QGONSY', 'conversation_id': None, 'data': {'text': 'COMMAND_EXPIRE_HIT', 'inactive_text': "This HIT is expired, please return and take a new one if you'd want to work on this task.", 'type': 'COMMAND'}}>
On new ack: (<websocket._app.WebSocketApp object at 0x7fd85db757b8>, '{"type":"new packet","content":{"id":"A226B246YXJDNS_6375e253-fb76-42cb-af54-dbc60350f310","sender_id":"A226B246YXJDNS","receiver_id":"[World_convai2:self_1553147054]","assignment_id":"3UWN2HHPUZCS9YHAB3KWJAX5QGONSY","conversation_id":"o_c617cc83-1da1-469a-b0d3-76fc5b4f210a","type":"ack","data":null}}')
Closing channel A226B246YXJDNS_3UWN2HHPUZCS9YHAB3KWJAX5QGONSY
Send packet: Packet <{'id': 'A226B246YXJDNS_b56f66ae-3f75-4173-b8f1-ad975fb2fb06', 'type': 'message', 'sender_id': '[World_convai2:self_1553147054]', 'receiver_id': 'A226B246YXJDNS', 'assignment_id': '3UWN2HHPUZCS9YHAB3KWJAX5QGONSY', 'conversation_id': None, 'data': {'text': 'COMMAND_INACTIVE_HIT', 'inactive_text': 'You disconnected in the middle of this HIT and the HIT expired before you reconnected. It is no longer available for completion. Please return this HIT and accept a new one if you would like to try again.', 'conversation_id': 'o_c617cc83-1da1-469a-b0d3-76fc5b4f210a', 'agent_id': 'A226B246YXJDNS', 'type': 'COMMAND'}}>
Send packet: Packet <{'id': 'A226B246YXJDNS_5480eaf4-665b-4b80-93fa-df89810dfc36', 'type': 'message', 'sender_id': '[World_convai2:self_1553147054]', 'receiver_id': 'A226B246YXJDNS', 'assignment_id': '3UWN2HHPUZCS9YHAB3KWJAX5QGONSY', 'conversation_id': None, 'data': {'text': 'COMMAND_INACTIVE_HIT', 'inactive_text': 'You disconnected in the middle of this HIT and the HIT expired before you reconnected. It is no longer available for completion. Please return this HIT and accept a new one if you would like to try again.', 'conversation_id': 'o_c617cc83-1da1-469a-b0d3-76fc5b4f210a', 'agent_id': 'A226B246YXJDNS', 'type': 'COMMAND'}}>
``` 



Hi @andreamad8, sorry I didn't get a chance to actually debug this today, however there's definitely something amiss. The server is explicitly expiring the HIT right after the worker is confirmed as having entered the world but before anything else happens. Tomorrow I'll try to replicate this behavior.
Hi, @JackUrb thanks a lot for your help. 

Andrea
Alright I've taken a pass at the forward paths that could possibly be causing the bug on your end. Somewhere between the request to change to an onboarding conversation here:
https://github.com/facebookresearch/ParlAI/blob/a5eaf0863acfbaf97766fb4f170706258096bdc8/parlai/mturk/core/mturk_manager.py#L768
and the agent actually being put into an onboarding world here:
https://github.com/facebookresearch/ParlAI/blob/a5eaf0863acfbaf97766fb4f170706258096bdc8/parlai/mturk/core/mturk_manager.py#L779
a call to `force_expire_hit` is coming through. My best guess for where this is happening is here:
https://github.com/facebookresearch/ParlAI/blob/a5eaf0863acfbaf97766fb4f170706258096bdc8/parlai/mturk/core/mturk_manager.py#L626
Can you insert the following lines after line 615 and then run again?

```
print(agent.get_status(), conversation_id, agent.conversation_id)
print(self.assignment_to_onboard_thread)
print(self.worker_manager.conv_to_agent)
```

If this confirms my thought, it's likely that this is being caused by a race condition in which the frontend is sending the same alive packet (while waiting for an ack) at the same time that the system is expecting a completely different alive. In this case, that leads to degenerate behavior as the system treats it as a new alive.

It seems you've stumbled into what should be a rather rare edge case (as I have never encountered it or been able to reproduce it) with some amount of consistency, which is great for me to be able to fix it for everyone. I'm putting together something that should fix it, but I'll need the above output to confirm.

Resolving patch:
diff --git a/parlai/mturk/core/mturk_utils.py b/parlai/mturk/core/mturk_utils.py
--- a/parlai/mturk/core/mturk_utils.py
+++ b/parlai/mturk/core/mturk_utils.py
@@ -7,6 +7,7 @@
 import boto3
 import os
 import json
+import re
 from datetime import datetime
 
 from botocore.exceptions import ClientError
@@ -374,7 +375,9 @@ def expire_hit(is_sandbox, hit_id):
 def setup_sns_topic(task_name, server_url, task_group_id):
     # Create the topic and subscribe to it so that our server receives notifs
     client = boto3.client('sns', region_name='us-east-1',)
-    response = client.create_topic(Name=task_name)
+    pattern = re.compile('[^a-zA-Z0-9_-]+')
+    filtered_task_name = pattern.sub('', task_name)
+    response = client.create_topic(Name=filtered_task_name)
     arn = response['TopicArn']
     topic_sub_url = \
         '{}/sns_posts?task_group_id={}'.format(server_url, task_group_id)
diff --git a/parlai/mturk/core/socket_manager.py b/parlai/mturk/core/socket_manager.py
--- a/parlai/mturk/core/socket_manager.py
+++ b/parlai/mturk/core/socket_manager.py
@@ -210,6 +210,11 @@ def __init__(self, server_url, port, alive_callback, message_callback,
         self.last_sent_heartbeat_time = {}  # time of last heartbeat sent
         self.last_received_heartbeat = {}  # actual last received heartbeat
         self.pongs_without_heartbeat = {}
+        # TODO update processed packets to *ensure* only one execution per
+        # packet, as right now two threads can be spawned to process the
+        # same packet at the same time, as the packet is only added to this
+        # set after processing is complete.
+        self.processed_packets = set()
         self.packet_map = {}
         self.alive = False
         self.is_shutdown = False
@@ -443,7 +448,13 @@ def on_message(*args):
                 pong_conn_id = packet.get_receiver_connection_id()
                 if self.last_received_heartbeat[pong_conn_id] is not None:
                     self.pongs_without_heartbeat[pong_conn_id] += 1
+            elif packet_id in self.processed_packets:
+                # We don't want to re-process a packet that has already
+                # been recieved, but we do need to tell the sender it is ack'd
+                # re-acknowledge the packet was recieved and already processed
+                self._send_ack(packet)
             else:
+
                 # Remaining packet types need to be acknowledged
                 shared_utils.print_and_log(
                     logging.DEBUG,
@@ -458,6 +469,10 @@ def on_message(*args):
                 # acknowledge the packet was recieved and processed
                 self._send_ack(packet)
 
+                # Note to self that this packet has already been processed,
+                # and shouldn't be processed again in the future
+                self.processed_packets.add(packet_id)
+
         def run_socket(*args):
             url_base_name = self.server_url.split('https://')[1]
             protocol = "wss"
diff --git a/parlai/mturk/tasks/convai2_model_eval/run.py b/parlai/mturk/tasks/convai2_model_eval/run.py
--- a/parlai/mturk/tasks/convai2_model_eval/run.py
+++ b/parlai/mturk/tasks/convai2_model_eval/run.py
@@ -110,6 +110,7 @@ def main():
                 agent_qualifications.append(MASTER_QUALIF_SDBOX)
             else:
                 agent_qualifications.append(MASTER_QUALIF)
+        mturk_manager.ready_to_accept_workers()
         mturk_manager.create_hits(qualifications=agent_qualifications)
 
         if not opt['is_sandbox']:
@@ -127,7 +128,6 @@ def run_onboard(worker):
             world.parley()
             world.shutdown()
         mturk_manager.set_onboard_function(onboard_function=run_onboard)
-        mturk_manager.ready_to_accept_workers()
 
         def check_worker_eligibility(worker):
             return True


Test patch: