Repository: facebookresearch/ParlAI

Issue:
Wizard of Wikipedia/mturk_evaluation_task fails to run (looks for perz0na data)
**Bug description**
I am getting a memory error while running the Mechanical Turk task for evaluating models trained on the Wizard of Wikipedia task.

**Reproduction steps**
Enter steps to reproduce the behavior:
Execute the run.py from /ParlAI/projects/wizard_of_wikipedia/mturk_evaluation_task

**CLI logs:**
```
/home/namanahujia/chatbots/ParlAI/parlai/agents/transformer/transformer.py:18: UserWarning: Public release transformer models are currently in beta. The name of command line options may change or disappear before a stable release. We welcome your feedback. Please file feedback as issues at https://github.com/facebookresearch/ParlAI/issues/new
  "Public release transformer models are currently in beta. The name of "
[ optional arguments: ] 
[  ag_shutdown_time: 120 ]
[  auto_approve_delay: 86400 ]
[  datapath: /home/namanahujia/chatbots/ParlAI/data ]
[  human_eval: False ]
[  max_choice_time: 300 ]
[  max_resp_time: 240 ]
[  max_turns: 10 ]
[  mturk_log: data/mturklogs/2019-04-23-11-36.log ]
[  only_masters: False ]
[  range_turn: 3,5 ]
[  unique_workers: False ]
[ ParlAI Model Arguments: ] 
[  dict_class: None ]
[  init_model: None ]
[  model: None ]
[  model_file: None ]
[ Mechanical Turk: ] 
[  allowed_conversations: 0 ]
[  block_qualification: None ]
[  count_complete: False ]
[  disconnect_qualification: None ]
[  hard_block: False ]
[  heroku_team: None ]
[  hobby: False ]
[  is_sandbox: True ]
[  local: False ]
[  log_level: 20 ]
[  max_connections: 30 ]
[  max_hits_per_worker: 0 ]
[  max_time: 0 ]
[  max_time_qual: None ]
[  min_messages: 0 ]
[  mturk_log_path: /home/namanahujia/chatbots/ParlAI/logs/mturk ]
[  num_conversations: 1 ]
[  reward: 0.05 ]
[  task: None ]
[  tmp_dir: None ]
[  unique_qual_name: None ]
[  unique_worker: False ]
[  verbose: False ]
[ TorchAgent Arguments: ] 
[  add_p1_after_newln: False ]
[  betas: (0.9, 0.999) ]
[  delimiter:   ]
[  embedding_projection: random ]
[  embedding_type: random ]
[  fp16: False ]
[  gpu: -1 ]
[  gradient_clip: 0.1 ]
[  history_size: -1 ]
[  label_truncate: None ]
[  learningrate: 0.0008 ]
[  lr_scheduler: reduceonplateau ]
[  lr_scheduler_decay: 0.5 ]
[  lr_scheduler_patience: 3 ]
[  momentum: 0 ]
[  nesterov: True ]
[  no_cuda: False ]
[  nus: (0.7,) ]
[  optimizer: adamax ]
[  person_tokens: False ]
[  rank_candidates: False ]
[  split_lines: False ]
[  text_truncate: None ]
[  truncate: 1024 ]
[  update_freq: -1 ]
[  use_reply: label ]
[  warmup_rate: 0.0001 ]
[  warmup_updates: -1 ]
[ TorchRankerAgent: ] 
[  candidates: batch ]
[  cap_num_predictions: 100 ]
[  encode_candidate_vecs: False ]
[  eval_candidates: inline ]
[  fixed_candidate_vecs: reuse ]
[  fixed_candidates_path: None ]
[  ignore_bad_candidates: False ]
[  init_model: None ]
[  train_predict: False ]
[ Transformer Arguments: ] 
[  activation: relu ]
[  attention_dropout: 0.0 ]
[  data_parallel: False ]
[  dropout: 0.0 ]
[  embedding_size: 300 ]
[  embeddings_scale: True ]
[  ffn_size: 300 ]
[  learn_embeddings: True ]
[  learn_positional_embeddings: False ]
[  memory_attention: sqrt ]
[  n_heads: 2 ]
[  n_layers: 2 ]
[  n_positions: None ]
[  n_segments: 0 ]
[  normalize_sent_emb: False ]
[  reduction_type: mean ]
[  relu_dropout: 0.0 ]
[  share_encoders: True ]
[  use_memories: False ]
[  variant: aiayn ]
[  wrap_memory_encoder: False ]
[ Dictionary Arguments: ] 
[  bpe_debug: False ]
[  dict_endtoken: __end__ ]
[  dict_file: None ]
[  dict_initpath: None ]
[  dict_language: english ]
[  dict_lower: False ]
[  dict_max_ngram_size: -1 ]
[  dict_maxtokens: -1 ]
[  dict_minfreq: 0 ]
[  dict_nulltoken: __null__ ]
[  dict_starttoken: __start__ ]
[  dict_textfields: text,labels ]
[  dict_tokenizer: re ]
[  dict_unktoken: __unk__ ]
[ Wizard Transformer Ranker Arguments: ] 
[  chosen_sentence: False ]
[  knowledge_dropout: 0.7 ]
[  knowledge_truncate: 50 ]
[  legacy: False ]
[  use_knowledge: True ]
[ WizardRetrievalInteractive Arguments: ] 
[  debug: False ]
[  get_unique: True ]
[  num_retrieved: 7 ]
[  responder_model_file: None ]
[  retriever_model_file: None ]

********************************************************************************
Thank you for using ParlAI! We are conducting a user survey.
Please consider filling it out at https://forms.gle/uEFbYGP7w6hiuGQT9
********************************************************************************

[building data: /home/namanahujia/chatbots/ParlAI/data/models/wikipedia_full/tfidf_retriever/model.tgz]
[ downloading: http://parl.ai/downloads/_models/wikipedia_full/tfidf_retriever/model.tgz to /home/namanahujia/chatbots/ParlAI/data/models/wikipedia_full/tfidf_retriever/model.tgz ]
Downloading model.tgz: 100%|███████████████| 8.14G/8.14G [05:53<00:00, 23.0MB/s]
unpacking model.tgz
[ warning: overriding opt['remove_title'] to False (previously: True )]
/home/namanahujia/chatbots/ParlAI/parlai/core/agents.py:416: UserWarning: WARNING: Neither the specified dict file (test_ret.dict) nor the `model_file`.dict file (/home/namanahujia/chatbots/ParlAI/data/models/wikipedia_full/tfidf_retriever/model.dict) exists, check to make sure either is correct. This may manifest as a shape mismatch later on.
  'on.'.format(old_dict_file, new_opt['dict_file'])
Traceback (most recent call last):
  File "run.py", line 252, in <module>
    main()
  File "run.py", line 100, in main
    bot = create_agent(start_opt)
  File "/home/namanahujia/chatbots/ParlAI/parlai/core/agents.py", line 581, in create_agent
    model = model_class(opt)
  File "/home/namanahujia/chatbots/ParlAI/projects/wizard_of_wikipedia/interactive_retrieval/interactive_retrieval.py", line 35, in __init__
    self._set_up_retriever(opt)
  File "/home/namanahujia/chatbots/ParlAI/projects/wizard_of_wikipedia/interactive_retrieval/interactive_retrieval.py", line 73, in _set_up_retriever
    self.wiki_map = json.load(open(wiki_map_path, 'r'))
  File "/home/namanahujia/anaconda3/envs/parlai/lib/python3.7/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/home/namanahujia/anaconda3/envs/parlai/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
MemoryError
```


Discussion:
Interesting, how much RAM does your machine have? I would think the index of wikipedia would fit in memory, but maybe not if your machine is small.
RAM is 32 GB. So, that should not be a problem
It’s difficult to say because this appears to be crashing on loading Wikipedia. Indeed, keeping a full copy of Wikipedia in memory (even with sparse vectors) requires a large amount of memory.

It’s unclear to me that the reason for this failure isn’t just that it’s simply too much. It’s very possible that it’s a bug, but I would not be surprised that loading a large transformer retrieval model plus the tfidf Wikipedia model could easily run over 32gb.

I do notice that the responder model isn’t protected by shared, but the Wikipedia retriever is. Can you add some print statements to the init of the interactive retrieval model to see every time it’s making calls to set up retriever and set up responder
I just ran it myself. It seems to top out at around 47gb, and then crashes with the below crash. This should be fixed but is a separate issue. I suspect that it would continue to grow after loading these files, but probably would top out under 64gb.

```python-traceback
Traceback (most recent call last):
  File "projects/wizard_of_wikipedia/mturk_evaluation_task/run.py", line 252, in <module>
    main()
  File "projects/wizard_of_wikipedia/mturk_evaluation_task/run.py", line 136, in main
    topics_generator = TopicsGenerator(start_opt)
  File "/private/home/roller/working/parlai/projects/wizard_of_wikipedia/mturk_evaluation_task/worlds.py", line 74, in __init__
    self.load_topics()
  File "/private/home/roller/working/parlai/projects/wizard_of_wikipedia/mturk_evaluation_task/worlds.py", line 77, in load_topics
    with open(self.topics_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/private/home/roller/working/parlai/data/wizard_of_perzona/wizard_data_no_full_splits/topic_splits.json'
python projects/wizard_of_wikipedia/mturk_evaluation_task/run.py  222.87s user 66.86s system 59% cpu 8:08.65 total
```

Some extra information:
```bash
$ ls /private/home/roller/working/parlai/data/wizard_of_perzona/
ls: cannot access '/private/home/roller/working/parlai/data/wizard_of_perzona/': No such file or directory
$ ls /private/home/roller/working/parlai/data/wizard_of_wikipedia/
data.json  test_random_split.json  test_topic_split.json  topic_splits.json  train.json  valid_random_split.json  valid_topic_split.json
```

Resolving patch:
diff --git a/projects/wizard_of_wikipedia/mturk_evaluation_task/worlds.py b/projects/wizard_of_wikipedia/mturk_evaluation_task/worlds.py
--- a/projects/wizard_of_wikipedia/mturk_evaluation_task/worlds.py
+++ b/projects/wizard_of_wikipedia/mturk_evaluation_task/worlds.py
@@ -2,10 +2,11 @@
 # This source code is licensed under the MIT license found in the
 # LICENSE file in the root directory of this source tree.
 from parlai.core.agents import create_agent_from_shared
+from parlai.core.worlds import validate, MultiAgentDialogWorld
 from parlai.mturk.core.agents import TIMEOUT_MESSAGE
 import parlai.mturk.core.mturk_utils as mutils
-from parlai.core.worlds import validate, MultiAgentDialogWorld
 from parlai.mturk.core.worlds import MTurkOnboardWorld
+from parlai.tasks.wizard_of_wikipedia.build import build
 
 from joblib import Parallel, delayed
 import json
@@ -67,18 +68,21 @@
 
 class TopicsGenerator(object):
     def __init__(self, opt):
+        self.opt = opt
         self.topics_path = os.path.join(
-            opt.get('datapath'),
-            'wizard_of_perzona/wizard_data_no_full_splits/topic_splits.json'
+            opt['datapath'],
+            'wizard_of_wikipedia/topic_splits.json'
         )
         self.load_topics()
 
     def load_topics(self):
+        if not os.path.isfile(self.topics_path):
+            # download the wizard of wikipedia data
+            build(self.opt)
         with open(self.topics_path) as f:
             self.data = json.load(f)
-        self.seen_topics = self.data['train']['train_mtdo_topics']
-        self.unseen_topics = self.data['valid']['valid_mtdo_topics'] + \
-            self.data['test']['test_mtdo_topics']
+        self.seen_topics = self.data['train']
+        self.unseen_topics = self.data['valid'] + self.data['test']
 
     def get_topics(self, seen=True, num=3):
         if seen:


Test patch: