Repository: facebookresearch/ParlAI

Issue:
Failed to Reproduce Human Evaluation of Controllable Dialogue with Pretrained Model
**Bug description**
I failed to reproduce the human evaluation of the controllable dialogue with pre-trained model. The dialogue log in `evaluation_logs/` folder. 

**Reproduction steps**
For the first dialogue in `evaluation_logs/baseline_model.jsonl`
*MetaData*
- model_persona: "i am terrified of clowns due to childhood trauma.\ni enjoy working with my hands and love gardening.\ni just got a job working in a gas station.\ni was adopted at a young age to an older couple.\nmy father keeps trying to set me up on blind dates."
- dialog:[{"speaker":"model","text":"Hello, how are you today?"},{"speaker":"human_evaluator","text":"hello! im doing well. how are you"},{"speaker":"model","text":"I am well, thank you. What do you do for a living?"},{"speaker":"human_evaluator","text":"i work part time at burger king making minimum wage. what about you"},{"speaker":"model","text":"I work in a gas station."},{"speaker":"human_evaluator","text":"what do you do for fun"},{"speaker":"model","text":"I work in a gas station. What about you?"},{"speaker":"human_evaluator","text":"i work at burger king"},{"speaker":"model","text":"What do you do for a living?"},{"speaker":"human_evaluator","text":"i live with my parents but i work part time in fast food"},{"speaker":"model","text":"What city are you from?"},{"speaker":"human_evaluator","text":"i am from boston. where are you"}]

Step 1. Open interactive chat with baseline agent.
```
python projects/controllable_dialogue/interactive.py \
-mf models:controllable_dialogue/convai2_finetuned_baseline
```
Step 2.
Enter your Message: `your persona: i am terrified of clowns due to childhood trauma.\nyour persona: i enjoy working with my hands and love gardening.\nyour persona: i just got a job working in a gas station.\nyour persona: i was adopted at a young age to an older couple.\nyour persona: my father keeps trying to set me up on blind dates.\n__SILENCE__`
Step 3.
Get agent response: `Hi how are you doing?`. Which does not match the evaluation log `Hello, how are you today?`

For all the 111 dialogues in `evaluation_logs/baseline_model.jsonl`, I can only reproduce 44 of them. 

**Expected behavior**
When the pre-trained model is given the same model persona and human response, I expect to reproduce the dialogue in `evaluation_logs`. 

**Logs**
Please paste the command line output:

The command line output is slightly different from the original script, where I compare the model output with the `evaluation_logs/baseline_model.jsonl`. 

The first dialogue in `evaluation_logs/baseline_model.jsonl`:
```
{'text': 'your persona: i am terrified of clowns due to childhood trauma.\nyour persona: i enjoy workingwith my hands and love gardening.\nyour persona: i just got a job working in a gas station.\nyour persona: i was adopted at a young age to an older couple.\nyour persona: my father keeps trying to set me up onblind dates.\n__SILENCE__', 'episode_done': False}
[PERSON_2]: Hi how are you doing?
[REFDATA]: Hello, how are you today?  
Warning! Mismatched
[Human]: hello! im doing well. how are you  
[PERSON_2]: I am good thanks for asking
[REFDATA]: I am well, thank you. What do you do for a living?  
Warning! Mismatched
[Human]: i work part time at burger king making minimum wage. what about you  
[PERSON_2]: I work in a gas station
[REFDATA]: I work in a gas station. 
Warning! Mismatched
[Human]: what do you do for fun
[PERSON_2]: I work at a gas station
[REFDATA]: I work in a gas station. What about you? 
Warning! Mismatched
[Human]: i work at burger king  
[PERSON_2]: What do you do for a living
[REFDATA]: What do you do for a living?  
Warning! Mismatched
[Human]: i live with my parents but i work part time in fast food 
[PERSON_2]: What city are you from?
[REFDATA]: What city are you from?
```

I can reproduce the second dialogue.

The third dialogue in `evaluation_logs/baseline_model.jsonl`:
```
{'text': 'your persona: i am a professional wrestler.\nyour persona: i enjoy being around people.\nyour persona: i like to workout a a times a week.\nyour persona: i volunteer in a homeless shelter.\nyour persona: in my spare time i do volunteer work.\n__SILENCE__', 'episode_done': False}
[PERSON_2]: Hello, how are you today?
[REFDATA]: Hi how are you today  
Warning! Mismatched
[Human]: im good and how are you  
[PERSON_2]: I am doing well. Just got back from a long workout.
[REFDATA]: I am good do you work  
Warning! Mismatched
[Human]: yes i do work and what about you  
[PERSON_2]: I am a professional wrestler.
[REFDATA]: I am a professional wrestler  
Warning! Mismatched
[Human]: thats cool i work online  
[PERSON_2]: What do you do online?
[REFDATA]: What do you do online  
Warning! Mismatched
[Human]: i work for a company to answer phones   
[PERSON_2]: That sounds like a lot of fun.
[REFDATA]: What do you do online  
Warning! Mismatched
[Human]: i  just told you  
[PERSON_2]: What do you do online?
[REFDATA]: What do you do for a living  
Warning! Mismatched
```

**Additional context**
I tried to roll back to different code-base versions and got no luck. 



Discussion:
Hmm so I think it must be some sort of mismatch in something, because we have a [test](https://github.com/facebookresearch/ParlAI/blob/master/tests/nightly/gpu/test_controllable.py) that verifies these pretrained models don't deviate over time. That hasn't fired, so I'm reasonably confident that it isn't a new bug, but some mismatch between the settings at the time we collected those longs, and these experiments.

One thing that stands out to me is that in MTurk we had some [normalization of punctuation and casing](https://github.com/facebookresearch/ParlAI/blob/c5bd36d065532695d31b3f4a76cff21347a100ca/parlai/utils/strings.py#L13) on both humans and models to remove surface level discrimination.

Normalizing the model output:
https://github.com/facebookresearch/ParlAI/blob/c5bd36d065532695d31b3f4a76cff21347a100ca/projects/controllable_dialogue/mturk/worlds.py#L405
https://github.com/facebookresearch/ParlAI/blob/c5bd36d065532695d31b3f4a76cff21347a100ca/projects/controllable_dialogue/mturk/worlds.py#L474


Normalizing the human:
https://github.com/facebookresearch/ParlAI/blob/c5bd36d065532695d31b3f4a76cff21347a100ca/projects/controllable_dialogue/mturk/worlds.py#L422
https://github.com/facebookresearch/ParlAI/blob/c5bd36d065532695d31b3f4a76cff21347a100ca/projects/controllable_dialogue/mturk/worlds.py#L466
https://github.com/facebookresearch/ParlAI/blob/c5bd36d065532695d31b3f4a76cff21347a100ca/projects/controllable_dialogue/mturk/worlds.py#L491

However, the *model still observes its own responses non-normalized.*

One problem with this is that the model still seems to have mismatches on the very first turn. I'm taking at look at this.
Okay I found it. We have a postprocessing script that took the raw dumps from the MTurk runs and put them in a readable and standardized format. In that script, it had roughly `personas = sorted(personas)`. This was useful because it helped us normalize the personas in some of the [ACUTE-Eval experiments](https://arxiv.org/abs/1909.03087). However, it made the logs not reproducible (since the order of the personas matters to the model).

I've reprocessed them without sorting them, and will upload those new logs shortly, as well as the verification script I wrote.
@stephenroller Thanks for the prompt reply. Looking forward to the new logs!

IMHO, reproducibility of these logs is very beneficial to the whole research community.  As far as I know, this is the *only* open-source dialog project that did comprehensive human evaluation and also released both the evaluation logs and pre-trained models. I really appreciate it!
I really appreciate you pointing out the bug and posting a thorough bug report.

Resolving patch:
diff --git a/projects/controllable_dialogue/tasks/build.py b/projects/controllable_dialogue/tasks/build.py
--- a/projects/controllable_dialogue/tasks/build.py
+++ b/projects/controllable_dialogue/tasks/build.py
@@ -17,7 +17,8 @@ def build(opt):
     dpath = os.path.join(opt['datapath'], FOLDER_NAME)
     # version 1.0: initial release
     # version 1.1: add evaluation logs
-    version = '1.1'
+    # version 1.2: add reproducible evaluation logs
+    version = '1.2'
 
     if not build_data.built(dpath, version_string=version):
         if build_data.built(dpath):
@@ -40,6 +41,12 @@ def build(opt):
         build_data.download(URL_ROOT + fname_evallogs, dpath, fname_evallogs)
         build_data.untar(dpath, fname_evallogs)
 
+        # and the reproducible logs.
+        # for more info see https://github.com/facebookresearch/ParlAI/issues/2855
+        fname_evallogs = 'evaluation_logs_reproducible_v1.tar.gz'
+        build_data.download(URL_ROOT + fname_evallogs, dpath, fname_evallogs)
+        build_data.untar(dpath, fname_evallogs)
+
         print("Data has been placed in " + dpath)
 
         build_data.mark_done(dpath, version)


Test patch: