opened
stephenroller
username_0
Hi there,

Thanks for the excellent paper of [Takanobu et al. (2020)](https://arxiv.org/pdf/2005.07362.pdf). I thoroughly appreciate the systematic comparison across so many points, and especially the call out of mismatch between single turn performance and overall performance.

One question I have is about the source code. Right now, the top line README for this repo shows a very similar table to Table 1 of the paper, but with very different results (and indeed, quite a few different systems, with some disagreeing). Furthermore, you have similar tables for 2, 3, etc in the README, but with different metrics.

- How the table in the README different from the one presented in the paper?
- Why the mismatch of reported metrics? Can one use ConvLab-2 to produce the rest of the metrics reported in the paper?
- The paper calls out that the code was all open source in February, while this repo became available in May. Perhaps I should be looking at a different repo?
- If there are two repos, which one do you anticipate receiving more support? I assume this one, as it aims to be a standardized platform.

Best wishes from the [ParlAI](https://github.com/facebookresearch/ParlAI) team & collaborators

Thanks,
Stephen
Questions about "Is your goal oriented model performing well"
issue
created
zqwerty
username_1
Thanks for your interest. The paper of Takanobu et al. (2020) use [ConvLab](https://github.com/ConvLab/ConvLab) platform. This repo updated data, algorithms, the simulator, the evaluator, etc. This repo will receive more support.
comment
created
stephenroller
username_0
Great, thanks much!
comment
closed
stephenroller
username_0
issue