Question: How are models evaluated in this human-machine communication game?
Answer: by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews
Alternative answer: efficiency of a communication scheme $(q_{\alpha },p_{\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords, accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence