TASK DEFINITION: In this task, you are given a part of an article. Your task is to generate headline (title) for this text. Preferred headlines are under fifteen words.
PROBLEM: We investigate evaluation metrics for endto-end dialogue systems where supervised labels, such as task completion, are not available. Recent works in end-to-end dialogue systems have adopted metrics from machine translation and text summarization to compare a model’s generated response to a single target response. We show that these metrics correlate very weakly or not at all with human judgements of the response quality in both technical and non-technical domains. We provide quantitative and qualitative results highlighting specific weaknesses in existing metrics, and provide recommendations for future development of better automatic evaluation metrics for dialogue systems.

SOLUTION: How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

PROBLEM: In this paper we present methods for attacking and defending k-gram statistical analysis techniques that are used, for example, in network traffic analysis and covert channel detection. The main new result is our demonstration of how to use a behavior’s or process’ k-order statistics to build a stochastic process that has those same k-order stationary statistics but possesses different, deliberately designed, (k + 1)order statistics if desired. Such a model realizes a “complexification” of the process or behavior which a defender can use to monitor whether an attacker is shaping the behavior. By deliberately introducing designed (k + 1)-order behaviors, the defender can check to see if those behaviors are present in the data. We also develop constructs for source codes that respect the k-order statistics of a process while encoding covert information. One fundamental consequence of these results is that certain types of behavior analyses techniques come down to an arms race in the sense that the advantage goes to the party that has more computing resources applied to the problem. Points of view in this document are those of the authors and do not necessarily represent the official position of the sponsoring agencies or the U.S. Government. V. Crespi is with the Department of Computer Science, California State University at Los Angeles, Los Angeles CA, 90032 USA. email: vcrespi@calstatela.edu. Crespi’s work was partially supported by AFOSR Grant FA9550-07-1-0421 and by NSF Grant HRD-0932421. G. Cybenko is with the Thayer School of Engineering, Dartmouth College, Hanover NH 03755. email: gvc@dartmouth.edu. Cybenko’s work was partially supported by Air Force Research Laboratory contracts FA8750-10-1-0045, FA8750-09-1-0174, AFOSR contract FA9550-07-1-0421, U.S. Department of Homeland Security Grant 2006-CS-001-000001 and DARPA Contract HR001-06-1-0033 A. Giani is with the Department of EECS, University of California at Berkeley, Berkeley CA 94720. email: agiani@eecs.berkeley.edu. Giani’s’s work was partially supported by U.S. Department of Homeland Security Grant 2006-CS001-000001 and DARPA Contract HR001-06-1-0033 when she was a Ph.D. student at Dartmouth April 28, 2011 DRAFT ar X iv :1 10 4. 50 71 v1 [ cs .L G ] 2 7 A pr 2 01 1 CRESPI, CYBENKO, GIANI 2

SOLUTION: Attacking and Defending Covert Channels and Behavioral Models

PROBLEM: Recent works on end-to-end neural network-based architectures for machine translation have shown promising results for English-French and English-German translation. Unlike these language pairs, however, in the majority of scenarios, there is a lack of high quality parallel corpora. In this work, we focus on applying neural machine translation to challenging/low-resource languages Turkish and low-resource domains such as parallel corpora of Chinese chat messages. In particular, we investigated how to leverage abundant monolingual data for these low-resource translation tasks. Without the use of external alignment tools, we obtained up to a 1.96 BLEU score improvement with our proposed method compared to the previous best result in Turkish-to-English translation on the IWLST 2014 dataset. On Chinese-toEnglish translation by using the OpenMT 2015 dataset, we were able to obtain up to a 1.59 BLEU score improvement over phrase-based and hierarchical phrase-based baselines.

SOLUTION:
On Using Monolingual Corpora in Neural Machine Translation