Question: What has been used as the pre-training objective on pre-training for NLP in existing work?
Evidence: 
Antisocial behavior. Antisocial behavior online comes in many forms, including harassment BIBREF30, cyberbullying BIBREF31, and general aggression BIBREF32. Prior work has sought to understand different aspects of such behavior, including its effect on the communities where it happens BIBREF33, BIBREF34, the actors involved BIBREF35, BIBREF36, BIBREF37, BIBREF38 and connections to the outside world BIBREF39.
Post-hoc classification of conversations. There is a rich body of prior work on classifying the outcome of a conversation after it has concluded, or classifying conversational events after they happened. Many examples exist, but some more closely related to our present work include identifying the winner of a debate BIBREF40, BIBREF41, BIBREF42, identifying successful negotiations BIBREF21, BIBREF43, as well as detecting whether deception BIBREF44, BIBREF45, BIBREF46 or disagreement BIBREF47, BIBREF48, BIBREF49, BIBREF50, BIBREF51 has occurred.
Our goal is different because we wish to forecast conversational events before they happen and while the conversation is still ongoing (potentially allowing for interventions). Note that some post-hoc tasks can also be re-framed as forecasting tasks (assuming the existence of necessary labels); for instance, predicting whether an ongoing conversation will eventually spark disagreement BIBREF18, rather than detecting already-existing disagreement.
Conversational forecasting. As described in Section SECREF1, prior work on forecasting conversational outcomes and events has largely relied on hand-crafted features to capture aspects of conversational dynamics. Example feature sets include statistical measures based on similarity between utterances BIBREF16, sentiment imbalance BIBREF20, flow of ideas BIBREF20, increase in hostility BIBREF8, reply rate BIBREF11 and graph representations of conversations BIBREF52, BIBREF17. By contrast, we aim to automatically learn neural representations of conversational dynamics through pre-training.
Such hand-crafted features are typically extracted from fixed-length windows of the conversation, leaving unaddressed the problem of unknown horizon. While some work has trained multiple models for different window-lengths BIBREF8, BIBREF18, they consider these models to be independent and, as such, do not address the issue of aggregating them into a single forecast (i.e., deciding at what point to make a prediction). We implement a simple sliding windows solution as a baseline (Section SECREF5).
Pre-training for NLP. The use of pre-training for natural language tasks has been growing in popularity after recent breakthroughs demonstrating improved performance on a wide array of benchmark tasks BIBREF53, BIBREF54. Existing work has generally used a language modeling objective as the pre-training objective; examples include next-word prediction BIBREF55, sentence autoencoding, BIBREF56, and machine translation BIBREF57. BERT BIBREF58 introduces a variation on this in which the goal is to predict the next sentence in a document given the current sentence. Our pre-training objective is similar in spirit, but operates at a conversation level, rather than a document level. We hence view our objective as conversational modeling rather than (only) language modeling. Furthermore, while BERT's sentence prediction objective is framed as a multiple-choice task, our objective is framed as a generative task.

Pre-training for NLP. The use of pre-training for natural language tasks has been growing in popularity after recent breakthroughs demonstrating improved performance on a wide array of benchmark tasks (Peters et al., 2018; Radford et al., 2018). Existing work has generally used a language modeling objective as the pre-training objective; examples include next-word prediction (Howard and Ruder, 2018), sentence autoencoding, (Dai and Le, 2015), and machine translation (McCann et al., 2017).
Answer: A language modeling objective.