JESSI is trained using only the datasets given on the shared task, without using any additional external data. Despite this, JESSI performs second on Subtask A with an F1 score of 77.78% among 33 other team submissions. It also performs well on Subtask B with an F1 score of 79.59%.

Organizers observed four main challenges: (a) sparse occurrences of suggestions; (b) figurative expressions; (c) different domains; and (d) complex sentences. While previous attempts  ,  ,  made use of human-engineered features to solve this problem, the goal of the shared task is to leverage the advancements seen on neural networks, by providing a larger dataset to be used on data-intensive models to achieve better performance. This paper describes our system JESSI (Joint Encoders for Stable Suggestion Inference). JESSI is built as a combination of two neural-based encoders using multiple pre-trained word embeddings, including BERT  , a pre-trained deep bidirectional transformer that is recently reported to perform exceptionally well across several tasks. The main intuition behind JESSI comes from our finding that although BERT gives exceptional performance gains when applied to in-domain samples, it becomes unstable when applied to out-of-domain samples, even when using a domain adversarial training  module. This problem is mitigated using two tricks: (1) jointly training BERT with a CNN-based encoder, and (2) using an RNN-based encoder on top of BERT before feeding to the classifier.

in-domain) and achieved an F-Score of 79.59\% on Subtask B (i.e. out-of-domain), even without using any additional external data.

Experiments	In this section, we show our results and experiments. We denote JESSI-A as our model for Subtask A (i.e., BERT INLINEFORM0 CNN+CNN INLINEFORM1 Att), and JESSI-B as our model for Subtask B (i.e., BERT INLINEFORM2 BiSRU+CNN INLINEFORM3 Att+DomAdv). The performance of the models is measured and compared using the F1-score.

ThisIsCompetition at SemEval-2019 Task 9: BERT is unstable for out-of-domain samples	This paper describes our system, Joint Encoders for Stable Suggestion Inference (JESSI), for the SemEval 2019 Task 9: Suggestion Mining from Online Reviews and Forums. JESSI is a combination of two sentence encoders: (a) one using multiple pre-trained word embeddings learned from log-bilinear regression (GloVe) and translation (CoVe) models, and (b) one on top of word encodings from a pre-trained deep bidirectional transformer (BERT). We include a domain adversarial training module when training for out-of-domain samples. Our experiments show that while BERT performs exceptionally well for in-domain samples, several runs of the model show that it is unstable for out-of-domain samples. The problem is mitigated tremendously by (1) combining BERT with a non-BERT encoder, and (2) using an RNN-based classifier on top of BERT. Our final models obtained second place with 77.78\% F-Score on Subtask A (i.e.

Conclusion	We presented JESSI (Joint Encoders for Stable Suggestion Inference), our system for the SemEval 2019 Task 9: Suggestion Mining from Online Reviews and Forums. JESSI builds upon jointly combined encoders, borrowing pre-trained knowledge from a language model BERT and a translation model CoVe. We found that BERT alone performs bad and unstably when tested on out-of-domain samples. We mitigate the problem by appending an RNN-based sentence encoder above BERT, and jointly combining a CNN-based encoder. Results from the shared task show that JESSI performs competitively among participating models, obtaining second place on Subtask A with an F-Score of 77.78%. It also performs well on Subtask B, with an F-Score of 79.59%, even without using any additional external data.

Introduction	Opinion mining  is a huge field that covers many NLP tasks ranging from sentiment analysis  , aspect extraction  , and opinion summarization  , among others. Despite the vast literature on opinion mining, the task on suggestion mining has given little attention. Suggestion mining  is the task of collecting and categorizing suggestions about a certain product. This is important because while opinions indirectly give hints on how to improve a product (e.g. analyzing reviews), suggestions are direct improvement requests (e.g. tips, advice, recommendations) from people who have used the product. To this end,  organized a shared task specifically on suggestion mining called SemEval 2019 Task 9: Suggestion Mining from Online Reviews and Forums. The shared task is composed of two subtasks, Subtask A and B. In Subtask A, systems are tasked to predict whether a sentence of a certain domain (i.e. electronics) entails a suggestion or not given a training data of the same domain. In Subtask B, systems are tasked to do suggestion prediction of a sentence from another domain (i.e. hotels).

Acknowledgement	This research was supported by the MSIT (Ministry of Science ICT), Korea, under (National Program for Excellence in SW) (2015-0-00910) and (Artificial Intelligence Contact Center Solution) (2018-0-00605) supervised by the IITP(Institute for Information & Communications Technology Planning & Evaluation)

Joint Encoders for Stable Suggestion Inference	We present our model JESSI, which stands for Joint Encoders for Stable Suggestion Inference, shown in Figure FIGREF4 . Given a sentence INLINEFORM0 , JESSI returns a binary suggestion label INLINEFORM1 . JESSI consists of four important components: (1) A BERT-based encoder that leverages general knowledge acquired from a large pre-trained language model, (2) A CNN-based encoder that learns task-specific sentence representations, (3) an MLP classifier that predicts the label given the joint encodings, and (4) a domain adversarial training module that prevents the model to distinguish between the two domains.

Question: What datasets were used?
Answer: datasets given on the shared task, without using any additional external data