Introduction
Opinion mining BIBREF0 is a huge field that covers many NLP tasks ranging from sentiment analysis BIBREF1 , aspect extraction BIBREF2 , and opinion summarization BIBREF3 , among others. Despite the vast literature on opinion mining, the task on suggestion mining has given little attention. Suggestion mining BIBREF4 is the task of collecting and categorizing suggestions about a certain product. This is important because while opinions indirectly give hints on how to improve a product (e.g. analyzing reviews), suggestions are direct improvement requests (e.g. tips, advice, recommendations) from people who have used the product.
To this end, BIBREF5 organized a shared task specifically on suggestion mining called SemEval 2019 Task 9: Suggestion Mining from Online Reviews and Forums. The shared task is composed of two subtasks, Subtask A and B. In Subtask A, systems are tasked to predict whether a sentence of a certain domain (i.e. electronics) entails a suggestion or not given a training data of the same domain. In Subtask B, systems are tasked to do suggestion prediction of a sentence from another domain (i.e. hotels). Organizers observed four main challenges: (a) sparse occurrences of suggestions; (b) figurative expressions; (c) different domains; and (d) complex sentences. While previous attempts BIBREF6 , BIBREF4 , BIBREF7 made use of human-engineered features to solve this problem, the goal of the shared task is to leverage the advancements seen on neural networks, by providing a larger dataset to be used on data-intensive models to achieve better performance.
This paper describes our system JESSI (Joint Encoders for Stable Suggestion Inference). JESSI is built as a combination of two neural-based encoders using multiple pre-trained word embeddings, including BERT BIBREF8 , a pre-trained deep bidirectional transformer that is recently reported to perform exceptionally well across several tasks. The main intuition behind JESSI comes from our finding that although BERT gives exceptional performance gains when applied to in-domain samples, it becomes unstable when applied to out-of-domain samples, even when using a domain adversarial training BIBREF9 module. This problem is mitigated using two tricks: (1) jointly training BERT with a CNN-based encoder, and (2) using an RNN-based encoder on top of BERT before feeding to the classifier.
JESSI is trained using only the datasets given on the shared task, without using any additional external data. Despite this, JESSI performs second on Subtask A with an F1 score of 77.78% among 33 other team submissions. It also performs well on Subtask B with an F1 score of 79.59%.
Joint Encoders for Stable Suggestion Inference
We present our model JESSI, which stands for Joint Encoders for Stable Suggestion Inference, shown in Figure FIGREF4 . Given a sentence INLINEFORM0 , JESSI returns a binary suggestion label INLINEFORM1 . JESSI consists of four important components: (1) A BERT-based encoder that leverages general knowledge acquired from a large pre-trained language model, (2) A CNN-based encoder that learns task-specific sentence representations, (3) an MLP classifier that predicts the label given the joint encodings, and (4) a domain adversarial training module that prevents the model to distinguish between the two domains.
Experiments
In this section, we show our results and experiments. We denote JESSI-A as our model for Subtask A (i.e., BERT INLINEFORM0 CNN+CNN INLINEFORM1 Att), and JESSI-B as our model for Subtask B (i.e., BERT INLINEFORM2 BiSRU+CNN INLINEFORM3 Att+DomAdv). The performance of the models is measured and compared using the F1-score.
Conclusion
We presented JESSI (Joint Encoders for Stable Suggestion Inference), our system for the SemEval 2019 Task 9: Suggestion Mining from Online Reviews and Forums. JESSI builds upon jointly combined encoders, borrowing pre-trained knowledge from a language model BERT and a translation model CoVe. We found that BERT alone performs bad and unstably when tested on out-of-domain samples. We mitigate the problem by appending an RNN-based sentence encoder above BERT, and jointly combining a CNN-based encoder. Results from the shared task show that JESSI performs competitively among participating models, obtaining second place on Subtask A with an F-Score of 77.78%. It also performs well on Subtask B, with an F-Score of 79.59%, even without using any additional external data.
Acknowledgement
This research was supported by the MSIT (Ministry of Science ICT), Korea, under (National Program for Excellence in SW) (2015-0-00910) and (Artificial Intelligence Contact Center Solution) (2018-0-00605) supervised by the IITP(Institute for Information & Communications Technology Planning & Evaluation)

Question: What datasets were used?
Answer: datasets given on the shared task, without using any additional external data

Question: How did they do compared to other teams?
Answer: second on Subtask A with an F1 score of 77.78% among 33 other team submissions, performs well on Subtask B with an F1 score of 79.59%