Venue: AKBC 2019 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents TRE, a Transformer based architecture for relation extraction, evaluating on two datasets - TACRED, and a commonly used Semeval dataset.

Overall the paper seems to have made reasonable choices and figured out some important details on how to get this to work in practice.  While this is a fairly straightforward idea and the paper doesn't make a huge number of innovations on the methodological side, however (it is mostly just adapting existing methods to the task of relation extraction).

One point that I think is really important to address: the paper really needs to add numbers from the Position-Aware Attention model of Zhang et. al. (e.g. the model used in the original TACRED paper).  It appears that the performance of the proposed model is not significantly better than that model.  I think that is probably fine, since this is a new-ish approach for relation extraction, getting results that are on-par with the state-of-the-art may be sufficient as a first step, but the paper really needs to be more clear about where it stands with respect to the SOTA.
review_title: Review of Improving Relation Extraction by Pre-trained Language Representations

final_score: rating: 7: Good paper, accept
review_content: This paper presents a transformer-based relation extraction model that leverages pre-training on unlabeled text with a language modeling objective.

The proposed approach is essentially an application of the OpenAI GPT to relation extraction. Although this work is rather incremental, the experiments and analysis are thorough, making it a solid contribution.

Given that the authors have already set up the entire TRE framework, it should be rather easy to adapt the same approach to BERT, and potentially raise the state of the art even further.

In terms of writing, I think the authors should reframe the paper as a direct adaptation of OpenAI GPT. In its current form, the paper implies much more novelty than it actually has, especially in the abstract and intro; I think the whole story about latent embeddings replacing manually-engineered features is quite obvious in 2019. I think the adaptation story will make the paper shorter and significantly clearer.

review_title: Incremental but solid contribution

final_score: rating: 5: Marginally below acceptance threshold
review_content: This article describes a novel application of Transformer networks for relation extraction.

CONS:
- Method is heavily supervised. It requires plain text sentences as input, but with clearly marked relation arguments. This information might not always be available, and might be too costly to produce manually. 
Does this mean that special care has to be taken for sentences in the passive and active voice, as the position of the arguments will be interchanged?

- The method assumes the existence of a labelled dataset. However, this may not always be available. 

- There are several other methods, which produce state of the art results on relation extraction, which are minimally-supervised. These methods, in my opinion, alleviate the need for huge volumes of annotated data. The added-value of the proposed method vs. minimally-supervised methods is not clear. 

PROS:
- Extensive evaluation
- Article well-written
- Contributions clearly articulated

review_title: Application of existing method for relation extraction

Meta-review: metareview: Current SOTA on TACRED uses precomputed syntactic and semantic features. This paper proposes to replace this pipeline with a pretrained Transformer with self-attention. This pretrained model is further fine-tuned to do the TACRED relation extraction. The reviewers like the paper and I am happy with the overall discussion. I believe the pretrained model could be useful for other relation extraction tasks, so I am accepting this with a slight reservation.

As noted by Reviewer 3, this pretrained model requires supervised annotations. It would be useful if the paper could add a discussion on the following questions:

1. Why is the supervised data required for pretraining a viable option than syntactic and semantic features? The latter are task-agnostic, so I believe they will be readily available for many languages.

2. How hard is it to create pretraining data vs. supervised relation extraction data?
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)