Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose to extend the standard Transformer architecture by allowing it to attend over factual information, represented as a large memory of dense representations of entity mentions. The resulting architecture is made up of two parts: a mention encoder, which is used to build up the "mention memory"; and the transformer model augmented with attention over the memories. These are pre-trained in two stages for efficiency reasons. The model is evaluated on two claim verification datasets and two QA datasets, showing convincing performance against comparable methods.
main_review: Strengths:
* While it's true that this approach combines aspects of previous work and is therefore not extremely novel, the combination is interesting and well motivated, and the paper is technically sound. This approach reminds me a lot of REALM and RAG, with the crucial difference that the retrieved knowledge isn't concatenated to the inputs but rather directly comes in the form of dense representations. This has a series of advantages, which the authors explain well in section 3 – no length constraint, less computationally expensive.
* The paper is easy to follow. The appendices provide useful information on the pretraining procedure.
* Performance on four standard tasks is convincing and the choice of baselines is reasonable.

Weaknesses:
* While pretraining is described very clearly, with full details of hyperparameters given in an appendix, details on the finetuning procedure are very light. 
* I would have liked to see a part of section 3 dedicated to discussing generative models like RAG.

Typos:
* p10: "we have release"
summary_of_the_review: A strong, well written paper describing a novel approach for incorporating knowledge into a transformer encoder. The main components are not highly novel from a technical point of view, but their combination is.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: __Method:__ This paper proposes a new approach to integrate knowledge sources into Transformer-based models. Concretely, entity mentions found in English Wikipedia (approx. 150M -- these mentions are linked to Wikipedia entities) are encoded into __Mention Memory__, which consists of key vectors and value vectors.  The knowledge representations in a __Mention Memory__ are accessed from a __TOME__ block, which is a stack of transformer blocks with a __memory attention__ layer. In this layer, a mention in the input sequence is converted into a “knowledge-injected” vector representation which is given by a weighted sum of the mention vectors (i.e., value vectors). The entire model (TOME) is pretrained in two stages. First, the Mention Encoder is pretrained, and the Mention Memory is generated. Here, the Mention Encoder is trained with a small-scale TOME architecture (Batch-TOME) using the masked LM loss and the coreference loss. Next, the Mention Memory is fixed, and other parameters in the TOME model are updated. The TOME model is trained with a masked LM task as well as an entity prediction task. This output layer allows us to use the TOME model for downstream tasks such as TriviaQA and Complex WebQA. 

__Evaluation:__ This approach is evaluated on two tasks: claim verification (HoVer, FEVER) and QA (TriviaQA, CWQ) and compared with various baselines (EaE, RAG, REALM etc.). In the claim verification experiments, the TOME models (both 1 and 2 blocks) outperform all baselines on HoVer, which require reasoning using multiple sources. On FEVER, the TOME models outperform baselines except RAG. In the QA experiments, the TOME model with 2 blocks outperforms all baselines on CWQ. On TriviaQA, the TOME models outperform similar models such as EaE and much larger generative LMs (T5) but underperform the retriever-based models. Additionally, the authors perform analysis on retrieved passages, memory sizes, and performance on unseen entities in a QA task.  

main_review: ### Pros
- This work extends the past work on knowledge-augmented LMs and proposes an approach to encode knowledge sources in a more fine-grained way. 
- This architecture maintains explicit memories for mention spans in the knowledge source, so it still retains some interpretability.
- Strong results on HoVer and CWQ. 
- The method and the training details are clearly explained.
- The authors will release the code and model checkpoints.   

### Cons 
- The model performance is inconsistent across different datasets. The TOME model shows strong results on HoVer but underperforms RAG on FEVER. Many FEVER claims are simple definitive sentences (e.g., single hop relations) and can be found in English Wikipedia, so it’s understandable that end-to-end retriever-based models might perform well. I wanted to see more analysis on this point (why is the number lower?). Similarly, it did well on CWQ, but the TriviaQA numbers are behind the retriever-based models. I think instance-level error analysis would be helpful to articulate underlying issues (from data or a model). 
- Although the authors promise to release their code and checkpoints, I’m wondering if people with limited resources (e.g., researchers in academia) can use this model. In Appendix B, it says that the TOME model is finetuned on 32 TPU. From the current version of the paper, it’s unclear if this model can be fine-tuned on a fewer number of TPUs (or GPUs?).

### Questions / Suggestions
- It seems that the pretraining data is annotated using off-the-shelf tools (NER, EL). I’m wondering about the trade-off between the entity coverage (e.g., adding more mention spans on top of the hyperlinks) and the data quality (cascading errors from the NER, EL tools). 
- If  the TOME model is only able to train on and answer 84% of examples, it might be interesting to see the baseline performance using the same train/eval sets. 
- There might be other choice for claim verification such as FM2 (https://arxiv.org/pdf/2104.04725.pdf) , which consists of higher quality claims compared to FEVER (but the data size is smaller).
- The Fact as Experts paper is also related: https://arxiv.org/pdf/2007.00849.pdf

### Minor
- Use “English Wikipedia” in the main text (Bender rule).
- Passage length is denoted as $T$ in Section 2 Notation, but $L$ is used in Appendix A. 

summary_of_the_review: I am leaning towards acceptance. The way TOME infuses textual knowledge into Transformer models is new. This approach actually integrates a passage retriever into a transformer-based LM and potentially covers broad downstream tasks. But, I think the analysis on the experimental results can be improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes TOME, a Transformer that additionally performs cross-attention over contextual mention encodings (in 1 or 2 layers - TOME-1/TOME-2) along with the usual self-attention. It can be viewed as extending EaE (Fevry et al., 2020) from entities to mentions (aka., virtual KB - Dhingra et al., 2020). TOME is pretrained on 150m mentions extracted from Wikipedia by masked language modeling and other auxiliary objectives. It significantly outperforms EaE on fact checking (HoVer, FEVER) and entity-only QA (TriviaQA, CWQ), but lags behind explicit retrieve-and-read models like RAG, REALM, FiD.
main_review: STRENGTHS

- The idea to attend over mentions rather than entities during feedforward is ambitious and refreshingly novel in the context of existing methods that focus on retrieving KB entries.

- The paper develops a staged training scheme to make training feasible, which would be otherwise infeasible even with the compute resource the authors have.

- TOME clearly outperforms EaE on the considered datasets with fewer parameters.


WEAKNESSES

- While TOME is certainly a neat extension of EaE, its main ideas are already pitched in previous works. Specifically, the benefits of virtual KBs (VKBs) over KBs, and performing multiple rounds of KB cross-attention within Transformer, are already highlighted in previous works. TOME's contribution is putting them together in a new model, but the main ideas individually are not a contribution of this work, so its novelty should be taken with a grain of salt.

- While I understand that TOME is small and fast, and larger explicit retrieval-based models have an advantage over TOME, it substantially lags behind the existing state-of-the-art models on both fact checking and QA. I don't think the performance shouldn't be a reason to dismiss the contributions of the work, but it does mean that TOME is not state-of-the-art. 

- I think the paper can do better in justifying the choice of architecture and objectives. It's certainly reasonable, but there's no analysis. We don't know the impact of different span encoding architectures or pretraining objectives. Given how expensive it is to train TOME I suppose understanding development choices is a bit challenging. 

summary_of_the_review: The paper presents a large-scale pretrained Transformer that attends to mention encodings (EaE+VKB). It improves over EaE but lags behind state-of-the-art.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an external knowledge-based pre-trained model that can leverage entity-wise knowledge embeddings. The mention encoder model encodes entity mentions occurred in an external corpus (i.e. Wikipedia) into embeddings as the memory. The TOME model retrieves most possible entity mentions from this memory and performs attention on them to generate aggregated embeddings, which will be integrated into the output representations of the current layer. To make training efficient, a two-stage training strategy is used where mention encoder is trained first and TOME is trained secondly. Evaluations are performed FEVER, HoVer, TriviaQA and ComplexWebQuestions, comparing with several baselines covering BERT, EaE, RAG and REALM, where TOME achieved consistent improvements on most of them.
main_review: Pros:
- a batch-wise mechanism is proposed to support efficient pre-training.
- entity-aware pre-training tasks are proposed to learn embeddings that can better embed entity knowledge.
- evaluations conducted are comprehensive and sold.

Cons:
- seems the proposed model still needs an NER to detect entity mentions from each input passage, which could lead to an unexcepted performance decrease when the model is used for out-domain inputs.
- I suggest to include more entity-insensitive NLP tasks in the paper, to verify that proposed entity-aware mechanism is not biased on entity-related tasks.
summary_of_the_review: The paper is clearly written and easy to follow. The motivation of the proposed method is intuitive. The model design is reasonable. The experiments are solid. In general, this is a good paper. One suggestion is to add more entity-insensitive task, to verify that the proposed model will not decrease its performances on such tasks.
review_title: null

Meta-review: The paper proposes TOME, which extends Transformer by attending to entity mention memory. Experiments are conducted on claim verification and QA.

Reviewers generally found the paper is solid. However, the novelty appears to be limited and is mainly in the combination of existing models.
Decision: Accept (Poster)