Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The contribution of this paper is two-fold. 
First, the authors create (and make available) IndicLink, a dataset of multilingual fact linking in 5 indian languages + English. The dataset is translated from English, from a portion of WebRED by annotators. The authors attempt to coin "multilingual fact linking" and propose a formalization of the problem. 
Second, the authors evaluate current bi-encoder and cross-encoder retrieval and reranking architectures for fact linking using an multilingual transformer model to encode facts (subject, predicate and object names) and additionally 
propose a system following the mGenre paradigm where in the standard bi/dual encoder + reranking cross-encoder, the reranking cross-encoder is replaced by a generative (constrained) decoder.
The authors propose an evaluation of their system on their dataset, against other current systems on all 5 languages and english and also perform various ablations of the model in order to determine the effect of the different components on the final results. 


Strong points: 
- Interesting system that shows promising results. 
- A new dataset that can be of interest to the community. 
- Relatively well written 

Shortcomings: 
 - Potential bias in the dataset construction due to the translation from English as pointed out by another reviewer. 
 - Poor justification for the choice of the specific Indian languages used in the dataset. Some of the languages belong to different sub-families, which is good, but the selection criteria aren't clear. For example if the criterion was the number of speakers, then surely, a language like Bangla would have to be included. Other European languages could be included too as they are available in the source dataset. 
 - The formalization for the task is difficult to parse and perhaps too arcane (doesn't follow usual conventions). As such the textual description is much easier to understand. 
 - No error analysis (an appendix shouldn't be used to finish the paper, the error analysis has as much a place in the paper itself as the ablation studies. 

Detailed comments and questions: 
- Please clearly state the definition of "fact" for the task of fact interlinking. "Fact" has many conflicting and very different definitions in related subfields of computer science, even though the venue is clearly oriented towards fact as a synonym for assertion in a knowledge base, it's still important to say so in the introduction. 
- Please improve the formalization, that's a very strange way of modeling binary relations (predicates are a binary relation between subject and object). The set of facts is clearly contained in the cartesian product of R \subseteq ExE, why isn't it defined as such? Equation 1 doesn't need to be so complex, if a fact is synonymous with triple, you can just state {F_i \in R| T_m \entails F_i}. 
Since there is no formal way of defining that a "fact" is contained or expressed in a textual fragment or an utterance, you could very well use the entailment symbol (just a suggestion, this is far from being the only way). It's also likely that a diagram would go a long way towards explaining the task in a precise yet clear manner. 
- What prevents the application of the methods to standard monolingual datasets for fact linking? An external benchmark beside the authors' own dataset would strengthen the approach. 
- Tables 3 onwards aren't placed optimally and are far from where they're commented.
- Equations 3 and 4 would again benefit from being represented schematically. 


review_title: Interesting task and systems for multilingual fact linking, some shortcoming 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies the problem of fact linking -- linking facts from a knowledge graph (KG) to a given sentence where the fact is mentioned. More specifically, recognizing the sparsity and skewness of KG entity and relation labels in terms of language coverage, this paper proposes to study the multilingual setting, where the input sentence is in an underrepresented language and one needs to link KG facts, often only expressed in more high-resource languages, to the input sentence. A new test set for 6 Indian languages is created by manually translating sentences from WebRED with professional translators. The paper also proposes a model for this task based on the Dual Encoder - Cross Encoder architecture, but unlike conventional Cross Encoder, it follows mGENRE and proposes a constrained autoregressive decoder to generate valid facts. It is shown that the proposed model outperforms re-ranking based Cross Encoders by a large margin on the new test set.

Strengths
- A new problem and dataset with an emphasis on underrepresented languages in NLP study
- Strong baseline model with autoregressive decoder for fact linking
- Empirical results seem promising

Weaknesses
- An obvious baseline is missing, where one uses some off-the-shelf machine translation tool to translate the English labels in the KG to the target language, which effectively turns the multilingual problem into monolingual. Discussion and demonstration of the inefficacy (if there's any) of that is needed.
- It's surprising that constrained generation leads to worse performance than free-form generation, and there lacks a convincing analysis and explanation for that. In principle, the constrained version should be strictly better than the free-form version, unless there are unnecessary constraints or something happens with the optimization. Since it's a major contribution of this paper, it merits more thorough discussion.

Minor
- Table 2, missing "K" in the Assamese row?
- Related work, "Fact Extraction", "whereas fact extraction system do not":  "system" -> "systems"
- I find it a bit odd to capitalize phrases like "Entity Linking", "Coreference Resolution", and "Multilingual Fact Linking" in the main text. Probably better to be lower-cased.
review_title: Good multilingual fact linking corpus with interesting model, but missing an obvious baseline

final_score: rating: 6: Marginally above acceptance threshold
review_content: === Summary ===

This paper considers the task of multilingual fact linking, where the goal is to link abstract representations of facts to their language-specific representations in multiple languages. The challenge is that although "worldly facts" themselves are language-agnostic, annotations of how they appear in text form (e.g., described in a sentence or named) are often restricted to only a few languages. The proposed task of multilingual fact linking seeks to link KG facts with their mentions in multilingual text, even when the label of the fact doesn't match the language of the text (e.g., matching <NYC, country, USA> to a Hindi text that expresses this).

=== Strengths ===

- The paper is mostly well-written. It's a bit hard to follow exactly what the task setting is (which components are in language A and which are in language B). It would be much clearer if a simple, formal definition was given in the introduction. It also would be good to perhaps highlight the MFL row in table 1 as "this paper".

- Overall, the task is well motivated and the introduced dataset, despite some limitations (see below), may be helpful for further multilingual development.

- The paper proposes a retrieval + generation model (ReFCOG) that performs well as compared to retrieval + reranking models, which provides a good starting point for the task.

=== Weaknesses ===

- One of the biggest downsides is that the evaluation dataset (IndicLink) is (a) limited in typological scope to a select family of languages, and (b) translated from English. The question of "why not translate" is addressed nicely in Section 3.2 of Clark et. al. 2020 (https://arxiv.org/pdf/2003.05002.pdf). Thus, there is a concern that the artifacts introduced by "translationese" may considerably hurt generalization performance to purely native text.

- The proposed ReFCOG model is composed of several existing components previously proposed in multi-lingual work, and seems more like a baseline than a purely "novel" approach. That's ok, as it's good to have documented performance of simple to slightly complex models when introducing a new dataset (but the messaging might be better off slightly altered to reflect that).

=== Questions ===

- Why does constrained generation *reduce* performance? It seems that by restricting generation to only valid facts, it wouldn't be filtering any beam candidates that reflect possible correct answers. So, for any unconstrained generation that would be correct, why would it not still be the top candidate from the constrained search? It would seem that the set of correct answers should be a perfect subset.

=== Missing references ===

- The related work could be more complete in included related work from multilingual fact retrieval, for example, Jiang et. al. (https://aclanthology.org/2020.emnlp-main.479.pdf).

review_title: Solid but limited multilingual fact linking resource

Meta-review: This paper proposed a valuable resource in underrepresented languages, hence an exciting contribution for AKBC community. The new dataset presents a multi-lingual fact linking task for 5 indian languages in addition to English. The paper proposes a retrieval + generation model (ReFCOG) that performs well as compared to retrieval + reranking models, which provides a good starting point for the task. Reviewers have presented excellent list of suggestions to improve the paper further, especially, adding simple MT baseline (or a discussion if not possible in this setup), including systematic error analysis in the main paper, clarifying novelty of the proposed approach. I urge authors to address these in the updated draft of this paper.


Decision: Accept