Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

The paper investigates intersection of federated learning and distant supervision of knowledge graphs from texts. The main innovation is a simple yet empirically effective denoising rule that selects only the sentences deemed to be the most reliable for learning in each round of training.


Strong points
* The method is very simple to implement therefore it might be widely adopted as a baseline.
* The denoising step is specific only for federated learning and distant supervision. It can be applied to many other domains as well (and not just for relation extraction).

Weak points
* The paper would benefit from further proofreading. 
* Some uncertainty about experimental results, namely repeatability of the runs and methodology of hyperparam selection (more in questions section).


Recommendation

* Pre rebuttal: I like the simplistic approach however in current form I am leaning towards rejecting the paper due to my uncertainty in the empirical evaluation. However I am willing to change my evaluation if these questions are resolved.

* Post rebuttal update:  Several of my concerns about clarity of the experimental section were addressed. Therefore I increased my score and now I am inclining towards accepting the paper.

Questions

* Methodology of obtaining the empirical results isn't clear from the text. Section 4.1 mentions a single held out test set. However then it isn't clear how hyperparam sweep discussed in Sec 4.2 was done. If there is only a single test set were the hyperparams selected on the same test set that is later used to report results in Fig 2 and 3? I would consider that a flaw. Or is there a second validation set (that isn't mentioned in the text explicitly) used to tune hyperparams?

* Curves in Fig 2 and 3 seem to be from just one run for each setting. Doing more runs with different random initializations and reporting mean (or top k results) + confidence intervals would increase confidence in these results.


Possible improvements

* "Lazy MIL almost does not leak the corpus information in each platform" --- can you make this statement more precise? 

* Baseline attention models are always forced to spread their attention among sentences in the local bag even if they are all irrelevant. What if the attention module is allowed to output "all sentences are irrelevant" option (this is sometimes called 'sentinel', https://arxiv.org/abs/1612.01887). This might work as learnable denoising that deals with the issue that some local bags don't include any relevant examples. 

* Add references for all datasets in section 4.1.
* Alg 1: are lines 6 and 7 needed? they seem to be contained by line 8.
* Alg 1, line 11: this piece of Python in pseudocode might not be readable by everyone, I would explain it in plain words.
* Alg 2: Hyperparams --- "E is the number" (what number?)

Some language issues:

* leads to catastrophic repro- ducibility -> is not reproducible
* is a large biomedical with -> biomedical dataset with?
* KB is public available -> publicly available
* the token representation is represented as -> the token is represented as?
* The overall of Lazy MIL is illustrated in Algorithm 1. --- overall design of? overview of?
* "to extract the high-level sentence representation from three segments of CNN outputs, and the boundaries of segments are determined by the positions of the two entities." -> I would first define what segments are and only after that how they are used in NN, here the order is reversed and I find it more difficult to follow.
* Konečný (different accentation in the last character)
review_title: Simple and probably efficient idea, some questions about evaluation 

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper addresses relation extraction problem for distributed platforms for privacy concerns. The authors propose to leverage federated learning with denoising techinques for better performance. The relation extractor is set to the conventional piece-wise CNN and the main contributions focus on dealing noises in the distributed settings.

Pros:
* A new method for relation extraction in the federated learning to address privacy concerns.
* Best performance compared to baseline models.

Cons:
* Lack of novelty. This method follows the federated learning paradigm with few improvements. The main contribution is limited to the proposed lazy multiple instance learning, which is not specific to relation extraction problem. If this is not specified for relation extraction, more tasks are expected to demonstrate it is a general algorithm.
* Experiments are insufficient. First, it is doubtful whether the i.i.d. setting is a practical assumption for relation extraction, because in the real world the quality of corpus in different platforms may vary drastically. Second, given the i.i.d. setting, the authors can repeat their experiments to reduce randomness, because the proposed denoising strategy may only perform well in situations that useful sentences are allocated to only few platforms. The authors may provide more analysis on the distribution of contributions from platforms.
* Writing could be improved. In section 3.3, what does the ``value v^i'' refers to? Also, if v^i > v^j, then the id^i-th sentence in the platform i is selected...''. Does it mean that at every round only one sentence from a platform is selected? What if for a certain relation, there are many promising candidate sentences from one platform?
review_title: A good problem but improvements are expected

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This work focuses on investigating distant supervision for relation extraction task within federated learning paradigm. The proposed Lazy Multi-instance Learning approach identifies reliable sentences for the same entity pairs across platforms to denoise distantly supervised data and perform relation extraction.  

The proposed approach has been applied on two relation extraction datasets and has reported gains. 


Strengths:
- The core of distant supervision for relation extraction is to denoise sentences for building a supervised classifier. The problem is well investigated in NLP community. However, this work focuses on modeling the distant supervision problem in real-world environment addressing data decentralization, ownership and privacy. I buy the idea of applying the existing methods or extend the NLP applications in federated setup. 
- the paper is well written, clearly motivation and addressed real-world problem 
- interesting work: blend of distant supervision and federated learning for NLP task

Weaknesses:
- the paper lacks novelty in terms of methodology 
- Distant supervision for relation extraction (PCNN and MIL) and the federated learning methodologies used in this work have been inspired from existing works. This works combines the two.
- the experimental setup needs more clarity 


Questions:
1. Assumption made: Sentences with same entity pairs must scatter across platforms. How would the Lazy MIL system bootstrap without such an assumption (no same entity pairs)?
2. Assumption made: The number of output classes are same therefore the \Theta parameter (including output-layer weights connecting softmax) is used across platforms. In real world, it is not the case. Each platform (or customer) may have different relation types. How would you aggregate/distribute the output layer parameters?
3. How to obtain a minimum viable global \Theta? How to init this without having the actual training data in Master server?  
4. In equation 2, the v^i is not controlled by any threshold. Will it denoise if its value is too low (say <0.50) and it is still the highest among all the platforms? 
5. In section 4.1 in data partitioning, how do you ensure the number of relation types remains same across all platforms?

Additional comments: 

- Include an ablation study analyzing scores due to different values of K.

Experimental setup  unclear:
- how do to init \Theta in global model?
- What is the held-out data set for local and global model training?

Results:
- the experimental results show noticeable gains. It is surprising as the NYT dataset is well investigated in distant supervision settings. 
- As the training data is split across platforms, the overall system is decoupled into several local models. In essence, the overall performance in federated settings should deteriorate or remain competitive to the baseline models due to 'no joint' training on the overall corpora as well as due to (somewhat) lossy aggregation. Please provide a detailed reasoning about the noticeable gains achieved.   

Reproducibility
- no code available
review_title: Interesting work, Lacks novelty and unclear experimental setup

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper explores relation extraction with distant supervision in the federated setting and focuses on handling the label noise problem from automatic distant supervision by Multiple Instance Learning-based methods and proposes Lazy MIL. 

For a specific triple (h, r, t) in KB, values (probability with relation r) for sentences containing h and t will be calculated locally in each platform, and such best value and index in each platform will be uploaded to the master server. A master server decides the most reliable sentence and broadcast such information to all activated platforms in this round.

Strengths: 

+ This paper considers relation extraction in the federated setting, which is a new direction in this area;
+ The mentioned label noise problem is a real problem in distant supervision, and it’s intuitive that this problem exists in the federated setting;
+ Experiments show the effectiveness of the proposed method;

Weaknesses:

- The contribution of the paper is weak (in the context of the expectations of ICLR) 
- For baselines in experiments, it’s not clear how to use baselines in the federated setting. For example, for “keep the other modules unchanged and only replace the denoising module” in the last sentence of Sec. 4.3, it’s confusing for me that how to replace this part. Is the part for broadcasting denoising information? 
- Except for the experiment results, it’s not clear that if the proposed method is specific to the federated setting, and what are the federated-setting-specific designed compared with other baselines?

Questions:

In Sec. 4.4, “We believe the reason is that our denoising method can hinder false-positive instances from poisoning local models.” Does this mean that other baselines can’t hinder false-positive instances?
review_title: An interesting paper but only contributes to a limited field.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper introduced a new federated scenario for distantly supervised relation extraction where extractions come from multiple private resources. And therefore a global model cannot directly access all the data simultaneously. 

To protect data privacy, this paper assigned each resource a local model for separate training. In order to apply the at-least-one sentence bag denoising technique, at the beginning of each round, local models will calculate scores of sentences in the local bag, compare scores with other resources and then generate the training data. After each round of parallel training, local models will be synchronized with the global model via a weighted average algorithm. By only sharing scores rather than the full text, data privacy is protected. Experiments over two datasets show promising results.

####################

Reasons for score:

I vote for marginally negative. Overall, this paper brought up an interesting challenge for this NLP sub-task. However, it is not clear to me the necessity of compromising accuracy while doing parallel computing, and model innovation is slightly weak. 

####################

There are two comments from a performance-driven perspective:

1. Is there a specific reason for this task to do parallel computing? From my understanding (if correct), if parallel training is not required, we can avoid major information loss from delayed denoising and averaging step for model integration. In another word, if we only train one local platform at a time (and denoising frequently), and then synchronize model parameters to the next platform for further training, the performance should be close (or equal if using ONE model) to the performance upper bound when K=1.  If parallel computing is the key, then it is necessary to analyze the speed and performance trade-off.

2. It is necessary to use pre-trained language models like MTB (matching the blank) paper as the encoder because improvement over the current baseline could possibly be mitigated by a stronger encoder.


review_title: This paper tackles a new scenario for distant supervision relation extraction where extractions come from multiple private resources thus global training is not available.

Meta-review: The paper studies Distantly Supervised Relation Extraction(DSRE) in distributed settings. Though DSRE has been studied in Centralized setting it has not been studied in distributed platform. This  paper leverages the federated learning setup for this problem and proposes to use Lazy MIL for this purpose.  The paper  identifies the main challenge as label noise but does not attempt to characterise the severity of the problem vis-a-vis the centralised setup.  Though intuitive but a formal approach would have helped in understanding the importance of the derived results better. 

Decision: Reject