Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This work proposes a method to learn how to aggregate weak supervision sources in the context of sequence labeling. In particular, the model has two main steps: i) it learns a source dependent transformation and ii) it learns a mechanism to combine them. 

In general, the paper is well organized but it is not easy to follow. It was not clear to me how the authors combine the source dependent representation in Section 4.2 with the aggregation phase presented in Section 4.3. In particular, it is confusing to me how the model combines during the training Eq. 3 and Eq. 5. Similarly, it is not clear to me how in Eq. 6 the model can calculate attention coefficients based only on information from the sentence embedding (h^(i)).

This work assumes a BLSTM-CRF architecture as a baseline, but it does not explore alternative approaches, such as transformer.   

In terms of the experiments, authors evaluate the resulting model using two application settings: combining noisy crowd annotations (AMT) and unsupervised cross-domain model adaptation. Results are encouraging, the proposed method is able to outperforms several recent works in terms of F1 metric for the case of AMT and accuracy for the case of cross-domain adaptation. Qualitative results also shows reasonable performance. The supplemental material also include an ablation study.

In summary, the proposed method is interesting and results seem to be encouraging, however, there are parts of the proposed method that are not clear to me. I rate the paper as weak reject. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 
## Updated review

I have read the rebuttals. The new version of the paper is clearer and the new baseline experiments are a good addition. 

## Original review

This paper presents an approach to train a neural networks-based model for sequence modelling using labels from different sources. The proposed approach explicitly models the annotator and uses an attention model to select the best aggregation method. The model is used on two scenarios: learning with crowd annotation and cross-domain adaptation. For the first scenario, noisy annotators are simulated with models trained on subsets of the data, the proposed model is compared with related works and is shown to achieve the highest f-1 score. For the second scenario, different domains in three NLP tasks are used and the model is shown to yield the best performance.

I think this paper should be accepted, for the following reasons:
- The approach is novel as far as I can tell, and the approach of learning to aggregate labels is significant, as it could also be applied to tasks where inter-annotators agreement is a problem.
- The experiments are convincing and show the potential of the proposed approach. 
- The comparison with related works is thorough.

Detailed comments
- I don't understand the notion of "normalized expertise" in Section 5.4, can the authors briefly describe it in the paper ?
- The paper is not easy to read, for instance the first paragraph of Section 5 contains critical information to understand the experiments, maybe it should be moved the Section 4 and developed more, typically in two subsections "Application to crowd annotation" and "Application to cross-domain" for example.
- Typos:
    - Section 2, 3rd paragraph "for traget corpora" -> "target"
    - Same paragraph: "Yang & Eisenstien (2018) represented" -> "represents" to be consistent
    - Section 4.1: "BiLSTM-CRF" -> "BLSTM-CRF"
 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
This paper proposes ConNet, a new label aggregation method for sequence labeling tasks, including crowd-annotation and cross-domain model adaptation. The model consists of a decoupling phase, which learns annotator-specific transforming matrices A, and an aggregation phase with an attention module. Extensive experimental results demonstrate the superiority of the proposed model over baselines. 
The paper is generally well-written and easy to follow, and the results seem convincing, so I think it can inspire other works on this topic. My main concern on the paper is its generalization. Crowdsourcing usually involves lots of annotators, and some of them only give very few labels. In these situations, the proposed model introduces lots of new parameters (A, Q), which may cause difficulty during training. So the positive results in Fig 3(b) are very important to dispel my worry, which requires more explanation. 
Below are some detailed questions:
-      How do you calculate the sentence embeddings h_i during training?
-      Have you introduced some regularization terms on A and Q?
-      What are the most important hyper-parameters for this model, and how to tune?
-      Can this method be extended to new tasks other than sequence labeling?
-      Can you compare your method with the aggregation method used in on-the-job learning paper [1]?
-      92.33 in tab 2 shouldn’t be bold.
 
[1] Werling K , Chaganty A , Liang P , et al. On-the-Job Learning with Bayesian Decision Theory[J]. Computer Science, 2015.

review_title: Official Blind Review #3

Meta-review: While the revised paper was better and improved the reviewers assessment of the work, the paper is just below the threshold for acceptance. The authors are strongly encouraged to continue this work.
Decision: Reject