Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary
This paper proposes a prototype matching network (PMN) to model transcription factor (TF) binding motifs and TF-TF interactions for large scale transcription factor binding site prediction task. They utilize the idea of having a support set of prototypes (motif-like features) and an LSTM from the few shot learning framework to develop this prototype matching network. The input is genomic sequences from 14% of the human genome, each sequence in the dataset is bound by at least one TF. First a Convolutional Neural Network with three convolutional layers is trained to predict single/multiple TF binding. The output of the last hidden layer before sigmoid transformation is used as the LSTM input. A weighted sum of similarity score (sigmoid of cosine similarity, similar to attention mechanism of LSTMs) along with prototype vectors are used to update the read vector. The final output is a sigmoid of the final hidden state concatenated with the read vector. The loss function used is difference of a cross-entropy loss function and a lambda weighted prototype loss function. The latter is the mean square error between the output label and the similarity score.  The authors compare the PMN with different lambda values with CNN with single/multi-label and see marginal improvement in auROC, auPR and Recall at 50% FDR with the PWM. To test that PWN finds biologically relevant TF interactions, the authors perform hierarchical clustering on the prototypes of 86 TFs and compare the clusters found to the known co-regulators from the TRRUST database and find 6 significant clusters. 


Pros:
1. The authors utilize the idea of prototypes and few shot learning to the task of TF-binding and cooperation. 

2. Attention LSTMs are used to model label interactions. 

Just like CNN can be related to discriminative training of PSSM or PWM, the above points demonstrate nicely how ideas/concepts from the recent developments in DL can be adopted/relate (and possibly improve on) to  similar generative modeling approaches used in the past  for learning cooperative TF binding.

Cons:

1. Authors do not compare their model’s performance to the previously published TF binding prediction algorithms (DeepBind, DeepSEA). 
2. The authors miss important context and make some inaccurate statements: TF do not just “control if a gene is expressed or not” (p.1). It’s not true that previous DL works did not consider co-binding. Works such as DeepSea combined many filters which can capture cooperative binding to define which sequence is “regulated”. It is true this or DeepBind did not construct a structure a structure over those as learned by an LSTM. The authors do point out a model that does add LSTM (Quang and Xie) but then do not compare to it and make a vague claim about it modeling interactions between features but not labels (p. 6 top). Comparing to it and directly to DeepSee/Bind seems crucial to claim improvements on previous works. Furthermore, the authors acknowledge the existence of vast literature on this specific problem but completely discard it as “loose connection to our TFBS formulation”. In reality though, many works in that area are highly relevant and should be discussed in the context of what the authors are trying to achieve. For example, numerous works by Prof. Saurabh Sinha have focused specifically on joint TF modeling (e.g. Kazemian NAR 2011, He Plos One 2009, Ivan Gen Bio 2008, MORPH Plos Comp Bio 2007). In general, trying to lay claims about significant contributions to a problem, as stated here by the authors, while completely disregarding previous work simply because it’s not in a DL framework (which the authors are clearly more familiar with) can easily alienate reviewers and readers alike.  

3. The learning setup seems problematic:
3a. The model may overfit for the genomic sequences that contain TF binding sites as it has never seen genomic sequences without TF binding sites (the genomic sequences that don’t have CHIP peaks are discarded from the dataset). Performance for genome wide scans should definitely include those to assess accuracy.
3b. The train/validation/test are defined by chromosome. There does not seem to be any screening for sequence similarity (e.g. repetitive sequences, paralogs). This may inflate performance, especially for more complicated models which may be able to “memorize” sequences better. 
4. The paper claims to have 4 major contributions. The details of second claim that the prototype matching loss learns motif like features is not explained anywhere in the paper. If we look at the actual loss function equation (12), it penalizes the difference between the label and the similarity score but the prototypes are not updated. The fourth claim about the biological relevance of the network is not sufficiently explored. The authors show that it learns co-bindings already known in the literature which is a good sanity check but does not offer any new biological insight.  The actual motifs or the structure of their relations is not shown or explored.
5. PWN offers only marginal improvement over the CNN networks 


review_title: This paper presents a method to learn motifs and cooperative binding structure from data for multiple TF binding (ChIP-Seq).

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes an approach for transcription factor binding site prediction using a multi-label classification formulation. It is a very interesting problem and application and the approach is interesting. 

Novelty:
The method is quite similar to matching networks (Vinyals, 2016) with a few changes in the matching approach. As such, in order to establish its broader applicability there should be additional evaluation on other benchmark datasets. The MNIST performance comparison is inadequate and there are other papers that do better on it. 
They should clearly list what the contributions are w.r.t to the work by Vinyals et al 2016.
They should also cite works that learn embeddings in a multi-label setting such as StarSpace.

Impact:
In its current form the paper seems to be most relevant to the computational biology / TFBS community. However, there is no comparison to the exact networks used in the prior works DeepBind/DeepSea/DanQ/Basset/DeepLift or bidirectional LSTMs. Further there is no comparison to existing one-shot learning techniques either. This greatly limits the impact of the work.

For biological impact, a comparison to any of the motif learning approaches that are popular in the biology/comp-bio community will help (for instance, HOMER, FIMO).

Cons:
The authors claim they can learn TF-TF interactions and it is one of the main biological contributions, but there is no evidence of why (beyond very preliminary evaluation using the Trrust database). Their examples are 200-bp long which does not mean that all TFs binding in that window are involved in cooperative binding. The prototype loss is too simplistic to capture co-binding tendencies and the combinationLSTM is not well motivated. One interesting source of information they could tap into for TF-TF interactions is CAP-SELEX (Jolma et al, Nature 2015).

One of the main drawbacks is the lack of interpretability of their model where approaches like DanQ/DeepLift etc benefit. The PWM-like filters in some of the prior works help understand what type of sequence properties contribute to binding events. Can their model lead to an understanding of this sort?

Evaluation:
The empirical evaluation itself is not very strong as there are only modest improvements over simple baselines. Further there are no error-bars etc to indicate the variance in their performance numbers.
It will be useful to have a TF-level performance split-up to get an idea of which TFs benefit most.

Clarity:
The paper can benefit from more clarity in the technical aspects. It is hard to follow for anyone not already familiar with matching networks. The objective function, parameters need to be clearly introduced in one place. For instance, what is y_i in their multi-label framework?
Various choices are not well motivated; for instance cosine similarity, the value of hyperparameter epsilon.
The prototype vectors are not motif-like at all -- can the authors motivate this aspect better?

Update: I have updated my rating based on the author rebuttal
review_title: This work proposes an approach for transcription factor binding site prediction using a prototype matching network. While the approach is interesting, the work needs further improvements to be appealing to the ICLR audience.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors of this manuscript proposed a model called PMN based on previous works for the classification of transcription factor binding. Overall, this manuscript is not well written. Clarification is needed in the method and data sections. The model itself is an incremental work, but the application is novel. My specific concerns are given below.

1. It is unclear how the prototype of a TF is learned. Detailed explanation is necessary. 

2. Why did the authors only allow a TF to have only one prototype? A TF can have multiple distinct motifs.

3. Why peaks with p-value>=1 were defined as positive? Were negative classes considered in the computational experiments?

4. What's the relationship between the LSTM component in the proposed method and sparse coding?

5. The manuscript contains lots of low-end issues, such as:
5.1. Inconsistency in the format when referring to equations (eq. equation, Equation, attention LSTM, attentionLSTM, t and T etc);
5.2. Some "0"s are missing in Table 3;
5.3. L2 should be L_2 norm; 
5.4. euclidean -> Euclidean; pvalue-> p-value;
5.5. Some author name and year citations in the manuscript should be put in brackets;
5.6. The ENCODE paper should be cited properly, ("Consortium et al., 2012" is weird!) ;
5.7. The references should be carefully reformatted, for example, some words in the references should be in uppercase (e.g. DNA, JASPER, CNN etc.), some items are duplicated, ...

Comments for the revised manuscript: I decide to keep my decision as it is. My major and minor concerns are not fully well addressed in the revised paper. 
review_title: This work may have some value in applications, but contains lots of issues.

Meta-review: This paper proposes an approach for predicting transcription factor (TF) binding sites and TF-TF interaction.  The approach is interesting and may ultimately be valuable for the intended application.   But in its current state, the paper has insufficient technical novelty (e.g. relative to matching networks of Vinyals 2016), insufficient comparisons with prior work, and unclear benefit of the approach.  The reviewers also had some concerns about clarity.  
Decision: Reject