Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presentes a small extension to the Neural Transformer model of Vaswani et al 2017:
the multi-head attention computation (eq. 2,3):
head_i = Attention_i(Q,K,W)
MultiHead = Concat_i(head_i) * W = \sum_i head_i * W_i

is replaced with the so-called BranchedAttention (eq. 5,6,7,4):
head_i = Attention_i(Q,K,W)       // same as in the base model
BranchedAttention = \sum_i \alpha_i max(0, head_i * W_i * kappa_i * W^1 + b^1) W^2 + b^2

The main difference is that the results of application of each attention head is post-processed with a 2-layer ReLU network before being summed into the aggregated attention vector.

My main problem with the paper is understanding what really is implemented: the paper states that with alpha_i=1 and kappa_i=1 the two attention mechanism are equivalent. The equations, however, tell a different story: the original MultiHead attention quickly aggregates all attention heads, while the proposed BranchedAttention adds another processing step, effectively adding depth to the model.

Since the BranchedAttention is the key novelty of the paper, I am confused by this contradiction and treat it as a fatal flaw of this paper (I am willing to revise my score if the authors explain the equations) - the proposed attention either adds a small amount of parameters (the alphas and kappas) that can be absorbed by the other weights of the network, and the added alphas and kappas are easier/faster to optimize, as the authors state in the text, or the BranchedAttention works as shown in the equations, and effectively adds depth to the network by processing each attention's result with a small MLP before combining multiple attention heads. This has to be clarified before the paper is published.

The experiment show that the proposed change speeds convergence and improves the results by about 1 BLEU point. However, this requires a different learning rate schedule for the introduced parameters and some non-standard tricks, such as freezing the alphas and kappas during the end of the training.

I also have a questions about the presented results:
1) The numbers for the original transformer match the ones in Vaswani et al 2017, am I correct to assume that the authors did not rerun the tensor2tensor code and simply copied them from the paper?
2) Is all of the experimental setup the same as in Vaswani et al 2017? Are the results obtained using their tensor2tensor implementation, or are some hyperparameters different?

Detailed review:
Quality:
The equations and text in the paper contradict each other.

Clarity:
The language is clear, but the main contribution could be better explained.

Originality:
The proposed change is a small extension to the Neural Transformer model.

Significance:
Rather small, the proposed addition adds little modeling power to the network and its advantage may vanish with more data/different learning rate schedule.

Pros and cons:
+ the proposed approach is a simple way to improve the performance of multihead attentional models.
- it is not clear from the paper how the proposed extension works: does it regularize the model or dies it increase its capacity?
review_title: Small tweaks ot the multi-head attention from "Attention is all you need"

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper describes an extension to the recently introduced Transformer networks which shows better convergence properties and also improves results on standard machine translation benchmarks. 

This is a great paper -- it introduces a relatively simple extension of Transformer networks which only adds very few parameters and speeds up convergence and achieves better results. It would have been good to also add a motivation for doing this (for example, this idea can be interpreted as having a variable number of attention heads which can be blended in and out with a single learned parameter, hence making it easier to use the parameters where they are needed). Also, it would be interesting to see how important the concatenation weight and the addition weight are relative to each other -- do you possibly get the same results even without the concatenation weight? 

A suggested improvement: Please check the references in the introduction and see if you can find earlier ones -- for example, language modeling with RNNs has been done for a very long time, not just since 2017 which are the ones you list; similar for speech recognition etc. (which probably has been done since 1993!).

Addition to the original review: Your added additional results table clarifies a lot, thank you. As for general references for RNNs, I am not sure Hochreiter & Schmidhuber 1997 is a good reference as this only points to a particular type of RNN that is used today a lot. For speech recognition there are many better citations as well, check the conference proceedings from ICASSP for papers from Microsoft, Google, IBM, which are the leaders in speech recognition technology. However, I know citations can be difficult to get right for everybody, just try to do your best. 
review_title: review of "Weighted transformer network for machine translation"

final_score: rating: 6: Marginally above acceptance threshold
review_content: TL;DR of paper: they modify the Transformer architecture of Vaswani et al. (2017) to used branched attention with learned weights instead of concatenated attention, and achieve improved results on machine translation.

Using branches instead of a single path has become a hot architecture choice recently, and this paper applies the branching concept to multi-head attention. Weirdly, they propose using two different sets of weights for each branch: (a) kappa, which premultiplies the head before fully connected layers, and (b) alpha, which are the weights of the sum of the heads after the fully connected layers. Both weights have simplex constraints. A couple of questions about this:

* What is the performance of only using kappa? Only alpha? Neither? What happens if I train only of them?
* What happens if you remove the simplex constraints (i.e., don't have to sum to one, or can be negative)?
* Why learn a global set of weights for the branch combiners? What happens if the weights are predicted for each input example? This is the MoE experiment, but where k = M (i.e., no discrete choices made).
* Are the FFN layer parameters shared across the different heads?
* At the top of page 4, it is said "all bounds are respected during each training step by projection". What does this mean? Is projected gradient descent used, or is a softmax used? If the former, why not use a softmax?
* In Figure 3, it looks like the kappa and alpha values are still changing significantly before they are frozen. What happens if you let them train longer? On the same note, the claim is that Transformer takes longer to train. What is the performance of Transformer if using the same number of steps as the weighted Transformer?
* What are the Transformer variants A, B, and C?

While the results are an improvement over the baseline Transformer, my main concern with this paper is that the improved results are because of extensive hyperparameter tuning. Design choices like having a separate learning rate schedule for the alpha and kappa parameters, and needing to freeze them at the end of training stoke this concern. I'm happy to change my score if the authors can provide empirical evidence for each design choice
review_title: Improved results but unsure about details

Meta-review: The paper proposes a modification to the Transformer network, which mostly consists in changing how the attention heads are combined. The contribution is incremental, and its novelty is limited. The results demonstrate an improvement over the baseline at the cost of a more complicated training procedure with more hyper-parameters, and it is possible that with similar tuning the baseline performance could be improved in a similar way.
Decision: Reject