Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a Transformer based that employs relaxed inductive bias compared tgo Tanner graph-based methods while utilizing domain knowledge compared to standard model-free decoders based on fully-connected graph.

strengths_and_weaknesses: This paper's main strength is the well-defined scope of the problem. 

The paper's main weakness is that the proposal’s benefits were not studied thoroughly. For examples, I have a few questions below

questions: - Figure 3: Is the multi-head self-attention block still necessary? Given that we’re looking for the localized bits that meet each row of the parity check (PC) matrix, will single-head self-attention suffice? In other words, if you permute the rows of the PC matrix, you will get the same result, indicating that the problem has some sort of equivariance property that the authors may utilize in the Transformer design

- Figure 2: This is a good illustration; it would be fascinating to see how the attention maps visualized after the training and how they compare to the restrictive Tanner graph; it would also be interesting to see if the mask-based decoders genuinely help attention to more targeted interactions or not 

- Ln 147: Meant Algorithm 1?

limitations: - Ln 154-157: Is the complexity reduction in 4.2 from code-aware self-attention adequate enough to apply the method to longer LDPC codes, such as greater than 512, say 1K or 4K lengths? If it’s not, the authors should have stated that clearly as a limitation or future work of the study
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - The paper presents a novel Transformer based generic decoding procedure for typical linear error correction code families (LDPC, Polar etc..). 
- The method does achieve SOTA performance on mid block length codes (~100-200)
- The key novel contribution of the work was the introduction of positional reliability embeddings and the attention mask, both of which (particularly the mask) uses some clever information theoretic domain understanding. These ideas improve the convergence and hence the performance of the model
- One key selling point of the paper is that the method is general enough to be applied to any linear code without any modification. 
strengths_and_weaknesses: Strengths: 
- The paper presents a novel way of using transformers and self attention to achieve strong results in terms of the BER
- The architecture isn't just using a ML-model and blindly using it for the decoding problem. The training uses some clever ideas from the code construction to improve the convergence.

Weaknesses:
- The explanation was overall good, but in some places a bit lacking. More detailed comments at the end
- The authors briefly mention that the runtime/power numbers might be sub-par as compared with non-ML methods.. still it might be great to see the comparison to know where the field is at. 

Specific suggestions:
- A couple of typos.. line 117 propriety -> property
- $y_b$ -> although clear, it is not defined in the manuscript
- The postprocessing an pre-processing is not clear.. please describe more clearly. For example: it is not clear to me (without reading the cited reference, what you mean by "The post-processing step plugs back the vector elements of y...."
- The use of mask is quite nice, and the authors claim that it reduces the complexity by 84%.. does this reduction in complexity lead to reduction in FLOPS/run-time.. or just a symbolic reduction which leads to lower complexity and better convergence? Please clarify
questions: - The use of mask is quite nice, and the authors claim that it reduces the complexity by 84%.. does this reduction in complexity lead to reduction in FLOPS/run-time.. or just a symbolic reduction which leads to lower complexity and better convergence? Please clarify

- The intuition behind positional reliability encoding is intuitively clear to me, at least for the elements corresponding to |y_i|.. can the authos explain the intuition behind the encoding for the syndrome bits? 

- Comparison with List-SC decoding for polar codes might be interesting. 
limitations: No societal limitations
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers a very important and interesting topic: the decoding of error-correcting codes. 
Over the past few years, there has been a lot of work on applying learning to the decoding problem. Various neural architectures have been considered as well, starting from feedforward networks to recurrent and convolutional neural networks. There also have been a long line of work which introduced learnable parameters to the existing decoder architecture, some of which are considered as a baseline in this paper. 

There has not been a transformer-based channel decoder yet; training a transformer that can successfully decode error-correcting codes is empirically quite challenging. This paper addresses the challenge by utilizing the knowledge of codes in a way; using the appropriate masking and modulating the embedding by syndrome values. This approach is novel and interesting - so I would like to advocate the acceptance of this paper in that regard. 

strengths_and_weaknesses: 
(Strength) 
As mentioned in detail above, the paper successfully demonstrates a transformer-based channel decoder for the first time. 

(Weakness) 
Despite the idea being novel and the results being promising, I have two major concerns. 

The first is whether the comparison between the BP families and the transformer families is fair. Given that the transformer-based decoder is not a purely data-driven decoder, we would expect to see gains compared to the existing decoder. For example, L-layered augmented BP decoders typically have complexity similar to the L-layered BP decoders. Hence, the complexity of the augmented BP decoder is similar to the complexity of the traditional BP decoder but is superior in terms of reliability. [Hyper-BP might be computationally expensive due to the 'hyper network' which selects the weights. However, there are other neural augmented BP algorithms that do not include the weight selection module but are pretty good.]

Second, the presentation and ablation studies can be further improved. For example, section 6.1 and the description of Figure 5 lack details. Also, I wonder if any similar ablation study can be done for the polar or LDPC or BCH codes. Section 6.2 is interesting and insightful. In Section 3, providing the number of operations would be more informative (and potentially mention vanilla augmented-BP algorithms as well). 

For these reasons, my current score is 4, but I am really at the exact borderline. I would like to listen to the authors; I will go through the rebuttal and fix my score afterwards. 

--- 
After the rebuttal: The new experiment results (e.g., Appendix G) are exactly are very informative and satisfactory. The reliability gap between the transformer-based decoders and neural BP decoders does not seem huge. (It is a bit hard to interpret the gain from the negative logarithm of the BER, and the SNR gain might be more indicative.) Nevertheless, I still really like the idea of utilizing the transformer architecture for decoding, and the updated manuscript is comprehensive. I'd updated my score accordingly. 
questions: I'd be curious to see the comparison between the reliability of neural-augmented BP and the transformer  (vanilla, no weight adaptation network) -based decoder when their complexities are similar. 

I'd be also curious to see the performance of these decoders for non-Gaussian channels. Is there a difference in terms of robustness? 
limitations: The authors mention that the complexity could be potentially improved by several techniques.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies neural decoding of (linearly coded) transmissions in the physical communication layer. It proposes a novel architecture that has three main features: it is based on transformers, contains a scaled element-wise embedding of the input, and has an adapted-mask self-attention mechanism. The proposed architecture is compared empirically to existing neural decoders and is shown to outperform them with higher decoding success and less computational complexity.
strengths_and_weaknesses: Strengths:
----------------

Strength 1: The presented method is the first transformer-based decoder and it also gives an efficient architecture.

Strength 2: The proposed method has better decoding performance than other neural network based approaches. 

Strength 3: The proposed method allows to input expert knowledge to the system and has lower complexity than other neural network based methods.


Weaknesses:
---------------------

Minor Weakness 1: Only AWGN channel is considered in this work. This may be in line with other neural decoder works, but it would be important to consider also wireless channels with fast fading and e.g., with interference, which change the noise statistics. There is a possibility that the learned methods might provide even further gains in these scenarios by learning the noise correlation statistics.

Minor Weakness 2: Especially with wireless fast fading channels, other modulations than BPSK should be considered, at least 16QAM.

After the author feedback (rebuttal):
------------------------------------------
I have reviewed the author feedback and they have done good job in clarifying most of the questions the reviewers have posed. I have changed the score accordingly. 



questions: Question 1: in related work, a dichotomy between model-based on model-free was presented and the authors of the present work place theirs’ in the model-free category. Since the present work uses expert knowledge in how the adapted-mask is generated, one could argue that it belongs somewhere in the middle of these categories. Could the authors elaborate on this?

Question 2: Would there be a way to compare neural decoders to other non-learned SOTA decoders? Would it be possible to do more quantitive comparison as well?
limitations: See above.
review_title: null

Meta-review: metareview: This paper is part of a popular line of research aiming to apply neural network concepts to the decoding of error-correcting codes. The main novelty consists in the introduction of an architecture based on transformers. The authors provide convincing and thorough numerical results comparing the BER and the complexity of the proposed approach with various baselines. Such results apply to codes in the short to medium block-length range (from 32 to 128 bits). 

The reviewers have expressed a number of concerns in their initial reports. After the rebuttal stage, most of these concerns have been resolved. The reviewers Nt2o and MQDw have particularly appreciated the additional numerical results provided by the authors (BP baselines, non-Gaussian channels, other modulations and SCL decoder for polar codes). This is also explicitly pointed out in the updated reviews.

In summary, there is clear consensus towards accepting the paper.  After my own reading of the manuscript, I agree with this assessment and I am happy to recommend acceptance. As a final note, I would like to encourage the authors to include in the camera ready the additional experiments and discussions mentioned in the rebuttal. 

confidence: Certain
Decision: Accept