Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a character-length-constrained sentence summarization model by extending the length constraint from word-level to character-level in inference. The major technical part follows NAUS [Liu et al, ACL’22] which is a BERT-encoder with a CTC decoder. The main contribution is extending the NAUS’s word length constraint from word level to character level, which is formalized as a standard 0-1 knapsack problem with a DP solver. The experiments on Gigaword headline generation and DUC2004 datasets demonstrate the performance over several truncation models in terms of ROUGE scores. 
strengths_and_weaknesses: Pros:
The write-up is good, and the paper is easy to follow

Cons:
1. The character-level constraint for summarization is not well-motivated. In common sense,  character counts are positively correlated with character counts, especially considering most words share similar character counts as the long-tail distribution. 

2. Limited technical novelty. The model heavily follows NAUS’s ([Liu et al, ACL’22]) DP-based word-length-constraint method. The idea of extending from word-level to character-level is trivial. 

3. Lacks comparison with the non-truncation model, i.e., NAUS.
questions: Please see Limitations.
limitations: 1. The bucket grouping trick (lines 142-147, page 4) has some limits: (1) it would not reduce the theoretical complexity, i.e., quadric remaining as the 2D programming; (2) it may suffer from heavy collisions, as the skewed long-trail distributed character counting makes most words share the same bucket. 

2. The idea and the technical part seem quite similar to NAUS (both using DP solution), it would be better to clearly state what the main novelty is compared with word-level constraints (not only with 0-1 knapsack problem) and in what points make methodologically is challenging.

3. Besides the truncation model, the method should be compared with the non-truncation model, i.e., NAUS. 

Minor issue:
In Theorem 1. “See the appendix for the proof.” -> no appendix found.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors proposed a method to control the summary length by characters in non-autoregressive sentence summarization models. The proposed method utilizes dynamic programming-based decoding to follow the length limitation as an imposed constraint. Since considering all possible states in the decoding is intractable, the authors separately consider the length states by relying on buckets. Experimental results showed that the proposed method could control the summary length by keeping ROUGE scores. Furthermore, human evaluation results also showed that the proposed method outperformed the truncation-based length controlling.
strengths_and_weaknesses: Strength:
- Exact length control based on dynamic programming.
- Length can be controlled while maintaining summary performance.

Weakness:
- Lack of exact solution due to the use of buckets for the proposed method to run in practical time.
- Lack of human evaluation for the proposed method and conventional AR methods.
- Controlling summary length by a non-autoregressive model has already been conducted in the deletion-based sentence compression task. [1]

[1] Wang, Liangguo, et al. "Can Syntax Help? Improving an LSTM-based Sentence Compression Model for New Domains." Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017.
questions: - In experiments, you prepared four buckets to reduce computational time in decoding. However, the inference time of a single bucket shown in Figure 2 seems practical. How did you decide on the bucket size in the experiments?
- Comparison with the autoregressive models in the unsupervised setting showed that NACC achieved comparable ROUGE scores with the AR methods. It indicates that NACC is effective in terms of informativeness. On the other hand, the current setting lacks the comparison between the proposed and AR methods in terms of fluency. Is there any information about this point?
limitations: The proposed method NACC is on the basis of non autoregressive decoding, which may be inferior to autoregressive decoding in fluency.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper investigates the problem of length-controlled sentence summarization and proposes NACC, a Non-Autoregressive summarization model with Character-level length Control based on a knapsack-like algorithm. The authors conduct an evaluation of the Gigaword headline generation and DUC2004 datasets in both supervised and unsupervised settings and the experimental results show that the proposed framework can obtain comparable results to the baseline methods.

strengths_and_weaknesses: Strength:
1. Length-controlled text summarization is a useful task in practice especially meets the headline generation tasks.
2. The motivation and assumption are reasonable. A knapsack-like problem setting and non-autoregressive framework are interesting. The proposed method is straightforward and easy to be implemented. 
3. The experimental comparisons and analysis are sufficient and human evaluation is also conducted.

Weakness:
1. This work only conducts experiments on Gigawords and DUC2004, more experiments on more datasets are more concrete.
2. From the results tables we could observe that the results are much worse than some traditional auto-regressive methods. For example, on the test set of Gigawords, previous works can obtain some better results: 36, 17, and 33 for R-1, R-2, and R-L respevtively. Therefore I am wondering about the real quality of the generated sentences, though the length is a restricted condiction, the quality is more crucial.
questions: na
limitations: The authors  addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a dynamic programming algorithm based on the Connectionist Temporal Classification (CTC) model for controlling the  summary length in text summarisation tasks. This is important in many tasks such as headline generation. Experiments on DUC2004 and Gigaword showed the effectiveness of this method. This paper might be more appropriate for NLP conferences. 

strengths_and_weaknesses: Strengths:
* Leverage the CTC method for controlling the summary length in text summarization. 

Weakness:
* For evaluation ROUGE is considered limited, some other semantic focused metrics like BertScore may also be considered.
* DUC-2004 and Gigaword are small news datasets, the summaries are more lying in the first few sentences, it would be better if the authors can try larger datasets from other domains, e.g. arXiv scientific paper title generation.
questions: For unsupervised text summarisation, there is also a length control mechanism, e.g. do clustering for sentences and choose the centroid or do text compression for each cluster, so the number of clusters is proportional to the summary length. See [1]. Apart from the truncate methods, it would be better to add such baseline methods. 


[1] Zhao, J., Liu, M., Gao, L., Jin, Y., Du, L., Zhao, H., Zhang, H. and Haffari, G., 2020, July. Summpip: Unsupervised multi-document summarization with sentence graph compression. In Proceedings of the 43rd international acm sigir conference on research and development in information retrieval (pp. 1949-1952).
limitations: * The domain of the datasets are limited to news.
* Some experiments with large pre-trained models (e.g. T5, BART) and CTC may be considered.
review_title: null

Meta-review: metareview: This paper addresses sentence compression by controlling the output by the number of characters. The proposal relies
on a non-autoregressive generation model interfaced with a dynamic programming algorithm where lengths are divided into buckets
for efficient inference. The proposed algorithm has advantages over previous methods, both computationally and in terms
of results. The authors have addressed the reviewers comments, and honestly discussed shortcomings and limitations. 
It is also worth pointing out that they even have conducted a human evaluation study to assess the completeness of the summaries
produced by their length-controlled model. I would urge the authors to evaluate additional aspects of the output such as content, and
faithfulness to the input (i.e., are there hallucinations). Also the comparison should include an AR system and the gold upper bound. 
confidence: Certain
Decision: Accept