Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces a very large dataset for text summarisation purposes derived from GitHub. The dataset contains the commit title and the description text of commits and the idea is that the title represents a summary of the description text. The main contributions of this paper are that the dataset if very large compared to other text summarisation datasets and also that it is specific to software engineering domain. For these reason it is a novel dataset and a useful addition to the field. However, the fact that the dataset has been derived from commit messages limits its usefulness outside of this domain. 
strengths: The main strengths of this paper are that it introduces a novel and very large dataset for the text summarisation task in the software engineering domain. The addition of ethical considerations and potential limitations of the study are a good addition to the paper.
weaknesses: The main limitation of this paper is that the dataset contains data form a niche domain and it is questionable what is the usefulness of the dataset for building text summarisation models outside of this domain.
correctness: For the best of my knowledge the dataset has been constructed in a correct and sound way.
clarity: The paper is well written and has a clear structure. 
relation to prior work: The related work section contains sufficient and relevant references.
additional feedback: All in all I liked the paper very much and it introduces a nice new dataset for the text summarisation task. I worry for the usefulness of the dataset as it contains data from a very narrow domain.
review_title: Large-scale dataset for text summarisation derived from commit messages

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper releases an automatically constructed dataset for commit message summarization. The dataset pairs a long form commit message on github with a summary of the same message with the goal to generate the summary given the message.

strengths: The dataset presented is novel and seems freshly collected -- the dataset makes a contribution to a space lacking a dataset like the one presented here. The idea of meaning preservation for the summaries is also interesting and novel.

weaknesses: The main limitation of the work to me is the dataset splits for train, development, and evaluation. The current dataset makes a random split based on the commit hash, i would recommend making time and project sensitive splits instead. Splits can be setup so that no commits from the same project overlap between the different splits, similarly commits from the past can be used as training data with the future serving as test.

Update post discussion period: The authors promise addition of a train/test split and experimental results with no leakage across splits.
correctness: The constructed dataset does not seem to have undergone any human validation in its construction phase beyond a filtering of projects and commits. I would have liked to see a human evaluation of messages and summaries. If not the whole dataset then atleast of the test set alone. 

Update post discussion period: The authors labelled ~100 message-subject pairs for whether the subjects were valid summaries and found ~80% of them to be summaries, they also state that filtering can allow up to 90% of pairs to be summaries. This is as a reasonable initial exploration of how clean the dataset is. These levels of noise though not a problem for training are likely a problem for reasonable evaluation. A manually filtered test/dev set to facilitate clean evaluation would be a useful contribution of this work. 

clarity: The paper is largely well written although some of the sections could benefit from some consolidation. For example:
Section 5, L162-186: Consider listing and describing the "baselines" in a latex description environment.
Section 6: It would be helpful if "concept", "term", "core term" were used more consistently. Potentially also reducing the number of different terms used to refer to the same items.
relation to prior work: Well placed with respect to broader literature.
additional feedback: NA.
review_title: Paper about large scale summarization dataset requiring some polishing.

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The authors crawled code commits from Github and reformed it as a text summarization task. 

strengths: - The main contribution of the work is they crawled the data and do filtering/cleaning of such data.
weaknesses: - The hypothesis that "title/subject is the summary of the commits" is not very motivated and persuasive.
- Very weak data analysis and comparison to existing work, for examples, what is the ratio that title is included in the commits? what is the min, average, max number of words (not charactors) in titles and in commits? What is the overalapping and similarity among commits in the same post? 
- No qualitative study. The data seems to be quite noisy in my opinion when I check a few of them manually.
- Experiment-wise, it is not quite clear that what is the relationship between commits. For example, do we necessarily need to use all the commits to generate the title, or maybe just 1-2 commits are already fine. 

correctness: - I have doubts about the quality of the dataset as a summarization task.

clarity: - There are many aspects that the paper can be improved. For examples, many paragraphs in the paper are too short (e.g., line 19-20, line 23-25, line 108-110).
- Spending too much space talking about cleaning and filtering, which is less important than the data analysis itself.  
relation to prior work: - Weak comparison to previous summarization dataset work.
additional feedback: N/A
review_title: Why subjects should be summary of commits?

Meta-review: Reviewers agree that this is an interesting topic but worry about the utility of the dataset is limited and also has some ethical issues. Also, it will require a series of additional work to make it ready for publication, for example:
(1) how to justify: "title/subject is the summary of the commits"
(2) more data analysis
(3) human evaluation of dataset's quality.
I’m inclined to reject the paper.
Decision: Reject