Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: null
review_content: As mentioned by the previous reviewers, I believe that a data statement/datasheets/factsheet etc would be good to accompany this work. In addition, I have the following comments below.

“Inspired by the construction of ImageNet [12], we selected 67K unique English nouns from WordNet [36] yielding a set of diverse keywords (called “topics”) to specify our search queries”

Papers like [1] show some of the issues with this approach, including derogatory images and labels in ImageNet and Tiny Imagenet. This resulted in the Tiny Imagenet dataset being taken down. What was done here to ensure that such categories are not included? I see that you have filtered out twitter/facebook etc below. Also note that automatic filtering out of certain words can in itself be harmful to members of marginalized groups [2]. 

“In total, we extracted 2831 properties from DBpedia that we use as semantic types.”
How did the authors decide to use these specific properties? What are some potential issues with these properties (e.g. see above regarding works pertaining to WordNet) and how were they mitigated? What are the potential issues of annotating columns using models such as FastText pertained on common crawl? 

Have the authors done an analysis of the topic subsets besides the 10 mentioned in the paper? Given the scale of the dataset I am concerned about the same issues as mentioned in [1]. As mentioned in [2], curating datasets and documenting them is key. Have the authors attempted to create data statements (or data sheets) for this dataset? My understanding was that it was required for this NeurIPS track but I might have misunderstood.

[1] Birhane, Abeba, and Vinay Uday Prabhu. "Large image datasets: A pyrrhic win for computer vision?." 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2021.

[2] Bender, Emily M., et al. "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?🦜." Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 2021.


review_title: Additional comments after reading the prior discussion 

final_score: rating: null
review_content: The ethical concerns mentioned throughout the reviews and comments, which concern PII, licenses, responsible use, skews, and harmful content, have been reasonably well-addressed given the state-of-the-art in automatically handling these issues within datasets.

I agree with the previous ethics reviewer that a factsheet/datasheet/data card documentation would be a boon; this is also where the authors could further share the concerns raised and how they have addressed them (which could be a model for other dataset releases, IMO).

The largest remaining issue I see is that this dataset is shared as fully extracted content, rather than as urls.  When data is constructed as a set of urls, connected to the original source, then it is much easier for people who don't want their data in the dataset to simply delete their original sources and have that percolate to any further use of the dataset.  Ideally, the part of this dataset that is the table itself with columns, rows, cell values and a header with the original column names, could instead be a *link* to this content, where the content behind each link undergoes regular auto-population updates from the original sources that still return valid content.  This is a lot more work that opens up a lot of further issues (such as whether earlier versions of the dataset used to train deployed models should still be "around" in some way), so is a longer-term goal.

Although this dataset includes ethically problematic content, including the use of extracted (rather than linked) content, and imperfectly identified PII, etc., the authors have addressed the issues as best they could given current state of the art and the dataset's intended and foreseeable uses.  The dataset appears to be releasable if the work is accepted, but I encourage the authors to consider a url-only release in the future.
review_title: An Official Ethics Review

final_score: rating: null
review_content: This is a comment by one of the official ethics reviewers.

The technical reviewers have rightly pointed out several concerns with the dataset. The license issue has been addressed by the authors already by conducting an additional filtering. That's great.

However, on the point of encoding bias, the authors state that "In our revision, we emphasize the corrective maintenance that we will carry out to ensure no sensitive information or explicit biases are present in GitTables." The future tense is the problem. Mitigating the release of sensitive information and mitigating unwanted biases is not so easy. I do not feel comfortable with the dataset being released and its paper published until the authors have taken some concrete actions and the results are seen.
review_title: An official ethics review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors in this paper present Gittables, which is a corpus of tabular data they have scraped from GitHub, annotated that large corpus of data analyzed, and presented this data for public use. The authors compared the dataset fairly with the present availability of corpora and have highlighted the importance of Gittables in a clear and concise manner. 
strengths: * Gittables does solve some problems authors have highlighted in the related work section.
* Design principles described in section 3.1 are clearly identified and fulfilled by Gittables 
* Authors have compared the distribution shift for this paper with the existing corpora and have presented that Gittables is a novel addition. Gittables can be helpful for training better models and will be helpful for the community.
weaknesses: * The authors have mentioned that they only scraped tables from public repositories, not open-sourced repositories. I think that Public repositories on Github are still subject to copyright law and according to GitHub's official policy. In the search query authors provided on the footnote of page 4, authors have queried all repositories instead of properly licensed repositories. Though authors have addressed this issue in supplemental materials. 

* Authors have not mentioned the consequences of using the CSV files out of context. (this issue is addressed in the supplemental material though)
correctness: * There are no major issues with the correctness of the paper.
clarity: The paper is very well written. Analysis and motivation behind the paper are very well described.
relation to prior work: The authors have clearly described how this work differs from prior work and have described novel contributions in the field from their work.
additional feedback: No additional comments
review_title: An interesting and large corpus scraped from Github

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes GitTables, a large-scale dataset of relational tables (1.7M), mined from GitHub. The corpus is annotated with > 2K semantic types using distant supervision. 

The authors performed details analysis on GitTables, and found it has a topical coverage and content structurally different from web table corpora, thus potentially providing complementary benefits in future applications.
strengths: - The proposed dataset, GitTables, is a relatively large-scale dataset (1.7M) up to date. The authors proposed an interesting approach of using distant supervision to annotate its semantic columns.

- The table is automatically extracted from Github, and based on the analysis this dataset shows a topical coverage and content structurally different from the web table corpora, thus could yield benefits in future applications.

- The authors presented detailed analysis on the topical distribution, syntactic and semantic types of column annotations, and demonstrated one use-case of semantic column type detection using GitTables.
weaknesses: - I think the authors should present a better analysis on how GitTables can be used. It is briefly discussed in Section 4.4, but it is not surprising that a model trained on web table corpora does not generalize very well on GitTables, since the topic distribution is different. I think it would be more convincing if the authors can show a use-case which is achievable by GitTable (given its wider coverage over certain topics) but not by web table corpora.

- A discussion on the responsible use of GitTables should be included. Given the skewed topic distribution of GitTables, will it yield any issues when training a model on this corpus? E.g., what kind of applications will GitTables be suitable for, and for what applications should we not use GitTables?
correctness: Mostly correct.
clarity: Yes the paper is well-written.
relation to prior work: The relation to prior work is discussed adequately.
additional feedback: Please add a discussion on the ethics/responsible use of GitTables. Is there any bias towards certain topics given the topic distribution on Github could be skewed?
review_title: A large-scale dataset on github relational tables

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: [Edited Jul 19, 2021]: 
Thanks to the authors for their response and for updating the submission. I think the revisions help to address the concerns raised in my review and the other reviews. After revisions, I continue to recommend acceptance.

-----

This paper describes a large-scale scraped dataset of CSV files from Github for training systems that represent offline database tables.
strengths: Significance of contribution: Large. The paper makes a very convincing argument that this dataset is quite critical for extending high-capacity table-related models beyond simple HTML tables to offline relationship database tables. Specifically, the authors argue prior work has focused heavily on tables such as those found in Wikipedia that lack "ID" columns, large numbers of rows, and business-relevant content. GitTables represent an opportunity to extend the "table modeling" researching to many more domains.

Relevance to broader research community: Potentially high. The submitted manuscript describes in great detail a variety of specific applications, particularly at developing models that understand column semantics, are trained on relationship databases, etc. However, I imagine this dataset may be useful for a variety of other reasons, and there may be great interest in curating some subsets (e.g. for statistics, specific domain areas, etc.)

Ethical concern: The main paper does not discuss ethical (or legal) concerns in great depth but the supplemental materials do.
weaknesses: Relevance to broader research community: The submission as is may be difficult for the broader research community to use. Specifically, it seems a major requirement for this dataset to be used more widely will be the curation of smaller subsets. Many researchers may be concerned with training models or attempting to perform observational analyses on a massive dataset. However, this is certainly addressable and not necessarily a strong argument against publication in the paper's current form.

Ethics: a major weakness of the paper as submitted is a rather weak discussion of concerns around (1) licensing and (2) negative impacts of working with large, uncurated datasets (c.f. concerns raised in this high-profile paper: https://dl.acm.org/doi/10.1145/3442188.3445922). See below for more.
correctness: The dataset preparation choices all seem reasonable. The paper includes fairly clear design principles which guide the choices. No major concerns with correctness.
clarity: The paper itself is very well written. Also of note is that the GitTables website is very cleanly presented, which will be relevant to users of the dataset.
relation to prior work: Comparison with prior work is well written. It focuses specifically on "table corpora", but this seems quite reasonable relative to the contribution of this paper.

For future work, it may be interesting to compare GitTables with a large number of other dataset hosting services that also have many relational database-esque CSV files (e.g. Kaggle, Dataverse, etc.) but this is likely quite out of scope of the current submission.
additional feedback: Thanks to the authors for this submission!
review_title: This paper describes a large-scale scraped dataset of CSV files from Github for training systems that represent offline database tables.

Meta-review: Due to concerns raised by the ethics reviewers, we are not able to accept your submission at this stage. However, we warmly welcome you to update your manuscript according to the ethical comments and recommendations and resubmit to Round 2.
Decision: Reject