Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: - FACT - a dataset of an integer sequence dataset consisting of data from both organic and synthetic sources
- a variety of tasks designed to evaluate integer sequence models
- a collection of the performance of different baseline models on the FACT
strengths: Integer sequencing is applicable to many domains from data compression to communication to biochemistry. The authors introduced FACT - a dataset of an integer sequence dataset consisting of data from both organic and synthetic sources. FACT comprises of processed and generated data from OEIS. FACT provides an easier/better/cleaner access to OEIS.
weaknesses: The two novel contributions of the paper seem to be:
1) Processing and generating synthetic data from OEIS.
2) Designing different task types including sequence classification, sequence similarity, next sequence-part prediction, sequence continuation, and sequence element unmasking.

However, these two novel contributions are not very significant. FACT comprises of 341,000 processed entries from OEIS. The authors then extended the dataset by synthetically generating sequence branches. However, the author did not include details on how the data is generated (refer to section below). And processing an existing dataset is not a significant contribution. 

The five different task types are common in many sequence modeling domains such as language. They have also been used in and are popular in language processing, e.g. predicting the next word in a sentence. Thus, the five different task types are not completely new. 


correctness: Yes
clarity: - “Our initial experiments with the baseline models (cf. Section 4) further confirmed that much of the data did not reach the critical mass of  information necessary for reliable use in machine learning applications. We hence systematically extended the dataset by synthetically generated sequence branches while abiding by the structure and nature of the stem encyclopedia entries and providing carefully engineered automatic annotations wherever possible. ” 
    - Examples of data generation?
    - How much of the data was generated/extended
- Synthetic data generation summary needs more details:
    - How exactly is it generated 
- “In this section we present 5 types of tasks in order of difficulty. For each task, our dataset is split into three parts: training, validation, and test set. The training set consists of only synthetic sequences, the validation and test sets consist of a combination of samples from the OEIS dataset and the synthetic dataset.”
    - Why not include some OEIS in the training as well?
- In Table 1, “we also notice a general trend among all models to perform better on synthetic data than on the organic OESI sets”
    - What does this say about the quality of synthetic data?
relation to prior work: Yes
additional feedback: N/A
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces a dataset of integer sequences comprising data from both real-world and synthetic sources. It additionally provides a library named FACTLIB to process and generate additional data. The dataset includes several types of task, namely sequence classification, sequence similarity, next sequence-part prediction and sequence continuation. For each task a performance metric is given.
strengths: * The datasets is well-curated and contains a large number of tasks
* A wide range of baselines is provided and evaluated on the dataset
weaknesses: * Baseline models already achieve reasonably strong performance, which may limit the impact of the dataset
* The experiments should include few-shot learning scenarios for completeness and better assessment of the relevance of the dataset
* The paper does not report human performance
correctness: The claims in the paper are correct and the dataset is constructed in a sound way
clarity: The paper is well written
relation to prior work: To the best of my knowledge, the paper provides a good discussion of related work in its area
additional feedback: No additional feedback
review_title: Interesting line of research

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: **Update after rebuttal**: the authors did a good job in addressing the main shortcomings of this paper and benchmark, in my opinion. I have increased my score from 5 to 7 based on the revised paper, dataset and documentation.

----

The paper presents a dataset and 5 benchmarking tasks for learned abstraction and reasoning over integer sequences. The dataset comprises real and synthetic sequences and presents 5 tasks, being: classification, next-sequence-part-prediction, continuation, similarity and unmasking. A performance comparison of multiple classic and neural machine learning models is presented.
strengths: **S1**: The paper introduces a new dataset and benchmark for a set of tasks for reasoning over integer sequences. The proposed tasks, such as next-sequence-part-prediction and sequence continuation, are interesting and well-suited for machine learning models.

**S2**: The dataset seems carefully curated and is described well, albeit mostly in the appendix.

**S3**: Various machine learning models are evaluated on the benchmark tasks.

weaknesses: **W1**: The benchmark implementation seems not usable as-is due to missing documentation (see documentation) and lack of reproducibility. The latter is mainly caused by the lack of fixed seeds or dataset splits for train/test/validation purposes of machine learning models (see correctness).

**W2**: The motivation of this benchmark is, in part, to identify (high-level) abstractions explaining the integer sequences, the benchmark models and results towards that end are not well explored which restricts the value of this paper. Beyond classification/regression performance regarding the reasoning tasks, what insights are machine learning models expected to surface and/or what applications should they facilitate? Which open challenges does the benchmark identify, e.g. from the classification/regression mistakes? No symbolic methods were considered as baselines weakening the motivation for machine learning models. It seems good to illustrate such insights in the experimental section.

**W3**: The main paper does not provide sufficient detail regarding 1) the dataset, 2) interpretation of the baseline and model performances. The dataset contents and preprocessing/feature procedures are covered in the appendix, but its corresponding section in the main paper lacks important details needed to understand it on a high level. Although performances are generated for a large number of models, interpretation is very limited. This benchmark, therefore, reveals limited insights.
correctness: The dataset seems to be constructed correctly and detailed preprocessing steps are provided in the appendix (it is strongly recommended to include some of the key details in the main paper). The benchmark results may not be reproducible from the scripts, given that the dataset splits are non-deterministic (random sampled without fixed seed) and not reproducible. The lack of fixed dataset splits for train/test/validation of machine learning models jeopardizes the comparability of future evaluations. The benchmark should not be accepted without fixed seeds or dataset splits. The reported results should also stem from the same dataset splits.

clarity: The dataset is explained very (too?) briefly in the main paper, making the high-level construction, preprocessing and annotation of the datasets unclear. The performance comparison is also mainly covered in the appendix, which limits the insights from reading the main paper.
relation to prior work: It is not clear from the documentation (a ReadMe on GitHub) how one could use the GitHub repository to benchmark their machine learning models using the provided code and data. Clear guidance describing which functions to call etc., are needed to make the benchmark usable. One way of implementing this is by writing a readme, an example could be the AutoML benchmark: https://github.com/openml/automlbenchmark#readme, https://openml.github.io/automlbenchmark/index.html. Details for reproducing the experimental results as reported in the paper should be provided as well.

additional feedback: None.
review_title: New datasets for integer sequence reasoning and abstraction with basic benchmarking across 5 tasks

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes a new dataset called FACT, the Finitary Abstraction Comprehension Toolkit. The FACT dataset is composed of integer sequences and challenges a test taker on a variety of tasks, including sequence classification, sequence similarity, next sequence part prediction, sequence continuation, and sequence unmasking. The authors create the dataset by combing the OEIS public repo and designing several grammar structures to extend the dataset. In evaluation, the authors exam both neural and classical classifiers / regressors on the FACT datasets.
strengths: The work introduces an interesting suite of tasks on machines' abstraction ability and number sense. Number abstraction is an important topic that has been barely investigated and it's nice to see the authors working on this challenging domain.

The work is also well-grounded leveraging the OEIS database and structured grammars to generate data.

The work is nicely presented with examples and processing details. And it's good to see the work open-sourced. 
weaknesses: One and maybe the most serious concern of mine is on the goal of this dataset / benchmark. What do you expect out of the machine performance on it? On one hand, we already have various symbolic regression methods that can handle tasks regarding numbers and their relations. If you can have a symbolic regression method solve most of the dataset then what is the point of having it? If it cannot, then why not directly curate another symbolic regression benchmark? On the other hand, I actually suspect if an average person can solve the number sequence problems. We might be familiar with Fibonacci numbers but I'm not sure if an average person can quickly identify the underlying relations. Especially those related to other scientific domains, like the chemistry example in Line 113. If an average person cannot, then shall we expect a machine to be able to?

The manuscript mentions about multi-solutions to a problem. But I'm not sure if I understand it correctly. What if there are equally parsimonious explanations? Say, 1, 3, 5, 7 could be either interpreted as an odd number sequence but also a prime number sequence. Higher-order polynomials are also discussed. How many subsequences (or how long of a sequence) do you think is appropriate to supply to humans to make the distinction?

On the evaluation part, it seems machines are pretty good at it. Is it because machines actually learn the relations in numbers or is it because the dataset is so large that statistical patterns have been observed in the training set? Can you possibly perform tests on generalization to tell the reason? Also, we have recently witnessed the potential of large language models. Can you show some examples that can or cannot be solved by, say, GPT-3, and analyze the results?

There may be incorrect references to Figure 3 in Line 70 and Line 82.
correctness: Yes.
clarity: Yes.
relation to prior work: The work is highly related to existing abstract reasoning problems in the following. And I think they should be properly discussed.

Raven's Progressive Matrices  
[1] Zhang, Chi, et al. "Raven: A dataset for relational and analogical visual reasoning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.  
[2] Barrett, David, et al. "Measuring abstract reasoning in neural networks." International conference on machine learning. PMLR, 2018.

The Abstraction and Reasoning Corpus  
[3] Chollet, François. "On the measure of intelligence." arXiv preprint arXiv:1911.01547 (2019).

Machine Number Sense  
[4] Zhang, Wenhe, et al. "Machine number sense: A dataset of visual arithmetic problems for abstract and relational reasoning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 02. 2020.
additional feedback: See weaknesses.
review_title: Review for FACT

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors contribute a large dataset of integer sequences (comprising OEIS sequences and synthetically generated sequences), a benchmark for learning the abstractions governing these sequences, and software tools to work with and extend both the dataset and the benchmark.
strengths: The package (dataset, benchmark, software) is well-designed overall and will help advance research in abstract reasoning with machine learning models, a task that is highly relevant to the NeurIPS community.
The software engineering thinking that went into the design of the library is appreciated.
weaknesses: Given that one of the advertised motivations behind the work is to help advance toward artificial general intelligence (AGI), the absence of few-shot learning scenarios in the benchmark is surprising.
See also "Correctness".
correctness: Overall, the dataset was constructed in a sound way, and the evaluation methods and experimental design of the benchmark are appropriate and performed correctly.

But:
- On judging sequence complexity by the length of the generating rule: The rule you use to generate a sequence might not be the only rule generating a sequence. Can you say anything about the length of your generating rule vs. the length of the shortest generating rule in existence?
- On your hierarchy of task difficulty: Sequence Continuation vs. Sequence Element Unmasking - unmasking _one_ element in the middle of a sequence appears (to me) to be easier than unmasking _one_ element at the end of a sequence, so the difficulty relationship you assume here is not entirely intuitive
- On the experimental setup and the reported baseline results: I might have missed something, but are you using a _single_ train-eval-test split? How is that split determined (beyond the details you give in the paper)? It might make sense to deliver _several_ train-eval-test splits with your benchmark, which would allow us to understand how performance varies across splits. In that spirit, I would have expected to see some indicators of variance in the result tables as well.
- In your result tables (if you choose to stick with the tables), make sure you _actually_ highlight _all_ of the best performers (you definitely didn't in Table 1!).
- When presenting results, please specify when higher is better and when lower is better in the captions, especially since you have _both_ cases.
- Similar to ARC (see "Relation to Prior Work"), you might want to consider also providing a _private_ test set against which external users can evaluate.
- What about few-shot learning? If you truly aim at making progress towards AGI, you might want to have a few-shot learning scenario (again similar to ARC), e.g., along the lines of "given a sequence s1, its continuation c1, and a sequence s2, generate the continuation of s2" (or something even more IQ-test-like). Humans don't need many examples to identify the simplest applicable pattern, so one could demand that AGI machines shouldn't, either.
clarity: The paper is structured well and written clearly.

I would prefer figures over tables to present the baseline results (in the main paper and in the Appendix), but I acknowledge that dumping results into tables is common practice.
relation to prior work: The paper discusses relevant related work and specifies clearly how FACT goes beyond the existing literature.

The work reminded me (in some of its spirit) of the Abstraction and Reasoning Corpus (ARC) (https://arxiv.org/abs/1911.01547; https://github.com/fchollet/ARC), which could be mentioned as related work, too.
additional feedback: - ll. 25–27: "sits at the centre of artificial intelligence research" - you're a bit overselling here, maybe tone it down a bit (e.g., "is a cornerstone" or so, as there can be multiple of those but usually only one center, and – see "Correctness" – you don't address few-shot learning)
- Figure 3: maybe mention the direction of the orderings in the caption (top to bottom and left to right); also, I assume that tasks that don't share a row or a column are incompatible, but you might want to make that clear in the text
- l. 160: "macro average F1 score" – I can guess what "macro" means here, but maybe you can make it a bit more explicit
- Tables 1–3, 6–9 could be more informative as figures
- Figure 4 is missing axis labels
- ll. 483/484: scoring from -2 to 2 would have been more intuitive
- ll. 545–552: You acknowledge that palindromicity is representation-dependent, which makes it stand out among the characteristics you define. Why did you include this anyways?
- ll. 556–562: I found this a bit confusing – e.g., does "primality" in your dataset now mean "consists only of primes" (ll. 541/542) or "is somewhat related to primes" (ll. 558–562)?

Typos and language things (only those that caught my eye while reading):
- l. 4 in the caption of Figure 1: "sub-shells" -> "sub-shells _of an atom_" (?)
- l. 4 in the caption of Figure 1: "is comprehended its original modality" -> "is comprehended _in_ its original modality"
- l. 2 in the caption of Figure 2: "in synthetic part" -> "in _the_ synthetic part"
- l. 2 in the caption of Figure 2: "whereas the size" -> "whereas the size_s_"
- l. 107: comma after "design" 
- l. 114: no "the" before "Occam's"
- l. 116: "multitude of disciplines of science" -> "multitude of _scientific disciplines_"
- l. 158: commas before and after "and as such"
- l. 159: commas before and after "for every sequence"; "where it belongs" -> "_to which_ it belongs"
- l. 166: "each other" -> "one another"
- l. 171: "mean square error" -> "mean square_d_ error" (twice)
- l. 180: "for model understanding of its inputs" -> "to test whether a model understands its inputs"
- l. 184: commas before and after "and as such"
- l. 195: comma after "sub-types"
- l. 203: "in" -> "on"
- l. 206: "current limitations of the existing methods" -> "limitations of existing methods"
- l. 209: "bench-marking" -> "benchmarking" 
- l. 209: "four" -> "4" (consistency)
- ll. 209/210: "transformer networks" sounds weird, maybe say "dense, recurrent, and convolutional neural networks, and transformers"
- ll. 211/213: "Ada Boost" vs "AdaBoost" - pick a spelling
- l. 218: comma after "our results", "the" before "category", comma after "level"
- l. 252: "the purpose" -> "that purpose"
- l. 261: "periodic functions" -> "periodic functions, which are"
- l. 280: "requires" -> "require"
- l. 282: "model's reasoning" -> "model reasoning"; "and aim" -> "and we aim"
- l. 286: "learning of abstractions" -> "learning abstractions"
- l. 324 (ref. 13): "ofinformation" -> "of information"
- l. 414: "easy to use" -> "easy-to-use"; comma after "example"
- l. 454: comma after "section"
- l. 481: comma after "To this end" (many commas missing in the Appendix after this point, not documented anymore below)
- l. 499: "returns" -> "return"
- l. 539: "withing" -> "within"
- ll. 541, 543, 553: line breaks after first word missing
- "non terminal" -> "non-terminal", globally (relevant for Appendix B)
- l. 623: no full stop after "function"
- l. 655: "for" before "which"
- l. 665: "four" -> "4"
- l. 749: "self contained" -> "self-contained"
review_title: Valuable dataset, benchmark, and software for research into learning abstractions.

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: summary and contributions: This paper proposes a set of benchmarks with the objective of determining certain properties of integer sequences such as whether they are finite, periodic, etc., identifying similar sequences (w.r.t. to their properties), detecting whether two sequences, s1 and s2, follow each other, identify the next entry of a sequence.

By breaking down the tasks of understanding integer sequences, they can investigate the potential of ML algorithms for each task individually and even stimulate the creation of models specific to one of these tasks. Overall, this could provide new insights for future research revolving this subject, even for symbolic models describing the sequence.
strengths: - A large and comprehensive set of benchmarks
- Thorough categorization of integral sequence properties
- A good starting baseline
- Division of the main goal into subtasks of varying difficulties
weaknesses: - The provided source-code (https://github.com/FACT-Development-Team/FACT) lacks an appropriate documentation on how to setup and use
- In this same line, there is no instructions on how to extend the benchmark and the ML methods available

correctness: yes
clarity: yes
relation to prior work: yes
additional feedback: none
review_title: A benchmark for the inference of Integer Sequences properties

Meta-review: metareview: This paper presents a large dataset of integer sequences and several tasks for evaluating modeling of those sequences.

The reviewers generally appreciated the careful curation of the dataset, the range of models explored, and the potential application domains.

GVg5 and GeCe also bring up a point about human performance, which I also interpret as asking about the identifiability of these sequences similar to a point raised by Bx7J. I am convinced by the authors' response to this point.

I agree with reviewer TiHT that it is disappointing not to see more engagement with symbolic understanding of these sequences. Figure 1 seems to set up an approach for dealing with these sequences via natural language descriptors of rules or abstractions, but this is not the problem being addressed in this paper. While I think this would be interesting to see covered, the work doesn't necessarily need to focus on it.

Overall, this paper is very thoroughly done and appears to substantiate its claims.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept