Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary and contributions: * This paper presents  a pilot study to build large scale EEG data that can be used for multi-class classification of use-centered tasks. The paper presents experiment design and data collection process for two EEG based BCI experiments:
     * Read-Write-Type:  typing vs. writing 
    * Think-Count-Recall: cognitive states like copying vs. answering vs. reading

* The authors propose the use of low-cost wireless hardware: the Muse headset
* The authors developed an online forum for discussion and use of this dataset
* Results of classification with the proposed new data and machine/deep learning models show a reasonable accuracy indicating the potential of this data collection method
strengths: * The authors pursue an important research direction where they try to collect a large amount of data using cheaper tools and simple experimental setting. This type of approach if successful  could help to quickly gather large scale data in medical community 
* Results show that these data achieve a reasonable accuracy ~ 70%
* The data will be provided upon request by validating the institution and research area of the requesting person
weaknesses: * While the motivation to gather medical data with cheaper, easier and faster approach is important the details provided in the paper are not enough to replicate the same data collection process. For example,
    * During data collection, how are the volunteers chosen, how long was each session, what kid of visualized feedback is send to the participants, how many EEG cycles were recorded, etc.
   * Data analysis: The details of  human-in-the-loop process is not provided. How should the EEG coach guide the end-user?
  * What are the details of the ML model? Was the data preprocessed before using it in a models. The dataset size is not clear. What is the threshold and what value of K was the best?

* The clinical significance of the present experiments and task are not explained in the paper so it is difficult to access how important the designed experiments are

* There is no comparison to data collected in clinical/research setting 

* The writing is not clear and there are many grammar issues.
   * there is no space around () at multiple places
   * Cross line figure [cite Figure]
correctness: * The main contribution of the paper is to provide an experimental design   and data collection pipeline to users to easily collect EEG data. However, the experiment steps are not clear which will make it difficult for reader to design and conduct other similar experiments?
* How are the participants picked? How important is it for the participants to be college students?
* Will continuous feedback to participants during the experiments impact the outcome of the experiments in anyway? Also, do you discard initial data points when the participants were not experienced?

clarity: The paper is not clear. It has many grammatical and minor issues. It needs to be revised throughout for clarity. Below I point out some issues:
* Think(T), count(C), recall(R) -> Think (T)
* Incorrect sentence: These datasetsQu et al. [2020b] suggest a future direction
* comma and spaces: Qu et al. [2020a]LaRocco et al. [2020]Lotte et al. [2018b].
* Can you please change qualitative description with quantitative numbers:low-cost -> what was the cost compared to clinical equipment's, interpretable feedback -> what feedback was provided?
* Please provide the link to the online forum
* Use sentence case in all figure captions

relation to prior work: * The authors provide some references and connect their work to these references at the beginning of each section. 
* It is not clear how the proposed TCR and RWT compare to existing experiments on cognitive tasks. What devices are used by present works could be added. 
* The size and format of this data in comparison to existing data is not available but it could be helpful to users.
* What deep/machine learning models do existing works use
additional feedback: I think the paper could be improved by adding quantitative details on dataset statistics and dataset collection process. The authors could also add additional ML/DL models to conduct these experiments and improve the baseline numbers.

Also, the paper needs to be revised for clarity, consistency and grammar.
review_title: The paper explores an important topic of gathering EEG data through cheap means and non-expert participants. However, I find the details on the dataset statistics , dataset collection and benchmarks models are not sufficient to allow readers to replicate these experiments. Also, the writing is not clear and grammatically correct. Hence, I think this manuscript needs a major revision before it is ready for publication.

final_score: rating: 3: Clear rejection
review_content: summary and contributions: This paper presents a framework for creating EEG datasets that can support BCI systems. Core components of the pipeline include experimental design, data collection, data analysis, and community forum. Two existing datasets are used as case studies for the framework.
strengths: EEG-based BCI systems have uses for many applications. Standardizing datasets and enabling the creation of large-scale datasets is likely critical for their development. 
weaknesses: Generally unclear what the significance and contribution of this paper is. 
correctness: Yes
clarity: The organization and clarity of writing could be improved. In general, the writing and formatting seems a bit unpolished. Also see "additional feedback" for some specifics. 
relation to prior work: Lacks significant discussion about the proposed framework’s relationship to prior work, it’s exact contributions, etc. For example, Sections 1.1 and 1.2 could use more context and explanation. It is not clear what is prior work and what (if any) is being proposed. 
additional feedback: Formatting of citations makes it difficult to read. 
Caption of Fig. 4 seems to be incomplete
Hard to tell how to read and what to take away from Fig. 6
review_title: Proposes framework for building large-scale EEG datasets, but significance of contributions is unclear

final_score: rating: 3: Clear rejection
review_content: summary and contributions: Update: 7/20/21 . Thank you for the revised submission. I do think that some of the missing information has been added to the manuscript, and I have improved the score to a 3, but I still think that the manuscript has a ways to go before being of suitable quality and documentation level for acceptance. There is a basic confusion over the exact nature of the dataset, and whether what is being presented is a dataset or a framework. I'm not sure a framework would be suited to this track. 

This submission introduces a dataset of EEG signals in humans performing different activities, intended for multi-class classification of tasks such as reading, writing, and typing. The authors introduce a machine learning benchmark on predicting the task humans are engaged in from the EEG. The ultimate goal of these datasets is to inspire new machine learning approaches for decoding behavior from EEG. As non-invasive neural recordings become ubiquitous, there is a greater need for such algorithms and datasets. I believe the dataset is from 20 EEG contacts sampled at 10 Hz, from 14 subjects, in 2-10 different tasks across six sessions, but am not absolutely sure due to the manuscripts clarity. Subjects are trained to use the EEG beforehand. The data is stored, in standardized EEG formats, but the nature of the metadata is unclear.  The data is only offered via written request. 

strengths: # Significance of the contribution.
Neural interfaces are becoming of increasing interest to industry and having large available datasets could be useful for students and researchers to tease out signal from noisy data. This research is well situated with respect to that goal. 



weaknesses: 
# Significance of the contribution, 
The work is unfinished and poorly documented, making it difficult to evaluate and follow. It is not clear what their dataset consists of (tasks and subjects), what algorithm was used for benchmarks, or how to get access to the dataset. The paper is poorly written, and it is not clear how many subjects and activities are included, and their extent. The paper does not discuss other EEG/ECoG datasets that may be relevant in the field. 


# Relevance to the broader research community.
Such a dataset, if of sufficient size, scope, and quality, could be of interest to the Neurips community. However I do not believe this dataset goes beyond already available datasets for inspiring algorithm design from brain signals. I am not sure what the novelty is. 

 # Accessibility and accountability,
The authors do not include information about research with human subjects in the author statement, e.g. discussion of wages, participant links, IRB approvals, or instructions given to patients. Numerous other fields in the checklist (eg. amount of compute, code reproducibility) were marked as 'yes', but I see no documentation of processing power. 


correctness: The dataset file structure is documented, but it is not otherwise clear what the data consists of, how many subjects there are, how many tasks and sessions there are, or if there is a metadata. It is not clear what the benchmark is, and if the design and methods of the algorithm are appropriate. 
clarity: No. I found the introduction and motivation sections easy to follow, but much of the rest was not. The figures were undocumented, discussed out of of order, and figure captions were left unfinished with author comments (Figure 4). The results section was difficult to follow and the methods behind the benchmarks are not clear. The reference formatting occludes the text, and the instructions for the checklist and submission were not commented out. 
relation to prior work: No. There is a discussion of past work on the specific tasks used, but there is not a discussion of different datasets for EEG and ECog. This is especially problematic, because such datasets are available in the BCI field (e.g. Miller, K.J. A library of human electrocorticographic data and analyses. Nat Hum Behav 3, 1225–1235 (2019); Schalk, G., McFarland, D.J., Hinterberger, T., Birbaumer, N., Wolpaw, J.R. BCI2000: A General-Purpose Brain-Computer Interface (BCI) System. IEEE Transactions on Biomedical Engineering 51(6):1034-1043, 2004; the Berlin BCI Competition; Kaya, M., Binli, M., Ozbay, E. et al. A large electroencephalographic motor imagery dataset for electroencephalographic brain computer interfaces. Sci Data 5, 180211 (2018)). There are likely available datasets in the more cognitive-EEG field, but there are not discussed, making it difficult to evaluate the novelty  of the approach. 
additional feedback: Even were the clarity issue with this dataset resolved, I am not convinced it is of sufficient breadth and interest for this conference.
review_title: EEG Dataset with insufficient clarity and documentation

Meta-review: This paper is about a dataset for EEG-based BCI tasks.  The reviewers found the topic interesting, but found that the dataset/paper was not sufficiently well-motivated or documented.
Decision: Reject