Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: The paper proposes a new software development methodology (Evident) for machine learning, data mining, and knowledge management. It discusses pitfalls of traditional software processes related to the waterfall and agile models and explains why they may not be sufficient for these fields. Further, the paper describes the main concepts related to Evident. The main idea centers around modeling the relationships between observations, hypotheses, and tests. Finally, the paper describes how this methodology may help to alleviate common pain points in ML, DM, and knowledge management.
strengths_and_weaknesses: Strengths:

S1- The paper aims to make a contribution to the important problem of improving development processes for ML and more. 

Weaknesses:

W1- It is difficult to judge the effectiveness of the contribution without any user studies with development teams that have applied Evident. At the very least, it would have been useful to see a few case studies and qualitative remarks on how teams could apply this in practice. This would instantiate the presented terminology (knowledges, containers, hypotheses, observations etc) with concrete tasks and terms of a given project.

W2- Often the paper makes strong claim regarding the ineffectiveness of Agile (or other processes like CRISP-DM) and the effectiveness of of Evident but the justifications are mostly abstract and sometimes handwaving potential benefits without a proper either theoretical or rigorous framework to prove these claims. 

W3- It is not clear how the authors distilled the presented challenges in Table 2 from related work. How did they decide to include or leave a challenge out of the table? Plus, there is a lot more work on methods and processes for ML by now that could also be considered. Some of the challenges are also very generic and they may as well apply to any software that may not include ML (e.g. dead code paths, Prototype Algo may be accidentally run in production causing damages).

Literature on challenges in ML software development:

Software Engineering for Machine Learning: A Case Study

Software 2.0 https://karpathy.medium.com/software-2-0-a64152b37c35

Machine learning testing: Survey, landscapes and horizons

“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI

An empirical study of common challenges in developing deep learning applications
questions: - Has this process been applied in practice? Are there any insights to share to this end?

- For future versions of the paper it may be useful to be more specific and highlight for which of the challenges in Table 2, Evident may work better than other methodologies and why. 
limitations: I do not think the authors have really described or mentioned the limitations of their approach. In contrary, the contributions are sometimes hyperbolically described claiming that the process can advance society, philosophy, and science, without any factual evidence.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper discusses problems (called pain points in the paper) in using the agile methodology for Data mining/Machine learning/knowledge management. It then presents a software development methodology for data mining tasks that can also be used for machine learning and knowledge management. Activities in a project are broken down into observations, hypotheses and tests that denote facts, knowledge and evaluation respectively. These serve as containers that can also used for storing and managing project related artefacts.
 
strengths_and_weaknesses: Strength:

- The paper identifies the need for having a software development methodology for DM/ML/KM
- The paper presents a summary of pain points for DM/ML/KM 

Weaknesses:

- The paper lacks scientific rigor. Although some of the ideas are interesting, there is no experimentation/evaluation to substantiate the proposed methodology
- The paper does not present any literature review of the area, that raises questions about whether the work in the paper is indeed a novel contribution


questions: - The authors have discussed the agile methodology and problems in its application for software development and artefact management for dm/ml/km. However, why compare with agile only? It would be better to compare with other DM methodologies that have been proposed. For example, the paper talks about the CRISP-DM, but lists limitations of this methodology briefly. No other methodology is mentioned. 

- How have goals and approaches been differentiated? For example, how is Individuals and interactions over processes and tools an approach and Responding to change over following a plan a goal

- The paper says only four principles are actionable, but lists only three

- The fact that agile is unfalsifiable is mentioned, and treated as a drawback. However, it is not clear which development methodologies  (if any) are falsifiable

- The paper says the scope of Agile is not defined. Again,  it would be good to mention whether all other methodologies have scope defined

- How have the pain points in Table 2 been identified? What is the last column?

- How do the  Fig 3 a), b) and c) represent induction, abduction and deduction? Some explanation is required

- Were experiments not carried out using Evident? At least something concrete is required, perhaps a case study with examples. Otherwise, the claims about alleviation of pain points in section 5 are not justified.

- The paper title mentions topology. Where is this discussed subsequently? 

- I am not sure I understand what is meant by the following:

"This paper discusses limitations in agile as a scientific claim"
"Beyond software development, Evident is illustrated to be applicable in many aspects of society"
"Although control experiment challenges or absence of quantitative project Agility measurements may explain no measurable scientific evidence,...."
"DM has not been regarded highly collaborative and scalable activities to deliver high throughput Knowledge"
"resulting in the under performance of production Algo"


limitations: As far as sections are concerned, the authors have mentioned pain points not alleviated under section 6.3.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes a project management/artifact management framework (Evident) in the form of a knowledge base topology. Throughout the paper, the authors discuss about the challenges presently faced by the AI community - including in knowledge representation and utilisation of traditional relational databases for ML/AI workflows. The paper is interesting and the problem being solved is relevant.
strengths_and_weaknesses: Weaknesses:
Very limited novelty in terms of the contributions of the paper. While the framework and discussions surrounding Evident are interesting, they do not add much substance from an AI perspective and from the project management perspective, it is not very clear how and why the proposed workflow is the way to go for the AI community. While the authors have described "pain points of DM, ML and KM", it is very theoretical with limited examples, limited practical scenarios and above all, limited interrelation with the substance related to AI over project management.

Strengths:
Interesting topic being discussed in the paper, very well written paper.


questions: This reviewer believes that the paper is very theoretical and oriented more towards the "project management" audience in comparison to the substance which an AI audience would be interested in. It would have been useful if this paper had more interaction with AI - beyond the very common pitfalls ("pain points") that the paper describes which are mostly well known in terms of consensus in the community. Also, without any user-studies or real-world use of this framework, the value which Evident can deliver is very hypothetically presented in the paper which makes it less rigorous. This reviewer suggests that paper would be more suitable for a journal relevant to AI and project management etc. in its current form.
limitations: Mostly the limitations have been addressed. However, there is not much added value or substance which the paper describes beyond the "Pain points not alleviated" described in Section 6.3 - the hypothetical nature of the Evident framework makes the paper read more like an essay (being very verbose) which affects the overall contributions outlined in the paper.
review_title: null

Meta-review: metareview: The paper identifies relevant issues in the current software development process for machine learning, data mining, and knowledge management, however it does not provide any practical evidence that the proposed directions can solve the identified issues. The content of the paper is more suited for a position paper than for a technical scientific paper, which is the target of NeurIPS. So, while recognising some value in the contribution of the paper, I believe its nature does not completely match the NeurIPS expectations, and it would be more suitable for conferences where position papers are one of the components of the technical program.
confidence: Certain
Decision: Reject