Evident: a Development Methodology and a Knowledge Base Topology for Data Mining, Machine Learning and General Knowledge Management
Authors: Mingwu Gao
Samer Haidar
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Data Mining
Machine Learning
Knowledge Management
Computer History
Software Engineering
Database
Sensing Hardware
Philosophy
Knowledge Sharing & Retention
Project Management

Abstract: Software has been developed for knowledge discovery, prediction and management for over 30 years. However, there are still unresolved pain points when using existing project development and artifact management methodologies. Historically, there has been a lack of applicable methodologies. Further, methodologies that have been applied, such as Agile, have several limitations including scientific unfalsifiability that reduce their applicability. Evident, a development methodology rooted in the philosophy of logical reasoning and EKB, a knowledge base topology, are proposed. Many pain points in data mining, machine learning and general knowledge management are alleviated conceptually. Evident can be extended potentially to accelerate philosophical exploration, science discovery, education as well as knowledge sharing & retention across the globe. EKB offers one solution of storing information as knowledge, a granular level above data. Related topics in computer history, software engineering, database, sensing hardware, philosophy, and project & organization & military managements are also discussed.

Evident: a Development Methodology and a Knowledge Base Topology for Data Mining, Machine Learning and General Knowledge Management
Anonymous Author(s)
Affiliation
Address
email
Abstract
Software has been developed for knowledge discovery, prediction and management for over 30 years. However, there are still unresolved pain points when using existing project development and artifact management methodologies. Historically, there has been lack of applicable methodologies. Further, methodologies that have been applied, such as Agile, have several limitations including scientific unfalsifiability that reduce their applicability. Evident, a development methodology rooted in the philosophy of logical reasoning and EKB, a knowledge base topology, are proposed. Many pain points in data mining, machine learning and general knowledge management are alleviated conceptually. Evident can be extended potentially to accelerate philosophical exploration, science discovery, education well as knowledge sharing retention across the globe. EKB offers one solution of storing information as knowledge, granular level above data. Related topics in computer history, software engineering, database, sensing hardware, philosophy, and project & organization & military managements are also discussed.
11
13
Introduction
16 Necessity the mother of invention, claimed Plato [ Deficient in rigorous scientific scrutinization as the statement is, major methodology evolutions in software development did not emerge until the emergence major computer innovations and thereafter elevated effort orchestration needs.
17
18
19 In the 1940s, digital programmable electronic computers revolutionized scientific calculation done previously with mechanical and analog computing machines (2). Assembly (1947)
20
3 4 and high level (1953) 5 6 programming languages rose to harness the unprecedented and ever-increasing
21
22
computing power, Eventually the term software was coined (1953) 77. Two Software Development Methodologies (SDMs) were proposed: 1. project breakdown of sequential phases, the essence of Waterfall (see Fig la), first presented no later than 1956 8 and 2. iterative and incremental SDM,
23
24
25 the essence of Agile (see Fib lb), first executed no later than 1957 19]
In the 1960s, operating systems (1962) emerged to orchestrate multiple computation tasks(10]. This
27 signified the shift of computer development from single-task specialized machines for military and
28 academia machines accessible the general public. The shift exemplified by The Mother of
29 All Demos (1968) which demonstrated many fundamental elements of personal computing for the first time (TT) and showed how software had evolved in both diversity and complexity. Meanwhile,
31 the first formal detailed diagram of the Waterfall methodology appeared in literature (See Fig 1a)
32 (1970) 212 and the name of Waterfall was ultimately coined (1976) [3]. Agile variants such as
33 evolutionary project management [14] and adaptive SDM [1] appeared in the early 1970s, although
34 no clear preference between Waterfall and Agile variants was found in the literature.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Table 1: Major Software Usage Evolutions and Methodology Developments
Period
Technology
Software Usage
Major Methodology Development
1940s
Programing Language
Scientific Calculation
First Waterfall variant presentation (1956) 8 first
Agile variant execution (1957) 19).
First detailed diagram of Waterfall idea [2), Waterfall name (1976)[3);; Agile variants: evolu- tionary project management [1] adaptive SDM [eally 1970s).
1960s
Operating System
Shifting to Applications
35
Waterfall standardized in military (1985) The Manifesto signed (2001) 224). Agile significantly more popular than Waterfall.
1980s
GUT & Internet
PC & Internet Applications
Knowledge Dis- covery, Prediction and Management
Around 1990
Data Storage
NA
Requirements
Testing
Coding
Overall
Collaboration
DM
Analysis
Data
Deploy
Design
Knowledge Discovery
Design
Coding
DM
Review
Analysis
-Algo Production
Feedback Loop
ML
Testing
Management Tool KM Standardization
Depley
Requirements
Fig a) A typical Waterfall design diagram [12] b) An iterative, evolutionary incremental design cyele commonly viewed as Agile[9) and how DM can improve each design rycle by discoveriug Knowledge about user needs during Review.
Fig Relations among DM, ML. and KM their pain points
36
37 In the 1980s, personal computers entered households 16 followed by graphic user interface (GUT) (1983) [16]. The Internet Protocol Suite (TCP/IP) was standardized (1982) [7) and commercial
38
39
Internet service providers emerged (1989) [1]]19] user-user communications created tremendous software needs, while Waterfall was still widely
[9] Unprecedented user-computer interactions and
40
41
deployed in software development. United States Department of Defense issued a military standard describing Waterfall as the required military software development process (1985) [20] However, software user needs grew so fast that, the heavy Waterfall SDM failed to deliver in pace. Consequently, a number of light weight SDMs were proposed and practiced (1990s) 21 22 23), Eventually The
42
43
44
Manifesto for Agile Software Development (The Manifesto) 24 was signed by 17 practitioners of
46
weight SDM (2001) and became the facto SDM.
47 Around 1990, data storage capacities grew significantly and software usages in Data Mining (DM, defined knowledge discovery from data) reached the tipping point. While data and software's storage manners differ between Von Neumann and Harvard architectures, data storage capacity growth empowered software to discover knowledge supported by scientific evidence (defined as Knowledge) that people never had access to. Corporations started to analyze customers' behavior and make business decisions based on Knowledge (1990s) (55] The first DM methodology, Cross-Indusse Standard Process for Data Mining (CRISP-DI DDM) was conceived (1996) 26 271. However, CRISP.
48
49
50
51
52
53
54 DM and its variants appear more of a theoretical framework, offer little meaningful or actionable guidance, and therefore have not gotten much traction. In addition, CRISP-DM is concerned only
55
56 with DM, not Machine Learning (MI., defined as deliver an algorithm (Algo) for Knowledge prediction) or Knowledge Management (KM) in general for science, medicine, military and on. 58 Due to the absence of alternative methodologies(sec Table 1), Agile is still being offered up for DM, ML, and KM [28 29 with questions being asked about its appropriateness 30
60 This paper discusses limitations in Agile as a scientific claim and why may not address the current
61 pain points of DM, ML and KM, which are later summarized. Evident along with Evident Knowledge
82 Base (EKB) proposed as project development and artifact management methodology. Evident's potential in alleviating many current pain points is demonstrated conceptually. Unalleviated pain

---

64
points and future work to fulfill the potential are also discussed. Beyond software development, Evident is illustrated to be applicable in many aspects of society. EKB is demonstrated as one potential infrastructure to store information as Knowledge, granular level above data.
65
66
2
Agile ambiguity and unfasifiability
Most people regard Agile as iterative, evolutionary and incremental software development [9] (sce Fig 1b) and many claim to be Agile practitioners. However, Agile empirical evidence is mixed and 70 hard to find 32 30] while no measurable scientific evidence has been found at all. Although control experiment challenges
68
71
absence of quantitative project Agility measurements may explain no measurable scientific evidence, concerns remain with the ambiguity with which Agile's 's approaches and scope are defined in The Manifesto.
72
2.1 Agile approaches are vaguely defined in The Manifesto
75 The Manifesto includes values and 12 principles [24). The goal is crystal clear: to rapidly deliver quality software that meets user needs, but not so much can be found for how to get there. Most of the Values and Principles appear to be goals but not approaches (see Appendix); some are concerned with approaches but vaguely defined; only four principles actionable, which turn out to have relevance in how to implement iterative, evolutionary or incremental development. Agile Alliance, co-founded by some original signers of The Manifesto, defines Agile "an umbrella term for set
77
79
81 of frameworks and practices" from which Agile practitioners "figure out the right things to do given your particular context." " 33 Unfortunately, no actionable approaches are defined.
82
83
Therefore although there are numerous frameworks under the Agile umbrella [34 (35, it's impossible to determine if development practice is Agile and the claim of Agile practice becomes unfalsifiable. Because falsifiability is the standard evaluating scientific against non-scientific claims introduced by Karl Popper (36), Agile is not a scientific claim. Consequently, no observable scientific evidence can 87 prove or disprove Agile, because technically no one can determine if project is Agile or not in the first place. If Agile rollout "fails", Agile proponents always argue that the Agile rollout was not implemented correctly.
85
89
Meanwhile Agile practitioners cannot determine if they practice Agile correctly either. Projects 91 employing Agile frameworks such as Test Driven Development or Feature Driven Development may not even realize that the projects may not be adaptive to new user needs. People who are essentially practicing Waterfall may believe they are practicing Agile only because they implement each sequential Waterfall phase incrementally or simply use Scrum or Kanban.
93
95 In defense, some proponents claim Agile as a philosophy 37 38
Granted Agile's goal may fit into Axiology, one of Philosophy's 's four domains (the rest as Metaphysics, Epistemology and Logic) [39] concerned with what is good, it appears to be common understanding and offers little value when Agile's approaches are vaguely defined.
96
97
98
99
2.2
No scopes are defined in The Manifesto
100
"The right things do given your particular context" by Agile Alliance [33] are expected to be found within Agile's 's frameworks, otherwise Agile is not practiced right. With no scope defined, Agile
101
102
seems cover the scope of all softwares. However, some softwares have non-ineremental needs or
103
simply only one need, e.g. to solve one specific partial differential equation numerically. Their needs are either met or not all. No iteration or evolutionary Agile design cycles exist.
104
105
Agile is also not applicable for Knowledge discovery tasks such as DM. In a typical Agile development cycle (Fig 1b), Review phase is discover Knowledge about user needs, which can be done through DM. Therefore Agile should not be applicable to DM, one phase of its own design cycle. Table 2: Pain Points of DM, ML and KM.
106
107
Pain Points for DM
Struggles deliver fast with technical debt
abc
Project Progress not easily measurable
b
Uncertainty in project timeline
b
Overall
Activities not easily trackable or reproducible
bc
3

---

Not scalable in terms of both collaborator number and project maintenance
ab
Few general project design patterns
abc
Anti-patterns not uncommon
abc
No methodologies to orchestrate team of size commonly seen in software development
Collabor- ation
Few intuitive manners to divide task among team members
ab
Deficient common awareness needs for process improvement
Tasks completed or ideas explored by team members cannot be easily found and reproduced causing duplicated work.
bc
Data compromised in availability, accuracy and consistency during acquisition
X
Data preprocessing process not standardized such as data labeling. object detection (e.g. identify object pixels in images) causing unstable data depen- dency, cascade correction and uncertainty in project progress
Data
Underutilized data may take unnecessary resource
b
Data may presented different data type such as integer, float or string, causing unnecessary data dependency for Algo and experiments
Off-the-sheli models are available for DM automation. However, DM au- tomation has not become common practice.
abc
Inefficient in-house model code implementation not uncommon
b
No appropriate version control tools. Current version control tools such as git are designed to only keep the best version Algo/experiment available, while
abc
DM and ML need multiple versions available concurrently for reference
Knowledge
No easy solution to request leexible data storage, memory and computation capacity as needed. Hard drive, RAM, CPU and GPU are difficult allocate even on the cloud.
x
Discovery
ML Algo
Research
The use of other Algos' output as imput results in correction cascades Algo is a sequential computation process different from typical software applications with number of independent features. Difficult to assign one Algo development into multiple team members
X
Algo user may have no clear understanding about the Algo and deploys outside its scope.
Multiple programing language smell
x
Pain Points for ML
Significant efforts of research Algo migration into production
bc
Even more sigmificant efforts if production Algo is written in different
X
language than the language used in research, e.g. in embedded system For Algo analyzing sensor data such as cameras or bio-sensors, product grade data won't t be available for Algo research until sensor hardware designs are complete. Algo becomes the product release bottleneck, resulting in either sub-optimal production Algo or delayed product release.
x
Algo Production
Production data source is inconsistent with research data source
X
Algo production is often done by team members, most likely software engi- neers, who did not produce the Algo, causing misuse
abc
Algo needs to load data in real time during production but most often not in real time during research, causing unnecessary Algo code re- factoring Prototype Algo may be accidentally run in production causing damages
b
No straightforward way to organize codes repository for research and produc- tion team members work in the same repository
abc
Dead code path
b
Algo update workflow not straight forward
b
Feedback
Few clear pattern designs for monitoring Algo performance in production
b
Loop
Actions based on unseen data predicted by Algo may alter observed data
X
Pain Points for KM
Few tools or resource help people check Knowledge formed is well sup- ported by evidence, especially when evidence appears long after presumed
abc
Manage- ment Tool
Knowledge has been formed.
Knowledge dissemination among community has always been a challenge. Knowledges formed by different organizations are not easy to combine
abc
abc

---

Knowledge formed within organizations is not easy to share and retain. Knowledge has been recorded in sentences or articles. Few standardized ways abc
abc
Standardi- zation
to represent general Knowledge. a/b/c: Pain points that can be alleviated by Evident's character a, b or c. x: Pain points that cannot be alleviated by Evident.
3 Pain points for DM, ML and KM
109
Owing to the absence of applicable methodologies, pain points have been continuously reported for DM, ML and KM [40 41 42 43 (see Table 2) in the current big data era with explosive growth in data volume, variety and velocity.
110
111
112
DM & ML's Algo typically comprises a data computation flow (defined as a Model, supervised or unsupervised), such logistic regression, and its configuration, such logistic regression coefficients. DM employs off-the-shelf or in-house Models to discover Knowledge from data. ML compares Knowledges discovered by DM by candidate models and deploys the one that performs best with its configuration, as the production Algo, for Knowledge prediction in production. Therefore, DM's pain points still apply to ML. Meanwhile because DM and ML are special forms of KM, their pain points are also applicable for KM (see Fig 2).
113
114
115
116
117
118
Generally speaking, DM has not been regarded highly collaborative and scalable activities to deliver high throughput Knowledge. It's rare to see hundreds of contributors in a DM project, unlike for example some complicated open source software projects that, deliver promptly, efficiently and continuously for years or even decades 44 45). is also rare of mass Knowledge production in organized and standardized manner with high production yield for unit period, commonly seen in consumer products such as automobiles or toothpastes. DM often struggles to deliver Knowledge rapidly with technical debts in reproducibility, measurability, trackability. DM needs to not only handle artifacts of different modalities such as documents, codes and data, but also address computation and data storage resource requests potentially across multiple platforms. Raw or preprocessed data can be compromised in availability, accuracy and consistency. Data dependency and entangled models often cause cascaded correction and uncertainty in project planning. In addition, routine tasks such as quarterly or annual finance analysis mostly have not be automated. Tools and project management methodologies are highly in demand fulfill DM's potential and deliver values.
120
122
123
124
125
126
127
128
129
130
131
132 In ML, the team members, usually software engineers or product managers, that deploy an Algo in production to predict future data may not have produced the Algo and may misuse it. The Algo codes are often refactored sometimes in different programming languages, operating systems or even in fixed point instead of float point. If the Algo is be deployed on data acquisition product such as cameras or bio-sensors, no product grade data are available for Algo research until the sensor hardware design is finalized enable data collection. Algo research therefore becomes the bottleneck for product release. Frequently sub-optimal Algo is deployed to meet the deadline or projects become delayed. What's more, production data may come from different sources compared research data potentially caused by, e.g. sensor upgrade or downgrade, resulting in the under performance of production Algo. After Algo deployment, no straightforward way exists to monitor Algo performance or thereafter update Algo. Future Knowledge predicated by the deployed Algo may encourage Algo users to alter decisions, which leads to the formation of future data with unanticipated hidden feedback loops. In addition, prototype Algo or dead codes may be accidentally run in production potentially causing catastrophic consequences.
133
134
135
136
137
138
139
140
141
142
143
145
146
Al Although scientific methods have guided people to discover Knowledge and improve practices such as baseed medicine [46) and experiment based military development (47), people still form Knowledge that lacks in supporting evidence [48]. One possible reason is the unavailability of tools or resources to check if the Knowledge formed is well supported by evidence, especially when there is a significant time gap between the Knowledge formed and the appearance of supporting evidence, e.g. for long-term investments or corporation strategies. Knowledge dissemination and retention are also huge challenges among the community and organization [49]. Furthermore, Knowledge is mostly recorded in the form of articles. However, because articles writing has not been and probably
147
148
149
150
151
152
153
154
will never be standardized, Knowledge has not been able to be represented in standardized manner for definition, reference and storage. The same Knowledge recorded different sentences or even languages may be interpreted differently.
155
5

---

Evident: a project development and artifact management methodology
157
158
4.1
Definition and scope
159
Evident is methodology of project, including but not limited to software, development and artifact management for DM, ML and KM, characterized by
160
project development mimicking continuous process of logical reasoning in philosophy;
162
b.
project activities or artifacts are broken into containers of Observations, Hypotheses and Tests (collectively defined as Containers);
directional association constructions towards and only towards Test Containers represent Knowledge.
165
Observation is collection of facts. A Hypothesis is Knowledge be formed out of Observation. A Test a Hypothesis evaluation process using Observation to prove or disprove the Hypothesis with without confidence levels. Containers indicate Observations, Hypotheses and Tests only be added or removed as a block.
167
189
Ubs
Obs 3
Hypo 1
Obs
Ois 3
Obs 2
Iiypo
Test
Hypo
Tesl _1_1
Hypo
Test 21_2
Indduction Krowledge
Hlypo 3
Text 31
Test
Tea 2 3
Hlypo
Test 23
Abduction Knowledge
diffferont Indretion Knowlodoos brased on the sume gair of Tlyo
Deduetion Knewledge from Induction Knowlodge
Peduction Knowledge Fror. Abdacion Knowledgal
1x snn Cha
far
(h)
Fig. Induction, alxiuction and deduction Knnwledgge re reoresenned in Evidlont (Hypo: Hypothesis: Ohe Ohservation)
Model
Datan
Data
Model
Model
Experiment
Model
Research Experiment
Prediction Experiment
DM
ML. Receurch
ML Production
(ar
Fig 1DM, Ml. processes represented in Evidew a) 1MM mimic Knowledge induction in Fig 3a; 5)
ML. Reseurch and Production mimic Knowledge abduction and Knowledge dedluction in Fig 3b:
170
4.2
Knowledge representation
172
Loosely speaking, Test associated with a Hypothesis and an Observation represents induction Knowledge (see Fig 3a); Test associated with a Hypothesis set and an Observation represents abduction Knowledge (see Fig 3b), Hypothesis associated Test that is also associated with an induction abduction Knowledge Test represents deduction Knowledge or prediction (see Fig 3a&b). Deduction Knowledge becomes induction Knowledge once Observation proving or disproving deduction Knowledge is associated with Test, while stay deducted Knowledge if the associated Observation overlooks (fails to either prove disprove), the deduction Knowledge. Multiple Tests associated with the same pair of Hypothesis(es)) and Observation represent multiple different Knowledges based different evaluation metrics (c.g. profit maximization or cost minimization) or Observation usage strategies (e.g. cross-valin cross-validation grouping) (see Fig 3c).
173
175
177
179
181
182
DM may be regarded as Knowledge induction with data as Observation, model as Hypothesis to be evaluated and data analysis experiment as model test on data to form induction Knowledge with statistical confidence (see Fig 4a). Similarly, MI. is Knowledge abduction. An example is data analysis experiment that picks the best off-the-shelf or in-house model that best explains the data to form the Algo for production (see Fig 4b). Experiments employing different cost functions (c.g.
185
185

---

187
RMSE, AUC or correlation coefficients), statistical confidence levels or data allocation strategies for training and testing may result different Knowledges or Algos.
188
189
4.3
Project development
180
Evident project developments are intuitively broken down into two granular levels: Knowledges and Containers. Mimicking logical reasoning philosophy, each project period develops batch of independent Knowledges or Containers assigned to teams of various sizes to maximize unit time throughput (see Fig 5). The next batches of Knowledge Containers can be adaptively planned after period reviews or retrieved from backlogs. Evident is compatible with Kanban, Scrum or other development tools or frameworks for project planning and development of single Knowledge or
191
192
193
195
196
Container. tools or frameworks that do not compromise the project breakdown into Knowledge
197
and Containers are applicable.
Plam 0
Plam
Plan 2
Ols
Obs 2
Obs 3
Team Knowledge 11
Hypo 2
member
Hypo
Test 11
Test 12
TBD
Knowlodge 11a
Hypo
Text
Team 3
Hypo 2
Hypo
Obs
Hypo 2
Test
Test 22 1
Test 23
members
Test
Test 31
Teat 21
Ilypo
Test
Toxt 32
Tost
Poriod
Period 2
Poried 3
Fig. One exemplury lividess project development process produce Knowlodges Fig 3 a&l&ke Ciroled tasks take 1 team member 3 periods, or take a team of members pariod. Knowledge 11 ropresents Obs Hypo & Tost Fig 3a.
Fig. An exemplary EKB composed Hypo, Obs
Test containers that store induction. abduction and deduction Knowledges in Fig. 3 a&b&c and project dovelopment artifacts Fig continnuusly
188
199
4.4 Artifact management: EKB
Evident artifact management may build topology of relational Knowledge base, named EKB, composed of Evident Containers and directional associations (see Fig 6). Knowledges can be reproduced by Containers stored in EKB. Oversimplified
201
202
table, EKB columns represent Observations; rows represent Hypotheses; values represent Tests or Test to be done (TBD). A Test can only be associated with Observation, even if the associated Observation overlooks the Hypothesis(es) associated with the Test. Any new Observation proving, disproving or overlooking the same Hypothesis(es) occupies column EKB.
203
205
208
207
4.4.1
EKB stores containers and Knowledges continuously
208
When new Hypotheses, Observations or deduction Knowledge Tests are developed, new rows or columns of TBDs are inserted. A Test associated with Hypothesis(es) and a Observation can be stored in the designated row and column to represent different Knowledges.
209
210
An induction Knowledge Test is placed in the row of its associated Hypothesis and the column of its associated Observation (see Fig 3a&c & Fig 6). An abduction Knowledge Test is placed in the row of the Hypothesis best explaining the Observation (see Fig 3b & Fig 6). A deduction Knowledge is placed in the row of its associated Hypothesis and the column of the pending Observation (see Fig 3a&b & Fig 6). Once an Observation becomes available proving or disproving the Hypothesis, a deduction Knowledge becomes an induction Knowledge. Multiple Tests representing different
212
214
215
216
217
Knowledges can be placed in the same slot (see Fig 3c & Fig 6).
218
4.4.2
EKB supports relational database operations of Permutation And Join
219
EKB is similar to a relational database 50 with Observations as columns, Hypotheses rows and Tests as values, but with potential associations among values for deduction Knowledges. Relational database operations independent of values associations, such as Permutation (switching rows and columns) and Join (merging EKBs) can be implemented without compromise EKB; operations dependent on values associations such as Restriction (select rows), Projection (select columns) and Compositions (merge selected columns&rows from multiple EKBs) can only be implemented for EKBs storing only induction or abduction Knowledge and have associations among Tests.
220
222
224
7

---

228
EKBs are highly flexible for team collaboration and maintenance. Different team members working on different Containers can share one EKB as the common work space to improve efficiency. Multiple EKBs can be joined together without information loss so that Knowledges produced by different Advantages and pain points alleviated
227
229
teams or team members can be accumulated into one EKB. For EKBS storing only induction and abduction Knowledge, all relational database operations are applicable, so that team members can compose their own EKBs without keeping a potentially large team EKB on the local machines.
230
231
5
233
Inspired by the philosophy of logical reasoning, Evident is intuitive to understand and follow. Project activity and artifact containerization supports incremental as well as adaptive project planning and artifact pattern abstraction. Disentangling Hypotheses and Observations reduces unnecessary de- pendency, cascade correction and uncertainty in project planning. Knowledge representation in associations among artifacts can not only track Knowledge development, but also Knowledge devel- opment status (prove, disproved or overlooked), which improves project measurability, trackability and reproducibility. Overall Evident may help applicable projects deliver fast and at scale with many pain points alleviated in DM, ML and KM (see Table 2).
234
235
237
239
5.1 DM
Containerized Data and Models in Evident prevents unstable data dependency, model entanglement and cascade correction. Dead data and codes can be easily identified and removed. Standardized Models and Experiments encourage reuse of computationally efficient containers, support automatic DM. Different experiments may use different optimization target function on the same Model and Data to deliver different Knowledges for different users, e.g. Marketing vs Engineering managers.
243
245
247
Containerization is an alternative to the state of the art artifact version control, such as git, which keeps only the one version of the code or data in the workspace with historic versions saved as commits. Evident keeps all applicable versions available in the workspace for easy access. This may appear to use more storage space. However current version control tools all save version commits as snapshots [5I] demanding comparable storage space of Evident if a Evident equivalent number of versions are stored.
249
251
252
253
Evident granulates project activities into independent standardized Knowledge and Container levels, supports adaptive development, facilitates project planning among collaborators in teams of various sizes and reduces planning overhead. Artifacts are continuously stored in EKB, making project development measurable, trackable, reproducible and scalable. Meanwhile once Containers are produced, Knowledge or documentation reports can be generated automatically instead of manually. Evident accelerates DM delivery in both short-term and long-term.
254
255
256
257
5.2 MI.
A deployed Algo can be evaluated easily by re-applying the original Research Experiment on the production data. Different users involved in the deployment can understand the Algo's scope and origins easily by examining the research Experiment (see Fig 4b). Research Experiments can load data in real time as Production Experiment, so that both Experiments can inherit the same design patterns with statistical amalysis and evaluation metrics. The Production Experiment can report and examine the prediction performance at regular time intervals to detect production data pattern drift for either model reconfiguration or model replacement. Once model with its configuration is retired from production, the production data is containerized and associated with the Production Experiment, transforming the Production Experiment into Research Experiment and a deduction Knowledge for prediction into an induction Knowledge that is also preserved in EKB.
251
263
265
267
269
Because both research and production can operate on the same EKB, research and production team members can share the same workspace the way software engineers work on the same code repository, facilitating the model migration from research to production and efficient team collaborations.
271
8

---

273
5.3
KM
274
Knowledge formatting into design patterss of Containers provides a meaningful progress towards Knowledge standardization for improved definition, reference and storage compared to state of art sentences or articles. EKB with standardized Container templates may ofter potential tools for people
275
278
to examine the Hypotheses formed against evidence or Observations, facilitating sidence-bbased decision making and Knowledge development. EKB can not only facilitate Knowledge dissemination, accumulation and retention, but also label the development status of each Hypothesis proved, disproved or overlooked, a desirable design pattern for projects and Knowledge Management.
278
280
Discussions
281
282
6.1
Significance
Evident may advance many society domains such software, philosophy, science, business well as Knowledge sharing and retention across the globe, thanks to its applicability to general KMM. 284
EKB may make smaller impacts than relational data base [50), the invention of which created data base industry, as one solution to store information as Knowledge, a granular level above data.
286
287
6.2
Work do
288
Work needs to be done regarding Evident ergodicity over logical reasoning in philosophy. If proved, Evident can support all logical reasoning in philosophy. No evidence has existed prove disprove the claim. Evident ergodicity is overlooked, stated in Evident language, especially considering logical reasoning in philosophy may evolve.
289
290
291
292
Control studies need to be done to show Evident can truly provide value. No tools tailored to support Evident project development planning and EKB are available, although some existing tools applicable for use. Particularly the tools that allow unexpected alteration proof, easy access and visualization of Containers are in demand. More detailed discussions need to be done about how Evident help applicable projects with examples.
294
296
297
6.3
Pain points not alleviated
Evident offers no detailed guidance in development below Container level. For example, a Model or computation flow cannot be broken down further into smaller modules by Evident for incremental and adaptive development. Multiple languages smells and accidents running prototype Algo in production cannot be avoided by Evident either. In addition, Evident cannot control future observation alteration caused by decisions made by people based on Evident produced Knowledge.
299
300
301
303
Evident can only manage project artifacts of data codes but sensors or hardwares. Pain points caused in data acquisition such as availability, inaccuracy and inconsistency are out of Evident's scope. Evident is incapable of improving computation hardware resources allocation either.
304
305
6.4
More words about Agile
307
Due to Agile's ambiguity and unfalsifiability as a scientific claim, it might be a better practice to drop the term Agile and instead quote each framework currently under Agile on its own. Frameworks such as iterative and evolutionary development as well as Kanban are valuable although need to be employed discretionally. Practitioners should have better understood what exactly they were doing without being fuzzed by the buzzword Agile.
309
311
Conclusions
312
313
The paper proposes Evident as project development and artifact management methodology for DM, ML. and KM as well as EKB as a Knowledge base topology. Evident and EKB have been shown of great value to alleviate many unresolved pain points. Evident has the potential to facilitate the advancement of many aspects of society due to its utility in general Knowledge management, EKB may serve the infrastructure for storing information Knowledge, granular level above data.
314
316

---

318
A
Appendix
319
A.1
The following among the four Values and twelve Principles of The Manifesto 24 appear to be goals:
320
321
Value 4: Responding to change over following a plan;
322
Principle 1: Customer satisfaction by early and continuous delivery of valuable software.
Principle 2: Welcome changing requirements, even in late development.
324
Principle 3: Deliver working software frequently (weeks rather than months)
325
Principle 7: Working software is the primary measure of progress
326
Principle 8: Sustainable development, able to maintain constant pace
Principle 9: Continuous attention technical excellence and good design
328
Principle 10: Simplicity-the art of maximizing the amount work not done-is essential
329
A.2
The following in The Manifesto appear to be approaches but vaguely defined:
Value 1:
Individuals and interactions over processes and tools
Value 2: Working software over comprehensive documentation
332
Value 3: Customer collaboration over contract negotiation
333
Principle 5: Projects are built around motivated individuals, who should be trusted
334
Principle 11: Best architectures, requirements, and designs emerge from self-organizing teams
A.3
The following in The Manifesto appear to be actionable approaches but irrelevant of iterative, evolutionary or incremental development regarded as Agile by most people [9]
336
Principle 4:
Close, daily cooperation between business people and developers
338
Principle 6: Face-to-face conversation the best form communication (co-location)
339
Principle 12: Regularly, the team reflects on how to become more effective, and adjusts accordingly
References
[1] B. Jowett and L. Campbell, Plato's 's Republic the Greek text. Oxford At the Clarendon Press, 1894.
342
343
[2] J. J. P. Eckert and J. W. Mauchly, "Electronic numerical integrator and computer," " New York, NY, 1947.
345
[3] A. D. Booth and K. H. Britten, "General considerations in the design of an all purpose electronic digital computer," Tech. Rep., 1947.
347
[4] M. Campbell-Kelly, "The development of computer programming in britain (1945 to 1955)." Annals of the History of Computing, vol. 4, no. 2, pp. 121-139, 1982.
348
[5] P. Bentley, Digitized: The science of computers and how it shapes our world. New York: Oxford University Press, 2012.
350
66] D. E. Knuth and L. T. Pardo, "Early development of programming languages," A History of Computing the Twentieth Century, 1980.
352
353
77] R. R. Carhart, A Survey of the Current Status of the Electronic Reliability Problem. Santa Monica, CA: RAND Corporation, 1953.
355
18
Symposium on advanced programming methods for digital computers: Washington, D.C., June 28, 29, 1956. Office of Naval Research, Dept. of the Navy, 1956.
10

---

357
[9] C. Larman and V. R. Basili, "Iterative and incremental development: A brief history," Computer,
358
vol. 36, pp. 47-56, 2003.
359
[10] S. H. Lavington, A History of Manchester Computers. British Computer Society, 1998.
[1]] T. Bardini, Bootstrapping: Douglas Engelbart, Coevolution, and the Origins of Personal Computing. Standford CA: Stanford University Press, 2000.
361
382
[12] W. W. Royce, "Managing the development of large software systems," in Technical Papers of Western Electronic Show and Convention, 1970.
383
364
[13] T. E. Bell and T. A. Thayer, "Software requirementss Are they really a problem?" in Proceedings of the 2nd International Conference on Software Engineering. Washington, DC, USA: IEEE Computer Society Press, 1976.
366
[14] T. Gilb, Software Metrics.
Winthrop Publishers, 1976.
[15] E. A. Edmonds, "A process for the development of software for nontechnical users adaptive system," General Systems, vol. 19, p. 215-18, 1974.
389
[16] W. Isaacson, The Innovators: How Group of Hackers, Geniuses, and Geeks Created the Digital Revolution. SIMON and SCHUSTER, 2014.
371
372
[17] J. L. Pelkey, "The history of computer communications." April 2022. [Online]. Available:
373
374
[18] R. Clarke, "Origins and nature of the internet australia," Emergence: Complexity and
375
Organization, vol. 4, pp. 1990-1994, 2004.
376
[19] R. H. Zakon, "Hobbes' internet timeline 25." April 2022. [Online]. Available:
https//wwwwwzzkooooooooooeeeetee
378
[20]
"Military standard: Defense system software development by department of defense," Jun 1985.
379
[21] J. M. Kerr and R. Hunter, Inside RAD: How to Build a Fully Functional System in 90 Days or Less. McGraw-Hill, 1993.
381
[22] R. Nagel and R. Dove, 21st Century Manufacturing Enterprise Strategy: An Industry-Led View. Diane Pub Co, 1991.
383
[23] A. Presley, J. Mills, and D. Liles, "Agile aerospace manufacturing." 1995.
384
[24]
M. Beedle, A. van Bennekum, and A. Cockburn, "Manifesto for agile software development," April 2001. [Online]. Available: https:/agiilmaaiffsttooo .org/
386
[25]
D. Power, "A brief history of decision support systems," April 2022. [Online]. Available:
387
[26]
R. Wirth and J. Hipp, "Crisp-dm: Towards standard process model for data mining." " Proceed- ings of the 4th International Conference on the Practical Applications of Knowledge Discovery and Data Mining, 2000.
389
390
391
[27]
IBM, "Crisp-dm help overview," April 2022. [Online]. Available: hutps///wwwibmm.comddcc. htw.://wwwe.ttta.aoioooidiihiiaa
393
[28]
Microsoft, "Agile development of data science projects." April 2022. [Online]. Available:
394
[29]
"Agile data science by data science process alliance," April 2022. [Online]. Available:
396
[30] I. Lee, "6 reasons why think agile data
science work." April 2022. [Online]. Available: htnps//owwrrdsaassceee..com
does not
398
t-reasons-why. thintthgiieee-e--eee data- science- ceseencc--oes--eeeee not- workwoocdo666060b55555

---

[31] E. Yan, "Data science and agile (what works, and what doesn't," April 2022. [Online]. Available: https//euggeneyan.co com/ www.wwooto///////oooooo/oo/o///e writing/data- science- and and-agile-wee soele--hat-veee works and what-doesnt/
[32] T. Dyba and T. Dingsøyr, "Empirical studies of agile software development: A systematic review," Information and Software Technology, vol. 50, no. 9, pp. 833-859, 2008.
403
333
"Agile 101 by agile alliance," April 2022. [Online]. Available: https:/www.agiialliacce..ggg a01le1017
405
[34] P. Abrahamsson, O. Salo, J. Ronkainen, and J. Warsta, "Agile software development methods: Review and analysis." 2017.
407
[35] A. L. Fruhling and A. E. Tarrell, "Best practices for implementing agile methods: A guide for department of defense software developers," Information Systems and Quantitative Analysis Faculty Publications, vol. 27, 2007.
409
411
[36] K. Popper, The Logic of Scientific Discovery. London and New York: Routledge, 2002.
[37] S. Braams, "The software development landscape: A rationalization of agile software development as a strategy in the face of organizational complexity," April 2022. [Online]. Available: http://esay.uuwettenn/8//77
413
415
[38] S. Forum, "Agile
methodology or
framework or philosophy."
April 2022. [Online]. Available:
417
418
[39] Schick, Doing Philosophy: An Introduction Through Thought Experiments. Megraw-Hill,
419
2009.
420
[40] D. Sculley, G. Holt, D. Golovin, Davydov, Phillips, D. Ebner, Chaudhary, M. Young, J.-F. Crespo, and D. Dennison, "Hidden technical debt in machine learning systems," in Advances in Neural Information Processing Systems, vol. 28. Curran Associates, Inc., 2015.
422
[41] H. H. Nithya Sambasivan, Shivani Kapania, ""everyone wants to do the model work, not the data work": Data cascades high-stakes ai," 2021.
424
425
[42] T. J. Sebastian Schelter, Felix Biessmann, "On challenges in machine learning model manage- ment," IEEE Data Eng. Bull., 2018
426
427
[43] F. Kumeno, "Sofware engneering challenges for machine learning applications: A literature review," Intelligent Decision Technologies, vol. 13, no. pp., vol. 13, no. pp. 463 463-476, 2019.
428
[44]
"Python," April 2022. [Online]. Available: https://www.pyhhon.org
430
[45] "Ubuntu," April 2022. [Online]. Available: https://ubuntu.com/
[46]
D. L. Sackett, "Evidence-based medicine," Seminars in Perinatology, vol. 21, no. 1, pp. 3-5,
432
1997.
[47] S. Spear and T. Hone, "Succeeding in periods of change," Proceedings U.S. Naval Institute, March 2022.
434
[48] LessWrong, "Welcome to lesswrong!" April 2022. [Online]. Available: wwwp./eswwoog.coo//sssshhhhhaals hwtp://wwwwi.aaaaaaaaaaaaaaaaaae WaDs/wellome--ne
https:
436
[49] E. M. Rogers, Diffusion of innovations, 5th ed. New York, NY: Free Press, 2003.
438
[50]
E. F. Codd, "A relational model of data for large shared data banks," Commun. ACM, vol. 13, 6, p. 377-38777 1970.
440
[51] Git, "Git internals git objects," April 2022. [Online]. Available: hutps//boook.iiiscc.coom
12

---

Checklist
442
443
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] - You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
444
445
446
447
Did you include the license to the code and datasets? [Yes] See Section ??.
448
Did you include the license the code and datasets? [No] The code and the data are proprietary.
449
450
Did you include the license to the code and datasets? [N/A]
451
Please do not modify the questions and only use the provided macros for your answers. Note that the
452
Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
453
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
496
457
(b) Did you describe the limitations of your work? [Yes]
458
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
460
461
2. If you are including theoretical results...
462
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
463
(b) Did you include complete proofs of all theoretical results? [Yes]
464
3. If you ran experiments..
465
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [N/A]
466
487
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
468
469
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
470
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
472
473
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
475
(b) Did you mention the license of the assets? [N/A]
476
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
478
(d) Did you discuss whether and how consent was obtained from people whose data you'l rre using/curating? [N/A]
479
480
(e) Did you discuss whether the data you are using/curating contains personally identifiable
481
information or offensive content? [N/A]
482
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
484
485
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
486
487
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
488
13


Reviews:
[{"review_id": "c1s8TX0Xzbn", "reviewer": "Reviewer_P1e5", "summary": "The paper proposes a new software development methodology (Evident) for machine learning, data mining, and knowledge management. It discusses pitfalls of traditional software processes related to the waterfall and agile models and explains why they may not be sufficient for these fields. Further, the paper describes the main concepts related to Evident. The main idea centers around modeling the relationships between observations, hypotheses, and tests. Finally, the paper describes how this methodology may help to alleviate common pain points in ML, DM, and knowledge management.", "questions": "- Has this process been applied in practice? Are there any insights to share to this end?\n\n- For future versions of the paper it may be useful to be more specific and highlight for which of the challenges in Table 2, Evident may work better than other methodologies and why. ", "limitations": "I do not think the authors have really described or mentioned the limitations of their approach. In contrary, the contributions are sometimes hyperbolically described claiming that the process can advance society, philosophy, and science, without any factual evidence.", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nS1- The paper aims to make a contribution to the important problem of improving development processes for ML and more. \n\nWeaknesses:\n\nW1- It is difficult to judge the effectiveness of the contribution without any user studies with development teams that have applied Evident. At the very least, it would have been useful to see a few case studies and qualitative remarks on how teams could apply this in practice. This would instantiate the presented terminology (knowledges, containers, hypotheses, observations etc) with concrete tasks and terms of a given project.\n\nW2- Often the paper makes strong claim regarding the ineffectiveness of Agile (or other processes like CRISP-DM) and the effectiveness of of Evident but the justifications are mostly abstract and sometimes handwaving potential benefits without a proper either theoretical or rigorous framework to prove these claims. \n\nW3- It is not clear how the authors distilled the presented challenges in Table 2 from related work. How did they decide to include or leave a challenge out of the table? Plus, there is a lot more work on methods and processes for ML by now that could also be considered. Some of the challenges are also very generic and they may as well apply to any software that may not include ML (e.g. dead code paths, Prototype Algo may be accidentally run in production causing damages).\n\nLiterature on challenges in ML software development:\n\nSoftware Engineering for Machine Learning: A Case Study\n\nSoftware 2.0 https://karpathy.medium.com/software-2-0-a64152b37c35\n\nMachine learning testing: Survey, landscapes and horizons\n\n“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI\n\nAn empirical study of common challenges in developing deep learning applications", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KVkjLn5bEsL", "reviewer": "Reviewer_FzW8", "summary": "This paper proposes a project management/artifact management framework (Evident) in the form of a knowledge base topology. Throughout the paper, the authors discuss about the challenges presently faced by the AI community - including in knowledge representation and utilisation of traditional relational databases for ML/AI workflows. The paper is interesting and the problem being solved is relevant.", "questions": "This reviewer believes that the paper is very theoretical and oriented more towards the \"project management\" audience in comparison to the substance which an AI audience would be interested in. It would have been useful if this paper had more interaction with AI - beyond the very common pitfalls (\"pain points\") that the paper describes which are mostly well known in terms of consensus in the community. Also, without any user-studies or real-world use of this framework, the value which Evident can deliver is very hypothetically presented in the paper which makes it less rigorous. This reviewer suggests that paper would be more suitable for a journal relevant to AI and project management etc. in its current form.", "limitations": "Mostly the limitations have been addressed. However, there is not much added value or substance which the paper describes beyond the \"Pain points not alleviated\" described in Section 6.3 - the hypothetical nature of the Evident framework makes the paper read more like an essay (being very verbose) which affects the overall contributions outlined in the paper. ", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weaknesses:\nVery limited novelty in terms of the contributions of the paper. While the framework and discussions surrounding Evident are interesting, they do not add much substance from an AI perspective and from the project management perspective, it is not very clear how and why the proposed workflow is the way to go for the AI community. While the authors have described \"pain points of DM, ML and KM\", it is very theoretical with limited examples, limited practical scenarios and above all, limited interrelation with the substance related to AI over project management.\n\nStrengths:\nInteresting topic being discussed in the paper, very well written paper.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Fo127SzUtYX", "reviewer": "Reviewer_Fty3", "summary": "The paper discusses problems (called pain points in the paper) in using the agile methodology for Data mining/Machine learning/knowledge management. It then presents a software development methodology for data mining tasks that can also be used for machine learning and knowledge management. Activities in a project are broken down into observations, hypotheses and tests that denote facts, knowledge and evaluation respectively. These serve as containers that can also used for storing and managing project related artefacts.\n ", "questions": "- The authors have discussed the agile methodology and problems in its application for software development and artefact management for dm/ml/km. However, why compare with agile only? It would be better to compare with other DM methodologies that have been proposed. For example, the paper talks about the CRISP-DM, but lists limitations of this methodology briefly. No other methodology is mentioned. \n\n- How have goals and approaches been differentiated? For example, how is Individuals and interactions over processes and tools an approach and Responding to change over following a plan a goal\n\n- The paper says only four principles are actionable, but lists only three\n\n- The fact that agile is unfalsifiable is mentioned, and treated as a drawback. However, it is not clear which development methodologies  (if any) are falsifiable\n\n- The paper says the scope of Agile is not defined. Again,  it would be good to mention whether all other methodologies have scope defined\n\n- How have the pain points in Table 2 been identified? What is the last column?\n\n- How do the  Fig 3 a), b) and c) represent induction, abduction and deduction? Some explanation is required\n\n- Were experiments not carried out using Evident? At least something concrete is required, perhaps a case study with examples. Otherwise, the claims about alleviation of pain points in section 5 are not justified.\n\n- The paper title mentions topology. Where is this discussed subsequently? \n\n- I am not sure I understand what is meant by the following:\n\n\"This paper discusses limitations in agile as a scientific claim\"\n\"Beyond software development, Evident is illustrated to be applicable in many aspects of society\"\n\"Although control experiment challenges or absence of quantitative project Agility measurements may explain no measurable scientific evidence,....\"\n\"DM has not been regarded highly collaborative and scalable activities to deliver high throughput Knowledge\"\n\"resulting in the under performance of production Algo\"\n\n", "limitations": "As far as sections are concerned, the authors have mentioned pain points not alleviated under section 6.3. ", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The paper identifies the need for having a software development methodology for DM/ML/KM\n- The paper presents a summary of pain points for DM/ML/KM \n\nWeaknesses:\n\n- The paper lacks scientific rigor. Although some of the ideas are interesting, there is no experimentation/evaluation to substantiate the proposed methodology\n- The paper does not present any literature review of the area, that raises questions about whether the work in the paper is indeed a novel contribution\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject