Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: The submission presents SimbaML, an open-source tool that generates realistic synthetic datasets from ordinary differential equation-based models to supplement real-world data in machine learning models. SimbaML allows for investigating transfer learning, data augmentation, identifying the need for data collection, and benchmarking physics-informed ML approaches. SimbaML provides diverse functionalities covering data pre-processing, training, and evaluation of ML models.
strengths_and_weaknesses: ## Strengths

- The paper addresses a relevant and important problem: the need for large and diverse datasets for training machine learning models.
- The proposed framework, SimbaML, seems to be a comprehensive solution that integrates mechanistic models and machine learning for dataset generation and analysis.
- The paper includes two use cases that demonstrate the potential of SimbaML in identifying needs for data collection and augmenting scarce real-world data.
- I appreciate that the authors have been abundantly clear throughout the paper and in the appendix. They clearly communicate the goals of SimbaML as well as contrast it with other frameworks and tools.
- The work provides high-quality reproducible code.

## Weaknesses

- The paper assumes that the mechanistic models used to simulate the dynamical behavior of various systems are accurate and unbiased. However, these models are constructed based on assumptions and simplifications of the underlying physics, and they may not fully capture the complexity of real-world systems. SimbaML provides no way to do so and does not study how this impacts the functionality of their work.
suggested_changes: 
Overall, the submission is well-structured, concise, and clearly presents the SimbaML framework for integrating prior knowledge of ODE models into the ML process by synthetic data augmentation. However, the authors may consider making the following changes to improve the submission:

- The authors may consider adding more details about the SimbaML framework's technical aspects, such as the data pre-processing, training, and evaluation pipelines, to provide a better understanding of how the framework works.
- In Figure 1B, the figure could be more descriptive to provide a better understanding of the size of the dataset as well as the dataset used for each experiment.
- I have made sure to not actively look for the authors of this work, however, the code repository includes a link to PyPI which unfortunately reveals the authors. I have made sure to not search up the names and since this does not seem intentional as per recommendation this will be ignored.
review_title: null

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: The SimbaML tool presented in this paper unifies realistic synthetic dataset generation from ordinary differential equation-based models and the direct analysis and inclusion in ML pipelines. It provides a solution for training sophisticated machine learning models that require large datasets that are difficult or expensive to collect for many applications.
strengths_and_weaknesses: Strengths:
1. SimbaML can supplement real-world data with prior knowledge about system dynamics, which can improve the accuracy and efficiency of machine learning models.
2. The tool is open-source and written in Python, making it accessible to a wide range of users.

Weaknesses:
1. SimbaML requires prior knowledge about system dynamics, which may not always be available or accurate.
2. The scalability of the tool may be limited by the computational resources required to generate synthetic datasets from ordinary differential equation-based models.
3. There is a risk of overfitting to synthetic data, which may not accurately represent real-world scenarios.


suggested_changes: 1. Expand the tool's capabilities beyond ordinary differential equation-based models to include other types of mechanistic models.
2. Address scalability concerns by optimizing the computational resources required to generate synthetic datasets and analyzing large datasets efficiently.
3. Further explore the potential for overfitting and develop methods to mitigate this risk, such as incorporating real-world data into the synthetic dataset generation process.
review_title: null

Meta-review: metareview: The submitted paper presents SimbaML, an open-source tool that generates realistic synthetic datasets from ordinary differential equation-based models to supplement real-world data in machine learning models. SimbaML offers diverse functionalities covering data pre-processing, training, and evaluation of ML models, and can investigate transfer learning, and data augmentation, identify the need for data collection, and benchmark physics-informed ML approaches. This toolkit can serve a wide of applications and have a potential impact on this area.

While the paper addresses an important and relevant problem, the paper assumes the accuracy and bias of the underlying physics in the models used to simulate dynamical behavior without studying the impact on the functionality of SimbaML and the risk of overfitting to synthetic data. The authors are encouraged to further investigate this issue.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Invite to present (notable)