please public prompt for per step that {data cleaning }  is placeholder and another section must be public that i must fill it .
# Final reviewed {data cleaning } Prompt

**Title:** Data Cleaning Phase

---

**Introduction:**

- **Data Cleaning** involves identifying and rectifying errors or inconsistencies to ensure high-quality data for analysis.
- Essential for enhancing the accuracy and reliability of machine learning models.

---

**What?**

- Focuses on detecting and resolving data quality issues such as missing values, outliers, and inconsistencies.
- Prepares the dataset for feature engineering and model training.

**So What?**

- High-quality data is fundamental for accurate machine learning predictions.
- Prevents flawed data from leading to unreliable models.

**Now What?**

- Move to **Feature Engineering** to create meaningful features from the cleaned data.
- Proceed to **Model Training** with the refined dataset.

---

**Key Activities & Objectives:**

- **Handling Missing Values:** Deletion, Imputation (mean, median, mode), KNN.
- **Addressing Outliers:** Removal, Transformation (logarithmic, Box-Cox).
- **Ensuring Data Consistency:** Standardizing formats, correcting typos.
- **Objectives:** Improve accuracy, ensure consistency, enhance completeness.

---

**Methods:**

- **Techniques:** Imputation, transformation, standardization.
- **Tools:** Pandas, NumPy, SciPy, OpenRefine.

---

**Challenges:**

- Data quality issues, handling high-dimensional data, automation of cleaning processes.

my outline:
{
# Refined Data Science Project Outline

1. **Problem Definition (Dependency: Foundation for the entire project)**
    - **1.1 Define the Type of Problem:** Clearly articulate whether the problem falls under classification, regression, clustering, anomaly detection, or another category. This helps in choosing appropriate algorithms and evaluation metrics.
    - **1.2. State Business Objectives:** Clearly outline the goals to be achieved by solving this problem using data science. This provides a clear direction and helps measure the project's success.
    - **1.3 Determine Project Data Requirements:** Identify the specific data points needed based on the problem definition. Consider what data is available, what needs to be collected, and the potential challenges in acquiring this data.
    - **1.4. Identify Data Sources:** Determine where the required data will be sourced from. This might involve databases, APIs, web scraping, or external datasets. Evaluate the accessibility and reliability of these sources.
    - **1.5. Evaluate Data Quality:** Assess the completeness, accuracy, and consistency of the identified data sources. This initial assessment helps anticipate potential data cleaning needs and informs decisions about data acquisition and preparation.
2. **Data Acquisition & Preparation (Dependency: Relies heavily on the 'Problem Definition' phase)**
    - **2.1. Data Collection:** Gather the data from the identified sources. Implement robust data ingestion techniques to handle different data formats and potential data quality issues.
    - **2.2. Data Exploration & Understanding:** Analyze the collected data to gain insights into its characteristics and identify potential challenges.
        - **2.2.1. Descriptive Statistics:** Compute descriptive statistics (mean, median, standard deviation, etc.) to understand the central tendencies and distributions of your variables.
        - **2.2.2. Data Visualization:** Generate visualizations (histograms, scatter plots, box plots, etc.) to visually explore data patterns, relationships, and potential outliers.
        - **2.2.3. Correlation Analysis:** Explore relationships between variables using correlation matrices or scatter plots to identify potential predictors and multicollinearity issues.
        - **2.2.4. Document Findings:** Record all observations and insights gained during data exploration. This documentation is crucial for guiding subsequent steps.
3. **Data Preprocessing (Dependency: Directly builds upon 'Data Acquisition & Preparation')**
    - **3.1. Data Cleaning:** Cleanse the data to address the issues identified during exploration.
        - **3.1.1 Handling Missing Values:** Employ appropriate strategies for handling missing data points (deletion, imputation). Consider the nature of missingness (Missing Completely at Random (MCAR), Missing at Random (MAR), Missing Not at Random (MNAR)) and its potential impact on the analysis.
        - **3.1.2. Addressing Outliers:** Handle outliers cautiously. Investigate their causes (data entry errors, natural variation) and decide on the appropriate action (removal, imputation, transformation, or using algorithms robust to outliers).
        - **3.1.3. Correcting Data Inconsistencies:** Standardize data formats, resolve inconsistencies in data values, and ensure data type compatibility for the chosen algorithms.
    - **3.2. Feature Engineering:** Engineer new features or transform existing ones to improve model performance and interpretability.
        - **3.2.1. Feature Transformation:** Convert categorical features into numerical representations (one-hot encoding, label encoding) suitable for machine learning algorithms. Transform numerical features (logarithmic transformation, standardization) to improve model performance, especially for algorithms sensitive to feature scales.
        - **3.2.2. Feature Creation:** Generate new features from existing ones through mathematical operations (interactions, polynomials), aggregations, or domain-specific knowledge. Carefully evaluate the relevance and potential impact of new features on model complexity and performance.
        - **3.2.3. Feature Selection:** Select the most relevant features to use in model training. Apply feature selection techniques (filter methods, wrapper methods, embedded methods) to reduce dimensionality, prevent overfitting, and improve model interpretability.
4. **Data Splitting (Dependency: Utilizes the preprocessed data. This phase is crucial for model evaluation and generalization)**
    - **4.1. Split Dataset:** Divide the preprocessed dataset into three distinct subsets: training, validation, and test sets.
        - **Training Set:** The largest portion of the data, used for training the machine learning model.
        - **Validation Set:** Held out from the training set, used to evaluate model performance during training and tune hyperparameters to prevent overfitting.
        - **Test Set:** Completely unseen data, used after model training and hyperparameter tuning, to provide an unbiased final assessment of the model's ability to generalize to new data.
    - **4.2. Determine Split Ratios:** Select suitable data splitting ratios (e.g., 80/20 for train/test or 70/15/15 for train/validation/test). The optimal ratio depends on the size of the dataset and the specific problem.
    - **4.3. Address Data Leakage:** Ensure no data from the validation or test sets leaks into the training set. Data leakage can lead to overly optimistic model performance estimates during training that do not reflect real-world performance.
5. **Problem-Solving Methods (Dependency: Relies on the prepared data from the previous steps. This phase involves the core machine learning aspects.)**
    - **5.1. Model Selection:** Choose appropriate machine learning algorithms based on the problem type (classification, regression, etc.), dataset characteristics, and business objectives.
        - **5.1.1. Consider a Diverse Set of Algorithms:** Explore a range of suitable algorithms (e.g., linear regression, logistic regression, support vector machines, decision trees, random forests, neural networks) to avoid bias towards any particular method.
    - **5.2. Model Training:** Train the selected machine learning models using the training data.
        - **5.2.1. Hyperparameter Tuning:** Experiment with different hyperparameter settings (e.g., learning rate, regularization strength, tree depth) using techniques like grid search, random search, or Bayesian optimization to find the optimal combination that yields the best performance on the validation set.
    - **5.3. Model Evaluation:** Evaluate the performance of the trained models using appropriate metrics based on the problem type (e.g., accuracy, precision, recall, F1-score for classification; mean squared error, R-squared for regression).
        - **5.3.1. Cross-Validation:** Apply techniques like k-fold cross-validation to obtain more robust and reliable performance estimates, especially when the dataset is limited.
    - **5.4. Model Comparison:** Compare the performance of different trained models based on the chosen evaluation metrics and select the best-performing model for deployment.
6. **Result Improvement (Dependency: Builds on the insights gained from 'Problem-Solving Methods')**
    - **6.1. Iterate on Previous Phases:** Based on the model evaluation, re-evaluate and iterate on previous phases. This may involve collecting more data, refining features, adjusting the data splitting ratio, or trying different algorithms to improve model performance.
    - **6.2. Ensemble Methods:** Explore using ensemble methods (combining multiple models) to potentially enhance predictive performance. Techniques like bagging (e.g., Random Forests) and boosting (e.g., XGBoost, AdaBoost) can significantly improve accuracy.
    - **6.3. Handle Overfitting:** If the model performs significantly better on the training data than on the validation set, address potential overfitting by using regularization techniques (L1, L2, dropout), reducing model complexity, or gathering more data.
    - **6.4. Document Improvements:** Maintain detailed documentation of any changes or refinements made throughout this phase, along with their impact on model performance.
7. **Deployment and Monitoring (Dependency: The culmination of all previous phases)**
    - **7.1. Model Deployment:** Deploy the chosen model into a production environment where it can be accessed for making predictions on new, unseen data. This might involve creating an API, integrating the model into an existing system, or deploying it as a standalone application.
    - **7.2. Performance Monitoring:** Continuously monitor the performance of the deployed model in the real-world setting. Track relevant metrics to ensure the model maintains its accuracy and identify potential degradation in performance over time (model drift). Establish a system for retraining the model periodically with new data to account for changes in the underlying data patterns.

}