Package: mlops-shape-project
Welcome to Shape MLOps Project🌐OverviewThis project was initiated as part of Shape's Hard Skill Test for the Machine Learning Engineer role. The core objective was to take an initial script namedjob_test_challenge.pyand transform it into a more organized 📐, maintainable 🛠️, and production-ready codebase, all the while preserving its primary functionality. The envisioned transformation entailed the integration of best practices, robust documentation 📄, and a modularized (preferentially object-oriented 📦) design to render the code apt for product release.Key Tasks 🔑:Code Modularization and Refactoring: The original script needed an overhaul to reach a modular, and potentially object-oriented, structure 🔄.Documentation: Implementing clear and concise documentation 📚 to ensure smooth navigation and comprehension for fellow developers.Preservation of Output: Despite alterations in the code's structure and logic, the model's predictions (output) ought to remain consistent ✔️.Configuration File Modification: The structure ofartifacts/pipeline.jsoncwas amenable to changes to ensure it meshed seamlessly with the revamped code structure 🔧.Incorporate TODOs: The code housed multiple TODOs that had to be addressed and realized as part of the challenge 📝.Scalability Consideration: The reshaped code was anticipated to cater to hefty data scenarios, suggesting that the supplied sample data file might, in a real-world setting, symbolize a table spanning several gigabytes in a data lake 📊.Constraints and Considerations 🚫:The generated code should be portable and capable of executing flawlessly in an alternate Python environment 🐍.The provided "pickle" files were crafted using Python version 3.8.13 and Scikit-learn version 1.0.2. Though other libraries might be pivotal for diverse facets of the project, these particular versions were indispensable for the "pickle" files 📌.Intricate or time-intensive modifications, if not instantly implementable, should be signposted within the code as TODOs. These markers should also sketch out the strategic approach intended for their eventual fulfillment 🕰️.🚀Run the projectThe provided configuration appears to be from apyproject.tomlfile, which is a configuration file for thePoetrydependency management and packaging tool in Python. To install and execute this project, follow the steps below:Prerequisites:DockerPoetryPython1️⃣ Install Poetry:If you haven't installed Poetry yet, you can do so with the following command:curl-sSLhttps://install.python-poetry.org|python3-Or you can use other installation methods detailed in theofficial documentation.2️⃣ Clone the project (if you haven't already):Assuming the project is hosted on a Git repository, clone it:gitclonehttps://github.com/JuniorTorresMTJ/mlops_shape_project.gitcdmlops-shape-project3️⃣ Install project dependencies:Navigate to the project directory and install the dependencies using Poetry:poetryinstallThis command installs both the main dependencies and the dev dependencies of the project.4️⃣ Activate the virtual environment:Poetry creates a virtual environment for your project. Activate it with:poetryshell5️⃣ Build the Docker image:The exact command to run the project depends on its structure. Typically, there's a main Python script or module you'd execute to start the project. If there's a clear entry point, you can run:dockerbuild-tmlops-shape-project:latest.6️⃣ Build the Docker image:dockerrun-p8000:8000mlops-shape-project:latestAPI should now be running and accessible athttp://localhost:8000.Additional tasks:The provided configuration also contains some tasks defined withtaskipy. You can run these tasks using thepoetry run taskcommand. For example:To run the lint task:poetryruntasklintTo serve the documentation:poetryruntaskdoc7️⃣ Endpoints:✅ GET /dataPurpose:This endpoint retrieves the original dataset from a predefined file location.Usage:You simply send a GET request to this endpoint without any parameters.Example withcurl:curl-X'GET'http://localhost:8000/dataResponse:If successful, you'll receive a JSON representation of the dataset. The structure will be a dictionary with columns as keys and their associated data as corresponding values.✅ POST /process_dataPurpose:Processes data from an uploaded file and returns the processed output.Usage:Send a POST request to this endpoint with an attached file containing the data you want to process.Example withcurl:curl-X'POST'-F'file=@path_to_your_file.csv'http://localhost:8000/process_dataResponse:If successful, you'll receive processed data (like predictions) as a JSON response.✅ POST /predictPurpose:This endpoint takes input data in the form of a JSON object and returns a prediction based on the provided data.Usage:Send a POST request with a JSON payload representing the data for which you want a prediction.Example withcurl:`curl-X'POST'-H"Content-Type: application/json"-d'{"column1": "value1", "column2": "value2", ...}'http://localhost:8000/predict`Response:If successful, you'll receive a JSON response with the prediction status and the predicted class.✅ POST /predict_batchPurpose:Provides batch predictions based on the provided data.Usage:Send a POST request with a JSON payload containing multiple rows of data for batch processing.Example withcurl:curl-X'POST'-H"Content-Type: application/json"-d'[{"column1": "value1_1", "column2": "value1_2", ...}, {"column1": "value2_1", "column2": "value2_2", ...}, ...]'http://localhost:8000/predict_batchResponse:If successful, you'll receive a JSON response with the prediction status and the predicted classes for each input data row.For each of the endpoints, if there's an error, the API will return a clear message indicating what went wrong, along with a status code indicating the nature of the error. Always make sure to check the returned message and status code to understand any issues.📖Documentation:The documentation of this project is an essential component for understanding its structure, functionalities, and usage guides. We strongly believe that comprehensive documentation can speed up the adoption, contribution, and debugging processes of the project. The documentation was crafted usingMkDocs, a swift and straightforward tool for creating documentation websites from markdown files. It allows us to concentrate on the content while automating the process of converting Markdown into a structured and styled website.⚠️Documentation Link⚠️📑Overview of Project TechnologiesThis section offers a comprehensive understanding of the technologies incorporated into our project, shedding light on each tool in detail. Grasping these technologies is pivotal for any developer or stakeholder aiming to become acquainted with the infrastructure and the project code.Python3.8.10 🐍Description: Language used for application development.Poetry📦Description: A tool for dependency management and packaging in Python. Poetry provides a streamlined and consistent approach to handle Python projects and their dependencies, ensuring transparency and reproducibility throughout the process.Scikit-learn1.0.2 📊Description: A machine learning library in Python. Scikit-learn, together with Pandas, stands as an essential tool for data processing, analysis, and the deployment of machine learning models. Both libraries bring forth a vast set of features and are widely recognized in the community.Pandas^2.0.3 🐼Description: Used for data reading and transformations.Blue^0.9.1 🔵Description: Coding standard adopted for the project.isort^5.12.0 📑Description: A coding utility to organize imports in a consistent format.taskipy^1.11.0 🛠️Description: Facilitates development by creating tasks.pyarrow^12.0.1 🏹Description: Enables reading of .parquet data files.Docker🐳Description: Docker is a platform designed to develop, ship, and run applications within containers. It ensures our application and its dependencies are packaged into an isolated container, which enhances portability and ease of deployment across varied environments.Grasping the intricate details of these technologies will significantly augment the efficiency and effectiveness of engagement with the project. 🚀🌟📝Next StepsHere are some next steps that can be implemented in the future:1️⃣ API Creation using FastAPIDevelopment of an API usingFastAPI, a modern and fast framework for creating Python APIs. This will facilitate integration with other platforms and allow remote access to the project's resources.2️⃣ Validation with StakeholdersWe plan to conduct several feedback sessions with our stakeholders to ensure the project aligns with the expectations and needs of the end users.3️⃣ Monitoring SetupMonitoring the performance and functionality of the system is crucial to ensure its reliability and efficiency. It's important to implement tools and processes for keeping an eye on the system.4️⃣ Alert Setup for Slack or EmailTo ensure a swift response to any issues that may arise, we'll implement alerts that will send notifications to Slack or predetermined emails in case of system failures.5️⃣ Refactoring of thedataframe_checker.pyclassCurrently, thedataframe_checker.pyclass is responsible for validating input data. Initially, it wasn't built with PySpark, but ideally, its internal logic should be replaced withPySparkto optimize validation and preparation of data on large datasets.6️⃣ Scheduling with AirflowTo automate and orchestrate our workflows, we will useAirflow. It will assist us in defining, scheduling, and monitoring workflows programmatically.7️⃣ Model Registration in MLflowLastly, to track and manage our machine learning model iterations, we will useMLflowto register, version, and share models.💻CONTACT ME💻