# From DevOps to MLOps: Overview and Application to Electricity Market Forecasting
## 4 Case Study 1: Forecasting an Hourly Day-Ahead Electricity Market Using an MLOps Pipeline
### MLOps Runtime Environment  
As mentioned in Section 3.1, subsection 'Code', the first step is to ensure that the code is appropriately versioned. GIT was selected as the version control for this implementation, and GIT feature branching was followed. Figure 8 shows the commits and GIT branches. Two feature branches were created for the development of the solution, namely, \"feature/build-model\" and \"feature/ingest-data\". Additionally, one \"develop\" branch and one \"release\" branch were created. The develop branch was branched out from the main branch, and the feature branches were branched out of the develop branch. The feature branches were frequently merged back to the development branch, and all the feature branches were updated with the recent changes. Once the final code was merged back to the develop branch, the code was planned for release from the release branch. The code was versioned correctly, and a stable version of the code was always kept in the main branch after tagging correctly. Figure 7: Proposed MLOps pipeline. In this use case--MLOps level 2--CI/CD implementation was followed as explained at the end of Section 3. Hence the CI/CD should include source control, ML pipelines, test services, model services, feature storage, and deployment services. The first step in the CI/CD process is to create pipelines. These pipelines run in a sequence to implement the price forecast task. Moreover, the pipelines should provide a supportive environment, including runtimes and libraries. We selected PyTorch as the ML library and Python as the scripting language. Python libraries such as NumPy and Panda were installed in a virtual environment and loaded by the pipeline at the beginning to prepare the environment for the forecasting job. For this implementation, 10 pipelines were selected, and the modular code was created to run in each component. The required unit tests were also implemented and included along with data validation. The trigger was set to any commit changes in the SCM main branch; the pipelines would be triggered upon such changes, and the code would be deployed to generate the forecasting results. There is a variety of software available for creating CI/CD pipelines. Jenkins was selected as the CI/CD server because it is the most commonly used CI/CD server for DevOps. MLOps pipelines were incorporated using Jenkinsfile and Python. In this use case, we also used the DevOps process for implementing a web UI using Django, where the Django web framework was used for viewing the predicted result. All of the required software--including GIT, Jenkins, Python, Anaconda, and Docker--was installed on the CSC cloud. Figure 9 shows the pipelines created for the FCR-N market price forecaster. A new SCM commit change at the main branch would trigger the pipeline. The following pipelines were created in this example implementation:  
* Data ingestion;
* Data preparation;
* EDA;
* Model building;
* Deployment and monitoring pipelines with a few sub-pipelines. \"Data Ingestion--Data Fetch\" fetches the data from the Fingrid and FMI API. \"Data Ingestion--Store Raw Data\" pipeline stores these data in an SQL database, and PostgreSQL is used as the database, which stores the raw data. \"Data Preparation--Feature Selection\" performs the feature selection from the raw data. Different experiments have been performed in the past, based which this pipeline selects the features. \"Data Preparation--Data Validator\" validates the data and checks against different validation rules and data consistency, including null values, the presence of NaN, and data types. Along with this  
Figure 8: Commits in GIT showing branches, merging, and tags. code level, unit test cases are also executed to ensure code integrity. These data are then stored in a feature store for future use and analysis using the \"Data Preparation--Feature Store\" pipeline. Data analysis is performed automatically with the \"EDA\" pipeline to avoid data drift and other issues with data quality.