Basic Concepts of DevOps Core and Pipeline Building
By: Amit Chauhan
Tags: Machine Learning, Python, Analytics, Data Visualization, Programming

Basic Concepts of DevOps Core and Pipeline Building

Development and operational tasks for better SDLC management

Photo by air focus on Unsplash

DevOps is an application development practice that merges development tasks and operational tasks for a better software development lifecycle management, in the meanwhile also handling the frequent updates, bugs, and features of the application.

DevOps involves continuous development tasks like planning the code, coding, building the code, and testing it along with continuous operational tasks like releasing the code, deploying it, operating it, and monitoring it, followed by continuous integration of both the task sets.



DevOps core concepts:

The continuous build is a continuous and automated build process. It runs the added or modified codes.

Continuous Integration is the automated build and execution of at least unit tests to prove integration of new code with existing code but preferably integration tests (end to end). It is a practice of automatically building and unit testing an entire application frequently, ideally, on every source code check-ins, dozens of times a day, if necessary.

Continuous delivery is the additional practice of deploying every change to a production-like environment and performing automated integration and acceptance testing after it passes its build and unit tests.CD can involve deploying on a local server creating an environment with the assistance of docker containers or virtual machine sets to carry out production environment similar to integration tests. Testing usually would result in a deployed production similar environment. This lets us in validating the deployment process and checking and monitoring the functionality of the environment real world alike.

Continuous deployment extends the concept of continuous delivery to a level where any modification undertakes full automation testing and then is deployed to the production environment.

Build Pipelines in practice:

Flow Chart of Building pipeline

A build pipeline is a sequential combination of tools and operations that performs assigned tasks ranging from the source code to a deployed system. Common tools used for the source code repository include Git, Perforce, or Subversion. These tools provide services to store and handle configuration management for all the applications used hapcode, deploy scripts, infrastructure definitions. Its job is to handle the code, protect it, manage its versioning, and handle simultaneous commits on the code.

Build system

Then next is a build system, which monitors the repository, and triggers the code builds whenever a change is requested. Jenkins, Bamboo, and TeamCity are popular build systems. Circle CI and Travis CI are some of the available online services. Any build scripts you assemble and can run make will also do the job. But some functionalities like webhooks, off plugins, and other features that’ll help you optimize the process will be missed in those codes. Its job’s to build code and run unit tests. And, of course, to provide feedback and visibility into the build process.

Depending on what language you’re using, you may have a variety of compilation and build orchestration tools. If it’s a compiled language, you may need a c or go compiler. Interpreted languages may not have a compilation step and skip straight to packaging.

You might have a build definition file like a make file or a docker file, or perhaps you use something like Maven, Ant, or Gradle as a command-line build orchestration tool. This allows developers to build on their local machine and to keep as much code in source control and out of the build console configuration as possible.

You probably have a unit test harness, which may be a separate tool, but it’s invoked inside the build system. Unit tests or basic tests designed to verify a piece of code is doing the job it was intended to do. Unit tests don’t have dependencies on external systems. At most, they mock or stub out anything outside their codebase. They’re your first line of defense in figuring out whether your build is going to work as expected.

Packaging

The next step is packaging what you’ve built into an artifact. Part of this is usually language-dependent. For example, Java code is usually put into JAR files, and then those are combined up into a WAR or EAR file. But you may have a higher level packaging structure you prefer, like an RPM package or docker image. Once you’ve packaged the code, it goes into your artifact repository.

Some people just use a storage device or Amazon S3 for their artifacts. Others use tools built for the job like Nexus or Artifactory. Or there’s technology-specific repose, like docker registries or Puppet Forges. In the old days, a building engineer thought they were done here. But not so. You have some bits on a drive somewhere, not a service delivering value to customers, so your job isn’t done. You need a deployment tool to bring up a working instance for your service. You’ll use the same tool to deploy both your test and production environments.

Your deployment code is code, and like any other code, you want to find any issues in the build and test cycle, not when you’re deploying to production. So you run your deployment tool on a test environment, sometimes also called a CI environment, to get your service up and running.

Next, you perform integration testing, which is designed to exercise a real running service in a real environment and ensure that it’s working correctly. Then you have an end to end acceptance testing stage. This may include a manual testing component. Manual tests can be mostly dispensed with once you reach CD process maturity but will almost always predominate initially.

Then you use the same artifact that passed testing and the same deployment tool to deploy it to the production environment. Ideally, you use some of the same acceptance tests to test it there as well. There are many more types of tools you can layer onto your pipeline, test coverage measurement tools, linters, security testing tools, performance testing tools, but these are the basics. And nailing the basics is the key to getting to continuous integration and then if you want continuous deployment.

Reach me on my LinkedIn

Recommended Articles

2. Python Data Structures Data-types and Objects