fabiochiu/medium-articles huggingface dataset:
HPB MainNet V1.7.2 officially launched!
By: Hpb Global
Tags: Scalability, Ethereum, Virtual Machine, https://github.com/hpb-project/go-hpb/releases/tag/version_1.0.7.2, Hpb

HPB MainNet V1.7.2 will support the latest version of Ethereum Virtual Machine, making High Performance Blockchain (HPB) a straightforward alternative for some projects hit by congestion on other blockchains.

HPB MainNet V1.7.2 will be an upgrade, which is expected to take effect at the blockheight of 8685000 on 15 Dec 2020. All HPB Node operators will need to upgrade to the latest version in order to enjoy continued block rewards beyond this date.

【What’s New】

1. The latest virtual machines on HPB MainNet supports Ethereum’s virtual machine V1.9.22. CHAINID and SELFBALANCE opcode are added, and the RANDOM opcode in the old version were replaced by CHAINID;

2. Added JSON-RPC interface and query interface for hpb_chainId on HPB MainNet;

3. Resolved some abnormal issues that may occure when node enters into p2p connection mode.

To upgrade your HPB MainNet to the latest version, please refer to the upgrade guide here: https://www.hpb.io/question-136

To download the HPB MainNet V1.7.2 software, please visit: https://github.com/hpb-project/go-hpb/releases/tag/version_1.0.7.2

To thank your effort of upgrading the HPB MainNe to the latest version, those who have completed the upgrade can apply for a 150 HPB reward. Please fill in the reward application form here:http://fans.hpb.top/s/UZBZJvNxxx/
Handling 50K concurrent users
Tags: Scalability, Online Examination System, Erlang, Couchdb, Aws Ecs

Recently, one of our customers wanted to conduct 150,000 exams a day. This in itself is not a challenge. Please see my previous post on how we scaled up from 1K online exams a day to 100K a day .

The new customer wanted to conduct exams of 50K users simultaneously in a single slot of 1 hour, 3 such slots a day (150K exams a day). So, the concurrency required was 50K.

Artwork by Ansh Tiwatne http://twitter.com/ansht_

App & DB Clusters

Our application is stateless, every request is independent and complete. So the requests can end up on any of the servers in our app cluster. The problem therefore reduces to optimising the number of requests reaching the server and of course horizontally scaling the app server cluster.

AWS ECS allows us to quickly scale the app cluster based on load. This is easy and a solved problem.

In addition to the app cluster, we run a multi-node CouchDB cluster and we provision high-end EC2 instances depending on load. CouchDB clustering is simple and its performance is excellent.

Request optimisation

Our MCQ based online examination system is write heavy as every option of every question has to be saved along with bookmarks, invalid question markers, several timers, and other proctoring features. There wasn’t much scope for us to optimise the app in this part.

However, we optimised requests by loading questions in batches instead of loading them one after another. A good batch values was reached considering user bandwidth, question content (images), etc. We were able to reduce millions of requests a day.

Conclusion

Building a stateless app is crucial for scale.
Accelerating Framework Launch Under Ethereum Foundation Ecosystem Support Program
By: Cryptoeconomics Lab
Tags: Scalability, Ethereum, Plasma, Dapps, Blockchain

Since our first grant received from the Ethereum Foundation last October, we have been having non-stop exploration around Plasma and Optimistic Virtual Machine. The efforts are finally paying off as we are getting close to our product launch. To speed it up even further, we are very thankful for the foundation’s continuous support that has granted us to their Ecosystem Support Program for the second time!

Through those months, there have been gradual and promising improvements around Plasma, OVM, and Optimistic Rollup in the Ethereum community. As a team dedicated to Plasma for more than a year, we acknowledge that it has been simply too complicated for many to delve into the protocol and build something with it. Cryptoeconomics Lab is, therefore, working hard to ship a Dapps development framework that Plasma is already built-in. It’ll boost your productivity and equip your Dapps with Plasma’s scalability and Ethereum’s security.

The framework is based on a design concept called the OVM, which is an “optimistic implementation of EVM” as originally proposed by the ex-Plasma Group (Optimism). Since its underlying design is L2 scaling solution-agnostic, our framework is not limited to Plasma but rather has future possibilities to support the other scaling solutions. In the upcoming launch, our focus is on Plasma construction that we believe has a great potential to grow scalability exponentially, compared to other L2 scaling solutions.

Sadly, the ongoing coronavirus outbreak has caused event cancellation worldwide, yet we expect to be more active in hosting hands-on workshops and hackathons in 2020 and onwards. By then, major releases will be notified on Telegram and Twitter.

Plus, we are currently looking for a Smart Contract Developer who can help accelerate the framework development. The team is pretty flexible with your preferences of location and work style. Feel free to reach out to us either in any suggested ways below!

For more details, check our job openings (English will be ready soon).

And for any other inquiries and comments, come and join us on Twitter or Telegram (English or Japanese).

*Recommended Readings

Plasma Group (Optimism) “Introducing the OVM”

Cryptoeconomics Lab “A Gentle Guide to the OVM”
Starting a People Analytics Function: 3 Building Blocks
By: Jenna Eagleson
Tags: Scalability, People Analytics, Business, Data Science, Data Driven

A solid foundation to building a scalable People Analytics function. You’ve got to start somewhere, you might as well start here.

Photo by Alex Kotliarskyi on Unsplash

People Analytics (also known as HR Analytics, Talent Analytics, and Workforce Analytics) is essentially the utilization of employee-related data to drive business decisions.

And it’s all the rage.

Data is everywhere, and every second, every mili-second, we are collecting more of it. Business leaders have learned that harnessing their data to make business decisions can have huge implications for their bottom line.

What types of ad campaigns generate the most sales? What kinds of products have the highest profit margins? Where does our target demographic do most of their shopping?

Collecting data, analyzing it, and then making decisions based off of those insights has become an expectation. And as business leaders have grown increasingly data savvy, there is a hot demand for ways to apply these principals to employee-related decisions. In comes People Analytics.

If you want to see super cool examples, Google’s People Analytics team has made some of their most impactful projects openly available.

Okay, so you’re bought in. You are convinced that People Analytics is going to change the world, make your business a million times more efficient, and enable your employees to be the most engaged employees on the planet. You want your organization to reap the benefits of this new field. Welcome to the cool kid’s table.

But where do you start?
The Benefits of Modern Continuous Delivery with Spinnaker and OpenShift
Tags: Cd Pipeline, Openshift, Continuous Delivery, Ci Cd Pipeline, Spinnaker

The Benefits of Modern Continuous Delivery with Spinnaker and OpenShift OpsMx Mar 4·1 min read

Continuous Delivery (CD) is the key DevOps practice is delivering software updates be it change requests, bug fixes, configuration changes, new feature updates, and so on.

Continuous Delivery ensures software is released speedily, safely, and sustainably to the end-users i.e. customers, be it for a large-scale distributed system, a complex production environment, an embedded system, or an app.

The goal of Continuous Delivery (CD) is to make deployments safe and predictable while achieving the ability for faster release cadence. The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production.

The Continuous Delivery Pipeline represents the workflows, activities, and automation needed to guide and manage a new piece of functionality from the ideation to an on-demand release of value to the end-user.

Read our blog on “The Benefits of Modern Continuous Delivery with Spinnaker and OpenShift” where we introduce you to the popular Continuous Delivery tool — Spinnaker, discuss its architecture, various installation methods including OpsMx Spinnaker Operator for RedHat OpenShift and its benefits.
Gitlab child pipeline with dynamic configuration in 5 minutes
By: Borislava Kirova
Tags: Cicd, Gitlab, Gitlab Ci, Continuous Delivery, Continuous Integration

I was extremely excited when this feature was introduced. It saved us tons of hardcoded configurations and hours of maintaining them because of their dynamic nature. It also helped us distribute better our jobs over the runners that we had.

This feature is very useful in cases of testing or building with different environment variables.

In the example that I’m showing, I have a testing framework and 2 different environment variables: currency and language. If I want to run the same test job for every language, I can just create a gitlab-ci.yml config and list all of the languages. Easy right!

But what if the list of languages is dynamic?! In that case, even for an environment that supports only 2 languages, I will be running unnecessary tests. This is a waste of time, and resources.

Let’s see how to create a child pipeline based on a dynamically generated config in just 2 steps.

1. The gitlab-cy.yml

In the GitLab configuration file we have:

a generation job

and a trigger job.

The generation job will execute a script that will produce the child pipeline config and then store it as an artifact. Then the trigger job will read the stored artifact and use it as configuration for the child pipeline.

stages:

- setup

- triggers

- test



generate-config:

stage: setup

script:

- node ./bin/generate-langs-config.js

artifacts:

paths:

- jackpots-langs-gitlab-ci.yml





trigger-langs-sanity-test:

stage: triggers

trigger:

include:

- artifact: langs-sanity-gitlab-ci.yml

job: generate-config



2. Config generation script

The generation script is just a js file. Settings can be fetched from backend API or local project configurations can be used as well.

The final result of the script should be a new gitlab-ci.yml that will be stored as an artifact.

const { writeFileSync} = require('fs');

const langs = ['en', 'it', 'de', 'da', 'se', 'fr'];



let config = `

stages:

- test

`;



for(let lang of langs) {



config += `

${lang}-sanity-test:

stage: test

script:

- npm run test -- --lang=${lang}

when: 'always'

tags:

- test

`;



}

writeFileSync('./langs-gitlab-ci.yml', config, {encoding: 'utf-8'});

}



parseGames();

And that's it! For more complex projects, multiple child pipelines can be added.

Conclusion

In large projects, the configuration files can easily become messy and hard to work with. Child pipeline feature facilitates job distribution and makes complex pipelines look clean and understandable.

And the dynamically generated config is very powerful when targeting dynamic content and environments.

Amazing features, very easy to implement just in a few minutes. If you deal with complex architectures and dynamic environments you should definitely consider using them.

Gitlab Docs:
Automate your BigCommerce Theme Configuration for Different Environments
By: Patrick Puente
Tags: Github Actions, How To, Bigcommerce, Automation, Continuous Delivery

The Solution

This approach uses the same GitHub workflow file that Rachael describes in her article and introduces a new .js file that Github will execute during deployment. The .js file will read a value from the repo’s secrets or environment variables and use it to populate a setting in the theme’s config.json. You can then reference the setting from config.json in your theme’s HTML template files with Handlebars as part of the {{theme_settings}} object.

Prerequisites

Stencil CLI

a Stencil theme

a Github account

a Github repo for your theme

If you do not have Stencil CLI or a theme, see the BigCommerce documentation for more information on configuring those items.

If you do not already have a Github account, sign up for one for free here: https://github.com/signup.

If you do not already have a repo for your theme, create one or begin by forking the Cornerstone theme at https://github.com/bigcommerce/cornerstone.

Write the Deployment Script

In your theme directory, create a new file. I named mine deploy.js. The filename will be important later, so take note of it now. In deploy.js, add the following simple Node.js script that writes an environment variable as a custom setting named mySetting in the theme’s config.json file:

Next, update the theme’s package.json file to declare a new script that will execute your deployment script. In this example, the script’s name is config. Add the following to the scripts object in package.json:

Example in the context of the entire package.json file: https://gist.github.com/hatertron3000/896ed8ca7e90084d2788e6016345b239#file-package-json-L73

Test the Deployment Script

To test the deployment script, run the following command which will execute the script that you configured in package.json in the previous step using an environment variable that you set in the command:

After you run the script, check config.json where you should now see settings.mySetting with the value you set in the previous command:

Configure Actions Secrets

In your Github repo, navigate to Settings > Secrets and create three secrets:

MY_SETTING: Any text value that you want to use to demo your deploy script

Any text value that you want to use to demo your deploy script STENCIL_ACCESS_TOKEN_PRODUCTION: The API access token used to initialize Stencil CLI with the stencil init command

The API access token used to initialize Stencil CLI with the stencil init command STENCIL_STORE_URL_PRODUCTION: The storefront URL for the BigCommerce including protocol. Example: https://store.example.org

Note: You could use an Environment to store the MY_SETTING variable if preferred. When I created this example, the repo was private so Environments were not available. If your repo is public or if you have a Github Enterprise account, you may use an Environment instead. To learn more about Github Environments, see the Github documentation: https://docs.github.com/en/actions/reference/environments

Create the Workflow
What are GitHub Actions and How to Use Them
By: Yann Mulonda
Tags: Front End Development, Github, Continuous Delivery, Continuous Integration, Software Development

Intro to GitHub Actions

Introduction

In this article, I’m going to share and demo a solution that could help offload some of the toolings to GitHub. This is called GitHub Actions. The way it works is that you create actions in your repositories by creating one or more text files. These are called workflows.

Workflows can handle common build tasks, like continuous delivery and continuous integration. That means that you can use an action to compress images, test your code, and push the site to your hosting platform when the master branch changes.

One interesting usage of Actions is in conjunction with tools like Bit (Github). For example, let’s say your team has a shared component collection on Bit. Your repo uses a few of these shared components. You can configure Bit to automatically send a PR whenever a component gets published with a bumped version. Using Actions, you can set up a CI to run on every such event.

This makes it possible to check your team’s shared components pass integration tests, across all projects.

Sharing components and syncing updates on Bit

Read more about it, here:

You can also have tasks that run on a specific timeframe, or that control what happens when somebody interacts with the GitHub repository itself.

So, when someone makes a comment on a pull request, it can send you a note or you can run an action when somebody stars your project.

Actions can run in a variety of platforms, including Linux, macOS or Windows, and these will run on virtual machines or containers. That means you can test your code in different environments. You can even run matrix workflows so you can test on multiple platforms simultaneously.

Understanding workflows

So, the main concept at the heart of GitHub actions is the workflow; this is done by creating a series of YAML files that contain a series of actions that determine your workflow in your repositories. These YAML files are stored in a special folder in the .github/workflows folder inside your repository. Here is an example:

At the moment, you can create up to 20 yaml files at this time, and each of the workflow files will respond to a specific event. In addition to that, each one of these workflows can have a number of jobs that perform different actions, and this is also where you specify the platform that you want to run the action under.

So, tod emo this; I’m going to use my Analog clock project which is build with simple Javascript vanilla. I made my project repository a template. You can go ahead and create a repository from this template and have something to play around with.

You can do this on your own projects if you want to, but it’s actually better to start off by creating a template repository or a brand new repository to play around with actions.

So let’s go ahead and add actions. The first time you create a workflow, Github provides some templates that you can use to create your first workflow based on your project repository. In this case, I’m going to add a Simple Workflow action.
Automating Laravel deployment using Ansible
By: Roelof Jan Elsinga
Tags: Laravel, Continuous Delivery, Deployment, Continuous Integration, Ansible

If you, like me, have been deploying changes manually to any of your websites consistently for months, if not years, you know that this is a repetitive task. Usually, you pull your changes from your version control system (VCS), run a few tasks to install production dependencies and/or compile them, cache your configuration, and reload some kind of service. It’s usually the same few steps with a few optional steps, in case you need to run database migrations for example.

What is Ansible?

You know that if something is repetitive, you can automate it. This is where Ansible comes in. Originally, Ansible is a tool to help with server orchestration and repeat tasks reliably on an infinite amount of servers. The best part of Ansible is that you don’t need to install anything on your remote machines. The only requirement is that you need to be able to connect to your remote machine through SSH. If you can do that, you can use Ansible.

You could compare Ansible to a large bash script that runs commands on the remote machine through SSH. The main difference between these two is that Ansible makes everything much easier and has built-in modules for abstracting many tasks. Pulling changes from GitHub, specifying only a repository and a destination folder is one of these modules. It makes writing tasks much quicker and easier.

Why is Ansible useful?

I mentioned that Ansible is originally used for server orchestration. As Ansible is essentially an easy to manage bash file, you can make it do anything you want to. This includes using it for deploying your websites, be it one 1 or 1000 servers. As long as you can use SSH on all of those servers, you can deploy on all of them.

As most of my websites are built using Laravel, I’ll provide a simple configuration to deploy your Laravel website to your server, migrate your database, cache your configuration, and clear your views cache. This is very basic, but it’s a starting point. This is not a tutorial, because frankly, I’ve just started out with using Ansible.

The basic configuration

Then you’ll need to create the secrets.yml file, you can use ansible vault:

ansible-vault create secrets.yml

Then fill it with these pieces of information:

github_user: your_username

github_token: your_github_access_token

To edit this file, you can use the following command

ansible-vault edit secrets.yml

This is still something I’m learning, so that’s why this is not a full-on tutorial. But by putting this out there, I’ve already learned a lot of new things about using Ansible.

There will definitely be more content about Ansible, because I’m already loving how easy it is and how many modules are built-in. When I know more about how it works in-depth, I will write a tutorial for it.

Posted on: June 7th, 2020
Kubernetes CKA Example Exam Questions Practical Challenge
By: Kim Wuestkamp
Tags: Docker, Continuous Delivery, Cka, Kubernetes, DevOps

The CKA exam is all about getting enough hands-on experience

Content

Overview and Tips

Be fast with Kubectl ≥ 1.18

You’ll spin up your own cluster

Every task has instructions at the beginning explaining how you set up the test environment using Vagrant and VirtualBox. This is a great way to explore Kubernetes deeply because you have full access to your own master and worker nodes.

You can also use the environment for more than just the provided scenarios, think and solve your own ones!

Why doing the CKA?

The CKA is hard but a rewarding certification. The amazing thing about it is that it’s hands-on and practical, which means no memorization or multiple-choice questions. During the exam, you can use the complete documentation at https://kubernetes.io/docs.

You’ll breathe kubectl and master your knowledge of Kubernetes Resources as well as administration tasks like cluster setup and troubleshooting.

You’ll be provided with a browser terminal and various scenarios for which you have 180 minutes time to solve.

More challenges

For a complete simulated CKA environment check out the:

More challenges on
Streamline: A super-efficient branching and CI strategy
By: Joe Brady
Tags: Continuous Deployment, Gitflow, Continuous Integration, Continuous Delivery, Branching Strategy

Mate, we all use GitFlow.

Indeed. And we all hate it sometimes, too.

How many times have you been on a project where your Pre-Prod and Staging environments have been so far behind your latest code sitting in the develop branch that you end up running demos from your Dev environment, or even on localhost ?

Here are a few examples of GitFlow-related “…said nobody ever”s:

“I really love moving my code around between develop , feature , bugfix , release , hotfix , master …” “Oh, you merged into develop ? Me too! I think our code will work together perfectly and not require any further testing.” “There’s a critical bug in Production? No problem. I am super-familiar with the hotfix pipeline and can guarantee you, Mr. Stakeholder, that it will work first time.”

Sometimes GitFlow is necessary. But perhaps there’s something better for the many, many times that it just felt like overkill and overhead?

Ned Stark says it best…

Mate, how about Trunk-based development?

Trunk-based development (https://trunkbaseddevelopment.com) is simpler than GitFlow, agreed! In fact, it provided the inspiration for Streamline. However, it lacks a robust strategy for continuous integration, and can be simplified even further with a few tweaks.

Streamline is an attempt to have the best of both worlds: simplicity and production-level robustness.

Come on then, how does it work?

The core concepts of Streamline are in the branching and continuous integration (CI) strategies. Continuous deployment (CD) will be covered in a later section.

Branching

To get started, you only need two types of branch:

One that represents the ‘single source of truth’, e.g. master or trunk

or One that represents ‘new code’, e.g. feature

For now, we will call the first branch trunk .

The branch type feature can be split into multiple branch naming conventions if desired, e.g. bugfix as well as feature , but ultimately this name will be the only difference between them.

Here’s the basic workflow:

A developer branches off trunk to feature/excellent-thing The developer makes their changes on feature/excellent-thing Once complete, the developer raises a pull request back into trunk Special and exciting CI happens (more detail later) The pull request is fast-forward merged (more detail later)

…That’s it! As you can probably guess, we will be relying on the CI to keep code quality high.

Continuous Integration

The CI process must have two key components:

An automated pipeline that is triggered when a pull request is raised or updated Merging the pull request into trunk is prevented until the following conditions are met:

The automated pipeline has passed

The feature branch is up-to-date with trunk

Both of these conditions are critical to the success of Streamline as an approach. Let’s explore each of them in more detail…

First merge condition: The CI pipeline has run and passed

The CI pipeline should:
Continuous Delivery for React Application Using Docker (part 1 of 2)
Tags: Continuous Delivery, Docker, Continuous Integration, React

Author – Roman Rodomansky, CTO at Ralabs.

What’s the Two-Part Piece About?

We’re practicing using CI/CD (continuous integration and continuous delivery/deployment) in many projects at Ralabs. With this tool, you get to save lots of time and effort on testing and delivering applications to end-users. We want to share our experience and the things we’ve learned on the best practices of CI/CD implementation in a project.

With this being said, the two articles, or rather two parts of one piece, introduce you to a guideline on how to set up Continuous Delivery for React project from scratch using Docker, AWS, and a couple of other 3rd party tools. You’ll also come across some useful articles related to the CI/CD implementation in general.

In this particular article, which is the 1st part, we tell about preparing the app foundation for future deployment. Basically, you start with running simple React app locally, then install Docker locally, setup EC2 server, and end up with preparing the Docker commands. Eventually, with our low-key tutorial, you’ll learn how to push your app’s image to remove the Docker registry.

The 2nd article, in turn, guides you in configuring the CI server, adding versioning and environment variables, applying the Continuous Deployment tool, and testing the achieved deployment flow. As a result, you’ll have automated deployment for any changes in the app.

If you have any questions after reading this article or want to gain a deeper insight into our experience, feel free to drop us an email at hello@ralabs.org. We’ll be happy to help you!

How Deployment Flow Should Look Like with CI/CD

CI and CD are the two important techniques widely used in DevOps.

CI is exceptionally popular in the Software Development world. It’s applied to any steps that can be automated in case of new changes, like building, testing, packaging etc.

From the software development perspective, all code changes are merged into a mainline, i.e. master, after the automatically created ones build.

In terms of costs, your team will need to spend time on automated tests for any new features or improvements. As a result of using CI\CD, you manage to save time on regression testing by minimizing the number of occurring bugs, since regression testing becomes automated.
Consumer Driven Contract testing on trunk based development pipelines using Pact
By: Çağatay Yücelen
Tags: Trunk Based Development, Consumer Driven Contract, Pipeline, Continuous Delivery, Pact

How Pact works?

Pact consists of a central server which is called as broker to store and keep track of contract files between consumers and providers.

A consumer, as its name, is a service which consumes a provider which is another service which provides the expected functionality. In order to verify the compatibility of integration between consumers and providers, pact expects us to create contract files.

Each consumer should create a contract file -which is basically a json file- specifying, given a request sent to specified provider, expected response fields, types, headers etc.

Pact provides tools for several languages to generate this contract files over unit tests of your client functions. We call this step as consumer test.

After creating contract files, we are versioning them and publishing it to pact broker to test it against provider later on. When contract is published to broker, we are ready to test if our provider is really compatible with this contract or not. We call this step as provider test. In this step, all contracts belongs to the provider is fetched from broker and validated against provider service in isolation.

What do you mean by “in isolation”? We are mocking all dependencies of provider service in this step, because we only want to validate the contract not the functionality. And this is one of the very points where Pact shines. To validate the compatibility between services, we don’t need to spin up all services together, we don’t need to depend on test data, basically avoiding time consuming and brittle end-to-end tests.

Provider test publishes the verification result back to broker to save it. By this way, we have all the compatibility matr between consumers and providers for each version of them. We can utilize this information to decide whether deploy the specific version consumer or not.
On the importance of Continuous Integration for GitOps
By: Vincent Behar
Tags: Gitops, Continuous Integration, Continuous Delivery, Kubernetes, Software Engineering

Why CI for gitops is not more common?

CI has been a common practice in software engineering for more than 10 years now. Hudson/Jenkins, which played a key part in popularizing it, is more than 15 years old. But still, in 2021 you’ll find plenty of git repositories used in a gitops workflow — representing an environment/infrastructure — without any CI pipelines, or with people pushing directly to the main branch, without creating pull requests first. Why is there such a gap between our code repositories and our environment repositories?

I think one of the reasons is that it’s not always the same people working on these 2 different kinds of repositories/codebase. You’ll often find software engineers working on “code repositories” and people with a more operational background working on the “environment repositories”. “devops” has been lost to the marketing/recruitment worlds, and SRE is not applied everywhere — yet. People lacking software engineering skills — or just without enough practice — don’t see CI as a mandatory prerequisite. Instead, they’ll just focus on different priorities, such as maintaining the production systems.

Another reason is that the main GitOps tools don’t promote CI. At all. Try to find any mention of Continuous Integration in the documentation of Flux or Argo CD. Oh sure, you’ll find something in the contribution guides, explaining how CI is configured to build and test those software, but nothing on how you can use it for your own need. For example, Flux has a feature to automatically create a new branch/commit whenever there is a new container image in a registry, along with a documentation guide to automatically create a pull request for it, but nothing on how to configure a continuous integration pipeline to validate it. On its homepage, it’s promising that you should “just push to Git and Flux does the rest”. Well, almost ;-) Note that I have nothing against these tools, I’m just pointing out that by not at least mentioning the benefits of CI, they are contributing to the abandonment of CI in the gitops world.

And last but not least, GitOps has often been compared and opposed to CIOps. I think that it might confuse people, thinking that you either do CI or GitOps. But not pushing to your target environment from your CI pipeline doesn’t mean you can’t use CI in your GitOps workflow…
Develop Serverless APIs With Speed and Confidence
By: Peter Njihia
Tags: AWS, Agility, Continuous Delivery, Serverless, Software Development

Serverless has brought about new horizons that were just not possible pre-cloud. A small team of developers, or even a single developer can develop a mature application in a fairly short time: “Maturity” here encompassing scaling as needs arise, security, availability, reliability, cost-effectiveness, etc. In an age where agility has a lot of weight in business, it’s worth taking notice of what can give you an edge. Serverless can. Notice the use of the word “can”, because it’s not a given that going Serverless will be fast, secure, scalable, cheap or even fulfill all f your business needs. So how do we get this edge?

It really falls on how you structure your solution, way before you write a single line of code. Because only then will you be able to build a reliable Serverless solution. A “Hello Serverless World” will not translate well to the real world. At the bottom of it, it’s all code, it’s still software engineering.

[Pauses for a minute and thinks about Low Code. Another day perhaps]

The obvious difference with serverless over “serverful??”, is that you are deploying your software on infrastructure mostly managed by your public cloud vendor. Your focus area is smaller.

The first mind-shift you need to make is quite contrary to convention: Build your software as if you want it to run anywhere. “Wait, you can’t do that! Most Serverless solutions are vendor-specific: AWS Lambda, Fargate, DynamoDB, Azure Functions!”. True, but to put it in another way: Be fixated on your business logic, and for you to be able to do that, you have to set up the foundation of your solution first, regardless of the destined platform. This is almost never explicitly mentioned in the Serverless marketing space, what you hear often, is to just dive in and write your business logic, we’ll handle the rest, relax. I strongly propose you spend some time laying some groundwork before jumping in. By doing so, you’ll actually find that your vendor lock-in concern is more manageable.

Enough pre-amble, let’s get down to it.
Build Your Docker Images Automatically When You Push on GitHub
By: Nassos Michas
Tags: Containers, Docker, Github, Continuous Delivery, Continuous Integration

Build Rules

To let Docker Hub know how and when to automatically build your images you can specify Build rules. You may have multiple rules that apply in parallel, effectively having multiple tags assigned to your images with just one git-push:

Build rules for tag-based and master branch automated builds

On the above figure we set up two different rules — let’s see what they do.

Tag rule

A “tag rule” allows Docker Hub to start building an image upon discovery of a new tag in your git repository. This should probably be your preferred way to build images for the official releases of your image.

As git tags can be arbitrary and contain anything a developer might choose, Docker Hub allows you to define a Regular Expression, in order to identify which part of a git tag should become part of the tag for the Docker image. In the example above we chose /^[0-9.]+$/ as the regular expression with which the image tag is extracted. Practically speaking, this tells Docker Hub that our git tags are eligible for automated builds, should be in the form of numbers with dots, and that should be the only text of the tag. The image tag that’s extracted and assigned to it is denoted by the {sourceref} attribute entered under Docker Tag.

Branch rule

A “branch rule” allows Docker Hub to start building an image upon activity on a specific branch. The typical use case here is to have such a rule to build a latest image based on the daily activity of your git repository. The branch to be monitored is entered under Source and the tag to be assigned to the Docker images built from it under Docker Tag.

Build your first image

As we now have everything in place, we only need to tag the current state of our git repository and push the tag back to GitHub; Docker Hub will do its magic and build an image for us:

Pushing a tag to trigger automatic build

Indeed, a couple of minutes later Docker Hub has automatically built and tagged the 1.0.0 image:

The first automated build of release 1.0.0

To also see our latest image being created automatically we need to push a new change in the master branch:

Pushing a change to master branch

Again, after a few minutes:

latest image built from master branch

Docker Hub provides you two additional screen on which you can monitor your builds:

Builds screen
Heroku Review Apps: Custom Domains
By: Gilbert Fu
Tags: DNS, Heroku, Continuous Delivery, Ruby on Rails, Custom Domain

The app.json file

An app.json must exist in your project directory to enable Review Apps. This configuration file specifies how each app should be built. At Clutter, our app.json file looks like this:

Sample app.json file with scripts specified

The app.json file gives us the ability to specify a postdeploy script to run after the release of the app has succeeded, and a pr-predestroy script that will deregister the domain when the Review App is destroyed.

The postdeploy script

In our postdeploy script, we utilize the open source Ruby gems dnssimple-ruby and platform-api to make the API calls for the domain and subdomain configurations.

The platform-api gem is provided by Heroku to interact with its platform via code, rather than manually. We use it to tell Heroku that Review Apps should not only be available at clutter-pr-NN.herokuapp.com, but also available at a custom domain pr-NN.clutter.com.

The dnssimple-ruby gem is provided by DNSimple to set up CNAME records via code, rather than manually. We use it to tell DNSimple to create a CNAME for the new custom domain pr-NN.clutter.com.

The resulting script looks like:

An example postdeploy script that will be executed through our app.json configuration

The pr pre-destroy script

Our pr-predestory script handles the cleanup of the CNAME record in our DNSimple account so that it is not polluted with old records.

A sample pr pre-destroy script that cleans up our CNAME records

Heroku Review Apps are a powerful tool to help our engineering and product teams here at Clutter continuously deliver new features. With the postdeploy script capability, we can continue to add more functionality to Review Apps such as third party integrations with other services.

If you enjoyed this article, please share with your friends. If you are looking to join an amazing team of Ruby developers, check our current openings at Clutter.
How To Speed Up Your GitLab CI Pipeline
By: Eduardo Lugo
Tags: Continuous Deployment, Gitlab Ci, Continuous Delivery, Continuous Integration, DevOps

When you first start using CI pipelines you get addicted to them. It’s easy to see why. After all you have made a great improvement to your process and it’s so fast! Soon you start doing more and more and you realize your pipeline is not so fast. It takes 30 minutes!

Waiting 30 minutes for a pipeline to be ready is boring. How can you get more speed? We went through several exercises and figured out a few things.

First let’s define what our pipeline looked like:

test ⇒ pack ⇒ build ⇒ plan ⇒ deploy ⇒ smoke

Now, let’s go over the 3 key things that changed to improve pipeline speed.

1. Artifacts

On the pack stage we generate some artifacts that need to flow to the next stage, build, which is where we build our container. If you are finding an error that goes something like “artifacts expired” on stages that don't use them, read on.

It turns out that GitLab CI by default grabs artifacts and passes them to the next stage, and the next stage, and the next stage. So if your pipeline is taking too long and your artifact is short lived then you could have extended the life of your artifact, like we did, and that was wrong!

In a big application, downloading the artifacts can mean a significant amount of time. What you want to do is:

(Obvious) If possible, reduce the number of artifacts. Mark the stages that don't use the artifacts with an empty dependencies array.

This was the biggest offender for us.

2. Cache

Caching for incremental compilation or even for external libraries also helps a lot. We added a cache for every branch with the “key” attribute. We then specified which modules had tests in them so no time was lost looking for test files elsewhere.

Beware! Having too much stuff in the cache can be a big offender too. Those get passed along stages similar to artifacts.

3. Parallel Stages

Our pipeline was huge, we had 5 stages, 2 of those had over 10 parallel jobs, so this step is not to tell you “do parallel jobs” but quite the contrary, beware of parallel jobs.

There is a limited number of parallel jobs a runner can take, so after about 4 all the others get paused and they sit around waiting for a slot. The job itself might be quick but you are adding additional time trying to make it parallel.

Also, setting up the job, downloading anything related to it, performing any before-script sections you might have, takes time. We found it helped to reduce the number of parallel jobs we had in all stages. Just making the most out of the environment that was available at the moment worked well.
5 Minute DevOps: GitFlow Best Practices
By: Bryan Finster
Tags: Gitflow, Continuous Delivery, Git, Continuous Integration, DevOps

I was asked recently about the best practice for using GitFlow with continuous integration. For those who do not know, this is GitFlow:

Post from 2010: https://nvie.com/posts/a-successful-git-branching-model/

There is no CI good practice that includes GitFlow. For continuous integration to meet the definition, these are the good practices:

Developers must implement all tests before they commit code. This is a non-negotiable and I will not work with developers who refuse to test. Professionals deliver working, tested code. Script kiddies don’t test.

tests before they commit code. This is a non-negotiable and I will not work with developers who refuse to test. Professionals deliver working, tested code. Script kiddies don’t test. Stand up CI automation and trigger builds for every pull request so that bad changes can be rejected.

Ensure that all tests are executed and linting and static code analysis is done for every PR.

Implement CI practices. All developers branch from the trunk, make changes, and submit PRs back to the trunk. The branches are removed in less than 24 hours.

You’re now doing Trunk Based Development. Welcome to your CI/CD journey!

The complexity of the CI automation will depend on how poorly the application is architected and the size of the development team. Monoliths with poor sub-domain boundaries will require much more complicated test design and test execution will take much longer. Evolve The architecture into independent, loosely coupled sub-domains to improve delivery speed, reduce testing overhead, and improve stability, resilience, and scalability.

GitFlow does not increase quality or value delivery. It delays quality signal feedback to the developers, incentivizes manual process, and is incredibly wasteful of time and resources. No modern development uses it.

Some developers have a religion built around GitFlow because it reduces typing (no feature flags needed, no branch by abstraction cleanup, etc.) and they don’t track their levels of re-work, lost changes, or conflict resolution. In 2010, GitFlow felt good. We could keep Master “clean”. That was almost a decade ago. Testing was still mostly manual. We were still on Java 6. NodeJS was barely a thing. Time to modernize. We don’t keep Master clean with added process. We keep it clean with automation.

Other examples:

Update March 2020: I asked the author of GitFlow if he could clarify the use cases to stop the GitFlow whack-a-mole and he very helpfully did so, Please thank him.
A Message to DBAs
Tags: Database Development, Continuous Delivery, Continuous Integration, Microservices

It’s time to get involved with the Continuous Pipeline.

As a DBA, I’m sure you’re not yet intimately involved in building out the Continuous Delivery pipeline. I’m here to tell you that you should be. As you already know, your impact to the software release process is no different than a developer’s. The question you need to ask yourself is, why aren’t my updates incorporated into the CI/CD pipeline along with all the other changes?

Part of the problem is what you are updating. In the development world, a binary can be easily copied over another older binary, no matter what version they are stomping on. That’s not the case with a database update. With these, you must modify the existing object, the database itself. Instead of a copy, you are running alter scripts, and they need to be changed or backed out of in a specific order.

In other words, it’s the nature of the beast that makes it problematic for you to participate in the CI/CD pipeline. While you certainly can use a CI/CD engine like Jenkins to execute your alter script as part of an automated process, your updates must be applied sequentially. You can’t just stomp over the top of your database like you can a binary. This means that the process must include the ability to track, incrementally, your database updates.

Now I know you knew this. I bring this up as we specifically address this problem. DeployHub is an open source Release Automation solution that performs continuous deployments driven by the CI/CD Pipeline. Included is a free version (based on the open source offering) that can track your incremental updates as part of a software application “package.”

With DeployHub, you create an Application Package based on Components. Components can be anything, including your database updates. When the software is released, the Application Package is interrogated based on the individual Components, and each Component is versioned, which allows DeployHub to track the differences between all Components in a software release.

When you create a database Component, you specify the alter script to move forward and to roll back. And there is more. DeployHub manages updates based on the Component versions and applies your database updates incrementally or rolls them back incrementally in the correct order. This ability is the real secret sauce of DeployHub. With us, you join in the automation game and start driving your database updates as part of the CI/CD pipeline.

To get started, sign up for DeployHub Team and help your developers build out a continuous deployment process that can handle your database updates. And yes, they can use their CI/CD pipeline to call DeployHub to do the important job of software updates with DB changes.

For other blogs on the subject, read more here.