Testing With Pipelines
By: Randal Kamradt Sr
Tags: Reactive Programming, Jenkins, Programming, Kubernetes, Java

Testing With Pipelines

Find the leaks and tighten the bolts

Photo by Daan Mooij on Unsplash

In my last article, I put together a pipeline that built and deployed three separate microservices. But I cheated: all of the testing stages were placeholders. Let’s add some tests.

Before you begin, we will eventually be tagging the GitHub repo, so if you want to follow along, and you haven’t been using your own repo, you might want to fork mine so you can include the tagging steps. The links to the two repositories are at the bottom of the article.

My thinking about tests has evolved quite a bit over the years. I used to be big on unit tests. When building monoliths, testing each part separately was very important to find bugs early, and the only way to test each part separately was through unit tests. Now in the microservice world, each component is able to run separately, and it can be tested separately and automatically in a (semi)-real environment. No more mocking up databases or HTTP calls; just deploy into the environment and test automatically every time you push code.

I think unit tests are still important, especially for reusable libraries. No one wants to use a library of code that isn't thoroughly tested. But with the MondoReacto project I’ve been working on, a vast majority of the code is provided by Spring and other third-party libraries. The only thing left for me to test is the unique logic of the microservice at hand.

I haven’t really talked about the MondoReacto project. It was my experiment in using reactive Java, specifically projectreactor.io (hence the name, MondoReacto). Reactive programming is a non-blocking, event-driven, and fluent style of programming. Anyone familiar with the Java Stream library introduced in Java 8 should feel right at home with the style. Here’s a snippet that reads an HTTP POST call and writes it to the message bus (in this case Kafka).

So what’s going on here? In reactive programming, we deal with Flux es and Mono s, but they’re very akin to Stream s. In the above snippet, we only see Mono s, which is a special case of Flux es that can handle at most one element.

Since this code takes a single element, payload , transforms it, sends it to Kafka, and returns Kafka’s status code, we can use Mono<Input> and Mono<String> as our inputs and outputs. Spring’s WebFlux library deals with Flux and Mono inputs and outputs and maps them to a non-blocking HTTP server (in this case Netty).

Our logic merely transforms the data from the input to the output, so it is relatively simple. We transform the Input to a String , a String to a ProducerRecord , and finally a ProducerRecord to a SenderRecord . We pass that into the Kafka sender as the final transformation, a Mono<SenderRecord> , and it returns a Flux<SenderResult> . Since we know we only have one record, we can transform that from Flux<SenderResult> to a Mono<SenderResult> via the publishNext operator and transform the SenderResult to a String suitable for returning from our post.

Anyone familiar with the Java Stream library knows that nothing happens until you have a terminal operator. The flows that are built up simply take the functions (transforms or filters) that are added to the flow and saves them for later processing. The same can be said for the Reactor library. It saves a list of transforms or filters, and then once the Flux or Mono is subscribed to, it passes units of data from one to the next. In our case, the subscription takes place in the Kafka receiver’s send method, which terminates the first flow and returns a second flow which is subscribed to deep in the bowels of Netty, so we never see either of the subscriptions.

The key to making this all run smoothly is that none of the map or filter methods in the pipeline are allowed to block: no database calls or file reads that might block on I/O. All calls must be asynchronous, and in order to wait for a response, we need to produce a Publisher, which the reactive system knows how to handle. Fortunately, there are a lot of libraries that work with this paradigm, such as the WebFlux HTTP controller and Kafka interface in the snippet above. In most cases, all we need to do is connect the pieces.

Let’s take a look at a test snippet. I’m using Cucumber.io for a testing platform that uses the Gherkin language. I’m going to test the first stage of the MonodoReacto project, which was described above, taking an HTTP POST and sending the data to Kafka. Here is my Gherkin description:

Feature: Incoming input is passed to message queue



Scenario: Good input is passed to message queue

Given a good input value

When the incoming service post is called

Then the return value should be 202

And the input value should be found on the message queue

Each of the steps should have an underlying code to execute it. Here is the step code for “When the incoming service post is called”:

And here is the code that’s called when the “And the input value should be found on the message queue” step is performed:

Again, this is a combination of filters and maps. I do wish there were a combination filter/map, which I use a lot to get rid of empty Optionals . The publishNext at the end implies that we are only looking for the next matching item, and the .filter(i -> i.equals(inputValue)) ensures that only matching items go through. The rest are discarded and the subscription is terminated after the first one is published.

What I would really like is a testing platform that is reactive so that each step would return a Mono with the data that would be sent to the next step. Since I don’t have that, I have to do a lot of blocking and guess about the timeouts for each.

So we have all this in place. (I’ll have the actual code linked at the bottom of this article.) Now we need to run it in the pipeline created in the last article. We’ll add building the tests into the Jenkins build pipeline. I’m going to build the app and the test in one step and then kick off the deploy pipeline. But first, I want a way to trigger the build when the code changes in true CI/CD fashion.

There’s a catch, though. Jenkins normally triggers from changes to the repository it is set up for. Our build pipelines are set up for the build repository. We could combine our build repository and code repository, but I like a clean separation of concerns. So I’m going to create a new pipeline that looks at the code repo, but does nothing but start the build pipeline. I’m also going to send myself a Slack notification that a build has been triggered. Here’s the SCM section:

Here is the Build Triggers section:

Notice I have to Poll SCM. My Jenkins host is behind my firewall, so I can’t get GitHub to trigger it automatically. It’s set to approximately every five minutes while I’m actively developing, but I’ll set it down or remove it once my heavy development is finished. I’m sure if every GitHub user in the world polled every five minutes, that would constitute a DDOS attack.

Finally, I’m going to trigger the build pipeline and send myself a Slack notification when a build is triggered. (You’ll need to install and configure the Slack plugin for that to work).

Next, I’m going to modify the build pipeline so that it builds the app and test images, and then triggers the deploy pipeline. Here is the first part:

The Build…Image and Start…Service stages will get duplicated for each of the other microservices: incoming-persist and incoming-read-service . I will also have to copy the desc/incoming-service-test.yaml to description yamls for the incoming-persist-test and incoming-read-service-test microservice. And finally, at the end of the Jenkinsfile , I’ll add another Slack notification.

post {

always {

slackSend (color: '#FFFF00', message: "Jenkins Build ${currentBuild.currentResult}: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]' (${env.BUILD_URL})")

}

}

Now, I will make a change to the MondoReacto project and get notified when the build starts and when it stops:

Oops! (sad trombone sound) At least you have a link to the build that failed. It turns out I had accidentally copied the git command to each stage, and it failed because it refused to overwrite the old repo. I removed all but the first git command, but that brings up another problem. These worker nodes hang around for a while and keep the repository around with them. We should be able to delete the repository at the end of the pipeline to prevent that from happening. But for now, I’ll just remove the extra git commands and delete the maven worker pod before restarting the pipeline. Run again and it works!

In addition, it kicked off three deployment pipelines, each staggered by about a minute.

Only the incoming service tests worked. As I said in my last article, the MondoReacto is a WIP. I’ll fix the issues in the other tests and won’t bore you with the details. The MondoReacto project was a learning project for me to figure out reactive programming, and there’s probably some naive usage in the tests.

Still to come: I want to tag the repository and the image when it moves through the pipeline so I know which version was the last to make it through the different stages. I also want more tests as it passes between stages, perhaps some performance tests. And I would like to have better logging with a nice front-end like Kibana.

But, for now, I have a pipeline to build, test, and deploy three separate microservices to three different environments, all triggered by a push to GitHub. The pipeline scripts for this article can be found here: https://github.com/rkamradt/MondoReactoBuild/tree/v1.2 and the code can be found here: https://github.com/rkamradt/MondoReacto/tree/1.1