The peculiar case of passing tests and failing coverage
By: Osher El-Netanany
Tags: Nodejs, Code Coverage, Debug, Ci, Mocha Js

The Story Begins

April 19th was a normal April day. I was working on few PRs to some soon-to-be open-source initiative, where one of them failed a build. Well, that’s usual — nothing to excite about — probably I did not run the lint locally and the builder stopped me — as it should.

It was the end of the day, and I’m not used to leaving holes behind. I opened the build report.

Then I saw.

And I heard the devil whisper through a smile — “welcome to voodoo land…”.

The Problem Observed

What the build report said is that all tests have passed, but the coverage report had failed.

The project’s tests use mocha. The project's coverage runs after the tests and uses nyc which runs mocha.

So how could the tests that have just run successfully fail?

Re-run the build reproduced the same results.

First: Analysis …uh, no. More like Pannic

The shock grew when I double-checked my local env — ubuntu, nodenv, node:12.

On my local env tests and coverage both passed. I did remember being happy with the coverage level before pushing, but I had to double-check.

My past started to haunt me. Many years ago I spent many a night debugging the transpiled code istanbul uses to collect coverage, shortly after I finished debugging js-minify and uglify. It was back in the days where JS minifiers and uglifiers have just got stable, but not stable enough to rid you of knowing what’s under the hood, and istanbul was gaining speed and popularity. grunt was not even born, and browserify was trendy.

I admit that these scars have a lot with my deep resentment to transpiled code. Framed as a necessary evil strictly contained in web-app production compilation, and nothing other. The further I keep it from my developers — the faster they go. That was then. But I’m sidetracking.

First Analysis

How come the other branches pass build? How come the master passes build? No. It’s not a docker thing, not an alpine thing, it’s not a node thing, not a Jenkins thing. It’s something local to all of them together: the changes in the specific branch, and the builder env — which is an node:12-alpine docker image.

First Goal

The first goal was to be able to have a local version I can play with. Something I can run locally and not wait for the builder. Moreover — I want to hack inside, hands-on.

I managed to do that with building locally the same docker image the builder uses, run it with the project mounted as a volume and play it from there. No IDE. No debugger. Just me and vi.

People who know me know that’s not what frightens me. As a sworn consologist I don’t debug with breakpoints and I’d take a basic text-editor over an IDE any day. But working in alpine using vi is a little too annoying… I never took the extra mile to master it. I just might some day…

Anyway, I ran the docker with docker run -it , I connected to it a few sessions with docker exec so I can have a shell for execution, a shell for an editor, and a shell for inspecting some extra files — and made myself at home.

Obviously — node-modules compiled for ubuntu are not compatible with alpine. So I had to remove them and install them from within the docker. Eventually, I had to do that many times, checking for every critical step, that what I got works both on ubuntu and in the node:12-alpine docker.

First local reproduction.

I reproduced locally a passing tests & coverage from the master

I successfully reproduced the failure from the broken branch

Now I had more information.

What the tests are doing there

The project is a command-line tool that acts as a container for project code. It consumes a configuration file that describes the structure of the project and project code for the project modules and does the heavy lifting around them in form of IOC manager, logging, handling low-level errors and process signals, and supervising graceful shutdown. It’s a part of an ecosystem of modular parts you express your BL over, where the project that failed CI is the shell entry point.

As such, the e2e tests (i.e. end-to-end tests) run after the unit-tests and proof that it runs well some good projects, and rejects with a friendly error when there’s an issue with the project code and/or configuration, and exits with correct exit-codes to indicate the life-cycle stage it failed in.

In every e2e test-case, the suite runs the CLI-tool with a test project as a fixture in a separate process using child_process.exec, collects it’s stdout and stderr , and makes assertions against them with respect to the expected behavior for that particular test-case.

broken stdout and stderr in e2e tests

In the builder, the tests run mocha with the default spec reporter, and the coverage runs it again with dot reporter. Already on the builder, I could speculate that the unit tests passed and problems started at the e2e stage, and I could validate that now — because on my local docker-image I forced a spec reporter on the coverage run.

I also could focus the run on the failing parts using --grep e2e . And last — the test suite takes an extra step just for this case I’ll need troubleshooting — and creates an e2e.run.log and an e2e.cover.log files with all the test-cases that were run, the arguments they are run with, the exit code they ended with, and the stdout and stderr they yielded.

What was clear is that in coverage mode the stdout and stderr were both completely empty…

The game commences

I noted the problem first appeared when I installed some dependency the PR had to integrate. The goal of the PR was to proof integration with a selection of reusable ecosystem components pluggable over this CLI-tool. Some of them were wraps of legacy code that work with monitoring tools such as newrelic.

My guts shrunk because my initial suspect was some c++ node-gyp compilation that looks different on Alpine, and does not work well in cover mode. I was horrified. Last time I had to troubleshoot something like that I did not win the battle by myself. I had then a lot of aid I could not summon that easily now.

After getting over the shock, and starting to think rationally — I started to analyze the process and the dependencies.

The Second Analysis

The start point was that without the added ecosystem dependencies it worked, and without it — specifically in the builder image — it did not, and that in the builder it passed in test mode, and failed in coverage mode.

We’re in a room in a maze with two doors (not including the one we came through).

Door 1 — Instrumentation

Just for cases like that, nyc lets you transpile the code so you can see how it looks. It’s done with:

nyc instrument <source> <target>

But I did not just want to look at it, I wanted to run the instrumented code myself and see how it behaves. You can even run in it in a debugger if you’re that kind of a guy. For this — there’s:

nyc instrument --in-place

which replaces every file it’s instructed to process with an instrumented version. When you’re done — you git checkout . and go back to normal.

So I ran nyc instrument --in-place , and then I ran mocha .

The tests passed!

Not good:

Failed to reproduce!!

I was gearing myself for the battle, blade in hand I opened the door where the dragon sits, and …heard crickets. Not here.

I wanted to slam in, but there was no dragon to fight. Was it hiding? How could I drive it out?

At that time I was too blunt to conclude decisively that I will not need to debug transpiled code. In post mortem I know it was evident.

But I did take a look at the go’old transpiled code, just to say

“I’m not frightened of you!”

and eventually decided I’m wasting time.

It did confuse me enough to put it aside for the moment. So I tried the other door.

Door 2 — Dependencies

The code itself in that legacy-code wrap is pure JS. So is the legacy component it wraps. However, it uses coralogix, newrelic, winston, and a lot more. Removing that specific component resulted with ~30 packages removed, (adding it added ~30), and there was clear evidence of C++ code compiled during npm install , one of them even compiled twice providing two different versions, one for the new log & monitor, and one for the legacy wrapper. Hmm. Ugly.

I analyzed the dependencies I got from Ubuntu and from Alpine, and started to lose myself there. It’s a swamp. Diff tools did not make it much easier. Not even when I extracted the out to where I have visual tools. Dispaired, I left for the night.

Next morning

In the morning after I polish-licked some docs, which always help me calm down. In docs (assuming you write them well) everything looks in control. Communicated. Clear. Human-friendly. Working. I love that.

I took a sniff of the swamp of the previous day and turned away in disgust.

I decided to try yet another way.

Door 3 — Backtracking

Step 1

I backtracked my steps and made sure that there was no code-change I did that could have caused it and verified the problem is in the dependencies:

I opened a new branch from master, removed the legacy wraps and its tests, but left the dependencies inside.

Passing locally

Passed in the docker.

<…gulp…> so I pushed the new branch to observe it on the builder.

The build passed.

Good. So far — my local docker seems good enough to represent the builder.

Step 2

I added the new wrapper to the legacy code. There were a bunch of pending tests waiting for that moment. I enabled them.

Passed locally

Passed in the docker

Passed in the builder

Good. Looked like progress.

Now I had to get rid of the unnecessary old dependencies.

Step 3 — the problem reproduced

I removed the legacy-project dependency.

Passed locally.

Failed in the docker, on the same empty std buffers.

Now what the project was telling me is that it does not want to let go of these old dependencies, even though it’s already using the new ones?

It wants to carry them on as a hunchback to the future, with all its old versions of newrelic, and winston and all?

I made sure that it passes with the old dependencies reinstalled back —and:

passed in the docker.

passed in the builder.

Observation confirmed.

But what’s the theory behind that? What does it mean?

There should be some offending dependency that goes rogue in the presence of the older dependencies.

I had to find out the offending one.

It’s not a dragon. This fight will not require high spells and special abilities. This is a devil that is defeated with patience and stamina.

Narrow Down

I knew that wrapper is just pure JS. So. I uninstalled it and installed each one of its direct dependencies, as observed in its package.json, and committed the changes to my own package.json.

Passed locally

Passed in the docker

Passed in the builder

Good.

Now I hoped I can isolate a single offending dependency, that if I install only it — it will pass, and without it — it will fail.

One by one I uninstalled all the second generation dependencies, making sure the coverage passes every time on my ubuntu host and in the docker alpine.

Make it comfortable to work

I was using in the docker session and in my host <up> + Enter reusing this command:

rm -rf node_modules && npm i && npm run cover -- -g e2e -b

which means — clear node_modules , install it from scratch, and run in cover mode only tests that match e2e , and bail on the first error.

I also modified the .nycrc to include no reporters.

The purpose was that it won’t try to save reports to disk or spit dirt to the screen, so I can focus on the mocha output and see it “fails successfully” —fails in the expected place with the expected error.

The devil is in the dependencies

The legacy code wrapper had

just one dependency — the legacy package. From that legacy package, I successfully isolated a single offender — newrelic@5.

From newrelic I successfully isolated a single offender — https-proxy-agent@3.

https-proxy-agent has two dependencies — base-agent and debug@3. I took a bad turn and spent a lot of time inside base-agent and it’s dependencies before I realized I’m looking at the wrong place.

Really? debug ? A simple pure-js debug tool? I never suspected it, but reality stroke in my lackwit face. Of course debug .

Evidently, debug is messing with stdout and stderr , and debug@4 does things differently. They just as well be competing over the same stdout and stderr . It happened to be that in that specific constellation of dependencies debug@3 got to be installed in the node_modules of the root project, while debug@4 landed as a private copy, inaccessible to other packages except its main user package.

So.

From https-proxy-agent I isolated debug@3. debug has just one dependency — ms.

When replacing debug with ms — the tests did not fail as expected.

ms is just a simple string manipulator, helping convert between strings expressing time and the number of milliseconds they represent.

The offender was debug .

I analyzed the last dependency tree that passed coverage.

The essentials are:

<the CLI tool>

+-- <the added new package>

| `-- newrelic@6.5.0

| `-- https-proxy-agent@4.0.0

| `-- debug@4.1.1

+-- debug@3.2.6 //<------------------- lock to 3.2.6 made coverage pass + ...

When the project does not depend on debug@3, then debug@4 is installed on the project’s root node_modules

When the project depends on debug@3 directly — then debug@4 is installed as a private copy of newrelic, accessible for https-proxy-agent.

At this stage, I decided that 2 extra rather slim dependencies should not be a show-stopper. For now…

The Resolve

That still does not explain why it runs in tests and not in cover mode. Here’s my best guess:

debug is probably involved in nyc and/or istanbul , and it’s possible that without nyc we get one version, and with it, we get to load two versions, where one of them loads first, and by the time the other loads — only one of them gets control of the stdout , and in coverage mode it happens to be the one the rest of the tool does not use.

I did not have time to debug that. Or create an isolated reproduction scenario.

So I locked to debug@3.

Passed locally

Passed in the docker

Passed in the builder

Good.

For now…

The weakness is that if somebody in the future will try to npm update npm update or upgrade debug explicitly — there’s a dangeon adventure waiting for him, and since there are no comments in package.json — I cannot leave a warning there that will jump with the offending dependency in a git-diff.

I filled a bug-report to debug, and that’s it. For now…

Aftermath

From a later point in time, I can say we got rid of debug entirely and devised a better means to debug.

The ecosystem around that runner has evolved to include a powerful logger which lets you control the verbosity per channel, and debug was eventually pushed out.

I still did not find the time to reproduce the issue in isolation as I promised in my bug report, and I just might will someday. But life pulls me hard in different directions, so for sincerity’s sake, I doubt it will happen.