How to access git metadata in CodeBuild when using CodePipeline/CodeCommit
By: Timothy Jones
Tags: Codebuild, Continuous Delivery, Git, Continuous Integration, Software Engineering

How to access git metadata in CodeBuild when using CodePipeline/CodeCommit

Hopefully your pipeline doesn’t disappear into the mist (Photo by JJ Ying on Unsplash)

Here’s the scenario: Attracted by the ease of using one cloud service for everything, you’ve got a build pipeline set up in AWS. You’re using CodeCommit for the source, a CodeBuild instance for your build target, and you’re using CodePipeline to hang it all together. Everything seems great- deployment into the rest of your AWS infrastructure is easy, builds kick off from commits, and everything is working well.

Until you decide you want to use the git metadata, like say using the output of git describe to figure out what version you’re building. When you try this, your build fails with:

fatal: not a git repository (or any parent up to mount point /codebuild)

If you came here from Google and you just want a solution, skip to the end of the post for a guide. If you’re interested in the gory details, read on.

Why doesn’t this work?

Turning to Google, you find this forum post from 2016. Linked at the bottom of that, is another forum post where an AWS engineer contributes:

CodePipeline downloads the source as a zip from the source provider rather than doing a Git clone, which means the .git folder won’t be retained and git commands like the one you’re running won’t work. Can you share more details about your use-case? It seems like your goal here is to produce a zip archive of your repository, which you could also achieve using something like the zip command. If you have other use-cases for retaining the .git folder I’d be interested in hearing more about them as this is a feature request we’ve heard from several customers.

You read that right. CodePipeline downloads the source and then repackages it without the .git directory before giving it to CodeBuild. I don’t know why it does this.

Another AWS user put this sign up, probably (Photo by Ken Treloar on Unsplash)

Can we use an environment variable?

Say we want to know what git describe --tags would have printed. CodeBuild provides some environment variables for us. Maybe we can use one of those to see the git metadata we want?

From the documentation, we have CODEBUILD_RESOLVED_SOURCE_VERSION , which is “the commit ID”. In one of my builds, I get:

CODEBUILD_RESOLVED_SOURCE_VERSION=80167249a7009aeff680573a9c40ab122176b361

That’s the git SHA of the most recent commit that’s being built. Potentially useful.

We also have CODEBUILD_SOURCE_VERSION , which appears to be the S3 ARN for the artefact that contained the zipped source. Not super useful.

The documentation also describes CODEBUILD_SOURCE_REPO_URL . Maybe I can use that to clone the repository — it would be a bit of a hack, but it would work. Here’s the documentation for it (emphasis added):

For CodeCommit and GitHub, this is the repository’s clone URL. If a build originates from CodePipeline, then this might be empty.

Sure enough, it’s empty in my build:

CODEBUILD_SOURCE_REPO_URL=

As far as I can tell, the documentation doesn’t say what “might” means in this context. Is it always empty? Sometimes? Only in certain configurations? I don’t know.

Don’t forget to get out and not know things in nature (Photo by Benjamin Davies on Unsplash)

Can we use an AWS API call?

I initially hoped that we could give the current commit hash to the CodeCommit CLI to determine the tag, but it’s fairly limited. Here’s the example response in the documentation for get-commit:

{

"commit": {

"additionalData": "",

"committer": {

"date": "1484167798 -0800",

"name": "Mary Major",

"email": "mary_major@example.com"

},

"author": {

"date": "1484167798 -0800",

"name": "Mary Major",

"email": "mary_major@example.com"

},

"treeId": "347a3408thisisanexampletreeidexample",

"parents": [

"7aa87a031thisisanexamplethisisanexample1"

],

"message": "Fix incorrect variable name"

}

}

We get the treeId , which is an internal Git concept, but we don’t get the git tag, even if the requested commit is directly tagged.

On the main code-commit cli page we can see there are some tag related API calls, but they are for AWS tags, not git tags.

I guess we’re going to have to clone the repository.

Not this kind of clone (Photo by Daniel Cheung on Unsplash)

Cloning the repository

I’m not the first to want to do this — some of the forum complaints linked at the start of the article worked around the limitation by cloning the repository. There’s also another article where the CodeCommit repository is cloned in CodeBuild here. However, I think the approach I used is a bit simpler.

Allow CodeBuild to clone your repository

Give CodeBuild the git credential helper

Then we need to:

Clone the repository at the right branch

Wind the repository back to the commit that is being built (otherwise you will have a race condition between commits and builds, which could result in the wrong git metadata being used)

Copy the .git directory to the source tree.

Here are the steps:

1) Allow CodeBuild to clone your repository

To do this, give your CodeBuild execution role the permission codecommit:GitPull . There’s a CloudFormation template at the end of this post, but here’s the policy statement:

{

"Effect": "Allow",

"Action": [

"codecommit:GitPull"

],

"Resource": [

"arn:aws:codecommit:*:*:<YOUR_REPOSITORY_NAME>"

]

},

2) Give CodeBuild the git credential helper

Start your buildspec.yml like this:

version: 0.2 env:

git-credential-helper: yes

That’s it. Now git clones will work in your CodeBuild agents.

3) Cloning the repository

A simple git clone <REPO_URL> in your buildspec will work, but you might not get the same source that is being used by your build — so we need to check out the right branch and git reset to the commit being built.

This is complex enough to be in a separate script which deals with this single concern and performs some error checking. Since I’ll want to do this on many projects, I’ve written a script you (or I) can drop in to any project. You can find it here, or download it with the following command:

Here’s a buildspec that uses the script and then prints the top of the git log:

version: 0.2 env:

git-credential-helper: yes phases:

build:

commands:

- scripts/codebuild-git-wrapper.sh <REPO_URL> <REPO_BRANCH>

- git log | head -100

(Note that you’ll need to provide your own values for REPO_URL and REPO_BRANCH).

Hopefully using your pipeline feels like this (Photo by Imani on Unsplash)

Seeing it in action

I’ve provided a CloudFormation template which contains an entire pipeline demonstrating the principle. If you want to see the whole pipeline in action, you can clone this repository and follow these instructions to get it up and running.

Summary

To get git metadata into CodeBuild/CodePipeline, follow these three steps:

Allow codecommit:GitPull in your CodeBuild role Put git-credential-helper: yes in the env part of your buildspec file Clone the repo, reset to the commit that matches CODEBUILD_RESOLVED_SOURCE_VERSION , then copy the .git directory to the build directory.

If you don’t want to write your own script to do the clone/reset/copy step, you can use this one I wrote for you.

If you want to see a whole pipeline in action, I’ve published an end-to-end sample pipeline here.