Anatomy of Modules in Go
By: Uday Hiwarale
Tags: Go, Golang Tutorial, Programming, Golang

Let’s first understand the mechanism of the Go modules, in theory. As we discussed some problems with Go’s traditional dependency management, there are few modifications we can easily propose.

First of all, we should be able to work from any directory, not just GOPATH which gives us the flexibility to relocate our source code anywhere.

which gives us the flexibility to relocate our source code anywhere. We should be able to install the precise version of a dependency package to avoid breaking changes.

We should be able to import multiple versions of the same dependency package. This is useful when our old application code is running with the old version of the dependency but we would like to use the new version of the dependency package for any new developments.

Like package.json in NPM, we need a file that can list the dependencies of our project. This is helpful because when you distribute your project, you don’t need to distribute your dependencies with it. Go can just look at this file and download the dependencies for you.

Well, the good news is, Go modules can do all these for you and much more. Go modules gives us a built-in dependency management system. But, let’s understand what a module is first.

A module is nothing but a directory containing Go packages. These can be distribution packages or executable packages.

💡 A module can also be treated as a package that contains nested packages. You can learn about nested packages from the packages lesson.

A module is also like a package that you can share with other people. Hence, it has to be a Git or any other VCS repository which we can be hosted on a code-sharing platform like Github. Hence, Go recommends

A Go module must be a VCS repository or a VCS repository should contain a single Go module.

repository or a repository should contain a single Go module. A Go module should contain one or more packages

A package should contain one or more .go files in a single directory.

Enough theory, let’s code now. Let’s create an empty directory anywhere but inside GOPATH. I am using nummanip directory to host my module and it will contain some packages to that to manipulate number data type.

As discussed before, since a Go module should be a VCS repository, I have created a Git repository on Github with below URL.

Now, the first step is to initialize the Go Module inside this directory. For that, we have go mod init <import-path> command.

This command creates go.mod file (which is like package.json file in NPM) that contains module import path and dependencies used by packages inside it. Also, let’s initialize the Git repository with the above GitHub remote URL.

In the first step, we initialized a Git repository and pointed it to the Github repository we just created. In the second step, we initialized a Go module and specified module import path in the command.

go mod init github.com/thatisuday/nummanip

Creating modules inside GOPATH is disabled by default. You will get this error if you initiate a go module inside $GOPATH → go: modules disabled inside GOPATH/src by GO111MODULE=auto; see ‘go help modules’ .

💡 This is a preventive measure to deprecate GOPATH in the near future. If you really need to create Go modules inside GOPATH, set GO111MODULE environment variable to on . This environment variable also controls how go get should behave inside GOPATH and outside of it. When GO111MODULE=off , the go get command simply clones the package from the master repository of the package and puts it inside $GOPATH/src . If GO111MODULE=on , then go get command puts the package code inside a separate directory and it supports versioning (discussed later in this article) From Go 1.13, GO111MODULE is set to auto by default which means go get command inside GOPATH will act as if GO111MODULE=off and when it is invoked outside GOATH, it will act as if GO111MODULE=on . You can use GO111MODULE=on go get <package>@<version> command to set the GO111MODULE environment variable in the command execution.

The above command created a go.mod file which contains the module import path and version of the Go this module was created on. The most difficult part is done and from here, we should focus on developing our packages.

We have created two packages inside our module. Currently, these are empty directories but soon we will put some code in there. calc package will export utilities to calculate something with numbers while transform will be used to transform numbers or number related data types (like an array of numbers).

⚠️ Since we are providing multiple packages through our module, we have created directories for each of them. But if want to just provide single package, you don’t need to create a separate directory for it. You can just write the package code inside the module directory (where go.mod file is). The only difference is how you import your package, which we will see later.

At this point, we are not sure if our module is a fully-fledged executable application or a distribution module with utility package(s). I always recommend keeping your reusable logic in separate packages and import it in your application code.

So to test our module and packages, we are going to create another module that will consume nummanip module packages. We won’t be publishing this test module as it is just for testing, hence we can init this module with a non-URL module name. We can use go mod init main to initialize the main package which will be the executable package.

💡 Go provide go test utility to test your packages with custom test suits. But that is a completely different topic which we will perhaps cover in upcoming tutorials.

We have created main module for testing purpose using the command go mod init main which created go.mod file. Also for simplicity, we have opened both main and nummanip module in VSCode as a workspace.

We have created math.go file inside calc package to provide a utility function to return the sum of two numbers.

Focus on package declaration part, package calc signifies that math.go belongs to calc package and since this package is in a separate directory under our module, package declaration has nothing to do with the module name. We have learned this inside the nested packages lesson.

💡 If our package code were inside the module directory, then package name would be same as the module name (in order to successfully resolve import statement).

Let’s publish our module. Publishing is just pushing your code to the remote VCS repository with a tag. But before we do that, we need to understand semantic versioning and git tags. So let’s get into it.

Semantic Versioning or SemVer is a universally accepted format to tag your release packages/modules etc. It has vX.Y.Z format where X is a major version number, Y is a minor version number and Z is patch version number (translated as vMajor.Minor.Patch).

For example, a package version 1.2.0 has the major version 1 , minor version 2 and patch version 0 . We bump only patch version when there is a small change in the package, minor in case new or enhanced functionality is added. When there is a major difference between the old version and the new version, we bump up the major version number resetting minor and patch version to 0 (for example 2.0.0 ).

💡Additional pre-release tags can be added as a suffix with release tag which goes like this x.y.z-rc.0 which means it is a release with release-component 0 (or x.y.z-beta.1 ). This is helpful to those who want to test the beta version.

Go specifies that, when there is an incompatibility between the new version and the old version, then the new version should be a major release. When a major version is released, Go treats it as a different module which we will see later in this tutorial and discuss why that is important.

As we know, a Git branch is nothing but a history of commits. Each commit has a unique identifier (commit hash) associated with them. At any specific commit, we can extract the state of the files in the repository.

When you want to release anything for public use, you need to provide a commit hash where the state of the files in the repository is stable and can be used in the production. But what we can also do is create an alias for the commit hash with the SemVer name. It’s called tagging.

Git provides two types of tags. The Lightweight Tag is simply a pointer to commit in Git History. While Annotated Tag is saved as a full object (which contains additional information like Tagger’s name, tag message, and other info) in the Git database. You can read about Annotated Tags here in depth because for simplicity, we will use lightweight tags.

Since we are releasing our module for the first time, we need to create a commit and push it to the remote repository. Then we will create a tag from the last commit (which will be the one we pushed) with a SemVer.

From above, we have created a commit and pushed it to the remote master branch. -f flag was used to forcefully push the local git history to the remote which is OK for the first-ever commit. Let’s create a Git Tag.

Since we are releasing the first version of our module, the SemVer should be v1.0.0 . When you are releasing a Go module, your SemVer tag must start with a lowercase v (for successful version resolution). When you create a Git tag, you need to push it to the remote repository with git push --tags .

GitHub provides information about tag inside the Releases section.

I have created app.go file inside main module to test our newly release Go module. Basically, we are importing the calc package from github.com/thatisuday/nummanip module to utilize Add method.

Since we know the import path of the module (and package in it), we can import directly in our code with a full URL path to the package which is github.com/thatisuday/nummanip/calc . To understand this, check out the nested packages lesson.

💡 If our package code were inside module directory itself, we would have imported it with github.com/thatisuday/nummanip only and we would have needed to use nummanip as the package variable to run nummanip.Add() .

Notice that until now, we haven’t installed this module yet using go get . Yes, to install a module, you can also use go get <package-name> command or use go get command to install all packages listed inside go.mod file.

But, when we run app.go using the command go run <file> , Go fetches latest version v1.x.x (explained later) of nummanip module. When Go successfully downloads the module, it also updates the go.mod file to save the downloaded dependency.

💡 This way, we do not need to tell other people which dependencies to install. Go can just look at go.mod file for dependencies needed by the module.

go.mod file

⚠️ You might ask, how Go resolves the package import URL as https://github.com/thatisuday/nummanip/calc returns 404 Not Found page. I couldn’t find exact documentation on it but my guess is that since GitHub is one of the recognized code hosting sites, it knows where the package is located and this documentation suggests that.

Now, there are many questions popping in our heads. When Go will install the module automatically? What’s the use of go get then? Where are the modules stored on the system?

Go modules are stored inside $GOPATH/pkg/mod directory (module cache directory). Seems like we haven’t been able to get rid of $GOPATH at all. But Go has to cache modules somewhere on the system to prevent repeated downloading of the same modules of the same versions.

When we execute the command go run or other Go commands like test , build , Go automatically checks the third party import statements (like our module here), and clones the repository inside module cache directory.

($GOPATH/pkg/mod)

We can see the cache directory has nummanip module tagged with version v1.0.0 to make import resolution simple. Go creates go.sum file to save checksums of the downloaded direct and indirect dependencies’ content.

go.sum file

Unlike package-lock.json file in npm which stores versions of the currently installed direct and indirect dependencies for 100% reproducible builds, go.sum file is not a lock file and but it should be committed along with your code in case your module is a VCS repository (explained here).

When you shared your module, go.sum plays an important role to ensure 100% reproducible builds by validating the checksum of each module.

Let’s add some patch version level functionality to our calc package.

Add is not variadic function

What we’ve changed in Add function is the ability to accepts multiple arguments using a variadic function argument (read tutorial). We then pushed a new commit with tag v.1.0.1 .

Let’s implement a newly improved Add function inside main module. But there is a gotcha here. Since Go already has nummanip module, go run or other commands will not fetch the newer version. To do that, we need to update our module manually (in the worst case, reinstall it).

update Go modules

To update Go modules present inside a module with go.mod file, use go get -u command. This command will update all the modules with the latest minor or patch version of the given major version (explained later). If you want to update only patch version even if the newer minor version is available, use go get -u=patch command instead.

If you need a precise version of a dependency module, you can use go get module@version command, for example, in our case to install v1.1.2 , we should use go get github.com/thatisuday/nummanip@v1.1.2 .

$GOPATH/pkg/mod

As you can see from the above screenshots, Go have downloaded the latest version v1.0.1 and saved in the module cache directory. This is a great way to manage dependencies on a global level, where multiple modules on the system can access different versions of the dependency.

Upgrading to a Major Version

Now the time has come to explain why Go only automatically resolves only v1.x.x versions of a dependency module or why go get -u updates only modules within the current major version.

It is generally accepted that a major version of a dependency is needed when there is a substantial change in how given dependency works. For example, Angular v1 is very different from Angular v2. They both works in very different ways.

Also, when a major version of a dependency is imported, it could lead to incompatibilities which means your old code might now work with the new major released version of the dependency.

Hence, if go get -u would have updated the dependency module to a major version for example, from v1.0.2 to v2.0.0 then our application might now have worked/built properly. Then how Go solves this issue?

Go says, when you update your module to a newer major version, technically it’s a different module as it’s not backward compatible. So module with v1.x.x is different package compared to v2.x.x . That means the consumer who has imported your module must manually install the module with the new major version using go get and specify the new import path.

So what’s the deal with a new import path? As we have go.mod file which specifies the module import path at the top, we need to change to something else. But don’t worry, the only modification we need is to append major version code (vX) to the import path so that consumer can import multiple versions of the same package. Let’s see that in action.

incompatible changes in the Add function

From the above example, we can see that, we added the functionality inside the function Add to check if arguments are greater than 2. If arguments passed are less than 2, then Add will return an error as its first return value, else sum as its second return value, which also makes sense.

This implementation of the function Add will not work with previous code as Add in version 1.x.x used to return only one value. Which means, our code has officially become backward-incompatible (not something to cheer for though). Hence, it’s fair to assume that the next release version should be v2.0.0 as it is a different package overall and Go is not wrong about that.

If you have noticed, we created a different branch v2 for the new major release. This will make thing easier to handle since we need to also support v1 in case some bugs or enhancement is needed there.

Now the most important thing is to update our go.mod file to add version prefix in the module import statement.

Go understands the vX syntax and successfully resolve the module import despite this confusing import path. vX syntax is not flexible, which means it should be precise with the released SemVer major version release number, for example, v2 for v2.x.x release.

(installing v2)

In the above example, we have installed a newer major release version of nummanip module. Since it’s a major release, we need to use go get command to manually install it. Since this newer release also has new import syntax, we need to use that too.

The annoying things could be aliasing your package. Since we are importing two different versions of the same package, we need to alias either one of them to resolve the same package name variable conflict. Hence, we aliased v2/calc package to calcNew and used it to access Add function.

go.mod file

module cache

You can also see that, go get command has added module entry to go.mod file and saved the newer version of the module inside the cache directory.

Run and Build Executable Module

When you are working on an executable module, you can run the module by using go run . command or run individual files using go run path/*.go command which I have explained in my packages tutorial.

When it comes to building a module, you can either use go build or go build . command which builds the module and outputs the binary in the current directory.

You can also use go install or go install . which installs (outputs) the module binary inside $GOPATH/bin .

💡 When it comes to non-executable modules like nummanip module in our case, unlike pre-go1.11, you can’t create package archives using go install command. This is because go install can’t predict the module version (SemVer) of a module from the Git history or Git tags. Also, Go Modules do not save packages as binary archives as explained in my packages tutorial.

Indirect Dependencies

We have used the “Indirect Dependencies” term before but haven’t explained yet. Well, Indirect Dependencies as they sound aren’t direct dependencies to our module. Direct dependencies are those which our module imports while indirect dependencies are those which direct dependencies import.

go.mod files note downs both direct and indirect dependencies and mark them using a trailing comment as shown in the below example.

(direct vs indirect dependencies)