Building Git in Elixir — Part 5 (Build Index)
By: Meraj Molla
Tags: Ruby, Git, Elixir, Software Development, Version Control

In the previous article, I focused on storing nested trees as Merkle trees so that egit can store nested directories. In this article, I will focus on writing building index so that in subsequent articles we can store changes incrementally in repository.

A Bit of Theory

Git Index is used to provide cache of all the blobs representing current state of the project. Git add command uses this information to incrementally save changes in repository. Index file caches a lot of information from filesystem, such as size of file, last modified time, file mode, etc that Git add command can use to speed up operations.

At this stage it’s a good idea to inspect index file that resides under .git folder. We have the directory structure as below —

If we do a git add and inspect index with hexdump command we can see —

For understanding this data format you can look at [5]. But in short it has —

— 12 byte header that contains —

4-byte signature (‘D’, ‘I’, ‘R’, ‘C’)

4-byte version number — in this case version is 2

32 bit number of entries in index — in this case 4

For each entry, it stores ten 4-byte numbers that’s obtained via call to stat() function, 20-byte SHA-1 of the blob that appears, 2-byte set of flags (that includes length of filename), filename padded with zeros such that total length is multiple of 8.

Stat() information stored in ten 4-byte numbers are —

32-bit ctime seconds

32-bit ctime nanoseconds

32-bit mtime seconds

32-bit mtime nanosecond

32-bit dev

32-bit ino

32 bit mode

32-bit uid

32-bit gid

32-bit file size

The index is ended by another 20-byte SHA-1 hash — which is hash of the entire index content itself so that Git can check integrity of index itself.

With git ls-files we can see all the files that are part of index —

For a single entry as below —

It’s easy to see first 12-byte header (underlined yellow), and last 20-byte SHA-1 (underlined red). Rest belong to the entry for hello.txt.

Focus of This Article

In this article, focus is to replicate this behaviour —

add new egit add command

command this command will build the index

Elixir Code Walkthrough

The finished code for this article is available here — https://github.com/imeraj/elixir_git

The README file in the repository contains instructions to build egit and how to use egit add command.

Build Index & egit add Implementation

cli.ex now include a new clause to handle egit add command —

Since we need the ability to store files and nested directories, all the paths are sent to the processing logic as a list —

To process new egit add command, there is a new module that looks like —

Here —

line 11–26 — processes the paths and for each path it calls Workspace.list_files() to list the files, reduces them into index. So once this block of code executes, we have a list of indices for all the paths given

— processes the paths and for each path it calls to list the files, reduces them into index. So once this block of code executes, we have a list of indices for all the paths given line 28–32 — combines those indices and get a flat list of all entries that goes inside index

— combines those indices and get a flat list of all entries that goes inside index line 34–35 — does the final writing of this single final index to file

Newly, added type Index does most of the heavy lifting —

Here —

line 10–13 — creates an entry for index (which is of new type Egit.Types.Index.Entry and updates the index by adding new entry

— creates an entry for index (which is of new type and updates the index by adding new entry line 107–123 — defines the to_s() function that builds the entry as described above in theory section

— defines the function that builds the entry as described above in theory section line 15–45 — defines write_updates() function that takes the final index and writes to file by first creating an index.lock file. The writes process is broken into three functions so that after writing each entry the SHA-1 value can be updated —

finish_write() writes the final hash for the entire index.

Once these writes are done, it renames index.lock to index so that we get our final index.

Take egit For a Spin

Finally, it’s time to take egit for a spin. I ran our newly added egit add command on the same directory structure mentioned in theory section above which gives the same results—

Conclusion

In part 5 of this article series, I have added egit add command to build index so that in subsequent articles I can build on that and support incremental update to files instead of storing entires contents using egit commit every time.

For more elaborate and in depth future technical posts please follow me here or on twitter.

References