Understanding the Fundamentals of Git | by Rachit Tayal | Towards Data Science

Git is a powerful tool, but it can be overwhelming especially for newcomers. Even for experienced developers, getting stuck in a merge or a rebase conflict is pretty common. Even with extensive blogs available, it can be sometimes tricky to identify the cause, ultimately ending up wasting our productive time.
There are a plethora of tutorials out there already, but most of them simply talk about high-level user commands, syntax, and how to use them abstracting out most of the internal details.
This article tries to uncover how Git works under the hood. But why should we even bother about that? Knowing how it works from the ground up can help a great deal in using the commands. Since it’s not about learning the commands, instead it’s about learning the conceptual model. So, instead of learning a bunch of commands, we will try to understand how the commands work. Once we start to understand it, we’ll be more and more comfortable and confident in using the commands, and eventually, we’ll start enjoying it. Besides, there is always a geeky pleasure in knowing the internals. With this, let’s demystify git.
At its very core, git is a simple key-value store, a simple structure that maps keys to values and this structure is persistent i.e. it’s stored in our disk. The values are just sequences of bytes. It can be any kind of content, be it a text file or binary file, etc. We can provide a value to git and git will calculate a unique key for it, which can be used later to retrieve the content.
As a demonstration, we’ll use the hash-object command, which takes some data and gives back the unique key. Since hash-object does not operate directly on string data, we’ll have to pipe the input and pass the — stdin flag to let hash-object know. (hash-object is one of the plumbing commands, it’s totally fine if it's not used by you before)
$ echo "Hello Sarah" | git hash-object --stdin0e170dcd3efae4d183c78dced6c5b9c7e13f70c5
The above-generated hash will be used as a key to store that content. It is generated using the SHA1 algorithm. To dig deep into SHA1, you can refer here. The important point to note is, for the same sequence of bytes, we’ll always get the same hash every time on every operating system. And it’s almost impossible that we’ll get the same hash for two different content.
Therefore, every piece of content in the git repository will have its own hash. Now, let’s try to store this content in git and see how it works. First, we’ll have to initialize a new Git repository:
$ git init testInitialized empty Git repository in /workdir/test/.git/$ cd test$ ls -a./    ../   .git/$ ls .git/objectsinfo/  pack/
Now, we have a .git directory to save our contents. If we examine the .git directory, it has a subdirectory called objects which is called the object database. This is where git saves all the objects for saving content, files, directories, etc.
Currently, there are no objects except for pack and info subdirectories created as part of initialization. Using the hash-object again with -w the flag will not only return the key but write that object to the .gitstore.
$ echo "Hello Sarah" | git hash-object -w --stdin0e170dcd3efae4d183c78dced6c5b9c7e13f70c5$ ls .git/objects/0e/   info/ pack/$ find .git/objects -type f.git/objects/0e/170dcd3efae4d183c78dced6c5b9c7e13f70c5
We can see, git has stored our data. A directory is created by the first two hexadecimal digits (0e) of the hash. A file is present inside it with the remaining characters of the hash. This is the naming scheme git follows. Our original data is present in that file. This file is called the blob of data.
A blob is a generic piece of content. Git added a small header and compressed the original content inside this blob. We can examine the content with the git cat-file command. Passing -t will tell us the type of content while the -p flag will print the contents of that object.
$ git cat-file 0e170dcd3efae4d183c78dced6c5b9c7e13f70c5 -tblob$ git cat-file 0e170dcd3efae4d183c78dced6c5b9c7e13f70c5 -pHello Sarah
Therefore, git can take any piece of content, generates a key for it as shown in the hash, and then persist the content in the repository as a blob.
We’ll now dive a little deeper into the object database. We’ll explore what and how are the various types of objects git stores.
There are 3 main types of objects that git stores:
Blob: This object as we have seen above stores the original content.
Tree: This object is used to store directories present in our project.
Commit: This object is created whenever a commit is made and abstracts all the information for that particular commit.
Let’s create an example project to understand each one in-depth. An app project. At the root of it, we’ll have a README.md file and a src directory. Inside src directory, we have a textfile and hello_world.py source file.
$ git init app$ cd app$ echo "To demonstrate GIT objects" > README.md$ mkdir src$ cd src$ echo "To demonstrate GIT objects" > textfile$ echo "print('hello world')" > hello_world.py$ tree ..├── README.md└── src    ├── hello_world.py    └── textfile1 directory, 3 files
Currently, there is nothing committed to our git repository, so let’s make the initial commit of adding these files and directory. Once the commit is made, we’ll inspect the objects directory to see the various objects created.
$ ls -a .git/objects./    ../   info/ pack/$ git add .$ git statusOn branch masterNo commits yetChanges to be committed:(use "git rm --cached <file>..." to unstage)new file:   README.mdnew file:   src/hello_world.pynew file:   src/textfile$ git commit -m "inital commit, adding README and src dir"$ git statusOn branch masternothing to commit, working tree clean$ git logcommit 67e94955f7d3fffd307d0f070e17d652800b503e (HEAD -> master)inital commit, adding README and src dir$ ls -a .git/objects./    ../   17/   63/   67/   75/   b9/   info/ pack/$ find .git/objects/ -type f.git/objects/67/e94955f7d3fffd307d0f070e17d652800b503e.git/objects/17/14900a9b496867b319101942b690af9e4d399e.git/objects/75/d9766db981cf4e8c59be50ff01e574581d43fc.git/objects/63/03057f97f1cb0222d3e916c3ab471e3c06b9b2.git/objects/b9/941014e9cc4cb12e3b1283de8846127f96e6d7
We see one commit in the commit history but Inside the objects directory, we see the various objects being created. Let’s inspect them one by one using the git cat-file command starting with the commit viewed in log command.
$ git cat-file 67e949 -tcommit$ git cat-file 67e949 -ptree 1714900a9b496867b319101942b690af9e4d399eauthor RachitTayal <rachit.tayal@gmail.com> 1624011001 +0530committer RachitTayal <rachit.tayal@gmail.com> 1624011001 +0530inital commit, adding README and src dir
As we can see above, the commit object contains a simple and short piece of text. Whenever a commit is made (using git commit), git generates this text, generates its hash, and then stores it pretty much the same way a blob is stored. The commit text contains all the metadata about the commit. Apart from Author, committer, date of commit, it has a hash for the tree object. This commit (171490) is pointing to the root directory of the project. Let’s fetch the contents of this tree object to understand it further.
$ git cat-file 171490 -ttree$ git cat-file 171490 -p100644 blob 6303057f97f1cb0222d3e916c3ab471e3c06b9b2 README.md040000 tree b9941014e9cc4cb12e3b1283de8846127f96e6d7 src
Just like a commit object, a tree object is also a piece of text. It contains the list of contents (represented by hashes) present in the directory. In this case, we have a blob and another tree with their corresponding names. blob is the README.md file present in the root while the tree is the src directory in the root.
Let’s use the git cat-file again to get the contents of the blob first.
$ git cat-file 630305 -tblob$ git cat-file 630305 -pTo demonstrate GIT objects
So, the contents of the blob are the same as what present inside our README.md file.
To summarize, a commit points to a tree (signifying root), and this tree points to a blob (README.md file), and another tree (src directory). The blob is just a piece of content present inside the README.md file. Let’s quickly inspect the other tree.
$ git cat-file b99410 -ttree$ git cat-file b99410 -p100644 blob 75d9766db981cf4e8c59be50ff01e574581d43fc hello_world.py100644 blob 6303057f97f1cb0222d3e916c3ab471e3c06b9b2 textfile
It has two blobs. One of the blobs stores the content of the hello_world.py file. While the other blob stores the content of textfile . The important thing to note is blob for textfile and README.md has the same hash because the contents of both the files are exactly the same. Therefore, instead of creating two separate copies of the same blob, git will store a single blob.
The overall object database can be depicted as follows:
Now, let’s make some changes in theREADME.md file and make a second commit.
$ echo "Added some change" >> README.md$ git add README.md$ git statusOn branch masterChanges to be committed:(use "git reset HEAD <file>..." to unstage)modified:   README.md$ git commit -m "updated README"$ git logcommit 5abd2e3d6903385c2d7f646811f84decd41bcdc7 (HEAD -> master)...commit 67e94955f7d3fffd307d0f070e17d652800b503e
Let’s peek inside the second commit using the cat-file command again.
$ git cat-file 5abd2e -tcommit$ git cat-file 5abd2e -ptree 62da8c3c87d1b7725dbcc4450d5ec25384a7614eparent 67e94955f7d3fffd307d0f070e17d652800b503eauthor RachitTayal <rachit.tayal@gmail.com> 1624019462 +0530committer RachitTayal <rachit.tayal@gmail.com> 1624019462 +0530updated README
Apart from the obvious fields, it also has a parent commit that points to the commit of the previous hash. Commits are linked in that fashion. Most commits have a parent. (exception is the first commit). The hash of the tree present (62da8c) in this commit is different from what the first commit was pointing to, which means it is pointing to a new tree.
Inspecting the contents of the new tree will yield us:
$ git cat-file 62da8c -p100644 blob 5ca3f4ffb43baf06df6e3fcb0a056cc6263cf32f README.md040000 tree b9941014e9cc4cb12e3b1283de8846127f96e6d7 src
We can see, it has another tree (src directory) and a blob (README.md) file. The blob does not have the same hash as in the previous commit because we have added changes to the README.md file. Hence a new blob is created.
However, the tree object has the same hash as the previous commit, since no changes were made, git will reuse the previous tree object. The file structure of the object database after the second commit will look like this:
With this, we got the overall gist of how the git object model works. Once this is absorbed, understanding further complex operations like merging, rebasing, and branching strategies become very intuitive. (which we’ll be covering in the next article).
We have seen how git does the content tracking based on key-value structure. Keys are based on hashes generated by the SHA1 algorithm. We have seen various plumbing commands like git hash-object and git cat-file to generate the hash and view the contents of the object file respectively.
Next, we saw what goes behind the scenes in the git object database whenever we make commits to our project. It highlighted the overall git object model, the various types of objects stored in git (blobs, tree, commits), and how the objects are interleaved together on further commits. This lays the foundation for understanding branching and merging related operations, and various other advanced concepts, which we will discuss shortly in the upcoming article. Till then keep exploring.