Package: gil
Gil (git links) toolGil is a git links tool to manage complex git repositories dependencies with
cycles and cross references.This tool provides a solution to thegit recursive submodules dependency problem.ContentsRequirementsSetupSampleUsageMotivation: git submodules vs git linksGit submodules managementRecursive submodules problemMotivation: git subtrees vs git linksGit subtrees managementSubtrees manual management problemRequirementsLinuxMacOSWindowsgitpython3Setupsudopip3install--prefix/usr/localgilSampleConsider we have the following git repository dependency graph:Where:Blue nodes are third-party componentsGreen nodes are components that we developEdges show components dependency with a branchRoot .gitlinks fileIn order to describe the sample model with git links we have to define root
.gitlinks file in your sample repository with a following content:# ProjectsCppBenchmarkCppBenchmarkhttps://github.com/chronoxor/CppBenchmark.gitmaster
CppCommonCppCommonhttps://github.com/chronoxor/CppCommon.gitmaster
CppLoggingCppLogginghttps://github.com/chronoxor/CppLogging.gitmaster# ModulesCatch2modules/Catch2https://github.com/catchorg/Catch2.gitmaster
cpp-optparsemodules/cpp-optparsehttps://github.com/weisslj/cpp-optparse.gitmaster
fmtmodules/fmthttps://github.com/fmtlib/fmt.gitmaster
HdrHistogrammodules/HdrHistogramhttps://github.com/HdrHistogram/HdrHistogram_c.gitmaster
zlibmodules/zlibhttps://github.com/madler/zlib.gitmaster# Scriptsbuildscripts/buildhttps://github.com/chronoxor/CppBuildScripts.gitmaster
cmakescripts/cmakehttps://github.com/chronoxor/CppCMakeScripts.gitmasterEach line describe git link in the following format:Unique name of the repositoryRelative path of the repository (started from the path of .gitlinks file)Git repository which will be used ingit clonecommandRepository branch to checkout(Optional) Relative path in the base repository(Optional) Relative path in the target repositoryEmpty line or line started with#are not parsed (treated as comment).CppBenchmark .gitlinks fileCppBenchmark .gitlinks file should be committed into the CppBenchmark project
and have the following content:# ModulesCatch2modules/Catch2https://github.com/catchorg/Catch2.gitmaster
cpp-optparsemodules/cpp-optparsehttps://github.com/weisslj/cpp-optparse.gitmaster
HdrHistogrammodules/HdrHistogramhttps://github.com/HdrHistogram/HdrHistogram_c.gitmaster
zlibmodules/zlibhttps://github.com/madler/zlib.gitmaster# Scriptsbuildbuildhttps://github.com/chronoxor/CppBuildScripts.gitmaster
cmakecmakehttps://github.com/chronoxor/CppCMakeScripts.gitmasterCppCommon .gitlinks fileCppCommon .gitlinks file should be committed into the CppCommon project and
have the following content:# ModulesCatch2modules/Catch2https://github.com/catchorg/Catch2.gitmaster
CppBenchmarkmodules/CppBenchmarkhttps://github.com/chronoxor/CppBenchmark.gitmaster
fmtmodules/fmthttps://github.com/fmtlib/fmt.gitmaster# Scriptsbuildbuildhttps://github.com/chronoxor/CppBuildScripts.gitmaster
cmakecmakehttps://github.com/chronoxor/CppCMakeScripts.gitmasterCppLogging .gitlinks fileCppLogging .gitlinks file should be committed into the CppLogging project and
have the following content:# ModulesCatch2modules/Catch2https://github.com/catchorg/Catch2.gitmaster
cpp-optparsemodules/cpp-optparsehttps://github.com/weisslj/cpp-optparse.gitmaster
CppBenchmarkmodules/CppBenchmarkhttps://github.com/chronoxor/CppBenchmark.gitmaster
CppCommonmodules/CppCommonhttps://github.com/chronoxor/CppCommon.gitmaster
zlibmodules/zlibhttps://github.com/madler/zlib.gitmaster# Scriptsbuildbuildhttps://github.com/chronoxor/CppBuildScripts.gitmaster
cmakecmakehttps://github.com/chronoxor/CppCMakeScripts.gitmasterUpdate the repositoryFinally you have to update your root sample repository:# Clone and link all git links dependencies from .gitlinks filegilclone
gillink# The same result with a single commandgilupdateAs the result you'll clone all required projects and link them to each other
in a proper way:Workingpath:~/gil/sample
...
Updatinggitlinks:~/gil/sample/.gitlinks
Updatinggitlinks:~/gil/sample/CppBenchmark/.gitlinks
Updategitlink:~/gil/sample/modules/Catch2->~/gil/sample/CppBenchmark/modules/Catch2
Updategitlink:~/gil/sample/modules/cpp-optparse->~/gil/sample/CppBenchmark/modules/cpp-optparse
Updategitlink:~/gil/sample/modules/HdrHistogram->~/gil/sample/CppBenchmark/modules/HdrHistogram
Updategitlink:~/gil/sample/modules/zlib->~/gil/sample/CppBenchmark/modules/zlib
Updategitlink:~/gil/sample/scripts/build->~/gil/sample/CppBenchmark/build
Updategitlink:~/gil/sample/scripts/cmake->~/gil/sample/CppBenchmark/cmake
Updatinggitlinks:~/gil/sample/CppCommon/.gitlinks
Updategitlink:~/gil/sample/modules/Catch2->~/gil/sample/CppCommon/modules/Catch2
Updategitlink:~/gil/sample/CppBenchmark->~/gil/sample/CppCommon/modules/CppBenchmark
Updategitlink:~/gil/sample/modules/fmt->~/gil/sample/CppCommon/modules/fmt
Updategitlink:~/gil/sample/scripts/build->~/gil/sample/CppCommon/build
Updategitlink:~/gil/sample/scripts/cmake->~/gil/sample/CppCommon/cmake
Updatinggitlinks:~/gil/sample/CppLogging/.gitlinks
Updategitlink:~/gil/sample/modules/Catch2->~/gil/sample/CppLogging/modules/Catch2
Updategitlink:~/gil/sample/modules/cpp-optparse->~/gil/sample/CppLogging/modules/cpp-optparse
Updategitlink:~/gil/sample/CppBenchmark->~/gil/sample/CppLogging/modules/CppBenchmark
Updategitlink:~/gil/sample/CppCommon->~/gil/sample/CppLogging/modules/CppCommon
Updategitlink:~/gil/sample/modules/zlib->~/gil/sample/CppLogging/modules/zlib
Updategitlink:~/gil/sample/scripts/build->~/gil/sample/CppLogging/build
Updategitlink:~/gil/sample/scripts/cmake->~/gil/sample/CppLogging/cmakeAll repositories will be checkout to required branches.If content of any .gitlinks changes it is required to rungil updatecommand
to re-update git links - new repositories will be cloned and linked properly!Commit, push, pullYou can work and change any files in your repositories and perform all git
operations for any repository to commit and contribute your changes.If you want to commit all changes in some repository with all changes in
child linked repositories you can do it with a single command:gilcommit-a-m"Some big update"This command will visit the current and all child repositories and perform
the correspondinggit commitcommand with provided arguments.Please note that gil command visit only child repositories that are described
in .gitlinks file no parent dependencies will be processed. For this reason
if you want to commit, pull or push all repositories the corresponding command
should be run in the root directory where root .gitlinks is placed.Pull, push commands works in a similar way:gilpull
gilpushBuildPlease investigate and follow links in the sample repository in order to
understand the logic how gil (git links) tool manages dependencies.Finally you can build sample projects with provided build scripts:~/gil/sample/CppBenchmark/build~/gil/sample/CppCommon/build~/gil/sample/CppLogging/buildAdditionally each module can be cloned and successfully build without root
repository. In this case local .gitlinks file will be used to resolve all
dependencies!UsageGil (git links) tool supports the following commands:usage: gil command arguments
Supported commands:
        help - show this help
        version - show version
        context - show git links context
        clone [args] - clone git repositories
        link - link git repositories
        update - update git repositories (clone & link)
        pull [args] - pull git repositories
        push [args] - push git repositories
        commit [args] - commit git repositories
        status [args] - show status of git repositoriesgil context- command will show the current git link context of the current directory;gil clone [args]- clone all repositories that are missed in the current context;gil link- link all repositories that are missed in the current context;gil update- clone and link in a single operation;gil pull [args]- pull all repositories in the current directory;gil push [args]- push all repositories in the current directory;gil commit [args]- commit all repositories in the current directory;gil status [args]- show status of all repositories in the current directory;Motivation: git submodules vs git linksGit submodules allow to get the same idea of repositories dependency, but with
a serious limitation - git submodules degrade to tree-like recursive structure
with repositories duplication.Git submodules managementGit submodules provides several operations to manage repositories dependency.Add submodule dependency# Add git submodule of 'master' branch into the relative directory 'modules/zlib'gitsubmoduleadd-bmasterhttps://github.com/madler/zlib.gitmodules/zlibInitialize submodules# Update all submodules in the current repositorygitsubmoduleupdate--init--recursive--remoteRefresh submodules from remote repositoriesgitpull
gitsubmoduleupdate--init--recursive--remote
gitpull--recurse-submodules
gitsubmoduleforeach"(git checkout master; git pull)"Synchronize submodules with remote repositoriesgitpull
gitsubmoduleupdate--init--recursive--remote
gitpull--recurse-submodules
gitsubmoduleforeach"(git checkout master; git pull)"gitadd--all
gitcommit-m"Submodule Sync"gitpushRemove submodule dependency# Remove git submodule from the relative directory 'modules/zlib'gitsubmoduledeinit-f--modules/zlib
rm-rf.git/modules/modules/zlib
gitrm-fmodules/zlibRecursive submodules problemGit submodules representation of the described sample repository is the
following:~/gil/sample/CppBenchmark
~/gil/sample/CppBenchmark/build
~/gil/sample/CppBenchmark/cmake
~/gil/sample/CppBenchmark/modules/Catch2
~/gil/sample/CppBenchmark/modules/cpp-optparse
~/gil/sample/CppBenchmark/modules/HdrHistogram
~/gil/sample/CppBenchmark/modules/zlib
~/gil/sample/CppCommon
~/gil/sample/CppCommon/build-DUPLICATE!
~/gil/sample/CppCommon/cmake-DUPLICATE!
~/gil/sample/CppCommon/modules/Catch2-DUPLICATE!
~/gil/sample/CppCommon/modules/CppBenchmark-DUPLICATE!
~/gil/sample/CppCommon/modules/CppBenchmark/build-DUPLICATE!
~/gil/sample/CppCommon/modules/CppBenchmark/cmake-DUPLICATE!
~/gil/sample/CppCommon/modules/CppBenchmark/modules/Catch2-DUPLICATE!
~/gil/sample/CppCommon/modules/CppBenchmark/modules/cpp-optparse-DUPLICATE!
~/gil/sample/CppCommon/modules/CppBenchmark/modules/HdrHistogram-DUPLICATE!
~/gil/sample/CppCommon/modules/CppBenchmark/modules/zlib-DUPLICATE!
~/gil/sample/CppCommon/modules/fmt
~/gil/sample/CppLogging
~/gil/sample/CppLogging/build-DUPLICATE!
~/gil/sample/CppLogging/cmake-DUPLICATE!
~/gil/sample/CppLogging/modules/Catch2-DUPLICATE!
~/gil/sample/CppLogging/modules/cpp-optparse-DUPLICATE!
~/gil/sample/CppLogging/modules/CppBenchmark-DUPLICATE!
~/gil/sample/CppLogging/modules/CppBenchmark/build-DUPLICATE!
~/gil/sample/CppLogging/modules/CppBenchmark/cmake-DUPLICATE!
~/gil/sample/CppLogging/modules/CppBenchmark/modules/Catch2-DUPLICATE!
~/gil/sample/CppLogging/modules/CppBenchmark/modules/cpp-optparse-DUPLICATE!
~/gil/sample/CppLogging/modules/CppBenchmark/modules/HdrHistogram-DUPLICATE!
~/gil/sample/CppLogging/modules/CppBenchmark/modules/zlib-DUPLICATE!
~/gil/sample/CppLogging/modules/CppCommon-DUPLICATE!
~/gil/sample/CppLogging/modules/CppCommon/build-DUPLICATE!
~/gil/sample/CppLogging/modules/CppCommon/cmake-DUPLICATE!
~/gil/sample/CppLogging/modules/CppCommon/modules/Catch2-DUPLICATE!
~/gil/sample/CppLogging/modules/CppCommon/modules/CppBenchmark-DUPLICATE!
~/gil/sample/CppLogging/modules/CppCommon/modules/CppBenchmark/build-DUPLICATE!
~/gil/sample/CppLogging/modules/CppCommon/modules/CppBenchmark/cmake-DUPLICATE!
~/gil/sample/CppLogging/modules/CppCommon/modules/CppBenchmark/modules/Catch2-DUPLICATE!
~/gil/sample/CppLogging/modules/CppCommon/modules/CppBenchmark/modules/cpp-optparse-DUPLICATE!
~/gil/sample/CppLogging/modules/CppCommon/modules/CppBenchmark/modules/HdrHistogram-DUPLICATE!
~/gil/sample/CppLogging/modules/CppCommon/modules/CppBenchmark/modules/zlib-DUPLICATE!
~/gil/sample/CppLogging/modules/zlib-DUPLICATE!Only 10 of 41 repositories are unique. Other duplicates each other multiple
times. Moreover some nested repositories included recursively! CppLogging
project has the third level of nesting (CppLogging -> CppCommon -> CppBenchmark).Each new level heavily increases the rate of duplication. As the result all
submodule management operations becomes slow and requires multiple executions
of synchronize script to pass changes from top level to the bottom. The count
of synchronizations in the worst case equals to recursive levels count.All this issues are solved with gil (git links) tool which allows to link all
required repositories together avoiding duplication. The tool also provides
operations to manage changes in all repositories with asimple easy to use
commands.Motivation: git subtrees vs git linksGit subtrees is an easy to use alternative to git submodules. It allows to nest
one repository inside another as a sub-directory.From the one side git subtrees are very lightweight with simple workflow. They
don't add any metadata files like git submodule does (i.e., .gitmodule) and
sub-project's code is available right after the clone of the super project is
done. Contents of the module can be modified without having a separate
repository copy of the dependency somewhere else.However they have several drawbacks. Contributing code back upstream for the
sub-projects is slightly more complicated. And developer should be very careful
of not mixing super and sub-project code in commits.Git subtrees requires precise manual management in case use have complicated
projects structure and lots of upstream commits to dependent projects.Git subtrees managementGit subtrees provides several operations to manage repositories dependency.Add subtree dependency# Add git subtree of 'master' branch into the relative directory 'modules/zlib'gitsubtreeadd-Pmodules/zlibhttps://github.com/madler/zlib.gitmaster--squashUpdate subtree from the upstream remote repositorygitsubtreepull-Pmodules/zlibhttps://github.com/madler/zlib.gitmaster--squashContribute changes from subtree back to the upstream remote repositorygitsubtreepush-Pmodules/zlibhttps://github.com/madler/zlib.gitmasterRemove subtree dependency# Remove git subtree from the relative directory 'modules/zlib'gitrm-fmodules/zlibSubtrees manual management problemInitially git subtrees lack of the samerepository duplicate problemwhich git submodules have. Therefore all repositories with multiple dependencies
will be duplicated and require manual and careful management.For instanceCppBenchmarkrepository presented four times in oursample:~/gil/sample/CppBenchmark
~/gil/sample/CppCommon/modules/CppBenchmark-DUPLICATE!
~/gil/sample/CppLogging/modules/CppBenchmark-DUPLICATE!
~/gil/sample/CppLogging/modules/CppCommon/modules/CppBenchmark-DUPLICATE!The first problem if you updatedCppBenchmarkand want to get the fix in all
dependent repositories you have to update them manually with the set ofgit subtree pullcommands:# Commit and push changes in CppBenchmark repositorycdCppBenchmark
gitcommit-a-m"Bugfix in CppBenchmark"gitpushcd..# Update CppCommon repositorycdCppCommon
gitsubtreepull--prefixmodules/CppBenchmarkhttps://github.com/chronoxor/test-CppBenchmark.gitmain--squash
gitpushcd..# Update CppLogging repositorycdCppLogging
gitsubtreepull--prefixmodules/CppBenchmarkhttps://github.com/chronoxor/test-CppBenchmark.gitmain--squash
gitsubtreepull--prefixmodules/CppCommonhttps://github.com/chronoxor/test-CppCommon.gitmain--squash
gitpushcd..The second problem if you found a bug inCppBenchmarkwhile debuggingCppLoggingand want to fix it in place, push the changes and update all
repositories properly. In this case you need to make the following steps:# Commit and push changes in CppBenchmark but from CppLogging repositorycdCppLogging
gitcommit-a-m"Bugfix in CppBenchmark (found in CppLogging repository)"gitpush
gitsubtreepush--prefixmodules/CppBenchmarkhttps://github.com/chronoxor/test-CppBenchmark.gitmain--squashcd..# Update CppBenchmark repositorycdCppBenchmark
gitpullcd..# Update CppCommon repositorycdCppCommon
gitsubtreepull--prefixmodules/CppBenchmarkhttps://github.com/chronoxor/test-CppBenchmark.gitmain--squash
gitpushcd..# Update CppLogging repositorycdCppLogging
gitsubtreepull--prefixmodules/CppBenchmarkhttps://github.com/chronoxor/test-CppBenchmark.gitmain--squash
gitsubtreepull--prefixmodules/CppCommonhttps://github.com/chronoxor/test-CppCommon.gitmain--squash
gitpushcd..As the result too much manual work should be done to make all repositories up
to date. Definitely all commands should be stored into scripts with care. And
this job requires some efforts and testing because it is very easy to make a
mistake. Git subtrees are good and easy to use for small repositories without
complicated hierarchy. But working with huge complicated repositories using
git subtrees is a big challenge as git subtrees do not suppot recursing
pull/push/update commands.