Package: git-filter-repo
git filter-repo is a versatile tool for rewriting history, which includescapabilities I have not found anywhere
else.  It roughly falls into the
same space of tool asgit
filter-branchbut without the
capitulation-inducing poorperformance,
with far more capabilities, and with a design that scales usability-wise
beyond trivial rewriting cases.git filter-repo is now recommended by the
git projectinstead
of git filter-branch.While most users will probably just use filter-repo as a simple command
line tool (and likely only use a few of its flags), at its core filter-repo
contains a library for creating history rewriting tools.  As such, users
with specialized needs can leverage it to quickly createentirely new
history rewriting tools.Table of ContentsPrerequisitesHow do I install it?How do I use it?Why filter-repo instead of other alternatives?filter-branchBFG Repo CleanerSimple example, with comparisonsSolving this with filter-repoSolving this with BFG Repo CleanerSolving this with filter-branchSolving this with fast-export/fast-importDesign rationale behind filter-repoHow do I contribute?Is there a Code of Conduct?Upstream ImprovementsPrerequisitesfilter-repo requires:git >= 2.22.0 at a minimum;some featuresrequire git >= 2.24.0 or laterpython3 >= 3.5How do I install it?git-filter-repo is a single-file python script, which was done to make
installation for basic use on many systems trivial: just place the
script into your $PATH.SeeINSTALL.mdfor things beyond basic usage or special
cases.  The more involved instructions are needed if youare working with a python3 executable named something other than "python3"want to install documentation (beyond the builtin docs shown with -h)want to run some of thecontribexampleswant to create your own python filtering scripts using filter-repo as a
module/libraryHow do I use it?See theuser
manual.If you prefer learning from examples:thesimple examplebelow may
be of interestthe user manual has an extensiveexamples
sectionthere is acheat sheet for converting filter-branch
commands,
which covers every example from the filter-branch manualthere is acheat sheet for converting BFG Repo Cleaner
commands,
which covers every example from the BFG websiteExternal sites also havealternative formats of the user
manualavailable, at least
for the most recent release.  This also may be beneficial if
htmlpreview.github.io starts hitting load limits.Why filter-repo instead of other alternatives?This was covered in more detail in aGit Rev News article on
filter-repo,
but some highlights for the main competitors:filter-branchfilter-branch isextremely to unusably
slow(multiple orders of magnitude slower than it should
be)
for non-trivial repositories.filter-branch is riddled with
gotchasthat can
silently corrupt your rewrite or at least thwart your "cleanup"
efforts by giving you something more problematic and messy than what
you started with.filter-branch isvery onerousto
usefor any rewrite which is even slightly non-trivial.the git project has stated that the above issues with filter-branch
cannot be backward compatibly fixed; they recommend that youstop
using
filter-branchdie-hard fans of filter-branch may be interested infilter-lamely(a.k.a.filter-branch-ish),
a reimplementation of filter-branch based on filter-repo which is
more performant (though not nearly as fast or safe as
filter-repo).acheat
sheetis available showing how to convert example commands from the manual of
filter-branch into filter-repo commands.BFG Repo Cleanergreat tool for its time, but while it makes some things simple, it
is limited to a few kinds of rewrites.its architecture is not amenable to handling more types of
rewrites.its architecture presents some shortcomings and bugs even for its
intended usecase.fans of bfg may be interested inbfg-ish, a reimplementation of bfg
based on filter-repo which includes several new features and bugfixes
relative to bfg.acheat
sheetis available showing how to convert example commands from the manual of
BFG Repo Cleaner into filter-repo commands.Simple example, with comparisonsLet's say that we want to extract a piece of a repository, with the intent
on merging just that piece into some other bigger repo.  For extraction, we
want to:extract the history of a single directory, src/.  This means that only
paths under src/ remain in the repo, and any commits that only touched
paths outside this directory will be removed.rename all files to have a new leading directory, my-module/ (e.g. so that
src/foo.c becomes my-module/src/foo.c)rename any tags in the extracted repository to have a 'my-module-'
prefix (to avoid any conflicts when we later merge this repo into
something else)Solving this with filter-repoDoing this with filter-repo is as simple as the following command:gitfilter-repo--pathsrc/--to-subdirectory-filtermy-module--tag-rename'':'my-module-'(the single quotes are unnecessary, but make it clearer to a human that we
are replacing the empty string as a prefix withmy-module-)Solving this with BFG Repo CleanerBFG Repo Cleaner is not capable of this kind of rewrite; in fact, all
three types of wanted changes are outside of its capabilities.Solving this with filter-branchfilter-branch comes with a pile of caveats (more on that below) even
once you figure out the necessary invocation(s):gitfilter-branch\--tree-filter'mkdir -p my-module && \git ls-files \| grep -v ^src/ \| xargs git rm -f -q && \ls -d * \| grep -v my-module \| xargs -I files mv files my-module/'\--tag-name-filter'echo "my-module-$(cat)"'\--prune-empty----allgitclonefile://$(pwd)newcopycdnewcopygitfor-each-ref--format="delete %(refname)"refs/tags/\|grep-vrefs/tags/my-module-\|gitupdate-ref--stdingitgc--prune=nowSome might notice that the above filter-branch invocation will be really
slow due to using --tree-filter; you could alternatively use the
--index-filter option of filter-branch, changing the above commands to:gitfilter-branch\--index-filter'git ls-files \| grep -v ^src/ \| xargs git rm -q --cached;git ls-files -s \| sed "s%$(printf \\t)%&my-module/%" \| git update-index --index-info;git ls-files \| grep -v ^my-module/ \| xargs git rm -q --cached'\--tag-name-filter'echo "my-module-$(cat)"'\--prune-empty----allgitclonefile://$(pwd)newcopycdnewcopygitfor-each-ref--format="delete %(refname)"refs/tags/\|grep-vrefs/tags/my-module-\|gitupdate-ref--stdingitgc--prune=nowHowever, for either filter-branch command there are a pile of caveats.
First, some may be wondering why I list five commands here for
filter-branch.  Despite the use of --all and --tag-name-filter, and
filter-branch's manpage claiming that a clone is enough to get rid of
old objects, the extra steps to delete the other tags and do another
gc are still required to clean out the old objects and avoid mixing
new and old history before pushing somewhere.  Other caveats:Commit messages are not rewritten; so if some of your commit
messages refer to prior commits by (abbreviated) sha1, after the
rewrite those messages will now refer to commits that are no longer
part of the history.  It would be better to rewrite those
(abbreviated) sha1 references to refer to the new commit ids.The --prune-empty flag sometimes misses commits that should be
pruned, and it will also prune commits thatstartedempty rather
than just ended empty due to filtering.  For repositories that
intentionally use empty commits for versioning and publishing
related purposes, this can be detrimental.The commands above are OS-specific.  GNU vs. BSD issues for sed,
xargs, and other commands often trip up users; I think I failed to
get most folks to use --index-filter since the only example in the
filter-branch manpage that both uses it and shows how to move
everything into a subdirectory is linux-specific, and it is not
obvious to the reader that it has a portability issue since it
silently misbehaves rather than failing loudly.The --index-filter version of the filter-branch command may be two to
three times faster than the --tree-filter version, but both
filter-branch commands are going to be multiple orders of magnitude
slower than filter-repo.Both commands assume all filenames are composed entirely of ascii
characters (even special ascii characters such as tabs or double
quotes will wreak havoc and likely result in missing files or
misnamed files)Solving this with fast-export/fast-importOne can kind of hack this together with something like:gitfast-export--no-data--reencode=yes--mark-tags--fake-missing-tagger\--signed-tags=strip--tag-of-filtered-object=rewrite--all\|grep-vP'^M [0-9]+ [0-9a-f]+ (?!src/)'\|grep-vP'^D (?!src/)'\|perl-pe's%^(M [0-9]+ [0-9a-f]+ )(.*)$%\1my-module/\2%'\|perl-pe's%^(D )(.*)$%\1my-module/\2%'\|perl-pes%refs/tags/%refs/tags/my-module-%\|git-ccore.ignorecase=falsefast-import--date-format=raw-permissive\--force--quietgitfor-each-ref--format="delete %(refname)"refs/tags/\|grep-vrefs/tags/my-module-\|gitupdate-ref--stdingitreset--hardgitreflogexpire--expire=now--allgitgc--prune=nowBut this comes with some nasty caveats and limitations:The various greps and regex replacements operate on the entire
fast-export stream and thus might accidentally corrupt unintended
portions of it, such as commit messages.  If you needed to edit
file contents and thus dropped the --no-data flag, it could also
end up corrupting file contents.This command assumes all filenames in the repository are composed
entirely of ascii characters, and also exclude special characters
such as tabs or double quotes.  If such a special filename exists
within the old src/ directory, it will be pruned even though it
was intended to be kept.  (In slightly different repository
rewrites, this type of editing also risks corrupting filenames
with special characters by adding extra double quotes near the end
of the filename and in some leading directory name.)This command will leave behind huge numbers of useless empty
commits, and has no realistic way of pruning them.  (And if you
tried to combine this technique with another tool to prune the
empty commits, then you now have no way to distinguish between
commits which were made empty by the filtering that you want to
remove, and commits which were empty before the filtering process
and which you thus may want to keep.)Commit messages which reference other commits by hash will now
reference old commits that no longer exist.  Attempting to edit
the commit messages to update them is extraordinarily difficult to
add to this kind of direct rewrite.Design rationale behind filter-repoNone of the existing repository filtering tools did what I wanted;
they all came up short for my needs.  No tool provided any of the
first eight traits below I wanted, and all failed to provide at least
one of the last four traits as well:[Starting report] Provide user an analysis of their repo to help
them get started on what to prune or rename, instead of expecting
them to guess or find other tools to figure it out.  (Triggered, e.g.
by running the first time with a special flag, such as --analyze.)[Keep vs. remove] Instead of just providing a way for users to
easily remove selected paths, also provide flags for users to
onlykeepcertain paths.  Sure, users could workaround this by
specifying to remove all paths other than the ones they want to
keep, but the need to specify all paths thateverexisted inanyversion of the repository could sometimes be quite
painful.  For filter-branch, using pipelines likegit ls-files | grep -v ... | xargs -r git rmmight be a reasonable workaround
but can get unwieldy and isn't as straightforward for users; plus
those commands are often operating-system specific (can you spot
the GNUism in the snippet I provided?).[Renaming] It should be easy to rename paths.  For example, in
addition to allowing one to treat some subdirectory as the root
of the repository, also provide options for users to make the
root of the repository just become a subdirectory.  And more
generally allow files and directories to be easily renamed.
Provide sanity checks if renaming causes multiple files to exist
at the same path.  (And add special handling so that if a commit
merely copied oldname->newname without modification, then
filtering oldname->newname doesn't trigger the sanity check and
die on that commit.)[More intelligent safety] Writing copies of the original refs to
a special namespace within the repo does not provide a
user-friendly recovery mechanism.  Many would struggle to recover
using that.  Almost everyone I've ever seen do a repository
filtering operation has done so with a fresh clone, because
wiping out the clone in case of error is a vastly easier recovery
mechanism.  Strongly encourage that workflow bydetecting and
bailing if we're not in a fresh
clone,
unless the user overrides with --force.[Auto shrink] Automatically remove old cruft and repack the
repository for the user after filtering (unless overridden); this
simplifies things for the user, helps avoid mixing old and new
history together, and avoids problems where the multi-step
process for shrinking the repo documented in the manpage doesn't
actually work in some cases.  (I'm looking at you,
filter-branch.)[Clean separation] Avoid confusing users (and prevent accidental
re-pushing of old stuff) due to mixing old repo and rewritten
repo together.  (This is particularly a problem with filter-branch
when using the --tag-name-filter option, and sometimes also an
issue when only filtering a subset of branches.)[Versatility] Provide the user the ability to extend the tool or
even write new tools that leverage existing capabilities, and
provide this extensibility in a way that (a) avoids the need to
fork separate processes (which would destroy performance), (b)
avoids making the user specify OS-dependent shell commands (which
would prevent users from sharing commands with each other), (c)
takes advantage of rich data structures (because hashes, dicts,
lists, and arrays are prohibitively difficult in shell) and (d)
provides reasonable string manipulation capabilities (which are
sorely lacking in shell).[Old commit references] Provide a way for users to use old commit
IDs with the new repository (in particular via mapping from old to
new hashes with refs/replace/ references).[Commit message consistency] If commit messages refer to other
commits by ID (e.g. "this reverts commit 01234567890abcdef", "In
commit 0013deadbeef9a..."), those commit messages should be
rewritten to refer to the new commit IDs.[Become-empty pruning] Commits which become empty due to filtering
should be pruned.  If the parent of a commit is pruned, the first
non-pruned ancestor needs to become the new parent.  If no
non-pruned ancestor exists and the commit was not a merge, then it
becomes a new root commit.  If no non-pruned ancestor exists and
the commit was a merge, then the merge will have one less parent
(and thus make it likely to become a non-merge commit which would
itself be pruned if it had no file changes of its own).  One
special thing to note here is that we prune commits which become
empty, NOT commits which start empty.  Some projects intentionally
create empty commits for versioning or publishing reasons, and
these should not be removed.  (As a special case, commits which
started empty but whose parent was pruned away will also be
considered to have "become empty".)[Become-degenerate pruning] Pruning of commits which become empty
can potentially cause topology changes, and there are lots of
special cases.  Normally, merge commits are not removed since they
are needed to preserve the graph topology, but the pruning of
parents and other ancestors can ultimately result in the loss of
one or more parents.  A simple case was already noted above: if a
merge commit loses enough parents to become a non-merge commit and
it has no file changes, then it too can be pruned.  Merge commits
can also have a topology that becomes degenerate: it could end up
with the merge_base serving as both parents (if all intervening
commits from the original repo were pruned), or it could end up
with one parent which is an ancestor of its other parent.  In such
cases, if the merge has no file changes of its own, then the merge
commit can also be pruned.  However, much as we do with empty
pruning we do not prune merge commits that started degenerate
(which indicates it may have been intentional, such as with --no-ff
merges) but only merge commits that become degenerate and have no
file changes of their own.[Speed] Filtering should be reasonably fastHow do I contribute?See thecontributing guidelines.Is there a Code of Conduct?Participants in the filter-repo community are expected to adhere to
the same standards as for the git project, so thegit Code of
Conductapplies.Upstream ImprovementsWork on filter-repo andits
predecessorhas also driven numerous improvements to fast-export and fast-import
(and occasionally other commands) in core git, based on things
filter-repo needs to do its work:git-2.28.0fast-import: add new --date-format=raw-permissive formatgit-2.24.0fast-export: handle nested tagst9350: add tests for tags of things other than a commitfast-export: allow user to request tags be marked with --mark-tagsfast-export: add support for --import-marks-if-existsfast-import: add support for new 'alias' commandfast-import: allow tags to be identified by mark labelsfast-import: fix handling of deleted tagsfast-export: fix exporting a tag and nothing elsegit-fast-import.txt: clarify that multiple merge commits are allowedgit-2.23.0t9350: fix encoding test to actually test reencodingfast-import: support 'encoding' commit headerfast-export: avoid stripping encoding header if we cannot reencodefast-export: differentiate between explicitly UTF-8 and implicitly
UTF-8fast-export: do automatic reencoding of commit messages only if
requestedgit-2.22.0log,diff-tree: add --combined-all-paths optiont9300: demonstrate bug with get-mark and empty orphan commitsgit-fast-import.txt: fix wording about where ls command can appearfast-import: check most prominent commands firstfast-import: only allow cat-blob requests where it makes sensefast-import: fix erroneous handling of get-mark with empty orphan
commitsHonor core.precomposeUnicode in more placesgit-2.21.0fast-export: convert sha1 to oidgit-fast-import.txt: fix documentation for --quiet optiongit-fast-export.txt: clarify misleading documentation about rev-list
argsfast-export: use value from correct enumfast-export: avoid dying when filtering by paths and old tags existfast-export: move commit rewriting logic into a function for reusefast-export: when using paths, avoid corrupt stream with non-existent
markfast-export: ensure we export requested refsfast-export: add --reference-excluded-parents optionfast-import: remove unmaintained duplicate documentationfast-export: add a --show-original-ids option to show
original namesgit-show-ref.txt: fix order of flagsgit-2.20.0update-ref: fix type of update_flags variable to
match its usageupdate-ref: allow --no-deref with --stdingit-1.7.3fast-export: Fix dropping of files with --import-marks and path
limitingfast-export: Add a --full-tree optionfast-export: Fix output order of D/F changesfast-import: Improve robustness when D->F changes provided in wrong
ordergit-1.6.4:fast-export: Set revs.topo_order before calling setup_revisionsfast-export: Omit tags that tag treesfast-export: Make sure we show actual ref names instead of "(null)"fast-export: Do parent rewriting to avoid dropping relevant commitsfast-export: Add a --tag-of-filtered-object option for newly
dangling tagsAdd new fast-export testcasesfast-export: Document the fact that git-rev-list arguments are
acceptedgit-1.6.3:git-filter-branch: avoid collisions with variables in eval'ed
commandsCorrect missing SP characters in grammar comment at top of
fast-import.cfast-export: Avoid dropping files from commitsgit-1.6.1.4:fast-export: ensure we traverse commits in topological order