Package: repotracer
📈 Repotracer: Watch your code changing over timeRepotracer gives you insight into the change going on in your codebase.It will loop through every day in the history of a git repository, and collect any stat you might ask it to. Eg:Typescript migration: count the number of lines of JS vs TSCount number of deprecated function callsMeasure adoption of new APIsIt compiles the results for each day into a csv, and also immediately gives you a plot of the data in csv.
It supports incremental computation: re-run it every so often without starting from scratch.Use it to watch:Percent of commits that touched at least one test, and count of authors writing testsCount number of authors who have used a new libraryThese are only the beginning. You can write your own stats and plug them into repotracer. If you can write a script to calculate a property of your code, then repotracer can graph it for you over time. For example you could run your build toolchain and counting numbers of a particular warning, or use a special tool.Repotracer aims to be a swiss army knife to run analytics queries on your source code.InstallationInstall withpip install repotracer.To run theregex_count,file_countandloc_countstats, you'll need to haveripgrep,fdandtokeiinstalled, respectively. On Macos you can install these with:brew install ripgrep fd tokeiRepotracer will look a config file either in$PWD/.repotracer/config.jsonor$HOME/.repotracer/config.json. If neither exists, it will create one in the latter location.repotracer add-statadd-statwill guide you through the process of configuring a repo, and adding a new stat.UsageA collection of commands for onboarding will come soon. In the meantime:repotracer run reponame statnamewill compute a single stat. The data will show up in./stats/repo/stat_name.csv, and a plot will be written to./stats/repo/stat_name.png.repotracer run reponamewill run all the stats for that repo. For now this makes separate passes for each commit, later it might do several stats for the same commit at a time.repotracer runwill update all stats in the config file.Stat typesMore documentation about the configuration options will come soon.regex_countruns ripgrep and sums the number of matches in the whole repo. Additional args can be passed to ripgrep by addingrg_argsin theparamsobject.file_countrunsfdand counts the number of files found.loc_countrunstokeiand counts the number of lines of code per language.The next stat will bescript, which will run any bash script the user will want, to allow for max customization.Stat optionsThe config format is JSON5, but currently comments are lost when the command updates the config file. I'm planning on moving to TOML to fix that, because the python TOML library supports it."repos" : {
    "svelte": {
      "url": "", // the url to clone the repo from
      "stats": {
        "count-ts-ignore": { // the name of the stat. Will be used in filenames
          "description": "The number of ts-ignores in the repo.", //Optional. A short description of the stat.
          "type": "regex_count", //
          "start": "2020-01-01", // Optional. When to start the the collection for this stat. If not specified, will use the beginning of the repo
          "path_in_repo": "2020-01-01", // Optional. Will cd into this path to run the stat
          "params": { // any parameters that depend on the measurement type
            "pattern": "ts-ignore",  //The pattern to pass to rigpgrep
            "ripgrep_args": "-g '!**/tests/*'" // any extra arguments to pass to ripgrep
          }
        },
      }
    }
}
``featuresStats: Regex countStats: File countStats: LOC count (broken down by language)Stats: Custom ScriptStats: option to measure at monthly cadence instead of dailyStats: Turn Betterer count files into statsRunner: Incremental runsRunner: Interleaved runs, only stream through repo once when collecting multiple stats on repoRunner: Parallel execution by running on many copies of the repoRunner: whenpath_in_repois set, onlygit checkoutthat portion of the the fsFix logging to not be so all or nothingDeploy to pypi on MR mergesDesign GoalsRepotracer is meant to achieve:Reliably collecting stats, in a reasonable amount of time. The idea is that a nightly job in CI will be running stat collection, so as long as it takes < 30 mins to collect all stats that should be ok. However we don't want it to be dog slow, as the occasional "interactive" use or end-user running it directly will also be supported.Flexibility for common use cases like counting regex matches, files, LOC. But not a huge config surface; if you want to tweak a command too much, just use ascripttype for your stat.Out of the box simple graphing support, but not too much. I don't plan on adding many plotting options to the configs.Nice starting DX. It should be easy to download the app, install it in a repo and get a nice plot of something you care about.Dev NotesThis is a side project with no reliability guarantees. It also is optimized for the author's productivity/engagement. It uses heavyweight "dev-friendly" libraries, and doesn't focus too much on code cleanliness.
The main priority is to get value now (the nice data/graphs), rather than build a timeless masterpiece.That doesn't mean it's meant as a filthy mess. Here are the main concepts:Repotracer manages a a collection ofStatobjects. These are specified by aStatConfig, and they mostly define some params + theMeasurement(ie the actual command to run, liketokei,ripgrep, a custom script, etc).AStatcan run itself, and that will update the csv for that stat.
From the POV a user, they care about running manyStats on a single repo,
so we aggregate those into aRepoConfig. This mainly defines where to download the repo. There is the idea that the repo storage could be pluggable, eg if
the repo is stored on NFS or something different.The overall config object isGlobalConfig, and it composes a couple basic parameters plus a list ofRepoConfigs.In theory a bunch of things can be made pluggable, but we'll wait until we need to swap anything out to define the interfaces.We use pandas to store & interface with the data, for ease of use. Pandas gives day-aggregation functions, and dataframe powers.