Package: labcrawler
LabCrawler byFrancis PotterNOTE: The project has changed. We intend to rewrite everything. In the meantime, enjoy limited functionality.InstallationInstall LabCrawler usingpipx.Installpipxfirst if necessary.pipxinstalllabcrawlerLater, to upgrade to the latest version of LabCrawler:pipxupgradelabcrawlerHost and tokenLabCrawler operations require a GitLab host (such as GitLab.com) and token.Download apersonal access tokenorgroup access tokenwith API access to the group(s) and project(s) you wish to query at the correct role level for the requested operation. For best practice desktop use, store the token in a password manager such as 1Password.LabCrawler uses theWizLib ConfigMachine framework, which supports several options for configuring the host and token (to be documented here).The default host isgitlab.comand the default behaviour is to request the token from the user at runtime.List ProjectsLists all the projects in a group and its subgroups.labcrawlerlistprojectsmy-groupList GroupsLists all the subgroups in a group and its subgroups.labcrawlerlistgroupsmy-groupMove projectsTransfer multiple projects between namespaces (groups/subgroups). Notespecific token requirements.The list of projects to be moved comes from stdin and must use the entire path with namespace. For best practice desktop use, apply a bash pipe.echo"my-group/my-project"|labcrawlermoveprojectsmy-new-groupOf course, the list of project paths can come from a file.cat"my-list-of-project-paths.txt"|labcrawlermoveprojectsmy-new-groupPiping allows the commands to chain.labcrawlerlistprojectsmy-group|labcrawlermoveprojectsmy-new-groupOr, edit the list before moving the projects.labcrawlerlistprojectsmy-group>tempfile.txt
emacstempfile.txt
cattempfile.txt|labcrawlermoveprojectsmy-new-groupLegacy documentstionUsingMeltanoandPandas.Examine a set of GitLab projects for usage of governance processes such as CI/CD, merge requests, merge request approvals, security scanning, and code review. This tool pulls project and group configuration and recent history data from GitLab and returns it in a CSV that can be imported to a spreadsheet.Assumes the use of the GitLab "ultimate" paid tier, as it looks for configuration settings such as merge request approvals, which are only available at that level.Remember, in GitLab, a "group" is like a portfolio or folder containing projects and subgroups. Currently, Labcrawler doesn't actually "crawl" - it only surveys projects that live directly in the designated groups.InstallationInstall LabCrawler usingpipx.Installpipxfirst if necessary.pipx install labcrawlerNote that LabCrawler installs its own dependencies, including Meltano, which might override themeltanocommand for an existing Meltano installation. It ought to work fine, but might contain a different version of Meltano itself.From your GitLab instance, you will also need apersonal access tokenorgroup access tokenwith API read access to the group(s) you wish to query. LabCrawler supports 2 options for token storage:For desktop use: Store the token in a password manager, then copy-and-paste it into LabCrawler when requested.For offline automated operation: Make the token available in the'GITLAB_PRIVATE_TOKENenvironment variable using an environment manager such as a.envfile or your CI platform's variables.UpgradeLater, to upgrade to the latest version of LabCrawler:pipx upgrade labcrawlerCheck the release notes for the new version - it might be necessary to runlabcrawler initagain to reload the Meltano tap and/or target.InitializationLabCrawler maintains a "workspace" for its config file and everything required for Meltano to operate. To set up the workspace, type:labcrawler initThe workspace should be good for the life of LabCrawler, though in rare cases a LabCrawler upgrade might require you to recreate it, so check the release notes when upgrading.The output from theinitcommand includes the location of the LabCrawler config file (labcrawler.json). Edit the file with the following information:api_url- URL to the root of the GitLab API, includinghttps://.groups- All the groups containing projects that you wish to examine. LabCrawler will not examine subgroups, so include them too.output_dir- Directory to store the CSV files that are generated; the default is based on the OS and might be sufficient.Meltano loadingLabCrawler loads data from the GitLab API into CSV files, which can then be examined using a spreadsheet application, independent script, or LabCrawler's built-in analyzer.Generating the CSV files requires two phases:meltandload.To run Meltano, issue the following command:labcrawler meltThe CSV files generated by Meltano will contain the verbatim fields from the GitLab REST API, so refer to theGitLab API Documentationto understand the fields. Data loaded includes:
-groups-projects-branches-merge_requests-project_members-group_members-usersCI configuration data loadingTo load LabCrawler-specific data, include CI configuration includes and committers, issue the following command:labcrawler loadTheloadcommand includes queries specific to GitLab CI configuration files, which are not covered by the Meltano tap.LabCrawler looks at each project's main GitLab CI configuration file. The default file is at the repo root with the name.gitlab-ci.yml, but the location can be overridden by the project maintainers, in which case it's read from theci_config_pathsetting at the project level. Sinceci_config_pathis now loaded by the Meltano tap, LabCrawler uses that information to query about the correct file.At this point, the main GitLab CI configuration file is the only file for which LabCrawler loads file-specific information.Theloadphase loads two specific types of information:includes- Paths to other GitLab CI configuration files from the same repo that are loaded into the main CI configuration file using theinclude:key. Useful to know where all the CI configuration lives, so it can be easily inspected.committers- Names and email addresses of developers who committed the most recent change to at least one line of the main CI configuration file, as per thegit blamecommand. Useful to know who has touched CI configuratoin most recently, so they can be easily questioned.Understanding includesTo list the "includes", LabCrawler downloads the CI configuration file itself from the default branch, parses the YAML, and identifies theinclude:key if it exists. The idea is to help understand where more CI configuration lives inside the project.Theinclude:keyhas some power, including the ability to include templates (shipped with GitLab itself) and CI configuration files from other projects and even from URLs outside of the GitLab instance. LabCrawler only lists local includes - that is, other CI configuration files in the same project that are used by the main CI configuration file.Understanding committersLabCralwer's "committers" list is based on the output from aGitLab API callthat emulates thegit blamecommand.Git differentiates between the author of a commit and the committer, both of which are stored by name and email address within the repo based on the developer's local configuration settings. Authors and committers are not authenticated by Git itself on the desktop.A push to GitLab might contain one or more commits. GitLab requires authentication and authorization for every push, and records the user who made the push in the database. Optionally, a pre-receive hook can be set up to confirm that the committers (inside the repo data) are authorized users. It's also possible to require signed commits. But GitLab has no way to authenticate the committers themselves, who might have committed code offline.So within the GitLab API, committer information comes from the Git repo itself, not from the users table. LabCrawler doesn't check other project-level configuration (such as push rules and the signed commit requirement) when reporting committer data. The discrepancy can result in surprises, such as commits by people not authorized to push to the repo, or even employees from other companies (or GitLab itself) showing up as committers.To investigate why an unknown committer appears in the committers data, use the GitLab web UI. Navigate to the project in question, identify the CI configuration file, and click the "Blame" button. You'll be able to see exactly which lines of code were touched by each committer, and with what commit message. It's also possible to use the "History" button to see the history of all commits to the file.Note that LabCrawler simply extracts the committers themselves from the API output, and discards the detailed information about which line each developer committed.TODO: Note about only loading one project's dataAnalysisLabCrawler generates simple CSV files in the output directory, which can be opened using Microsoft Excel or any other CSV application. Also, LabCrawler offers an easy-to-use analysis prompt based onpandas. To use the analysis:labcrawler analyzeThe above command:Reads the CSV file data into Pandas DataFrames (called "raw)Joins raw DataFrames into slightly-more-useful DataFrames with specific names for each datatypeOutputs how many rows are in each DataFrame, with the variable name for eachStarts a Python command-line Python interpreter for querying the resulting DataFrames using Pandas operations# List the projectsgitlab.projects# See what columns are available in any tablegitlab.projects.columns# View just certain columns from a tablegitlab.projects[['name','merge_method']]# See how many of each value are includedgitlab.projects.value_counts(['merge_method'])# Count the total of unmerged brancheslen(gitlab.branches.loc[~gitlab.branches['merged']])# View the number of unmerged branches by projectgitlab.branches.loc[~gitlab.branches['merged']].value_counts(['project_name'])# See who has access to a projectgitlab.project_members.query('project_name == "<project-name>"')[['user_username','access_level_name']]# See who has access to a groupgitlab.group_members.query('group_path == "<group-path>"')[['username','access_level_name']]# How many have each access levelgitlab.group_members.value_counts(['access_level_name'])# Who are the owners and maintainers?gitlab.group_members.query('group_path == "<group-path>" and access_level_name in ["Owner","Maintainer"]')[['username','access_level_name']]# Which projects use a non-default location for CI configuration?gitlab.projects.query("ci_config_path.notnull()")[['name','ci_config_path']]# Which projects have CI configuration at all?gitlab.ci_config_paths.query("main.notnull()")[['project_name']]# Projects without any CI configurationgitlab.ci_config_paths.query("main.isnull() and local_include.isnull()")[['project_name']]# Local CI configuration includesgitlab.ci_config_paths.query("local_include.notnull()")[['project_name','local_include']]# Who has edited the main CI configuration filegitlab.ci_config_committers[['committer_name','project_name']]LabCrawler sets themax_rowsattribute toNoneso you will see all the rows in default DataFrame output.For prettier output, we've added aneat()function.>>>neat(gitlab.projects[['name_with_namespace','merge_method']])name_with_namespacemerge_method-------------------------------------------------------------------------SteampunkWizard/WizLibmergeSteampunkWizard/FileWizmergeSteampunkWizard/LabCrawlermergeSteampunkWizard/BusymergeRoadmapWe maintain an informal roadmap of future functionality. Development of such functionality depends on contributions from the community, a generous donation, or us simply finding the time ;-).Put all these roadmap items into GitLab issues for visibility and sharingRedesign it to work within GitLab CI/CDUndo the hack that allows for large group listsFix the bug where users are being loaded into the CSV multiple times (possible Meltano tap-gitlab fix?)Fix everything to use theloggerinstead ofprint().Add a command (labcrawler info?) to get e.g. locations of config and output files.Load information aboutexternal main CI configuration filesLoad information about external included CI configuration filesLoad CI configuration includes recursively, so if an included CI config includes another one, it's loaded alsoPull committer information for included and external CI configuration files"Crawl" through subgroups at the beginning, to get all the groups and projects within a portfolioAdd one command to combinecrawl,melt, andloadEnable designation of groups and/or projects as options in the CLIProvide a Dockerfile for easy deploymentIntegrate with Jupyter Notebook to make querying easierGenerate useful reports (in GitLab Pages? Conflucence?)Read the CI configuration and smartly explain what it includesAdd (in reports, analysis, Jupyter, etc) clear explanations of what each field means (so users new to GitLab can understand more quickly)Provide a test bed for LabCrawler itself, including a GitLab instance with test groups/projectsContribute some or all of LabCrawler's custom queries back to tap-gitlabTroubleshootingIf theloadcommand seems to fail, try querying the GitLab API directly. The shell commands below assume that your token is already in theGITLAB_PRIVATE_TOKENenvironment variable.To make sure you can connect, and see the towen's user information:curl--requestGET--header"PRIVATE-TOKEN:$GITLAB_PRIVATE_TOKEN""$GITLAB_API_URL/api/v4/personal_access_tokens/self"Here's how to look up the raw blame data using CURL. Youmightneed to replace.gitlab-ci.ymlwith the actual path to the CI configuration file, and you also might need to change therefto the default branch.curl--requestGET--header"PRIVATE-TOKEN:$GITLAB_PRIVATE_TOKEN""$GITLAB_API_URL/api/v4/projects/<id>/repository/files/.gitlab-ci.yml/blame?ref=master"Finally, here's a crude API-driven way to list the files and directories at the root of a project:curl--requestGET--header"PRIVATE-TOKEN:$GITLAB_PRIVATE_TOKEN""$GITLAB_API_URL/api/v4/projects/<id>/repository/tree?per_page=999"Useful linksMeltano tutorialMeltano CLI referencetap-gitlab on GitHubGitLab API docsPandas DataFrame referenceDevelopmentMake sure you have Python 3.8+ and Pip installedMake a GitLab account and give it your SSH public keyFollow the steps below (YMMV, from memory, please update if I missed anything)git clone git@gitlab.com:steampunk-wizard/projects/labcrawler.git
cd labcrawler
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/freeze.txtThen to run it:python -m labcrawler init... and the other commandsThoughts on addingtap-github:Add it tomeltano.ymlSet up required env vars inlabcrawler.jsonRe-runinit