Package: gl-query
gl-queryis a tool to query the GitLab API for useful project and pipeline information.Installation / SetupTo install:pip install gl-queryGenerate a GitLab Personal Access TokenMake sure you store this token in a safe place. As a side-note, any results from thegl-querytool are dependent on both the permissions you give the token when you generate it as well as the access-rights you have on the GitLab instance.Create the~/.gl-query/config.cfgfile :[config]
# Must include "/api/v4/"
url = https://<GITLAB_HOST>/api/v4/
token = YOUR_GITLAB_PERSONAL_ACCESS_TOKENUsagePositional ArgumentsPositional arguments subject/verb commands for the CLI utility.
The format for position arguments is:gl-query <VERB> <SUBJECT> --<OPTIONAL_FLAGS>An example would be:gl-query get projectsVerbsVerbs are the first positional argumentCurrently, the only supported verb isgetas the current purpose of this program is to retrieve information from the GitLab API and output it. For future, it may be worth adding asetaction in order to make modifications to the state of the GitLab instance. Some proposed actions would be to :Trigger a pipelineModify user / group membership (access controls)Opening / Modifying / Closing Merge RequestsManaging Tags / ReleasesSubjectsSubjects are the second positional argument.Optional Arguments (Flags)There are multiple command-line flags compatibile withgl-query.--csv / -c<OPTIONAL>USAGE:gl-query get projects --csvThis flag takes no arguments. It simply tellsgl-queryto produce a CSV output of the query. When used without the--filenameflag it will write the CSV file to a default filename in this format:GL_PROJECT_ACTIVITY_SINCE_${DATE}.csv--filename / -f<OPTIONAL>USAGE:gl-query get projects --csv --filename all_projects.csvThe--filenameflag is simply the name of CSV file to write to, therefore, it is recommended to have a.csvextension. Avoid using relative paths if you do choose to specify the path. If--filenameflag is used, must use the--csvflag.--language / -l<OPTIONAL>USAGE:gl-query get projects --language ScalaSpecify a programming language if you would like to filter GitLab projects by language. The first letter of the language must be capitalized. If flag is not used, all languages will be queried by default.TODO: Exception Handling.--date-before<OPTIONAL>USAGE:gl-query get projects --date-before 2019-03-04Displays resultsbeforespecified date. The argument must follow the YYYY-MM-DD format.--date-after<OPTIONAL>USAGE:gl-query get projects --date-after 2019-03-04Displays resultsafterspecified date. The argument must follow the YYYY-MM-DD format.--has-pipeline / -p<OPTIONAL>USAGE:gl-query get projects -pThe flag takes no arguments and causes output to only contain results that have at least one associated pipeline. This is useful for viewing projects that have pipeline functionality.--silent<OPTIONAL>USAGE:gl-query get projects --lang java -c -f java.csv --silentThe flag blocks output to the console or standard out. This is good for security purposes (if this tool is being used in the GitLab CI/CD) or if the output is being written to CSV so you don't need console output.--scan-type<OPTIONAL>USAGE:gl-query get scans --scan-type sast --date-after 2021-08-02 -l java

gl-query get scans --date-after 2021-08-02 --scan-type "sast,srcclear,dependency-scan,srcclr" -l scalaThis flag must be used in conjunction with theget scansaction and outputs job information pertaining to the scan type(s). Currently there are three scan types supported for this query:sast(JS, Python, Go, Java, C++, sbt, Rust )dependency-scan(JS, Python, Go, Java, sbt, Rust)srcclearorsrcclr(JS, Python, Go, Java, C++, sbt, Rust )If you choose to select multiple scan types, remember tocomma-delimitthem surround them in double quotes as in the example above.--versionObtains version ofgl-queryutility. This should correspond with a Git Tag.Usage Examplesgl-query get project --project-id 3430
gl-query get projects -l scala --date-filter 2021-07-30 --csv --filename test.csv -p

gl-query get pipelines --project-id 4748
gl-query get pipeline --project-id 3430 --pipeline-id 247249Dev GuideUsefulcurlcommands for troubleshootingcurl --header "Authorization: Bearer <your_access_token>" "https://gitlab.gaikai.com/api/v4/projects"To getx-total-pages:curl -s -I --header "Authorization: Bearer <your_access_token>" "https://gitlab.gaikai.org/api/v4/projects/"Future WorkTesting (CI) : Integrategl-query_test.shscript into pipeline orpre-commithookImplement the conditional query param (date_beforeversusdate_afterdilemna)hereImplement GitLab Birthday in config fileForget scansaction, implement:Lookup by project-id / project-nameImplement custom exclude_projects functionalityAPI throttling -- backoff after a certain number of API Requests (need to find out GitLab API limits)Minimize / consolidate number of API callsInvestigate more robust / centralized exception HandlingFlags--top-level-group--cgei-subgroup--include-user-repositoriesImplement search functionality:https://gitlab.gaikai.org/api/v4/projects?search=nexusCreate a global query options function and child functions for each type of object (projects, pipelines, etc.)Integrate--csvoutput for all actionsDeveloper NotesManually Deploying to PyPibumpversion --current-version <CURRENT_VERSION> minor setup.py gl-query/__init__.py --allow-dirty

python3 setup.py sdist bdist_wheel

twine upload dist/gl-query-<VERSION_NUMBER>*
twine upload dist/gl_query-<VERSION_NUMBER>*Query Options / ParametersQuery parameters are generally of concern when executing listing operations (i.e. list projects, groups, users, pipelines, jobs, merge requests, etc...)Query parameters are generally not used when dealing with a specific project or pipeline.GLOBAL Query Parametersper_pagesortorder_byProjects Query Parametersarchived=falselast_activity_afterlast_activity_beforewith_programming_languageorder_by=last_activity_atPipelinesorder_by=updated_atyaml_errors=falsescope=finishedupdated_afterupdated_beforeTo implementpipeline_idfilter onget scans:Iterate through pipeline pages for given projectPiggy back off the get_pipeline method for job infoMaintain running sum of target scansUsing Date Filterstldr; Do not use bothdate_beforeanddate_afterinprojectsqueries.Scenario:Today's date is August 3, 2021Use Case: You are searching forsastscans between July 15, 2021 and July 30, 2021Project X has a pipeline which ran on July 15, 2021 with asastscan.If you execute agl-querywith--date-before 2021-07-30and--date-after 2021-07-15, this will only include projects that were last active
between July 15, 2021 and July 30, 2021 as it is using the&last_activityquery parameters.If Project X, was updated after2021-07-30for any reason, it will not show up in the query results since it's not between the specified date range, therefore excluding a perfectly legitimatesastscan from the results.The solution to this is tonot use boththe&last_activity_beforeand&updated_beforequery parameters when making requests toprojectsAPI endpoints respectively. Instead of using thedate_beforeparam, we will usejustthedate_afterparam. This will ensure that all projects and pipelines will be included in your query.Although this avoids the false negative problem, it significantly increases query time since it will queryALLprojects/pipelinesAFTERthe given date. It is possible to speed this up by implementing the following logic:Conditionally use either thedate_beforeordate_afterfor the query params.You will need to introduce a variable called GL_BDAY (basically the first day a project was created)if (DATE_BEFORE - GL_BDAY ) > ( TODAY - DATE_AFTER):
  Use date_after for `&last_activity_after` and `&updated_after` AND do not use date_before
else:
  Use the date_before for `&last_activity_before` and `&updated_before`This approach has it's own risks as GL project/pipeline activity may be skewed such that activity was low for the first couple months / year, so keep this into account if you choose to implement this.Useful QueriesAll Scans & All Languages since a certain dategl-query get scans --all-languages --scan-type "sast, dependency-scan, srcclr, srcclear"  --date-after 2021-07-04 --csv -f all_scans_since_07_04_21.csvNote, we query for bothsrcclearandsrcclrsince they are both used as job names in GitLab CI.Java SAST Scans ( w/--date-after)# Java SAST Scans (July 3 - August 3)
gl-query get scans --scan-type sast -l Java --date-after 2021-07-03RESULT:Total SAST Scan Jobs after 2021-07-03 : 576ALLsastScans ( w/--date-after)gl-query get scans --scan-type sast --all-languages --date-after 2021-07-03ALLdependency-scan( w/--date-after)gl-query get scans --scan-type dependency-scan --all-languages --date-after 2021-07-03ALLsrcclrorsrcclear( w/--date-after)gl-query get scans --scan-type srcclr --all-languages --date-after 2021-07-03gl-query get scans --scan-type srcclear --all-languages --date-after 2021-07-03[DANGEROUS] ALLsastscans EVER# This will look through every project since GL_BDAY (JANUARY 1, 2019) which could generate tremendous load against the GitLab instance

TODAY=$(date '+%Y-%m-%d')
gl-query get scans --scan-type sast --all-languages --date-before $TODAY