Package: cloudmage-gitutils
CloudMage GitUtils Python3 Utility and Reporting PackageTable of ContentsDescriptionRoad MapPython Version SupportPackage InstallationPackage DependenciesGitConfigParser ClassGitConfigParser Constructor ArgumentsGitConfigParser Attributes and PropertiesGitConfigParser Available MethodsGitConfigParser Class UsageGithubReports ClassGithubReports Constructor ArgumentsGithubReports Attributes and PropertiesGithubReports Available MethodsGithubReports Class UsageChangeLogContacts and ContributionsDescriptionIn part, the purpose of this library is to be able to either automatically detect a projects configured Git repository by searching for and parsing a .git/config file in a given file path, or to take an input consisting simply of the repositories URL (HTTP | Git formatted). The object has the repository url, provider, and any users configured within the git URL string. The Second part of this utility package is to provide a few pre-packaged reports that can be retrieved from Github such as an Open Pull Request report, showing all pull requests opened within a given repository user or organization namespace.Road MapCurrently this library gathers data from the Githubs API, however, future development will include Gitlab and Bitbucket repository API reporting as well.Python Version SupportThis library is compatible with Python 3.6 and higher. It may work with earlier versions of Python3 but was not tested against anything earlier then 3.6. As Python 2.x is soon to be end of life, backward compatibility was not taken into consideration.Package InstallationThis library has been published toPyPiand can be installed via normal python package manager conventions such aspiporpoetry.pip3installcloudmage-gitutilsPackage DependenciesThis package installs and imports the following python modules during installation:pyGithubprogresscloudmage.jinjautilsAdditionally the package takes advantage of the following built in python modules:ossysjsoninspectdatetimeGitConfigParser ClassThis class takes a directory path argument, which it uses as a target directory to search for a .git/config file. If a file is found, then the class will parse the URL from the config, and determines the git platform provider from the parsed URL path. This data is then used to return back an object instance with properties set to the parsed values.GitConfigParser Constructor ArgumentsThe following arguments can be used to instantiate a new object instance:pathA valid directory path where a.git/configfile can be found.Must be valid directory path, checked withos.path.exists()requiredtruetypestrverboseEnables verbose mode.   [true=enablefalse=disable]requiredfalsetypebooldefaultfalse(disabled)logRedirects object standard log messaging to provided log object.requiredfalsetypeobjdefaultNone(log to stdout, stderr if verbose=true)GitConfigParser Attributes and PropertiesThe following attributes/properties are available to an instantiated object instance. Any of the attributes or properties can be accessed with standard object dot notation as in the example:verbose_mode = GitConfigParserObj.verboseverboseVerbose setting that controls logging level within the object.   [true=enabled,false=disabled]returnstrueorfalse(enabled or disabled)typeboolinstantiated valuefalseurlVerbose setting that controls logging level within the object.   [true=enabled,false=disabled]returnsurl string->https://github.com/namespace/repositorytypestrinstantiated valueURL string parsed from .git/config in directory path specified during object instantiationproviderThe parsed provider (github.com,gitlab.com, orbitbucket.org) from a given URL stringreturnsProvider string->github.comtypestrinstantiated valueProvider string parsed from object url property during object instantiationuserIf a user was used in the config url, then the value of the user will be assigned to this propertyreturnsUser string->usernametypestrinstantiated valueUser string parsed from object url property during object instantiationlogThe class logger. Will either write directly to stdout, stderr, or to a lob object if passed into the object constructor during object instantiationreturnsLog Event Streamtypestrinstantiated valueLogs written to stdout, stderrGitConfigParser Available MethodsThe following methods are available to an instantiated object instance. Some of the methods are simply setter methods that run the logic required to discover and then set one of the above instance properties.verboseSetter method forverboseproperty that enables or disables verbose modeparametertyperequiredarg infoverbosebooltrueTrueenables verbose logging,Falsedisables itExamples:# Getter methodverbose_setting=GitConfigParserObj.verbose# Setter methodGitConfigParserObj.verbose=TruelogMethod to enable logging throughout the class. Log messages are sent to the log method providing the log message, the message type being one of[debug, info, warning, error], and finally the function or method id that is automatically derived within the function or method using the python inspect module. If a log object such as a logger or an already instantiated log object instance was passed to the class constructor during the objects instantiation, then all logs will be written to the provided log object. If no log object was provided during instantiation then alldebug,info, andwarninglogs will be written to stdout, while any encounterederrorlog entries will be written to stderr. Note that debug or verbose mode needs to be enabled to receive the event log stream.argtyperequiredarg infolog_msgstrtrueThe actual message being sent to the log methodlog_typestrtrueThe type of message that is being sent to the log method, one of[debug, info, warning, error]log_idstrtrueA string value identifying the sender method or function, consisting of the method or function nameExamples:defmy_function():__function_id=inspect.stack()[0][3]GitConfigParserObj.log(f"{__function_id}called.",'info',__function_id)urlSetter method forurlproperty that will search the object instances set directory path and look for a .git/config directory in that path. If found, then the setter method will parse the .git/config file and look for a URL line to parse and extract the URL. It will then update the object property with the parsed value. If this method is called post instantiation, then a valid directory path must be provided as an argument.parametertyperequiredarg infoconfig_pathstrtrueMust be a valid directory path. This value is checked byos.path.exists()and will write an error to the log if the provided argument directory path does not existExamples:GitConfigParserObj.url=/home/projects/my_project_directoryproviderSetter method forproviderproperty that will search the object instances updated url property for a valid git provider string. Currently the provider setter will look specifically for a value that matches one of[github.com, gitlab.com, bitbucket.org]. If this method is called post instantiation, then a valid git repository url must be provided as an argument. The provider property setter will parse either an http, https, git or ssh formatted URL string. During the parse operation, if a user is identified in the provider string such asuser@bitbucket.org, then the username will also be parsed and used to update theuserobject property.parametertyperequiredarg inforepository_urlstrtrueMust be a properly formatted URL string starting with one of[http, https, git, ssh]and must end in.gitExamples:GitConfigParserObj.provider="git@github.com:namespace/repository.git"GitConfigParser Class UsageGitConfigParser Default Instantiationfromcloudmage-gitutilsimportGitConfigParserProjectPath='/Projects/MyCoolProject'# Contains .git/config with# url = https://github.com/GithubNamespace/MyCoolProject-Repo.gitRepo=GitConfigParser(ProjectPath)repo_url=Repo.urlprint(repo_url)# https://github.com/GithubNamespace/MyCoolProject-Reporepo_provider=Repo.providerprint(repo_provider)# github.comrepo_user=Repo.userprint(repo_user)# NoneOptional Verbose Class Constructor Argument:When instantiating the class an optionalverboseargument can be provided. The argument expects a bool value of eitherTrueorFalse. By default verbose is set to False. Ifverbose=Trueis passed during object instantiation, then debug mode is turned on allowing the class to output DEBUG, INFO, and WARNING messages to stdout, and ERROR messages to stderr.GitConfigParser Verbose Instantiationfromcloudmage-gitutilsimportGitConfigParserProjectPath='/Projects/MyCoolProject'# Contains .git/config with# url = https://github.com/GithubNamespace/MyCoolProject-Repo.gitRepo=GitConfigParser(ProjectPath,verbose=True)repo_url=Repo.urlprint(repo_url)# https://github.com/GithubNamespace/MyCoolProject-Reporepo_provider=Repo.providerprint(repo_provider)# github.comrepo_user=Repo.userprint(repo_user)# None# Class DEBUG, INFO, and WARNING messages will be printed to stdout, and ERROR messages will be printed to stderrOptional Log Object:When instantiating the class an optionallogargument can also be provided. The argument expects an Logger object to be passed as an input. If passed then all DEBUG, INFO, WARNING, and ERROR messages will be printed to the standard log levels [log.debug(),log.info(),log.warning(),log.error()] and printed to the passed respective logger object method.GitConfigParser Log Instantiationfromcloudmage-gitutilsimportGitConfigParser# Define test log class# This is an example log object that simply appends any DEBUG, INFO and ERROR received class messages# to the respective log level list. Normally this would be a logger or custom log object.classLog(object):"""Test Log Object"""def__init__(self):"""Class Constructor"""self.debug_logs=[]self.info_logs=[]self.error_logs=[]defdebug(self,message):"""Log Debug Messages"""self.debug_logs.append(message)definfo(self,message):"""Log Debug Messages"""self.info_logs.append(message)deferror(self,message):"""Log Debug Messages"""self.error_logs.append(message)# Instantiate test log classGitLog=Log()ProjectPath='/Projects/MyCoolProject'# Contains .git/config with# url = https://github.com/GithubNamespace/MyCoolProject-Repo.gitRepo=GitConfigParser(ProjectPath,verbose=True,log=GitLog)repo_url=Repo.urlprint(repo_url)# https://github.com/GithubNamespace/MyCoolProject-Reporepo_provider=Repo.providerprint(repo_provider)# github.comrepo_user=Repo.userprint(repo_user)# NoneforitemsinGitLog.debug_logs:print(item)# Prints stored debug logsGithubReports ClassThis class will take an auth_token, and user/organization namespace name and run a query against the namespace for all open pull requests in any repositories living within that user/org namespace. Once the pull request data is returned, it can be then be used to generate the included HTML template report that will list the repository, pull request title, submitter, reviewers, creation date and days open count for each found repository.GithubReports Constructor ArgumentsThe following arguments can be used to instantiate a new object instance:verboseEnables verbose mode.   [true=enablefalse=disable]requiredfalsetypebooldefaultfalse(disabled)auth_tokenA valid Github personal access token used to search issues within the user/organization namespace account.requiredtruetypestr<br>logRedirects object standard log messaging to provided log object.requiredfalsetypeobjdefaultNone(log to stdout, stderr if verbose=true)GithubReports Attributes and PropertiesThe following attributes/properties are available to an instantiated object instance. Any of the attributes or properties can be accessed with standard object dot notation as in the example:verbose_mode = GithubReportsObj.verboseverboseVerbose setting that controls logging level within the object.   [true=enabled,false=disabled]returnstrueorfalse(enabled or disabled)typeboolinstantiated valuefalseauth_tokenProperty to set/re-set the Github Authentication Token required to make calls against Github APIs.returnsstring->0123456789109876543210typestrinstantiated valueProvided Token, If token not provided during object construction, use Obj.auth_token = ""repo_namespaceThe user / organization namespace that will be used as the search targetreturnsstring->CloudMagestypestrinstantiated valueNone, set with Obj.repo_namespace = ""is_organizationFlag to indicate if the provided namespace is an organization namespace (affects API calls).    [true=organization,false=user]returnstrueorfalse(organization or user) namespacetypeboolinstantiated valuefalsenotifyFlag to specify if, when running reports, a comment should be left on a repository that has exceeded the open days threshold.   [true=leave_comment,false=do_not_comment]returnstrueorfalse*(leave comments or do not leave comments)typeboolinstantiated valuefalseopen_pr_thresholdNumber value indicating a days threshold used for reports such as open pull requests. This value will be used to determine if a PR has exceeded desired days open.returnsint->5typeintinstantiated value5daystemplate_pathString value of the path where reporting templates are stored.returnsstr->/Path/to/template/filestypestrinstantiated value/{installation/path}/templateslogThe class logger. Will either write directly to stdout, stderr, or to a lob object if passed into the object constructor during object instantiationreturnsLog Event Streamtypestrinstantiated valueLogs written to stdout, stderrGithubReports Available MethodsThe following methods are available to an instantiated object instance. Some of the methods are simply setter methods that run the logic required to discover and then set one of the above instance properties.verboseSetter method forverboseproperty that enables or disables verbose modeparametertyperequiredarg infoverbosebooltrueTrueenables verbose logging,Falsedisables itExamples:# Getter methodverbose_setting=GitHubReportObj.verbose# Setter methodGitHubReportObj.verbose=TruelogMethod to enable logging throughout the class. Log messages are sent to the log method providing the log message, the message type being one of[debug, info, warning, error], and finally the function or method id that is automatically derived within the function or method using the python inspect module. If a log object such as a logger or an already instantiated log object instance was passed to the class constructor during the objects instantiation, then all logs will be written to the provided log object. If no log object was provided during instantiation then alldebug,info, andwarninglogs will be written to stdout, while any encounterederrorlog entries will be written to stderr. Note that debug or verbose mode needs to be enabled to receive the event log stream.argtyperequiredarg infolog_msgstrtrueThe actual message being sent to the log methodlog_typestrtrueThe type of message that is being sent to the log method, one of[debug, info, warning, error]log_idstrtrueA string value identifying the sender method or function, consisting of the method or function nameExamples:defmy_function():__function_id=inspect.stack()[0][3]GitConfigParserObj.log(f"{__function_id}called.",'info',__function_id)auth_tokenSetter method forauth_tokenproperty that will be used to make calls to Githubs API for authentication. The provided auth_token must be valid, and have access to read repositories and issues. An auth_token can be set during instance instantiation, using the property setter, or passed directly into a reporting method. This allows the versatility of using different tokens if necessary.parametertyperequiredarg infoauth_tokenstrtrueMust be a valid Github auth token. The search methods will throw a BadCredentials exception if an invalid token is provided.Examples:# Getter methodset_auth_token=GitHubReportObj.auth_token# Setter methodGitHubReportObj=GitHubReports(auth_token="0123456789109876543210")# Set using property setterGitHubReportObj.auth_token="0123456789109876543210"exportTOKEN="0123456789109876543210"GitHubReportObj.search_open_pulls(auth_token=$TOKEN)repo_namespaceSetter method forrepo_namespaceproperty that will be used to specify what repository namespace will be targeted for reporting. The repository namespace must be a string value, and a valid Github user or organization namespace. Therepo_namespacecan be set by the property setter method, or by being passed directly into the search method as a method argument.parametertyperequiredarg inforepo_namespacestrtrueMust be a valid Github user or organization namespace or reporting and/or any form of desired data collection will fail.Examples:# Getter methodnamespace=GitHubReportObj.repo_namespace# Setter methodGitHubReportObj=GitHubReports(auth_token="0123456789109876543210")# Set using property setterGitHubReportObj.repo_namespace="CloudMages"# Set within the reporting method callexportTOKEN="0123456789109876543210"GitHubReportObj.search_open_pulls(auth_token=$TOKENrepo_namespace="CloudMages")is_organizationSetter method foris_organizationproperty that will be used as a flag to determine if the provided repo_namespace is a user namespace or organization namespace. This information is relevant as the APIs that will be called to fetch data for or about the specified repository as such with repository or issues searchs will be affected by this flag. If an organization namespace is passed, but not signaled that the namespace is an organization, reports or other calls to the API could fail.Default value of this flag is set tofalse.parametertyperequiredarg infois_organizationboolfalseTruesignals organization,Falsesignals user namepaceExamples:# Getter methodOrgFlag=GitHubReportObj.is_organization# Setter methodGitHubReportObj.repo_namespace="CloudMages"GitHubReportObj.is_organization=TrueSearchIssues=GitHubReportObj.search_open_pulls()notifySetter method fornotifyproperty that will be used as a flag to publish a @mentioned comment on any open pull requests or issues that were found within the searched namespace if the open pull request or issue has been opened for a period of time exceeding the value set inopen_pr_threshold. If an open pull request or issue is discovered, and Obj.notify is set toTrue, then a message that specifically @mentions the issue submitter and all listed issue reviewers will be constructed and written on any offending issue as a comment indicating that the issue has been open in access of the threshold and should be reviewed and closed immediately. If the value is set toTrue, then messages will be sent to the log, but will not be published or commented on the repository issue itself.By Default this value is set toFalseparametertyperequiredarg infonotifyboolfalseTruesignals notify,Falsesignals write log onlyExamples:# Getter methodPublishComment=GitHubReportObj.notify# Setter methodGitHubReportObj.notify=TrueSearchIssues=GitHubReportObj.search_open_pulls()open_pr_thresholdSetter method foropen_pr_thresholdproperty that is used as the threshold metric for any open pull requests or issues discovered during namespace search. If a issue or pull request is open and the created date is greater then the value that is set for this flag in days old, then the issue or pull request will be added to reporting, and can be commented on by the search method to indicate that the pull request or issue has been open for an extended amount of time and should be reviewed and / or merged / closed immediately.By Default this value is set to5daysparametertyperequiredarg infodaysintfalseNumber of days threshold for pull requests / Github Issues.template_pathGetter only method fortemplate_pathproperty that is used to simply get the file location of where the modules HTML report templates are stored. This value can not, nor does not need to have its value altered as the included model templates will always be packaged in the /templates direcotry, held within the project root.By Default this value is {/path/to/this/module}/templates/parametertyperequiredarg infoNoneN/AN/ANo Parameter Value, this is a Getter method only.Examples:# Getter methodTemplatePath=GitHubReportObj.template_pathsearch_open_pullsThesearch_open_pullsreporting method will search a provided namespace for all open pull requests. For each open pull request item, the pull request Name, HTML URL, Title, Body, Submitter, Reviewers, Merge Data, Creation Data, Age, and Review States will be collected and returned back as a list of dictionaries. This data can then be used with the provided module template to render into an HTML report. The report will indicate by a green background any pull requests that have been approved and are awaiting either additional approvers or the submitter. The report will also indicate with a red background in the PR Days Open field if the pull request has been open longer then the configuredopen_pr_thresholdnumber of days.parametertyperequiredarg infoauth_tokenstrtrueMust be a valid Github auth token. The search methods will throw a BadCredentials exception if an invalid token is provided.repo_namespacestrtrueMust be a valid Github user or organization namespace or reporting and/or any form of desired data collection will fail.Examples:Open_PRs=GitHubReportObj.search_open_pulls(auth_token=$TOKENrepo_namespace="CloudMages")Execution Example:ConstructingsearchqueryforCloudMagesrepositorynamespace...OpenPullRequestThreshold:5daysNotificationsEnabled:FalseRepositoryNamespaceisOrganization:TrueVerboseModeEnabled:TrueInstantiatedGithubAPIConnectorObjectSearchResults:1openPullRequestswerereturned!OpenPRSearchreturned1resultsValidatingSearchResults...https://github.com/DaCloudMage/UnitTest-GitUtils/pull/1 Within the Open Days Limit.PrintingCollectedOpenPullRequestDataSet:[{"id":123456789,"repository":"UnitTest-GitUtils","repository_url":"https://github.com/DaCloudMage/UnitTest-GitUtils","number":1,"submitter":"user_1","reviewers":[],"link":"https://github.com/DaCloudMage/UnitTest-GitUtils/pull/1","title":"Open pr","body":"Test Open PR Report","created":datetime.datetime(2020,4,9,0,29,56),"age":datetime.timedelta(days=-1,seconds=66866,microseconds=735729),"age_days":1,"state":"open","is_merged":False,"merged":None,"mergable":True,"merge_state":"clean","merged_by":None,"review_count":0,"days_open_threshold":5}]1/1ofthereturnedsearchresultswereverifiedasopenpullrequests.GithubReports Class UsageGithubReports Default Instantiationfromcloudmage-gitutilsimportGitHubReportsimportos# Fetch exported TokenAUTH_TOKEN=os.environ['GITHUB_TOKEN']# Instantiate the objectOpenPRs=GitHubReports(auth_token=AUTH_TOKEN)OpenPRs.repo_namespace="CloudMages"OpenPrs.is_organization=TrueSearchOpenPRs=OpenPrs.search_open_pulls()print(len(SearchOpenPRs))Optional Verbose Class Constructor Argument:When instantiating the class an optionalverboseargument can be provided. The argument expects a bool value of eitherTrueorFalse. By default verbose is set to False. Ifverbose=Trueis passed during object instantiation, then debug mode is turned on allowing the class to output DEBUG, INFO, and WARNING messages to stdout, and ERROR messages to stderr.GithubReports Verbose Instantiationfromcloudmage-gitutilsimportGitHubReportsimportos# Fetch exported TokenAUTH_TOKEN=os.environ['GITHUB_TOKEN']# Instantiate the objectOpenPRs=GitHubReports(auth_token=AUTH_TOKEN,verbose=True)OpenPRs.repo_namespace="CloudMages"OpenPrs.is_organization=TrueSearchOpenPRs=OpenPrs.search_open_pulls()print(len(SearchOpenPRs))# Class DEBUG, INFO, and WARNING messages will be printed to stdout, and ERROR messages will be printed to stderrOptional Log Object:When instantiating the class an optionallogargument can also be provided. The argument expects an Logger object to be passed as an input. If passed then all DEBUG, INFO, WARNING, and ERROR messages will be printed to the standard log levels [log.debug(),log.info(),log.warning(),log.error()] and printed to the passed respective logger object method.GithubReports Log Instantiationfromcloudmage-gitutilsimportGitHubReportsimportos# Define test log class# This is an example log object that simply appends any DEBUG, INFO and ERROR received class messages# to the respective log level list. Normally this would be a logger or custom log object.classLog(object):"""Test Log Object"""def__init__(self):"""Class Constructor"""self.debug_logs=[]self.info_logs=[]self.error_logs=[]defdebug(self,message):"""Log Debug Messages"""self.debug_logs.append(message)definfo(self,message):"""Log Debug Messages"""self.info_logs.append(message)deferror(self,message):"""Log Debug Messages"""self.error_logs.append(message)# Instantiate test log classGitLog=Log()# Fetch exported TokenAUTH_TOKEN=os.environ['GITHUB_TOKEN']# Instantiate the objectOpenPRs=GitHubReports(auth_token=AUTH_TOKEN,verbose=True,log=GitLog)OpenPRs.repo_namespace="CloudMages"OpenPrs.is_organization=TrueSearchOpenPRs=OpenPrs.search_open_pulls()foriteminGitLog.debug_logs:print(item)# Prints stored debug logsChangelogTo view the project changelog see:ChangeLog: