Package: auditree-harvest
auditree-harvestThe Auditree data gathering and reporting tool.IntroductionAuditreeharvestis a command line tool that assists with the gathering and
formatting of data into human readable reports.  Auditreeharvestallows
a user to easily retrieve historical raw data, in bulk, from a Git repository
and optionally format that raw data to meet reporting needs.  Auditreeharvestis meant to retrieve and report on historical evidence from an
evidence locker.  It is, however, not limited to just processing evidence.  Any
file found in a Git repository hosting service can be processed by Auditreeharvest.PrerequisitesSupported for execution on OSX and LINUX.Supported for execution with Python 3.6 and above.Python 3 must be installed, it can be downloaded from thePythonsite or installed using your package manager.Python version can be checked with:python--versionorpython3--versionTheharvesttool is available for download fromPyPI.InstallationIt is best practice, but not mandatory, to runharvestfrom a dedicated Python
virtual environment.  Assuming that you have the Pythonvirtualenvpackage already installed, you can create a virtual environment namedvenvby
executingvirtualenv venvwhich will create avenvfolder at the location of
where you executed the command.  Alternatively you can use the pythonvenvmodule
to do the same.python3-mvenvvenvAssuming that you have a virtual environment and that virtual environment is in
the current directory then to install a new instance ofharvest, activate
your virtual environment and usepipto installharvestlike so:../venv/bin/activate
pipinstallauditree-harvestAs we add functionality toharvestusers will want to upgrade theirharvestpackage regularly.  To upgradeharvestto the most recent version do:../venv/bin/activate
pipinstallauditree-harvest--upgradeSeepip documentationfor additional options for usingpip.ConfigurationSince Auditreeharvestinteracts with Git repositories, it requires Git remote
hosting service credentials in order to do its thing.  Auditreeharvestwill by
default look for ausernameandtokenin a~/.credentialsfile.  You can
override the credentials file location by using the--credsoption on aharvestCLI execution. Valid section headings includegithub,github_enterprise,bitbucket,
andgitlab.  Below is an example of the expected credentials entry.[github]username=your-gh-usernametoken=your-gh-tokenExecutionCollate dataTo collate historical versions of a file from a Git repository hosting service
like Github, provide the repository URL (repopositional argument), the
relative path to the file within the remote repository including the file name
(filepathpositional argument) and an optional date range (--startand--endarguments).  You can also, optionally, provide the local Git repository path
(--repo-pathargument), if the repository already exists locally and you wish
to override the remote repository download behavior.harvestcollatehttps://github.com/org-foo/repo-bar/raw/baz/baz.json--start20191201--end20191212--repo-path./bar-repoFile versions are written to the current local directory whereharvestwas
executed from.File versions are prefixed by the commit date inYYYYMMDDformat.File versions are gathered with daily granularity.Only the latest version of a file for a given day is retrieved.If a file did not change on a date then no file version is written for that
date.  Instead the latest version prior to that date serves as the version of
that file for that date.If you don't provide a--startand an--endthen the latest version of a
file is retrieved.If you only provide a--startdate file versions from the start date to the
current date are retrieved.If you only provide an--enddate the latest version of a file for the end
date is retrieved.Generate report(s)To run a report using content contained in a Git repository hosting service
like Github, provide the repository URL (repopositional argument), the report
package (package), the report name (namepositional argument) and include
any configuration that the report requires (--config) as a JSON string.  You
can also, optionally, provide the local Git repository path (--repo-pathargument), if the repository already exists locally and you wish to override
the remote repository download behavior.harvestreporthttps://github.com/org-foo/repo-barauditree_arboretumcheck_results_summary--config'{"start":"20191212","end":"20191221"}'Getting report detailsTo see a full summary of available reports within any package (likeauditree-arboretum) do:harvestreportsauditree_arboretum--listTo see details on a specific report that include usage example do something like:harvestreportsauditree_arboretum--detailcheck_results_summaryReport developmentReports should be hosted with the fetchers/checks that collect the evidence for
the reports process. Withinauditree-arboretumthis means the code lives in the
appropriate provider directory.  Contributing common harvest reports are as follows:Adhere to the auditree-arboretum contribution guidelines -TODO add link.Reports go in the "reports" folder by provider.Create a python module with a class that extends theBaseReporterclass.TheharvestCLI will use the report module name as the name of the
report (sans the .py extention).Only one report class per report module is permitted.In the new report class the expectations are as follows:Provide a module level docstring that contains:A single line summaryA detailed description of the report that includes evidence/files being
processed and expected configurationAt least one usage exampleUse thecheck results summary report docstringas an example/template.harvestuses this docstring to display available reports and their
details to the user.Provide/Override thereport_filenameproperty to return the name of the
report (including extension).harvestuses this property to apply a report
template (if desired) and to determine which writer function to use when writing
the report to a file.  Use thecheck results summary report report_filename propertyand thePython packages summary report report_filename propertyas examples.Provide/Override thegenerate_reportmethod.  This is where you put your
evidence processing and report formatting logic.  Use thecheck results summary report generate_reportmethod as an example.harvesttakes the optional--configcommand line argument as a JSON
string when executing a report, converts it to a dictionary and attaches
it as theconfigattribute to your report object.  Use the report object'sconfigattribute in thegenerate_reportmethod if you plan to have report
specific configuration options.Your report object also has a method that retrieves an evidence file for
a given date. Use the report object'sget_file_contentmethod when
retrieving evidence from an evidence locker.Generating CSV reports:harvestuses the PythonCSV writerto write out the
report file. So be sure that yourgenerate_reportmethod returns a
list of dictionaries that adheres to the expectations of the PythonCSV writer.Generating reports from a Jinja2 template:Add a report template named the same as yourreport_filenameproperty with a.tmplextension.harvestwill start to look for
the template in the same directory as the report module.  So as long as
it exists within that directory structure,harvestwill find it.
Usepython_packages_summary.md.tmplas an example.harvestwill look for this template file as part of your report
processing and, if found, will pass yourgenerate_reportreturned
content through the template logic.Yourgenerate_reportreturned content should be a dictionary with
everything necessary for your report template to render the desired
report appropriately.The report template can access the "raw" content generated bygenerate_reportthrough a dictionary nameddataand also has
access to the report's attributes through thereportobject.
Usepython_packages_summary.md.tmplas an example.Generating reports without templates:You just want to generate report content directly fromgenerate_report?
No problem.  Just generate a string as the report content or a list of
strings as the rows of the report content andharvestwill do the rest.Custom report developmentIf you find that you have a specific reporting need that does not fit in as a commonharvestreport, no problem.  Just develop the report in a separate repo/project
following the same guidelines as above.  As long as the package is importable by
python and you tellharvestwhat package to look for your report(s) in via the CLI,
it will handle the rest.