Package: ansible-content-parser
ansible-content-parserOverviewansible-content-parseranalyzes Ansible files in a given source
(a local directory, an archive file or a git URL)
by runningansible-lintinternally,
updates Ansible files using theAutofix feature ofansible-lintand generates theftdata.jsonlfile, which is the training dataset for
developing custom AI models.BuildExecute thetoxcommand. Installable images are created under
thedistdirectory.InstallationPrerequisitesPython version 3.10 or later.UNIX OS, such as Linux or Mac OS.Note:Installation on Microsoft Windows OS is not supported.Procedureansible-content-parseruses a newer version ofansible-lintand its
dependent components. In order to isolate them from the existing
Ansible installations, it is recommended to installansible-content-parserin
a Python virtual environment with the following steps:Create a working directory and set up venv Python virtual environment:python -m venv ./venv
source ./venv/bin/activateInstallansible-content-parserfrom the pip repository:pip install --upgrade pip
pip install --upgrade ansible-content-parserAfter the installation is completed, verify thatansible-content-parserandansible-lintare installed correctly:ansible-content-parser --version
ansible-lint --versionA list of application versions and their dependencies are displayed.
In the output that is displayed, ensure that you have the same version ofansible-lint.Important:If there is a mismatch in the installedansible-lintversions, you cannot get consistent results from the content parser and ansible-lint.
For example, the following result shows a mismatch inansible-lintversions:$ ansible-content-parser --version
ansible-content-parser 0.0.1 using ansible-lint:6.20.0 ansible-core:2.15.4
$ ansible-lint --version
ansible-lint 6.13.1 using ansible 2.15.4
A new release of ansible-lint is available: 6.13.1 → 6.20.0If theansible-lintversions do not match, perform the following tasks:Deactivate and reactivate venv:deactivate
source ./venv/bin/activateVerify that theansible-lintversions match:ansible-content-parser --version
ansible-lint --versionFor example, the following output shows the same ansible-lint versions:$ ansible-content-parser --version
ansible-content-parser 0.0.1 using ansible-lint:6.20.0 ansible-core:2.15.4
$ ansible-lint --version
ansible-lint 6.20.0 using ansible-core:2.15.4 ansible-compat:4.1.10 ruamel-yaml:0.17.32 ruamel-yaml-clib:0.2.7Executionansible-content-parseraccepts two positional parameters (sourceandoutput)
with a few optional parameters.$ ansible-content-parser --help
usage: ansible-content-parser [-h] [--config-file CONFIG_FILE]
                              [--profile {min,basic,moderate,safety,shared,production}] [--fix WRITE_LIST]
                              [--skip-ansible-lint] [--no-exclude] [-v] [--source-license SOURCE_LICENSE]
                              [--source-description SOURCE_DESCRIPTION] [--repo-name REPO_NAME] [--repo-url REPO_URL]
                              [--version]
                              source output

Parse Ansible files in the given repository by running ansible-lint and generate a training dataset for Ansible
Lightspeed.

positional arguments:
  source                source, which can be an zip/tar archive, a git URL or a local directory
  output                output directory

options:
  -h, --help            show this help message and exit
  --config-file CONFIG_FILE
                        Specify the configuration file to use for ansible-lint. By default it will look for '.ansible-
                        lint', '.config/ansible-lint.yml', or '.config/ansible-lint.yaml' in the source repository.
  --profile {min,basic,moderate,safety,shared,production}
                        Specify which rules profile to be used for ansible-lint
  --fix WRITE_LIST      Specify how ansible-lint performs auto-fixes, including YAML reformatting. You can limit the
                        effective rule transforms (the 'write_list') by passing a keywords 'all' (=default) or 'none'
                        or a comma separated list of rule ids or rule tags.
  -S, --skip-ansible-lint
                        Skip the execution of ansible-lint.
  --no-exclude          Do not let ansible-content-parser to generate training dataset by excluding files that caused
                        lint errors. With this option specified, a single lint error terminates the execution without
                        generating the training dataset.
  -v, --verbose         Explain what is being done
  --source-license SOURCE_LICENSE
                        Specify the license that will be included in the training dataset.
  --source-description SOURCE_DESCRIPTION
                        Specify the description of the source that will be included in the training dataset.
  --repo-name REPO_NAME
                        Specify the repository name that will be included in the training dataset. If it is not
                        specified, it is generated from the source name.
  --repo-url REPO_URL   Specify the repository url that will be included in the training dataset. If it is not
                        specified, it is generated from the source name.
  --version             show program's version number and exitsourcepositional argumentThe first positional parameter issource, which specifies
the source repository to be used. Following three types of sources are supported:File directory.Archive file in the following table:File FormatFile ExtensionZIP.zipUncompressed TAR.tarCompressed TAR.tar.gz, .tgz, .tar.bz2, .tbz2, .tar.xz, .txzGit URL, e.g.git@github.com:ansible/workshop-examples.gitorhttps://github.com/ansible/workshop-examples.gitoutputpositional argumentThe second positional parameter isoutput, which specifies a writable
directory. If the directory already exists, it has to be
an empty directory. If it does not exist, it will be newly created with
the given name.ansible-content-parsercreates therepositorysubdirectory in theoutputdirectory and copies the contents of thesourcerepository
to it. The copied contents may be changed by during the execution
of the Content Parser.OutputsFollowing directory structure is created in the directory specified with theoutputpositional argument.output/
  |-- ftdata.jsonl # Training dataset
  |-- report.txt   # A human-readable report
  |
  |-- repository/
  |     |-- (files copied from the source repository)
  |
  |-- metadata/
        |-- lint-result.json     # Metadata generated by ansible-lint
        |-- sarif.json           # ansible-lint results in SARIF format
        |-- (other metadata files generated)ftdata.jsonlThis is the training dataset file, which is the main output ofansible-content-parser.It is in the JSONL format, each of whose line represents a JSON objectreport.txtThis is a human-readable report that provides the summary information of the run
ofansible-content-parser, which contains sections like:File counts per typeList of Ansible files identifiedIssues found by ansible-lintList of Ansible modules found in tasksNote: When the--skip-ansible-lintoption is specified, the first three sections do
not appear in the report.metadata directoryThis subdirectory contains a few files that contain metadata generated
in the Content Parser run.lint-result.jsonlint-result.jsonis created in themetadatasubdirectory
as the result of the execution
ofansible-content-parser. The file contains a dictionary, which
has two key/value pairs:filesThis is for the list of files that were found
in the execution. The format of each file entry is explained below.Each file entry is represented as a dictionary that contains following keysKeyDescriptionbase_kindMIME type of the file, for example,text/yamldirDirectory where the file resides.excException found while processing this file. It can be null.filenameFile namekindFile type, for example,playbook,tasksorrolenameFile name (Usually same asfilename)parentName of the parent, like a role. It can be nullroleAnsible role. It can be nullstop_processingIdentifies whether processing was stopped on this file or notupdatedIdentifies whether contents were updated byansible-lintFollowing shows an example of a file entry:{"base_kind":"text/yaml","dir":"/mnt/input/roles/delete_compute_node/tasks","exc":null,"filename":"roles/delete_compute_node/tasks/main.yaml","kind":"tasks","name":"roles/delete_compute_node/tasks/main.yaml","parent":"roles/delete_compute_node","role":"delete_compute_node","stop_processing":false,"updated":false}excludedThis is for the list of file paths, which were excluded in the secondansible-lintexecution because syntax check errors were found in those files on the first execution.
The files included in the list will not appear in the entries associated with thefileskey.Note:Ifansible-content-parseris executed with the--no-excludeoption, the second execution
does not occur even if syntax check errors were found on the first execution and
the training dataset will not be created.sarif.jsonThis is the output ofansible-lintwith the--sarif-fileoption.
Thereport.txtcontains a summary generated
from this file in the "Issues found by ansible-lint" section.