Package: import-yocto-bm
Synopsys Import Yocto Build Manifest - import_yocto_bm.py - v2.8INTRODUCTIONThis script is provided under an OSS license as an example of how to use the Black Duck APIs to import components from a Yocto project manifest.It does not represent any extension of licensed functionality of Synopsys software itself and is provided as-is, without warranty or liability.If you have comments or issues, please raise a GitHub issue here. Synopsys support is not able to respond to support tickets for this OSS utility.OVERVIEWTheimport_yocto_bm.pyscript is designed to import a Yocto project build manifest created by Bitbake. It replaces all previous scripts (includinghttps://github.com/matthewb66/import_yocto_build_manifest).It can be used as an alternative to the standard Yocto scan process for Black Duck provided within Synopsys Detect, and provides additional capabilities including checking recipes against the Black Duck KB, replacing recipe specifications where required and propagating locally patched CVEs to the Black Duck project.Version 2.1 provides an optional wizard mode to request additional information where it is not provided (use --nowizard to disable this mode for batch operation).YOCTO SCANNING IN BLACK DUCK USING SYNOPSYS DETECTThe default Yocto scan process for Black Duck is to determine Bitbake dependencies using Synopsys Detect (seeSynopsys Detect - scanning Yocto).You should NOT use Black Duck Signature scanning for whole Yocto projects (although Signature scanning is useful for custom recipes which may contain OSS which you want to identify). Signature scanning will not function accurately on Yocto projects because the code is modified locally by change/diff files and no reference copy of the changed code is available, meaning that Signature matching will not be able to determine the correct Yocto modified OSS components. Yocto projects are also very large and Signature scanning can consume large volumes of server resources without providing accurate matching.When used to scan a Yocto project, Synopsys Detect calls Bitbake to report the project layers and recipes, checking against the recipes reported at OpenEmbedded.org (refer tolayers.openembedded.orgto identify layers and recipes which will be mapped) and creating a Black Duck project containing the mapped components. Recipes in layers not reported at OpenEmbedded.org will not be imported into Black Duck, and you should consider using other Black Duck scan techniques to identify OSS within specific custom recipes referenced in the build. Note also that moving original OSS recipes to new layers will also stop them being reported.The resulting project will contain mapped recipes/layers including those used to create the image but which may not delivered in the built image. Recipe matching also requires that OSS components are used within standard recipes/revisions and within the original layer. If you copy a recipe to a new or different layer, it will not be matched in the Black Duck project by Synopsys Detect.To perform a Yocto scan using Synopsys Detect:Change to the poky folder of a Yocto projectRun Synopsys Detect adding the options--detect.tools=DETECTOR --detect.bitbake.package.names=core-image-sato(wherecore-image-satois the package name).New bitbake options have been added in Synopsys Detect 7.9 including the ability to locate the manifest files and remove build dependencies ( for example--detect.bitbake.dependency.types.excluded=BUILD- see (here)[https://community.synopsys.com/s/document-item?bundleId=integrations-detect&topicId=properties%2Fdetectors%2Fbitbake.html] ).WHY THIS SCRIPTThis script provides an alternative to scanning Yocto projects in Synopsys Detect.It should be considered in the following scenarios:Where many standard OpenEmbedded recipes have been moved to new layers (meaning they will not be matched by Synopsys Detect)Where OpenEmbedded recipe versions or revisions have been modified from the originalWhere a report is required for custom recipes so they can be scanned independentlyThe identification of locally patched CVEs is desiredThe script operates on a built Yocto project, by identifying the build manifest containingonly the recipes which are within the built imageas opposed to the superset of all recipes used to build the distribution (including build tools etc.) produced by Synopsys Detect by default. This script also optionally supports extracting a list of locally patched CVEs from Bitbake and marking them as patched in the Black Duck project.Recipes/layers are checked against the recipes maintained in the Black Duck KB which have been extracted from OpenEmbedded.org and those which match are added to the created Black Duck project.The script should be executed on a Linux workstation where Yocto has been installed and after a successful Bitbake build.If invoked in the Yocto build folder (or the build folder is manually specified using the -y option), then it will locate the license-manifest file automatically in the tmp/deploy hierarchy. Alternatively you can specify the license.manifest file as a script option or the default wizard mode will ask you to specify the file.If the Bitbake build was performed with the Yoctocve_checkclass configured, then the script will also locate the CVE log exported by CVE check, extract patched CVEs and set the remediation status of matching CVEs in the Black Duck project.The script requires access to the Black Duck server via the API (see Prerequisites below) unless the -o option is used to create the output scan file for manual upload.OVERALL SCAN PROCESS USING IMPORT_YOCTO_BMUsing this script will allow standard OSS recipes to be imported to the Black Duck project. It will also identify where standard OSS recipes have been moved to new or different layers, and match close revisions or versions which exist in the Black Duck KB.The proposed process to scan a Yocto project using this script is:Use the import_yocto_bm to identify original OSS recipes within the project and create a Black Duck project version.Add the--report rep.txtoption to export a report of the matched recipes. This will include reports categorised as follows:OK = recipe matched the Black Duck KB and will be included in the Black Duck projectREPLACED = recipe has been moved to a new layer and the script has referenced the original layer in the KB or the recipe was changed using a replacefile entry - will be included in the Black Duck projectREPLACED_NOREVISION = the script has replaced the revision to match the KB - will be included in projectREPLACED_NOLAYER+REVISION = recipe has been moved to a new layer with a new revision and the script has referenced the original layer and revision in the KB - will be included in the Black Duck projectNOTREPLACED_NOVERSION = the layer and recipe exist in the KB but the version does not - will not be included in the Black Duck projectNOTREPLACED_NOLAYER+VERSION = the recipe exists in the KB but the layer and version do not - will not be included in the Black Duck projectMISSING = recipe does not exist in the KB and will not be included in the Black Duck projectSKIPPED = recipe is not mapped in bblayers data (corrupt yocto project?) - skippedFor the NOTREPLACED_NOVERSION and NOTREPLACED_NOLAYER+VERSION recipes, you could consider using the--replacefile repfileoption to map to layers/recipes and version/revisions which exist in the KB, rerunning the script to import them. See the section REPLACING RECIPE NAMES below.For the MISSING recipes, these are almost certainly custom recipes containing either a mix of custom application code (potentially with OSS sources) or new OSS not obtained from Openembedded.org as standard OSS recipes. Identify the layers which contain mainly custom recipes, and use standard Black Duck signature scanning (and optionally snippet scanning) to search for modified OSS within these sub-folders only.Do not run a signature or snippet scan on the entire Yocto project- only use signature scanning for custom layers and recipes not mapped by this script. You can optionally combine the dependency and signature scans in the same Black Duck project.SUPPORTED YOCTO PROJECTSThis script is designed to support Yocto versions 2.0 up to 3.3 for building projects.OSS components from OpenEmbedded recipes maintained at layers.openbedded.org should be detected by the scan. Additional OSS components managed by custom recipes will not be detected.PREREQUISITESMust be run on LinuxPython 3 must be installed.A supported Yocto environment (version 2.0 to 3.1) should be loaded to the current shell (seePreconfigurationsection below). Alternatively use the --bblayer_out option to specify the output of thebitbake-layers show-recipescommand run separately.The Yocto project must have been pre-built with alicense.manifestfiles available.OPTIONAL: For patched CVE remediation in the Black Duck project, you will need to add thecve_checkbbclass to the Yocto build configuration to generate the CVE check log output. Add the following line to thebuild/conf/local.conffile:INHERIT += "cve-check"Then use the Yocto build command (e.g.bitbake core-image-satowhich will incrementally build without needing to rerun the full build, but will add the CVE check action to generate the log files.INSTALLATIONInstall the utility using pip - for example:pip3 install import_yocto_bmAlternatively, clone the repository and run directly using:python3 import_yocto_bm/main.pySTANDARD USAGERun the commandimport_yocto_bmwithout arguments to invoke the wizard to guide you through the required information and options.Theimport_yocto_bmparameters for command line usage are shown below:usage: import_yocto_bm [-h] [-p PROJECT] [-v VERSION] [-y YOCTO_FOLDER]
			[-t TARGET] [-o OUTPUT_JSON] [-m MANIFEST]
			[-b BUILDCONF] [-l LOCALCONF] [--arch ARCH]
			[-r repfile] [--cve_check_only] [--no_cve_check]
			[--cve_check_file CVE_CHECK_FILE] [--report rep.txt] [--nowizard|-wizard]

Import Yocto build manifest to BD project version

optional arguments:
  -h, --help            show this help message and exit
  --blackduck_url BLACKDUCK_URL
                        Black Duck server URL (also uses $BLACKDUCK_URL environment variable)
  --blackduck_api_token BLACKDUCK_API_TOKEN
                        Black Duck API token (also uses $BLACKDUCK_API_TOKEN environment variable)
  --blackduck_trust_cert
                        Black Duck trust server cert
  -p PROJECT, --project PROJECT
			Black Duck project to create (REQUIRED)
  -v VERSION, --version VERSION
			Black Duck project version to create (REQUIRED)
  -y YOCTO_FOLDER, --yocto_build_folder YOCTO_FOLDER
  			Yocto build folder (required if CVE check required or
			manifest file not specified)
  -o OUTPUT_JSON, --output_json OUTPUT_JSON
			Output JSON bom file for manual import to Black Duck
			(instead of uploading the scan automatically)
  -t TARGET, --target TARGET
			Yocto target (default core-poky-sato)
  -m MANIFEST, --manifest MANIFEST
			Input build manifest file (if not specified will be
			determined from conf files) - must be the 
			license.manifest file (not build.manifest)
  -b BUILDCONF, --buildconf BUILDCONF
			Build config file (if not specified 
			poky/meta/conf/bitbake.conf will be used)
  -l LOCALCONF, --localconf LOCALCONF
			Local config file (if not specified 
			poky/build/conf/local.conf will be used)
  -r REPFILE, --replacefile REPFILE
  			Replace file used to replace layer and recipe names
  --arch ARCH           Architecture (if not specified then will be determined
			from conf files)
  --cve_check_only      Only check for patched CVEs from cve_check and update
			existing project
  --no_cve_check        Skip check for and update of patched CVEs
  --cve_check_file CVE_CHECK_FILE
			CVE check output file (if not specified will be
			determined from conf files)
  --no_kb_check         Do not check recipes against KB
  --kb_recipe_dir KB_RECIPE_DIR
                        KB recipe data directory local copy
  --report rep.txt	If KB check is performed, produce a list of matched. modified and unmatched recipes
  --bblayers_out bbfile
  			Can be used to scan a build without access to the build environment - a file 
			containing the output of the command `bitbake-layers show-recipes '*'` instead of 
			calling the command directly as well as the license.manifest file to be specified 
			with -m. Also skips identification of recipe revisions (assumes all revisions are -r0)
  --wizard
            Start command line wizard (Wizard will run by default if config incomplete)
  --nowizard
            Do not start command line wizard even if config incomplete (batch mode)The script will use the invocation folder as the Yocto build folder (e.g. yocto_zeus/poky/build) by default (if there is abuildsub-folder then it will be used instead). The--yocto_folderoption can be used to specify the Yocto build folder as opposed to the invocation folder.The--projectand--versionoptions are required to define the Black Duck project and version names for inclusion in the json output file (and update CVE patch status).The--output_jsonoption can be used to specify an output file for the project scan. If specified, then the scan will not be uploaded automatically and CVE patch checking will be skipped.The--replacefileoption can be used to specify a layer/recipe/version replacement file (see REPLACING LAYER AND RECIPE NAMES below).The Yocto target and architecture values are required to locate the manifest and cve_check log files and will be extracted from the Bitbake config files automatically, but the--targetand--archoptions can be used to specify these manually.The most recent Bitbake output manifest file (located in thebuild/tmp/deploy/licenses/<image>-<target>-<datetime>/license.manifestfile) will be located automatically. Use the--manifestoption to specify the manifest file manually.The most recent cve_check log filebuild/tmp/deploy/images/<arch>/<image>-<target>-<datetime>.rootfs.cvewill be located automatically if it exists. Use the--cve_check_fileoption to specify the cve_check log file location manually (for example to use an older copy).Use the--cve_check_onlyoption to skip the scanning of the project and creation of a project, only looking for a CVE check output log file to identify and patching matched CVEs within an existing Black Duck project (which must have been created previously).Use the--no_cve_checkoption to skip the patched CVE identification and update of CVE status in the Black Duck project.PRECONFIGURATIONTheimport_yocto_bmscript calls Bitbake commands by default, so you will need to run the Yocto project config (change the location as required) before using the script - for example:cd /home/users/myuser/yocto_zeus/poky
source oe-init-build-envThis will set the Yocto environment and change directory into the Yocto build sub-folder.Alternatively, you can run thebitbake-layer show-recipescommand separately to create an output file, then use the--bblayers_out fileoption (changefileto the output file name) or specify the file in the wizard mode.If you want to upload the resulting project direct to Black Duck (and are not using the --output_json option), you specify the Black Duck server with URL & API_TOKEN either using command line options:--blackduck_url SERVER_URL
  --blackduck_api_token TOKEN
  --blackduck_trust_cert (specify if untrusted CA certificate used for BD server)OR by setting environment variables:BLACKDUCK_URL=https://SERVER_URL
  BLACKDUCK_API_TOKEN=TOKENWhereSERVER_URLis the Black Duck server URL andTOKENis the Black Duck API token.REPLACING RECIPE NAMESLayers and recipes extracted from the project are combined by Black Duck and used to lookup original OSS components athttps://layers.openembedded.org. If OSS components are moved from original layers to a new (custom) or different layer which is not shown athttps://layers.openembedded.organd which cannot be mapped by a close neighbour in the script, then they will not be mapped in the resulting Black Duck project.To reference a specific original component, you can use the--replacefile REPFILEoption to map OSS components back to original recipes.The replacefile option can also be used to remap new OSS component versions (not listed athttps://layers.openembedded.org) to previous versions which are listed.Example REPFILE content is shown below:RECIPE meta-customlayer/alsa-lib/1.2.1.2-r5 meta/alsa-lib/1.2.1.2-r0which will remap recipe and versionalsa-lib/1.2.1.2-r5in themeta-customlayertometa/alsa-lib/1.2.1.2-r0in teh Black Duck KB.EXAMPLE USAGECheck thePreconfigurationsection above before running the script.To run the utility in wizard mode, simply use the commandimport_yocto_bmand it will ask questions to determine the scan parameters.Use the option--nowizardto run in batch mode and bypass the wizard mode, noting that you will need to specify all required options on the command line correctly.Use the following command to scan a Yocto build, create Black Duck projectmyprojectand versionv1.0, then update CVE patch status for identified CVEs (will require the OE environment to have been loaded previously):import_yocto_bm -p myproject -v v1.0To scan a Yocto project specifying a different build manifest as opposed to the most recent one (will require the OE environment to have been loaded previously):import_yocto_bm -p myproject -v v1.0 -m tmp/deploy/licenses/core-image-sato-qemux86-64-20200728105751/package.manifestTo scan the most recent Yocto build in a different build folder location (not the current folder - will require the OE environment to have been loaded previously):import_yocto_bm -p myproject -v v1.0 --y $HOME/newyocto/poky/buildTo perform a CVE check patch analysis only (to update an existing Black Duck project created previously by the script with patched vulnerabilities) use the command (will require the OE environment to have been loaded previously):import_yocto_bm -p myproject -v v1.0 --cve_check_onlyTo create a JSON output scan without uploading (and no CVE patch update) use (will require the OE environment to have been loaded previously):import_yocto_bm -p myproject -v v1.0 -o my.jsonldTo create a JSON output scan without uploading (and no CVE patch update) without checking recipes against the KB recipe list downloaded from Github (will require the OE environment to have been loaded previously):import_yocto_bm -p myproject -v v1.0 -o my.jsonld --no_kb_checkIf you are unable to load the OE environment prior to running the utility, ensure you have the output of the commandbitbake-layers show-recipesin a file, and use the option--bblayers_out OUTPUT_FILEto specify the file OUTPUT_FILE containing this output, for example:import_yocto_bm -p myproject -v v1.0 -m poky/build/tmp/output/license.manifest --bblayers_out bblayers.txtCVEs from cve_check Versus Black DuckThe Yoctocve_checkclass works on the Bitbake dependencies within the dev environment, and produces a list of CVEs identified from the NVD for ALL packages in the development environment.This script extracts the packages from the build manifest (which will be a subset of those in the full Bitbake dependencies for build environment) and creates a Black Duck project.The list of CVEs reported bycve_checkwill therefore be considerably larger than seen in the Black Duck project (which is the expected situation).OUTSTANDING ISSUESThe identification of the Linux Kernel version from the Bitbake recipes and association with the upstream component in the KB has not been completed yet. Until an automatic identification is possible, the required Linux Kernel component can be added manually to the Black Duck project.UPDATE HISTORYV2.8Updated kb data & fixed bug refering to old fileV2.7Fixed issue with trustcert option usageV2.6Some wizard fixesV2.5Addressed API limits introduced in 2022.2V2.4Fixed another issue with replacefileV2.3Fixed issue with replacefile usageV2.2Various minor fixesV2.1Added extra information to report output fileAdded defaults to wizard mode & check for bd_trustcertV2.0Added new Wizard modeOther fixes to manage component version epochsUpdated KB dataV1.17 BetaUpdate to new KB dataV1.16 BetaFixed CVE patch process to use new Client API classV1.15 BetaChanged KB recipe download to use json formatted filesChanged --kb_recipe_file to --kb_recipe_dir to support json filesV1.14 BetaReworked replacefile logic and improved console reportingV1.13 Betareplaced the --debug option with --bblayers_out with the ability to define the input file.Fixed the --report option to use the specified report file (previously always wrote to report.txt)Changed the recipe matching logic to also look for KB matches without revisions (-rX) to support older Yocto versionsV1.12Minor bug fixesV1.11Fixed a minor issue with AUTOINC component versions.Previous VersionsVersions 1.10 and 1.9 added fixes for the KB recipe lookup, and a new report output showing the list of recipes matched, modified or not matchable (use--report reort.txtoption) as well as updating the list of KB recipes located in the data file.Version 1.8 will automatically download a list of all Yocto projects stored in the Black Duck KB and checks the recipes in the local project, remapping recipes to origin layers or revisions to reduce the number of non matched components in the resulting Black Duck project. This functionality can be skipped using the --no_kb_check option, or if the download of the KB data from Github is not possible within the script, the --kb_recipe_file option allows a local copy to be used.The DEPLOY_DIR and TMP_DIR values extracted from conf files now support the use of environment variables. Other fixes have been applied especially to the processing of the recipeinfo files and extraction of recipe revisions.