SRTool Developer Page

Category:SRTool
This page summarizes the Security Response Tool (SRTool) development process. We hope this will help you start contributing to the project. The SRTool is based on the Toaster codebase, so many of the process and debugging techniques apply.
Parent Wiki page: Contribute to SRTool
__FORCETOC__
== Design Philosophy ==
* Make the tool easy to use
** Support a web-based interface
** Support backend scripts
** Make the next-steps easy to find and understand
** On-line documentation
* Make the design modular
** It must be easy to add new CVE upstream data sources
** It must be easy to adapt the SRTool to a new orgainization
** It must be easy to add new reports and exports
* Make the design adaptable
** Use the Django migration feature for on-the-fly updates
** Make the data/interface table driven, remove hard coding as much as possible
* Minimize the data in the database
** Only put CVE summary information in the database
** Extract CVE details on-the-fly from the source data files
** Cache CVE data detail lookups for fast response
** Only put Defect summary information in the database
** Provide links to the Defect system, with its full defect data
* Use scripts to bulk load/update the database
** Scripts can update the database 10x faster than the Django DB interface
** Scripts support background cron-job driven updates
* Support data recovery
** The system will fail, so be prepared
** Provide data backup scripts, cron-job supportable
** Provide backup data recovery
** Make it easy to re-build the system from scratch
* Support audit trails
** Every user change should be trackable
** It must be easy to generate a report on a CVE's action history
* Support protected data
** There must be a validated user model so that changes can be tracked
** There must be a way to mark individual CVEs and there releated data private to specific users
== SRTool Startup ==
This section describes how the SRTool gets started and initialized.
=== SRTool Start-up High-Level ===
High level actions from the "bin/srt" startup script.
Description 	Call 	Data file 	Comments
Check if the webport is already used 	"manage checksocket ..." 	./.srtmain.pid 	If already used, warn and halt
Check the database and schema is up to date	"manage migrate ..." 	./srt.sqlite 	Create database if needed, assert migrations
Check database has fixtures and datasources	"manage checksettings ..."	Django/SRTool migration files	Apply fixtures, apply un-read datasource data
Start the server 	"manage runserver ..." 	./.srtmain.pid 	Start the server thread, open the webport
=== SRTool Start-up Details ===
Detailed actions from Django and "lib/*" startup scripts.
Description 	Code file 	Data file 	Comments
Check the database 	Check the database 	Check the database 	Check the database
Check if the webport is already used 	lib/srtmain/management/commands/checksocket.py	.srtmain.pid 	If not, 'bin/srt' warns and halts
Check if the database exists 	./bin/srt: "manage migrate" (Django) 	srt.sqlite 	If not, create empty database
Assert the Migrations 	./bin/srt: "manage migrate" (Django) 	lib/*/migrations/* 	Process migration files (not already added)
Setup Django Environment 	Setup Django Environment 	Setup Django Environment 	Setup Django Environment
Django default settings 	./bin/srt: "manage checksettings" (Django) 	lib/srtmain/settings.py 	Set the plug-ins, database connection, web connection, security, app priority
Apply the default fixture 	lib/orm/management/commands/checksettings.py 	lib/orm/fixtures/common.xml 	Set the default system database records
Apply the 'custom' fixture (if any) 	lib/orm/management/commands/checksettings.py 	lib/orm/fixtures/custom.xml 	Set the optional custom override system database records
Gather and sort the datasources 	lib/orm/management/commands/checksettings.py 	bin/*/datasource.json 	Common datasource is first, main app is last, rest sorted by 'key' value
Process the Data Sources 	Process the Data Sources 	Process the Data Sources 	Process the Data Sources
Process each datasource in order ... 	lib/orm/management/commands/lsupdates.py 	bin/*/datasource.json 	Add the datasource records, execute the 'init' function
1. process the initial 'common' datasource data	lib/orm/management/commands/lsupdates.py 	bin/common/datasource.json 	Add the common datasource records, execute the 'init' function
2. process the 'NIST' datasources 	lib/orm/management/commands/lsupdates.py 	bin/nist/datasource.json 	Fetch and add the NIST CVEs
3. process the 'MITRE' datasources 	lib/orm/management/commands/lsupdates.py 	bin/mitre/datasource.json 	Fetch and add the MITRE CVEs
4. process the other datasources 	lib/orm/management/commands/lsupdates.py 	bin/*/datasource.json 	Fetch and add the other data sources
5. process the master app datasources 	lib/orm/management/commands/lsupdates.py 	bin/yp/datasource.json 	Fetch and add the master app data source (e.g. 'yp')
5a. Define the external Defect Manager 	bin/yp/srtool_yp.py 	bin/yp/srtool_defect.py 	The 'init' function can read CVE defects and pre-populate the Vulnerabilities and Investigations
5b. Define the products 	bin/yp/srtool_yp.py 	bin/yp/yocto-project-products.json	The 'init' function can pre-populate the organization's Products
5c. Define the default users 	bin/yp/srtool_yp.py 	bin/yp/yocto-project-users.json 	The 'init' function can pre-populate the organization's users
6. process the final common datasource data 	lib/orm/management/commands/lsupdates.py 	bin/common/datasource.json 	Fetch and add the final data source (e.g. 'local cve', pre-score the 'new' CVEs)
Start the server 	Start the server 	Start the server 	Start the server
Start the server 	./bin/srt: "manage runserver" (Django) 	.srtmain.pid 	Start the server thread, open the webport
=== SRTool Start and Restart Notes: ===
* You can have as many SRTool instances as you wish (e.g. production, test, development, ...) as long as they use separate webports and separate directories (to avoid a collision on "./.srtmain.pid")
* Fixture values are always applied during starts and restarts. They use absolute record ID locations, and should be used for rock-solid bootstrap values that are fixed for the life time of the database.
* Datasource "srtsetting" values are like fixture values but use dynamic record locations, and are preferred for flexible and/or changeable data over the life of the database.
* The datasource "Init" functions are only executed once. This happens at the first startup-up, and any new data sources get processed at the next re-start.
== Major Components ==
=== Data Sources ===
The Data Source model is used to cleanly and dynamically connect external content (e.g. CVE sources) and managers (e.g. Defect Systems) to the SRTool.
Specifically, they define:
* Upsteam/external data
* Initialization actions
* Update actions
* Management actions
* Backup actions
Here is the datasource schema:
name 	description
srtsetting (normally defined in master organization datasource)	srtsetting (normally defined in master organization datasource)
SRTOOL_DOCUMENATION_URL 	The SRTool top bar documentation link
SRTOOL_LOCAL_LOGO 	The optional path to the top bar logo for the orgainization
SRTOOL_DEFECT_ADD 	Script: Add an existing defect to SRTool defect database, '--add defect'
SRTOOL_DEFECT_DEL 	Script: Delete an existing defect from SRTool defect database, '--del defect'
SRTOOL_DEFECT_NEW 	Script: create defect '--new program summary description priority components urllink'
SRTOOL_DEFECT_SAMPLENAME 	Displayed text schema of an example defect
SRTOOL_DEFECT_TOOL 	The registered script to manage defects
SRTOOL_DEFECT_URLBASE 	The url base lookup for the external defect tool
datasource 	datasource
key 	Sort key for this datasource entry
data 	Type of data (e.g. 'cve', 'database_schema', 'package_keywords', 'notify_categories', ...)
source 	Soure of the data (e.g. 'common', 'nist' ,'mitre', ...)
name 	Name for this Data Source
description 	Description of this Data Source
cve_filter 	If CVE datasource, prefix of included CVEs (e.g. "CVE-", "CVE-2018", ...)
init 	The init function for STARTUP, either a script call or a file path
update 	The updated function for UPDATES, either a script call or a file path
lastModifiedDate 	Date the last time this source was updated
update_frequency 	How often the UPDATE function should be called
update_time 	Dictionary to refine the update time (see below)
srtuser ('common' only) 	srtuser ('common' only)
name 	Name for this built-in SRTool user
email 	Email for this built-in SRTool user (typically the admin's or empty)
role 	The role for this user
is_staff 	This user is built-in (i.e. set to 'True')
srtuser_groups (Django user model, 'common' only) 	srtuser_groups (Django user model, 'common' only)
srtuser 	Name of above 'srtuser'
group 	Name of below 'group'
permissions (Django user model, 'common' only) 	permissions (Django user model, 'common' only)
content_type_id 	Unique record ID for this permission
codename 	Code Name for this permission (e.g. "Reader", "Admin", ...)
name 	Name for this permission
groups (Django user model, 'common' only) 	groups (Django user model, 'common' only)
name 	Name for this group (e.g. "Reader", "Admin", ...)
group_permissions (Django user model, 'common' only) 	group_permissions (Django user model, 'common' only)
group 	Group for this group (e.g. "Reader", "Admin", ...)
permission 	Above 'permission' name
=== Defect System Integration ===
The organization's Defect System integration script is expected to fulfill these requirements. The SRTool expects these actions to be defined the 'data source' that wraps the defect system integration script:
* Update command: this will perform the initial population of the SRTool of the CVE-affected defects in the Defect system
* Refresh command: this will perform a refresh of the SRTool of the CVE-affected defects in the Defect system, updating existing records and creating any new ones
* Create command: this will create a new defect, based on the parameters passed from the SRTool (Product, CVE, description, priority, category)
Implicit in the above features are the following. These will need to be researched and implemented against the specific Defect System:
* Ability to identify CVE-affected defects in the Defect system
** Some systems have the related CVE in a special field
** Some systems have the related CVE in a text field, for example in the defect's title/summary
* Ability to both lookup and create CVE-specific defects from Defect system
** For the major Defect Systems (e.g. Jive and Bugzilla) there are python libraries the support queries
** Otherwise, you will need to connect to/provide the specific supported API
The SRTool provides the following support:
* The file "bin/common/srtool_jira_template.py" is an example implementation for Jira
** It is based on Wind River's integration, whom also maintain and accept patches this template file
** See this link on how to leverage this template for your organization: Jira integration script support
* Here is how the SRTool registed products connect to their counter parts in the Defect System
** Each product record has a “defect_tags” member. This is a free form dictionary that can provide the values needed to map to the respective product records
** For example, the Jira python library expects a 'project' entry with a dictionary of the form {'key': product_defect_key}
** Here is an example mapping:
 # "bin/foo/foo_products.json" : Product JSON file:
 "defect_tags" : "{\"key\":\"FOO\"}",
 # "bin/foo/srtool_jira_foo.py": Defect tag translation
 product_defect_key = get_tag_key(product_defect_tags,'key')
 # "bin/foo/srtool_jira_foo.py": Jira query element
 issue_dict = {
 'project': {'key': product_defect_key},
* Here for example is how "srtool_jira_template" queries Jira for all CVE-related defects, once for each registered project, using the simple string matching method (where the CVE name is expected in the defect title):
 # searches current project's bug issues that contain "cve" in their text
 issues = jira.search_issues('project=%s AND text ~ "cve*" AND type = Bug' % product_defect_prefix, start_idx, block_size, False, 
* It is recommendation that admins define 'SRT_DEFECT_USER' and 'SRT_DEFECT_PASSWD' in the shell environment so that the credentials are never in a readable file. See these lines in the “srtool_acme_jira.py” script:
 srt_user = os.environ.get('SRT_DEFECT_USER')
 srt_passwd = os.environ.get('SRT_DEFECT_PASSWD')
=== Report/Export System ===
* Ability to export respective data from any page
** The "Export" button is available for all pages in the top bar.
** Most pages already have a default report setup, as seen in "lib/srtgui/reports.py"
** Additional pages can have a report defined via the class ReportManager() in "lib/srtgui/reports.py"
** The page's respective report is passed via "request.resolver_match.url_name"
* Ability to generate reports in multiple formats (e.g. text, CSV)
** Most pages already have a default "Text" and "CSV" format definded in "lib/srtgui/reports.py"
* Ability to easily add new reports and new output formats
** Examine "lib/acme/reports.py" to see how to easily extend and/or replace existing report implementations for custom master applications
=== Email Notification System ===
* The SRTool supports emailing notifications
** The current model has the SRTool manager review the notifications, who can then mark and send them
** The SRTool will support the automatic sending of appropriate emails (feature currently under design)
* The notifications can be:
** Manually created ones (e.g. see a Vulnerability or Investigation page)
** Automatically generated (e.g. an upstream CVE or attached Defect has changed priority of status)
* To enable the email system, these values must be defined:
** SRT_EMAIL_SMTP: The SMTP server IP address
** SRT_EMAIL_USER: The SRTool's email user account
** SRT_EMAIL_PASSWD: The SRTool's email user password
** SRT_EMAIL_FROM: The email 'from' address
* There are two ways to set these values
** They can be defined as SrtSetting values in the organization's "datasource.json" file (see "bin/acme/datasource.json_sample")
** They can be defined as environment variables (see "bin/dev_tools/srt_env.sh")
** Any of these SrtSetting values set to an empty string will result in the value being explicitly loaded from the environment
* NOTE: it is strongly suggested that the email password not be explicitly defined in the database
** It is preferable to instantiate "SRT_EMAIL_PASSWD" as an environment variable in the SRTool's execution shell, so that it is invisible to the outside
** To that point, in the above examples the SrtSetting "SRT_EMAIL_PASSWD" value is pre-defined as an empty string
* Here is the location of the SRTool email code:
** GUI: "lib/srtgui/views.py", in "xhr_notifications"
** Command line: bin/common/srtool_email.py"
* To bring up and test the email system, it is recommended that you start by directly using the command line tool to validate your settings, for example:
 $ bin/common/srtool_email \
 --from=road.runner@acme.com \
 --to=<your email address> \
 --subject="Test email" \
 --server="smtp.acme.com" \
 --user="rrunner" \
 --passwd="beepbeep" \
 --tls \
 --message="This is an email test" \
 [--verbose] \
 [--test]
* Most corporate email systems require the TLS flag. This is default for the SRTool.
=== Automatic and manual data source updates ===
* The SRTool has a facility to regularly refresh the upstream data, for example CVEs and Defect status
* The update job runs in the background, and is started and stopped together with the SRT web server
* Each data source can (a) select the update frequency, and (b) define the desired action in the 'update' field, typically a script call
* The update time can be refined with the 'update_time' filter value dictionary:
 update_frequency update_time (optional)
 ================ ======================================================
 MINUTELY = 0 "{\"minutes\":\"10\"}" # every ten minutes
 HOURLY = 1 "{\"minute\":\"10\"}" # at ten minutes past the hour
 DAILY = 2 "{\"hour\":\"2\"}" # at 2 hours after midnight
 WEEKLY = 3 "{\"weekday\":\"6\",\"hour\":\"2\"}" # day of week, hour
 MONTHLY = 4 "{\"day\":\"1\"\"hour\":\"2\"}" # day of month
 ONDEMAND = 5 "{}" # only on demand
 ONSTARTUP = 6 "{}" # on every SRTool start up
* The most recent status of the update scan is available in this file:
 $ cat update_logs/update_status.log 
 ====================
 Update: Date=2019-01-14 14:46:36,Filter='all',Force=False
 Skip 'Weekly archive database backup': update time not reached (2019-01-26 14:13:43)
 Skip 'Daily archive database backup': update time not reached (2019-01-15 14:13:43)
 Skip 'Score CVEs': update time not reached (2019-01-14 14:48:23)
 Skip 'MITRE 2016': update time not reached (2019-01-26 13:36:45)
 Skip 'MITRE 2019': update time not reached (2019-01-26 13:36:45)
 ...
* The developer can also perform direct updates
* Here is the list of manual update options
 $ # Display options (with the 'manage' options removed for clarity)
 $ ./bin/srt manage update -h
 SRT_MAIN_APP=yp
 usage: manage.py update [-h] [-v {0,1,2,3}]
 [--cron-start] [--cron-stop] [--list] [--run-updates]
 [--force]
 [--name-filter NAME_FILTER [NAME_FILTER ...]]
 [--verbose] [--trial]
 optional arguments:
 -h, --help show this help message and exit
 -v {0,1,2,3}, --verbosity {0,1,2,3}
 Verbosity level; 0=minimal output, 1=normal output,
 2=verbose output, 3=very verbose output
 --cron-start Start the SRTool background updater
 --cron-stop Stop the SRTool background updater
 --list, -l List data sources
 --run-updates, -u update scheduled data sources
 --force, -f Force the update
 --name-filter NAME_FILTER [NAME_FILTER ...], -n NAME_FILTER [NAME_FILTER ...]
 Filter for datasource name
 --verbose Debugging: verbose output
 --trial, -t Debugging: trial run
* See the list of available update data sources
 $ ./bin/srt manage update --list
 SRTool Update List:
 Data Source Name Frequency Offset Description
 =============== ======= ============== ========== ============================ ===========================================
 database_schema common SRT Schema OnStartup {} SRT database schema for command line tools
 backup_weekly common Weekly Backup Weekly {"weekday":"5","hour":"2"} Weekly archive database backup
 backup_daily common Daily Backup Daily {"hour":"7"} Daily archive database backup
 score_cves common Score Minute {"minutes":"10"} Score CVEs
 cve mitre MITRE Weekly {"weekday":"5","hour":"2"} MITRE 2016
 cve mitre MITRE Weekly {"weekday":"5","hour":"2"} MITRE 2019
 ...
* Update all sources (with optional "--force" option to update now)
 $ ./bin/srt manage update --run-updates [--force]
 BEGINNING UPDATING DATASOURCES... this MAY take a long time
 SRTool Update: time_now = 2019-01-14 20:18:12
 Update required	...	executing 'bin/common/srtool_common.py --score-new-cves NEW'
 0030: CVE-2019-0008
 ...
* The '--name-filter' can do matches again any of the 'Description','Name','Source','Data' fields
 $ # Update all 'nist' source types
 $ ./bin/srt manage update --run-updates -n nist
 $ # Update all 'cve' data types
 $ ./bin/srt manage update --run-updates -n cve
 $ # Update the specific "NIST 2019" source by its description
 $ ./bin/srt manage update --run-updates -n "NIST 2019"
* You can disable the automatic updater job, and start/stop it manually
 $ export SRT_SKIP_AUTOUPDATE=1
 $ ./bin/srt start
 --- or ---
 $ ./bin/srt start noautoupdate
 
 $ # Manually Start the update server job
 $ ... do work ...
 $ ./bin/srt manage update --cron-start
 $ ... do work ...
 $ ./bin/srt manage update --cron-stop
 $ ./bin/srt stop
=== Data backup and Recovery ===
* There are data sources that perform background backups
** The 'daily' backup save the data to 'backups/backup_$weekday'
** The 'weekly backup save the data to 'backups/backup_$year_$weeknum'
* The backed up data includes:
** The sqlite database
** The downloaded data files (but not the cache files)
** The downloaded attachment files
* To restore the data:
** Stop the SRTool server
** Delete or move the current sqlite file
** Copy the contents of the selected backup directory onto the SRTool base directory
** Start the SRTool server
== SRTool Code Organization ==
* All of the scripts are under the "/bin" directory
* All of the Django files are under the "/lib" directory
* All of the GUI files are under the "/lib/srtgui" directory
* All of the database management files are under the "/lib/orm" directory (object-relationhip-model)
* All of the user management files are under "/lib/user" and "lib/srtmain/templates"
* The GUI starts from "lib/srtmain/urls.py"
== Debugging the SRTool ==
1. Debugging Techniques
The same basic techniques for debugging Toaster also apply to the SRTool. See this link for details Debugging_Toaster.
2. Debugging Log Files
Debugging output:
Type 	Output Location
Start up errors 	All progress and error messages appear in the terminal shell
Django template and processing errors 	./srt_web.log
Normal SRTool exceptions and general execution messages	debug|info](msg)')
Short term debugging and trace messages 	./srt_dbg.log (via '_log(msg)')
3. Environment Variables
There are several environment settings that can be used to help the debug cycle and to present functional values.
To set the productions values, you can run the helper script:
 $ source ./srt_env.sh 
To set the debugging values, you can run the helper script:
 $ source ./srt_env.sh debug
Optional Environment Setting Overrides:
Name 	Purpose 	Default 	Production 	Debugging
SRT_PORT 	Default webport value 	9000 	9000 	9900
SRTDBG_LOG 	Log file for quick debug messages ("_log(...)") 	/tmp/srt_dbg.log	`pwd`/srt_dbg.log 	`pwd`/srt_dbg.log
SRTDBG_MINIMAL_DB 	On start up, only load 10 records from each source, useful for fast schema and functional testing	0 	0 	1
SRTDBG_SKIP_DEFECT_IMPORT	Skip loading the Defect data, useful for fast startup testing 	0 	0 	1
SRTDBG_SKIP_CVE_IMPORT 	Skip loading the CVE data, useful for fast startup testing 	0 	0 	0
SRT_SKIP_AUTOUPDATE 	Skip starting the background database updater job 	0 	0 	1
SRT_SMTP 	Email setup 	<empty> 	Example: smtp.acme.com	"
SRT_USER 	User name for SMTP and Defect server 	<empty> 	Example: srt_user 	"
SRT_PASSWD 	User password for SMTP and Defect server 	<empty> 	Example: srt_rocks! 	"
Notes:
* Holding the user name and password as environment variables is a good way to hide them from external users that may have read access to the SRTool file system
Core Environment Settings
Name 	Purpose 	Value 	Runtime
SRT_BASE_DIR	Root directory for the SRTool system	Directory that contains "bin/srt"	Set to environment by "bin/srt start"
Notes:
* The variable SRT_BASE_DIR is used by the SRTool runtime to provide the root directory for all of the SRTool resources, for example the database, data files, log files, and helper scripts.
* The variable SRT_BASE_DIR is used to define SRT_SQLITE_DEFAULT_DIR
* The variable SRT_BASE_DIR is also required for the Django/SRTool management functions. For such operations (like super user creation and model migrations), it is recommended to use "./bin/srt manage ..." to allow the "srt" tool to automatically and correctly prepare this and any other required environment variables and then invoke the requested management function. 
== Running Pylint on the SRTool Codebase ==
All submissions should first run a pylint test.
Here is an example on how to install and run pylint on an Ubuntu host:
 $ sudo apt install pylint3
 $ pip3 install pylint_django
Here is an example basic pylint command, selecting the "lib" directory:
 $ cd /path/to/srtool/clone
 $ PYTHONPATH=bin/:lib/ pylint3 --load-plugins pylint_django -E lib
Here is an example full feature pylint command, selecting the "bin" directory:
 $ cd /path/to/srtool/clone
 $ PYTHONPATH=./lib:./bin pylint3 --load-plugins pylint_django bin \
 --disable=C,R,unused-variable,unused-wildcard-import,redefined-outer-name,\
 unused-argument,fixme,bare-except,broad-except,redefined-builtin,\
 unnecessary-pass,logging-not-lazy,wildcard-import
The currently allowed exceptions are:
 W0603: Using the global statement (global-statement) 
 W0611: Unused ORM imported from srt_schema (unused-import)
== Quick Development Tools ==
A set of quick development assistance tools are available under "bin/dev_tools". To make them available, run:
 $ cp bin/dev_tools/* .
Here is a summary of their functions:
Name 	Function
./start.sh 	Script to start the SRTool. Edit the webport value for different directories/instances (production, development, ...)
./stop.sh 	Script to stop the SRTool
./restart.sh 	Script to restart the SRTool, useful for asserting run time code changes (models, views, tables, basically any change to a python file)
./recreate.sh 	Script to force a new SRTool from scratch, useful for startup/schema testing (especially when combined with "export SRTDBG_MINIMAL_DB=1")
./tail.sh [line_count]	Script to dump the tail from the many log files, useful in finding exception messages
source ./srt_env.sh 	Script to source the SRTool environment values, useful for seeing and updating the development overrides
./super.sh 	Script to set up a superuser account ("./bin/srt manage createsuperuser")
./lssrt.sh 	List of running SRTool instances, including their PID and webport
./showdb.sh 	Script to start a SQL GUI (e.g. 'sqlitebrowser'), useful for quick database content checking and minor adjustments
./masterapp.sh appname	Script to reset the master app (for example './masterapp.sh acme' and reset with './masterapp.sh yp')
There is also a basic sanity script to test the host settings, check that the database passed the startup and was initialized:
 $ ./bin/common/srtool_sanity_test.py -i
 Uname = #35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018 x86_64
 Django = (2, 2, 0, 'alpha', 0)
 Python3 = Python 3.5.2
 Sqlite3 = 3.11.0 2016-02-15
 Server PID = 8654
 Server Port = [8654](python3 /opt/dreyna_srt/srt_django20/lib/manage.py runserver --noreload 0.0.0.0:9993)
 CVEs = 64
 Users = 15
 Data sources= 25
 $
== Extending and customizing common files, template files ==
A 'srtool_patcher" script allows managing custom changes for your organization on top of the SRTool code base. There are two main workflows.
=== Patcher Workflows ===
* Use case #1: Extending common/template code with customized sections, with support for merging and upstreaming changes in the common/shared sections of the parent upstream files.
The initial use case is a shared Jira integration file that partners can extend to their particular installation. A working example is provided in the ACME directory:
 upstream: "bin/common/srtool_jira_template.py"
 custom : "bin/acme/srtool_jira_acme.py"
* Use case #2: Maintaining custom patches on top of regular files, with the ability to merge upstream changes and also to push your own contributions to the upstream sections.
A provided example is a customization of the "bin/srt" script for the ACME organization. See:
 bin/srt
 bin/acme/patcher/inplace/bin/srt
=== Patcher Model ===
* The custom sections are blocked off with comment tags (e.g. ACME). This is the model that Wind River has been using for years to maintain their patch set on top of Bitbake Toaster.
* There are two advantages of this model over (for example) the traditional pristine source plus patch tree model:
** By examining the code and the extracted patch, it is explicitly clear what are the intended changes
** Any changes that are not included in a custom section easy to spot, and are either (a) mistakes, or (b) general changes intended for the upstream code, for which it is now trivially easy to isolate and push using the below tools.
* Here are the custom section markers for the respective file types:
Python, shell script:
 ### ACME_EXTENSION_BEGIN ###
 ...
 ### ACME_EXTENSION_END ###
Javascript:
 /* ACME_EXTENSION_BEGIN */
 ...
 /* ACME_EXTENSION_END */
HTML:
 <!-- ACME_EXTENSION_BEGIN -->
 ...
 <!-- ACME_EXTENSION_END -->
JSON: note that JSON files do not support comment delimiters. To accommodate this, tagged name-value pairs are insert, with enumeration ("_%d") appended to insure each such entry is unique. Care must be taken with managing the trailing ',' since JSON is strict about this syntax.
 "ACME_EXTENSION_BEGIN_01" : "",
 ...
 "ACME_EXTENSION_END_01" : "",
* The Patcher also supports excluding sections of the common code.
** This feature should rarely necessary or used, given that almost all customizations can be additive (inserting additional functionality or overwriting variables), and all care should be taken to preserve the upstream code flow (or else the upstream code should be made more generic).
** The primary use case is if the common code is executing a function that causes undesired side effects for the customization. A second use case is to disable/replace a section of a template file instead of maintaining a duplicate in the main app directory.
** The tags are in this format. The content in the excluded section are commented, so that when it is cleaned those lines can be restored to the original.
 ### ACME_EXTENSION_EXCLUDE_BEGIN ###
 #<commented original code>
 ### ACME_EXTENSION_EXCLUDE_END ###
=== Patcher Management JSON File ===
* The script supports JSON files for pre-defined mappings. See:
 bin/acme/patcher.json
=== Patcher Usage overview ===
 Merge shared upstream code into a custom Jira script:
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json --merge-original
 Merge edits in script's common code areas back to upstream:
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json --merge-custom
 See if the shared common code sections have diverged from upstream:
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json --diff-original
 Extract the custom code sections, for examining and preserving custom changes:
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json --extract-custom-patch
 Stash then clean the custom code sections from a shared upstream file:
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json --clean-inplace
 Assert (merge) the custom code sections onto a shared upstream file:
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json --patch-inplace
=== Patcher Usage Examples ===
Example #1: CUSTOMIZING FROM A TEMPLATE FILE (e.g. Jira)
1. Make a change in the shared section of the custom file
 $ grep ACME_EXTENSION_BEGIN bin/acme/srtool_jira_acme.py | wc
 4 12 120
 $
 $ echo "#foo" >> bin/acme/srtool_jira_acme.py
2. Observe the difference from upstream, with custom sections skipped
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json --diff-original
 == DIFF from 'bin/common/srtool_jira_template.py' to clean version of custom file ===
 --- bin/common/srtool_jira_template.py	2019-01-17 19:01:51.375018816 -0800
 +++ bin/acme/patcher/srtool_jira_acme.py.clean	2019-01-17 19:08:32.779015082 -0800
 @@ -913,3 +913,4 @@
 
 srtool_basepath = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(sys.argv[0]))))
 main(sys.argv[1:])
 +#foo
 $
3. Export the change to upstream, observe that the custom sections were skipped
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json --merge-custom
 == Copying clean version of custom file to 'bin/common/srtool_jira_template.py' ===
 cp -f bin/acme/patcher/srtool_jira_acme.py.clean bin/common/srtool_jira_template.p
 $
 $ git diff bin/common/srtool_jira_template.py
 diff --git a/bin/common/srtool_jira_template.py b/bin/common/srtool_jira_template.py
 index 5f29355..57ca19b 100644
 --- a/bin/common/srtool_jira_template.py
 +++ b/bin/common/srtool_jira_template.py
 @@ -913,3 +913,4 @@ if __name__ == '__main__':
 
 srtool_basepath = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(sys.argv[0]))))
 main(sys.argv[1:])
 +#foo
 $
4. Restore upstream, push it back into the custom file, observe custom sections preserved
 $ git checkout bin/common/srtool_jira_template.py
 $
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json --merge-original
 Custom File: bin/acme/srtool_jira_acme.py
 Clean File: bin/acme/patcher/srtool_jira_acme.py.clean
 Patch File: bin/acme/patcher/srtool_jira_acme.py.patch
 * Preserving custom file as 'bin/acme/patcher/srtool_jira_acme.py.saved'
 cp bin/acme/srtool_jira_acme.py bin/acme/patcher/srtool_jira_acme.py.saved
 * Merging original file 'bin/common/srtool_jira_template.py' into custom file
 cp bin/common/srtool_jira_template.py bin/acme/srtool_jira_acme.py
 patch bin/acme/srtool_jira_acme.py bin/acme/patcher/srtool_jira_acme.py.patch
 patching file bin/acme/srtool_jira_acme.py
 $
 $ grep ACME_EXTENSION_BEGIN bin/acme/srtool_jira_acme.py | wc
 4 12 120
 $
5. If there is a problem merging the upstream code, for example a change is too close to a custom section, you will see something like:
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json --merge-original
 ...
 patch bin/acme/srtool_jira_acme.py bin/acme/patcher/srtool_jira_acme.py.patch
 patching file bin/acme/srtool_jira_acme.py
 Hunk #1 FAILED at 32.
 Hunk #2 succeeded at 78 (offset 1 line).
 Hunk #3 succeeded at 102 (offset 1 line).
 Hunk #4 succeeded at 761 (offset 1 line).
 1 out of 4 hunks FAILED -- saving rejects to file bin/acme/srtool_jira_acme.py.rej
 * ERROR: Merge failed, restoring previous custom file
 cp bin/acme/patcher/srtool_jira_acme.py.saved bin/acme/srtool_jira_acme.py
 $
The custom file is restored to its pre-patch state, and a reject file is available for review. At this point you can: for example
* Manually re-run the patch command shown in the output to get the working hunks applied
* Examine the reject file and manually apply the missing patch hunk.
Example #2: CUSTOMIZING INLINE FILES (e.g. "bin/srt")
1. Enable the example in-line custom patch
Edit the "bin/acme/patcher.json" file, and remove the "DISABLE" tag from the 'bin/srt' section.
2. Assert (merge) the customized version of "bin/srt"
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json --patch-inplace
Note that for this first time there is no managed patch file, so the customized version will simple be copied.
3. Observe that the custom sections are in place, one insertion and one exclude/replace:
 $ git diff bin/srt
 diff --git a/bin/srt b/bin/srt
 index 41753b7..af8a1fb 100755
 --- a/bin/srt
 +++ b/bin/srt
 @@ -18,6 +18,11 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see http://www.gnu.org/licenses/.
 
 +### ACME_EXTENSION_BEGIN ###
 +echo "Welcome to the SRTool, ACME Edition!"
 +echo ""
 +### ACME_EXTENSION_END ###
 +
 HELP="
 Usage: source srt start|stop [webport=<address:port>]
 Optional arguments:
 @@ -261,7 +266,17 @@ elif [ "$CMD" = "" ]; then
 exit 1
 fi
 
 -echo "The system will $CMD."
 +### ACME_EXTENSION_EXCLUDE_BEGIN ###
 +#echo "The system will $CMD."
 +#
 +### ACME_EXTENSION_EXCLUDE_END ###
 +### ACME_EXTENSION_BEGIN ###
 +#
 +# NOTE: Exclusions should be used only when necessary
 +# NOTE: The excluded lines are commented so that they are inactive but restorable
 +#
 +echo "The ACME SRTool system will now $CMD!"
 +### ACME_EXTENSION_END ###
 
 # Execute the commands
 case $CMD in
 $
4. Stash the custom version, restore the clean version
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json --clean-inplace
5. Verify that the custom section gone and the code is clean
 $ git diff bin/srt
 $
6. Make a change to the public section in the customized file
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json -I
 $ echo "#my public patch" >> bin/srt
7. Clean the customized file, observe that the public change remains
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json -i
 $ git diff
 [srtool]$ git diff bin/srt
 diff --git a/bin/srt b/bin/srt
 index 41753b7..4f9419c 100755
 --- a/bin/srt
 +++ b/bin/srt
 @@ -295,3 +295,4 @@ case $CMD in
 
 esac
 
 +#my public patch
 $
8. Make another change to the pubic section (as if an update was pulled), re-apply customization, observe that it still applies cleanly
 $ echo "#my second public patch" >> bin/srt
 $ ./bin/common/srtool_patcher.py -J bin/acme/patcher.json -I
 $
 $ grep ACME_EXT bin/srt | wc
 6 18 184
 $ tail bin/srt
 ...
 esac
 
 #my public patch
 #my second public patch
 $
9. Receiving from upstream any changes to common areas. If there is an upstream change to "bin/srt", you would:
* Run '--clean-inplace' to stash the customizations
* Run 'git pull' the get the upsteam changes
* Run '--patch-inplace' to merge the customizations
10. Pushing to upstream any changes to common areas. If you would like to commit a change to the common sections of "bin/srt", you would:
* Run '--clean-inplace' to stash the customizations
* Run 'git add/commit' the updated file
* Run '--patch-inplace' to re-merge the customizations
== Submitting Patches to the SRTool via GitLab ==
GitLab's hosted service has automatic repository mirroring, and it is very easy to fetch the lastest SRTool updates and share patch branches with the SRTool developers.
1) Open your gitlab.com account. You can create one if you do not have one already (they're free!).
2) Do the following:
* Select "New Project -> Import"
* Select "Project -> Repo by URL"
* Enter the git://git.yoctoproject.org/srtool URL
* Turn on Mirror Repository
This ensures that the master branch in the project is in sync with master at "git://git.yoctoproject.org/srtool", and then create your own branches for work.
3) Start your patch branch off of master with this suggested branch naming style:
 username/srtool/FeatureOrBug
4) Do Work
5) Test the changes. 
6) Rebase on master. It has probably changed while you were working (unless you are really really fast!)
7) Commit your change 
NOTE: The format of the commit message should be like this
 srtool: <module> <short one line summary>
 long(er) description
 [YOCTO #0000]
 Signed-off-by: First Last <name@domain.com>
Where YOCTO #0000 is the related bug number if there is one. Signed off by with your "git commit -s" credentials.
8) Share your branch's link with the SRTool maintainers, e.g.:
* david.reyna@windriver.com
* yocto-security@yoctoproject.org
Note: to use "yocto-security" you will first need to subscribe here: "https://lists.yoctoproject.org/listinfo/yocto-security"
9) NOTE: when the patch has been accepted upstream, you can refresh your master branch and delete your patch branch.
== Submitting Patches to the SRTool via poky-contrib ==
To contribute to the SRTool with this method you will also need authorization to write to the upstream yocto project repository "poky-contrib". Contact a member of the SRTool team for details.
1) Download the master branch of the SRTool
 git pull ssh://git@push.yoctoproject.org/srtool && cd srtool 
2) Add poky-contrib to the local repository you set up above
 git remote add poky-contrib ssh://git@git.yoctoproject.org/poky-contrib 
3) Fetch the poky-contrib branches
 git fetch --all 
4) Start your feature branch off of master, name style of branch is convention, but suggested. 
 git checkout -b username/srtool/FeatureOrBug origin/master 
5) Do Work 
6) Test the changes. 
7) Rebase on master. It has probably changed while you were working (unless you are really really fast!)
 git rebase origin/master 
8) Commit your change 
NOTE: The format of the commit message should be like this
 srtool: <module> <short one line summary>
 long(er) description
 [YOCTO #0000]
 Signed-off-by: First Last <name@domain.com>
Where YOCTO #0000 is the related bug number if there is one. Signed off by with your git commit -s credentials.
9) Push your feature branch to poky-contrib
 git push -u poky-contrib username/srtool/FeatureOrBug:username/srtool/FeatureOrBug
10) Send to the srtool-mailing list:
We accept patches on the Security mailing list ( yocto-security@yoctoproject.org ) by "git send-email".
e.g.
 $ git send-email HEAD^ 
11) NOTE: when the patch has been accepted upstream, you can clean up your poy-contrib branch with:
 git push -u poky-contrib :username/srtool/FeatureOrBug