Handling the workow of pkgsrc Security TeamAn introduction to nmh (new MH message system), a look to
the workow of pkgsrc Security Team and how to (possibly)
automate the automatable stus!
Leonardo Taccari<leot@NetBSD.org>
The NetBSD Foundation

pkgsrcCon 2018, July 7th 2018, Berlin, Germany

1 / 50Outline
An introduction to nmh (New Message Handler)
MH Mail System and nmh new MH message system
MH mailbox format
Practical look of a subset of nmh commands
pkgsrc Security Team tasks and workow
About pkgsrc Security Team
pkgsrc-security@ rotation list tasks
pkgsrc-security RT queue
pkg_admin and pkg-vulnerabilities
Automating the automatable stus of pkgsrc-security
Programmatic ways to access RT
Automating ticket handling via the MUA

2 / 50An introduction to nmh (New Message Handler)

3 / 50MH Mail System

I Initially developed in 1978 at RANDI MH two main design decisions:

I MH commands - the primitive operations on a message - areUNIX shell commands

I Each MH message is a normal UNIX le

I Ease (ab)using the handling of emails via Unix shell scripting

4 / 50nmh (new MH message system)

I Based on MH version 6.8.3
I Available under modified-bsd LICENSEI Very nice and friendly community!
I Intended to be a compatible drop-in replacement for MH
I Suite of simple single-purpose programs to send, receive, save,
retrieve and manipulate email messages
I Available in pkgsrc as mail/nmh

5 / 50nmh (new MH message system): MH mailbox format
I Every email message is just a le (e.g. 123) in a folder 1 (e.g.
pkgsrc-users)
I Most nmh commands operates on a folder (e.g.
+pkgsrc-users) and a range of messages (e.g. 123)
I Current folder can be omitted and is stored in the user's
context (`mhpath +`/`mhparam context`) 2
I Current message can be omitted and is stored in folder
mh-sequences le (`mhpath`/`mhparam mh-sequences`) 3
I Per-folder mh-sequences is also used to:
I Special Unseen-Sequence to mark unread messages
I Mark a range of messages with a user dened sequence

1

A directory in the le system
E.g. In /Mail/.context: Current-Folder: pkgsrc-users
3
E.g. In /Mail/pkgsrc-users/.mh_sequences: cur: 123
2

6 / 50Practical look of a subset of nmh commands
scan produce a summary listing of nmh messages
show display nmh messages
prev show the previous nmh message
next show the next nmh message
folder set current nmh folder
new report on nmh folders with new messages
fprev set current nmh folder to previous folder with newmessages
fnext set current nmh folder to next folder with newmessages
unseen scan any new messages in all nmh folders
pick search nmh messages
mark manipulate nmh message sequencescomp compose an nmh message
repl reply to an nmh message
7 / 50Subset of nmh commands: scan
scan produces a one-line-per-message listing of thespecied folder or messages. Each scan line contains themessage number (name), the date, the From: eld, theSubject eld, and, if room allows, some of the body ofthe message.

4

scan [-help] [-version] [+folder] [msgs] [-clear | -noclear] [-form
formatfile] [-format string] [-header | -noheader] [-widthcolumns] [-reverse | -noreverse] [-file filename]% scan +pkgsrc-changes last:5
N 68024 Jonathan Perkin Thu Jun 28 15:16 CVS commit: pkgsrc/textproc/ruby-nokogiri
N 68025 Takahiro Kambe
Thu Jun 28 15:33 CVS commit: pkgsrc/security/py-acme
N 68026 Takahiro Kambe
Thu Jun 28 15:34 CVS commit: pkgsrc/doc
N 68027 Jonathan Perkin Thu Jun 28 15:45 CVS commit: pkgsrc/www/libwww>N 68028 Greg Troxel
Thu Jun 28 19:00 CVS commit: pkgsrc/ham/rtl-sdr

4

From scan(1)8 / 50Subset of nmh commands: show
show lists each of the specied messages to the standard output (typically, the terminal).

5

show [-help] [-version] [+folder] [msgs] [-draft] [-showproc program][-showmimeproc program] [-header | -noheader] [-checkmime |-nocheckmime] [-concat | -noconcat] [switches for showproc orshowmimeproc]% show +pkgsrc-users 6391
Date:
Tue, 19 Jun 2018 16:20:24 +0200
To:
pkgsrc-users%netbsd.org@localhost
From:
Thomas Merkel <tm%netbsd.org@localhost>
Subject: pkgsrcCon 2018 in Berlin, 6.-8. July
Dear pkgsrc users and contributors,
this is a friendly reminder about the pkgsrcCon which takes place
in Berlin this year. If you like to present a talk, please send thetitle, slot duration and brief description for the website to
pkgsrcCon2018@NetBSD.org.
[...]

5

From show(1)9 / 50Subset of nmh commands: prev

prev performs a show on the previous message in thespecied (or current) folder. [...] This command is almost
exactly equivalent to show prev.

6

prev [-help] [-version] [+folder] [-showproc program] [-showmimeproc
program] [-header | -noheader] [-checkmime | -nocheckmime][switches for showproc or showmimeproc]

6

From prev(1)10 / 50

Subset of nmh commands: next

next performs a show on the next message in the specied (or current) folder.

[...]This command is almost

exactly equivalent to show next.

7

next [-help] [-version] [+folder] [-showproc program] [-showmimeproc
program] [-header | -noheader] [-checkmime | -nocheckmime][switches for showproc or showmimeproc]

7

From next(1)11 / 50

Subset of nmh commands: folder
When folder is given the -print switch (the default), it
lists: the current folder, the number of messages in it andtheir range (low-high), the folder's current message, andan indication of extra les, if any.

8folder [-help] [-version] [+folder] [msg] [-all | -noall] [-create |-nocreate] [-fast | -nofast] [-header | -noheader] [-recurse |-norecurse] [-total | -nototal] [-list | -nolist] [-push | -pop][-pack | -nopack] [-print] [-verbose | -noverbose]
folders is equivalent to folder -all% folderpkgsrc-changes+ has 68011 messages(1-68030); cur=68028.

% folder -fast
pkgsrc-changes% folder -all
FOLDER[...]
pkgsrc-bugs
pkgsrc-bulk
pkgsrc-changes+[...]# MESSAGES RANGE
has 9378 messages (has 5204 messages (has 68011 messages (; CUR(OTHERS)

1- 9379); cur= 9379.
1- 5204); cur= 5204.
1-68031); cur=68030.

TOTAL = 363611 messages in 144 folders.

8

From folder(1)12 / 50

Subset of nmh commands: newnew, in its default mode, produces a one-line-perfolder listing of all folders which contain messages in thespecied sequences, or in the sequence(s) listed in the prole entry Unseen-Sequence.

Each line consists of thefolder name, the total number of messages in the specied sequences, and a list of messages derived from the.mh_sequence le.9

new [-help] [-version] [sequences] [-mode mode] [-folders foldersfile]
fnext is equivalent to new -mode fnext
fprev is equivalent to new -mode fprev
unseen is equivalent to new -mode unseen% newnetbsd-source-changes
pkgsrc-changes
pkgsrc-wip-changes
total

9

From new(1)1. 40500
2. 68029-68030
1. 10349
4.

13 / 50

Subset of nmh commands: fprev

In fnext and fprev modes, new changes to the next orprevious matching folder, respectively.

10

new [-help] [-version] [sequences] [-mode mode] [-folders foldersfile]
fnext is equivalent to new -mode fnext
fprev is equivalent to new -mode fprev
unseen is equivalent to new -mode unseen% fprev
netbsd-source-changes

40500% fprev
pkgsrc-changes 68029-68030

10

From fprev(1)14 / 50

Subset of nmh commands: fnext

In fnext and fprev modes, new changes to the next orprevious matching folder, respectively.

11

new [-help] [-version] [sequences] [-mode mode] [-folders foldersfile]
fnext is equivalent to new -mode fnext
fprev is equivalent to new -mode fprev
unseen is equivalent to new -mode unseen% fnext
pkgsrc-changes 68029-68030% fnext
pkgsrc-wip-changes

11

10349

From fnext(1)15 / 50

Subset of nmh commands: unseenIn unseen mode, new executes scan sequences for eachmatching folder.

12

new [-help] [-version] [sequences] [-mode mode] [-folders foldersfile]
fnext is equivalent to new -mode fnext
fprev is equivalent to new -mode fprev
unseen is equivalent to new -mode unseen% unseen
1 unseen messages in netbsd-source-changes
N 40500 Kamil Rytarowski Fri Jun 29 11:33 CVS commit: src
2 unseen messages in pkgsrc-changes
N 68029 Jonathan Perkin Fri Jun 29 11:27 CVS commit: pkgsrc/pkgtools/pkgin
N 68030 Jonathan Perkin Fri Jun 29 11:28 CVS commit: pkgsrc/doc
1 unseen messages in pkgsrc-wip-changes
N 10349 Havard Eidnes
Tue Jun 26 22:42 Do away with use of pip in setup.py.

12

From unseen(1)16 / 50

Subset of nmh commands: pick
pick searches within a folder for messages with thespecied contents, and then identies those messages. Twotypes of search primitives are available: pattern matchingand date constraint operations.

13

pick [-help] [-version] [+folder] [msgs] [-reverse ...] [-and ...] [-or...] [-not ...] [-lbrace ... -rbrace] [--component pattern] [-ccpattern] [-date pattern] [-from pattern] [-search pattern][-subject pattern] [-to pattern] [-after date] [-before date][-datefield field] [-sequence name ...] [-nosequence] [-public |-nopublic] [-zero | -nozero] [-list | -nolist] [-debug]% pick -search 'CVE-' +pkgsrc-changes last:150
67887
67990
67992
68015
68019
% scan `pick -search 'CVE-' +pkgsrc-changes last:150`
N 67887 Thomas Klausner Sun Jun 24 10:16 CVS commit: pkgsrc/graphics/GraphicsMagick
N 67990 Maya Rashish
Tue Jun 26 21:49 CVS commit: pkgsrc/www/firefox52
N 67992 Maya Rashish
Tue Jun 26 23:29 CVS commit: pkgsrc/www/seamonkey
N 68015 Ryo ONODERAThu Jun 28 13:52 CVS commit: pkgsrc/www/firefox
N 68019 Ryo ONODERAThu Jun 28 14:04 CVS commit: pkgsrc/www/firefox60

13

From pick(1)17 / 50

Subset of nmh commands: mark
The mark command manipulates message sequencesby adding or deleting message numbers from folder-specicmessage sequences, or by listing those sequences and messages.

14mark [-help] [-version] [+folder] [msgs] [-sequence name ...] [-add |-delete] [-list] [-public | -nopublic] [-zero | -nozero]% scan `pick -from maya -and -search 'CVE-' +pkgsrc-changes last:150`
N 67990 Maya Rashish
Tue Jun 26 21:49 CVS commit: pkgsrc/www/firefox52
N 67992 Maya Rashish
Tue Jun 26 23:29 CVS commit: pkgsrc/www/seamonkey% mark -sequence needspullup `pick -from maya -and -search 'CVE-' +pkgsrc-changes last:150`% scan needspullup
N 67990 Maya Rashish
N 67992 Maya Rashish

14

From mark(1)Tue Jun 26 21:49 CVS commit: pkgsrc/www/firefox52
Tue Jun 26 23:29 CVS commit: pkgsrc/www/seamonkey

18 / 50

pkgsrc Security Team tasks and workow

19 / 50

Mission

The mission of pkgsrc Security Team is:I ensure that packages in pkgsrc are safe

I be sure pkgsrc users are aware of the known vulnerabilities

20 / 50

Who?
Current members of pkgsrc-security@ are:
I Alistair G. Crooks (<agc>)

I Daniel Horecki (<morr>)
I Thomas Klausner (<wiz>)
I Tobias Nygren (<tnn>)
I Ryo ONODERA (<ryoon>)
I Fredrik Pettai (<pettai>)I Jörg Sonnenberger (<joerg>)
I Leonardo Taccari (<leot>)
I Tim Zingelman (<tez>)

21 / 50

pkgsrc-security@ rotation list

Daniel Horecki <morr>, Tobias Nygren <tnn>, Ryo ONODERA<ryoon> and Leonardo Taccari <leot> are in the
pkgsrc-security@ rotation list.
I Each person is 'on' from Tuesday till Monday (once every 4
weeks)I Ensure that all tickets get handled ASAP
I reject the ones not aecting pkgsrc
I add entries to pkg-vulnerabilities
I inform the MAINTAINER (if any)

22 / 50

RT tickets and the pkgsrc-security queue

I Each vulnerability is handled via RT (Request Tracker)
ticketing system
I Public security feeds/MLs (e.g. NIST for CVEs) create newtickets on RT
I Every new ticket and/or RT comments are also received by
pkgsrc-security@ (as emails)

23 / 50

RT ticket statuses used by pkgsrc-security@

new new (usually unhandled) ticket
rejected duplicate issues and ones that do not apply to pkgsrc
resolved ticket that impacts pkgsrc and entry added to
pkg-vulnerabilities

24 / 50

Handling new tickets

I Is the ticket a duplicate?

I Mark its status as `rejected'
I Add a `duplicate' comment

I Does the ticket not apply to pkgsrc?

I Mark its status as `rejected' and
I Add a `No impact on pkgsrc' comment.

I Does the ticket apply to pkgsrc?
I Add an entry to pkg-vulnerabilities
I Mark its status as `resolved'
I Contact MAINTAINER (if any)

25 / 50

RT tickets (web interface)

Screenshot of new RT tickets for the pkgsrc-security queue

26 / 50

RT ticket #127438  CVE-2017-16068

Screenshot of RT ticket #127438, CVE-2017-16068

27 / 50

Updating multiple tickets on RT

Screenshot of updating multiple tickets on RT

28 / 50

pkg_admin(1) and vulnerabilities
pkg_admin(1) has several commands to inform users about
vulnerable packages installed on system.
audit print a list of all installed packages that containvulnerabilities On NetBSD, if thecheck_pkg_vulnerabilities option is set (it is bydefault 15 ) the daily(5) cron job will list all
vulnerability packages installed.
fetch-pkg-vulnerabilities fetch a new pkg-vulnerabilities
le 16 . This is disabled by default and can becongured via daily.conf(5) by adding
fetch_pkg_vulnerabilities=YES in/etc/daily.conf.
15
16

Please give a look to security.conf(5) if you are curious!
By default it is downloaded from ftp.NetBSD.org

29 / 50

pkg_admin audit in action% pkg_admin audit
Package pcre-8.42 has a denial-of-service vulnerability, see
https://nvd.nist.gov/vuln/detail/CVE-2017-11164
Package libxslt-1.1.32 has a insufficiently-random-numbers vulnerability, see
https://nvd.nist.gov/vuln/detail/CVE-2015-9019
Package jpeg-9c has a denial-of-service vulnerability, see
https://nvd.nist.gov/vuln/detail/CVE-2018-11813[...]

30 / 50

pkg-vulnerabilities
pkg-vulnerabilities is a TSV 17 that contains 3-uples:
package PKGNAME patterns 18
type of exploit (e.g. denial-of-service, buffer-overflow,
multiple-vulnerabilities, eol, . . . )URL URL that contains details about the vulnerability(often to nvd.nist.gov for CVEs)

17
Actually [ \t]SV!, i.e. awk '! /#/ { print $1, $2, $3 }' will
DTRT!
18
In case of doubt you can use pkg_admin pmatch pattern pkg that
returns true if `pkg' matches `pattern', e.g.
pkg_admin pmatch 'foo<1.0' 'foo-1.0' will return false.

31 / 50

Some numbers
Tickets handled in 2016
StatusTickets
rejected 10429
resolved 1367
stalled
0
Total
11796

Tickets handled in 2017
StatusTickets
rejected 23511
resolved 2847
stalled
2
Total
26360

I Number of vulnerable packages in pkgsrc head: 591 19
I Number of vulnerable packages in pkgsrc stable [2018Q1]:
624 2019
20

As of 2018-07-03 4:00 UTC
As of 2018-07-03 4:00 UTC

32 / 50

Automating the automatable stus of pkgsrc-security

33 / 50Request Tracker (RT) REST Interface

I Request Tracker (RT) provides a REST interface that permit
to programmatically access RT databases.
I Both devel/rt3 and devel/rt4 provides a rt Perl script
I Other package/modules exists for several programming
languages

34 / 50

rt: command-line interface to RT

I Perl script that can be used both non-interactively (directly
passing action when invoking it) or interactively (if no action
are passed, i.e. just by invoking it as `rt')
I Actions most commonly used:
list show a list of tickets
show show information about a ticket (description,
updates, comments)
edit modify elds of a ticket
comment add a comment to a ticket
help print help message

35 / 50

rt actions: list
Displays a list of objects matching the specied conditions. ("ls", "list", and "search" are synonyms.)

21

rt <ls|list|search> [options] "query string"
To show all ticket in the `new' status:% rt ls -s Status=new
127145: CVE-2017-17688 (airmail, emclient, ...)
127146: CVE-2017-17689 (airmail, emclient, ...)
127451: CVE-2018-0499 (xapian)127458: CVE-2018-10874 (ansible)
127470: CVE-2018-11489 (giflib, sam2p)
127472: CVE-2018-11490 (giflib, sam2p)
127504: CVE-2018-13033 (binutils)
127511: CVE-2018-13049 (php-glpi)
127517: CVE-2018-13066 (ming)
127537: CVE-2017-2615 (qemu)
127539: CVE-2018-10855 (ansible2)
127668: CVE-2018-13100 (linux)
127672: CVE-2018-13112 (tcpreplay)
127710: [SECURITY] [DSA 4238-1] exiv2 security update127735: CVE-2018-3750 (npm)21

From rt help list

36 / 50

rt actions: show
Displays details of the specied objects.

22

rt show [options] <object-ids>
To show information about ticket #127668:% rt show 127668
Date: Tue Jul 03 20:19:32 2018
From: rss@nvd.nist.gov
X-Queue: pkgsrc-security
Subject: [rt #ticket/127668] CVE-2018-13100 (linux)===> Ticket created by rss@nvd.nist.gov on Tue Jul 3 22:19:33 2018
An issue was discovered in fs/f2fs/super.c in the Linux kernel through 4.17.3,
which does not properly validate secs_per_zone in a corrupted f2fs image, as
demonstrated by a divide-by-zero error.
Home: [http://nvd.nist.gov/]
Link: [ https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2018-13100 ]
544638: untitled (300b)===> Outgoing email recorded by RT_System on Tue Jul 3 22:19:33 2018544640: untitled (606b)===> CVE IDs CVE-2018-13100 added by leot on Wed Jul 4 11:21:08 2018===> Subject changed from 'CVE-2018-13100' to 'CVE-2018-13100 (linux)'
by leot on Wed Jul 4 12:05:33 201822

From rt help show

37 / 50

rt actions: edit

Edits information corresponding to the specied objects.

23

rt edit [options] <object-ids> set field=value [field=value] ...
add field=value [field=value] ...
del field=value [field=value] ...

Ticket #127668 (CVE-2018-13100) does not aect pkgsrc so we
can close it:% rt edit 127668 set status=rejected# Ticket 127668 updated.

23

From rt help edit

38 / 50

rt actions: comment

Adds a comment (or correspondence) to the speciedticket (the only dierence being that comments aren't sent
to the requestors.)

24

rt <comment|correspond> [options] <ticket-id>#127668 had no impact on pkgsrc, so let's add a comment aboutthat:% rt comment -m 'No impact on pkgsrc' 127668# Message recorded

24

From rt help comment

39 / 50

Automating tickets handling from the MUA (or, putting
everything together!)I All (new) RT tickets and comments ends up in an MH folder
I The CVE ones from NIST have a CVE-[0-9]+-[0-9]+
pattern in the Subject:, we can automatically ll the
CVE IDs eld to RT
I All the ones that have no impact on pkgsrc can be marked
with a special sequence (e.g. `marked') and then automaticallymarked as `rejected' with a `No impact on pkgsrc' comment
I Usually when receiving CVE tickets no information about the
package is present in the Subject:, they can be marked witha PKGBASE sequence (e.g. `qemu') and then update thesubject of the ticket accordingly to ease further processing
when lling respective pkg-vulnerabilities entries
40 / 50

Automating tickets handling from the MUA (or, putting
everything together!)I Duplicate CVE tickets that are already in
pkg-vulnerabilities can be automatically rejected byparsing rt ls Status=new output and URL eld of
pkg-vulnerabilities matching the CVE-[0-9]+-[0-9]+
patterns
I Duplicate tickets in the pkgsrc-security queue can be easily
rejected similarly
I After marking CVE tickets as described an entry for
pkg-vulnerabilities can be populated with a template,e.g.:
PKGBASE-[0-9]+ TODO https://nvd.nist.gov/vuln/detail/CVE-<id>41 / 50

Filling CVE IDs in the ticket
Instead of doing that in shell scripting and rt, it is easier to use
wip/py-rt:
import rt
tracker = rt.Rt(RT_API_URL, basic_auth=(username, password))
tracker.login()
for ticket in tracker.search(Queue='pkgsrc-security', Status='new',
Subject__like='CVE', Format='s'):cves = re.findall('CVE-[0-9]+-[0-9]+', ticket['Subject'])
if cves:fields = { 'CF_CVE IDs': ' '.join(cves) }tracker.edit_ticket(ticket['id'].replace('ticket/', ''), **fields)

42 / 50

Closing marked tickets% scan +pkgsrc-security-rt marked
N 81635 National Vulnerab Wed Jul 04 20:18 [NetBSD.org #127747] CVE-2018-13144[...]>N 81641 National Vulnerab Wed Jul 04 20:18 [NetBSD.org #127753] CVE-2018-13145% scan -format '%{rt-ticket}' +pkgsrc-security-rt marked
NetBSD.org #127747[...]
NetBSD.org #127753% scan -format '%{rt-ticket}' +pkgsrc-security-rt marked | cut -d '#' -f 2
127747[...]127753% scan -format '%{rt-ticket}' +pkgsrc-security-rt marked |cut -d '#' -f 2 | xargs rt edit set status=rejected# Ticket 127747 updated.
[...]# Ticket 127753 updated.
% scan -format '%{rt-ticket}' +pkgsrc-security-rt marked |cut -d '#' -f 2 | xargs -n 1 rt comment -m 'No impact on pkgsrc'# Message recorded[...]# Message recorded

43 / 50

Report all `new' CVE duplicate tickets
The format of rt ls -s is 25 :<id>: <subject>
rt ls -s |
awk \
'
{
# Get rid of `:' in the ticket id
sub(/:$/, "", $1)}$2 ~ /CVE-[0-9]+-[0-9]+/ {id = $1
cve = $2
if (cves[cve])print idelse
cves[cve] = id}'
25

E.g.: 127750: CVE-2018-13139

44 / 50

Report all CVE tickets that are already in

pkg-vulnerabilities(cat $pkgvulnerabilities ; rt ls -s ) | awk \'# Slurp pkg-vulnerabilities CVE entries in vuln[]$3 ~ /CVE-[0-9]+-[0-9]+/ {pkgname = $1
type = $2
match($3, /CVE-[0-9]+-[0-9]+/)cve = substr($3, RSTART, RLENGTH)
entry[cve] = $0}

next

# Parse rt ls -s output$2 ~ /CVE-[0-9]+-[0-9]+/ {id = $1
match($2, /CVE-[0-9]+-[0-9]+/)cve = substr($2, RSTART, RLENGTH)}'

if (entry[cve]) {print id " " entry[cve]}45 / 50

Conclusion

I Reading/skimming tickets needs humans. . .
I . . . but at least when handling a lot of tickets scripting canease a lot!

46 / 50

Thanks

I pkgsrc Security Team and everyone who help keeping up todate pkg-vulnerabilities and requests pullups!
I pkgsrc Releng for their work!

47 / 50References I

Robert H. Anderson, Norman Shapiro, Tora K. Bikson, Phyllis
Kantar.
The Design of the MH Mail System.
https://www.rand.org/pubs/notes/N3017.html.
Request Tracker Wiki.
REST - Request Tracker Wiki.
https://rt-wiki.bestpractical.com/wiki/REST.
Alistair Crooks, Hubert Feyrer, The pkgsrc Developers.
The pkgsrc guide.
https://www.NetBSD.org/docs/pkgsrc/.

48 / 50References II

Thomas A. Limoncelli.
Manual work is a bug.
Queue, 16(1):20:1320:29, February 2018.
ISSN 1542-7730.
doi: 10.1145/3194653.3197520.
URL http://doi.acm.org/10.1145/3194653.3197520.

49 / 50

Questions?

50 / 50