Build & Release
How It’s Done
05.31.24

Wayne Mery — Release Manager, Daniel Darnell — Build Engineer

Table ofcontents

1.

The Pre-Release Process

2.

The Release Process

3.

The Post-Release Process4.

Summarized Timeline

5.

Links and Further Reading1 Release Channel

HelpfulTerminology

Used to determine which
release train you receive
updates from (Daily, Beta,Release/Monthly, ESR)

2 Balrog
Web app used to control
release sign-offs andupdate rates for eachrelease channel

3 CDN
Content Distribution
Network – Network ofservers that efficiently
distribute Thunderbird
updates to users

4 Taskcluster
Task execution frameworkused to build and release
Thunderbird

5 Treeherder
Web interface for
Taskcluster job steps

6 Ship-It
Web app used to create
releases from a given buildusing Taskcluster

7 Smoke test
Manual tests to verify that abuild is ready for delivery tousers

8 Uplift
Another term for backport,
which is the process ofapplying a specific software
patch from a newer version
or channel to an older
version or channel

The PreRelease
Process

1.1 Uplifting New Bugs
1.2 Writing Release Notes
1.3 Release CadenceUplifting New Bugs
How new patches get from Daily to Beta, ESR, and Release

Uplift Request
Uplift is usually requested by
the patch author• Request is assessed andapproved/denied byrelease manager• If a patch needs to bemodified for a different
release channel, a modified
patch is requested from theauthor

Uplifting Approved
PatchesOn or before the release dateof a beta or ESR, patches
approved for a release channel
are grafted into theappropriate branch• If a patch cannot bemerged cleanly or with asimple 3-way merge
correction, a patch revision
is requested from theauthor

After Uplifts
The newly-updated branch ofThunderbird is versioned for anew release• The branch is pinned to thenewest corresponding
version of Firefox• Either automatically or
manually, the version number
is updated• Links to each uplift on thenew branch are posted onbugs included in the releaseWriting Release Notes
How Release Notes are Picked, Written, and Organized

When Notes are Written

How Notes are Written

Several factors determine whether release notes
are written

Notes are ideally written to be informative butaccessible to all users• Does the bug impact end users?
• Is the bug a strictly internal change?
• Is the bug routine upstream synchronization?

• Describe the problem succinctly and in passivevoice• Organize the notes by various traits (from mostto least important)• Type - new, changed, or fixed• Area - mail (and news), address book, chat,calendar, or user interface• Feature - IMAP, CalDAV, global search, etc.

Release Cadence
How often each release channel sees a new release

Release channels each get updated at different intervals:• Beta – Roughly twice a week
• Merged once per month from daily• Release – Once per month, with subsequent point releases for approved bug fixes• Merged once per month from beta
• ESR – Once per year, with subsequent releases for approved bug fixes• Merged once a year from beta

TheRelease
Process

2.1 Life Cycle of a Release Candidate
2.2 Testing a New Release
2.3 Merge Day

Life Cycle of a Release Candidate
How a build becomes a release candidate, and becomes a release

PromotingPushing

ShippingTakes a build and creates andsigns release candidate forsmoke testing

Pushes the newly-promoted
build to the CDN, all but
releasing the candidate

Ships a release to the public• Builds L10n (and langpacks)• Build is pushed to candidates
directory of FTP and to“*-localtest” update channel• This is how smoke testers
acquire the build• Email is sent to appropriate
lists with release notes• Build is now available on
FTP as a new release• Release notes are madepublic• Files are copied from the"candidates" directory to the"release" directory• The release channel rule inBalrog is updated for the newrelease• Builds must be signed off
by release engineer andrelease manager• This allows regular users toget a small download therelease as an update

Testing a New Release
How new candidates are tested

Smoke Testing Email

Checking Documentation

Smoke testers are notified that a new candidate
is available to test

If a candidate passes smoke testing, release notes
and security advisories are checked for publicavailability and accuracy• Testers acquire build from“<channel>-localtest” release channel or FTP• Once tested, results are sent back to releasemanager• Betas are usually shipped and releasedsame-day, while ESR receives more testingprior to shippingMerge Day
What happens when we need to sync branches (daily to beta, beta to ESR, etc.)

Close Trees

Perform Merge

Open Trees and Release

The trees of the branches
being merged from andmerged to are closed toprevent changes

The “from” branch is merged
into the “to” branch using build
automation

If all goes well, the trees arereopened and the new releaseproceeds as a normal release• A dry run is performed firstto detect unforeseen
issues• Major version number ofbranch is increased• Email is sent out to keep
relevant mailing lists
apprised of merge
progress• Message is sent notifying
lists that the merge has been
completed

The PostRelease
Process

3.1 Monitoring Release Feedback
3.2 Common Complications

Monitoring Release Feedback
How feedback is gathered and potential issues are recognized

Various feedback and metrics are followed after a release to see what is being reported anddetermine if something needs to be done about potential issues:• Crash rate of new release• Newly-filed bug reports• Support requests at https://support.mozilla.org/• Distro (Linux) feedback• Social venues (reddit, mastodon, etc)
• Channel support group at https://thunderbird.topicbox.com/groups/beta• https://stats.thunderbird.net/• https://support.mozilla.org/kb/thunderbird-beta

Common Complications
What can go wrong and how we remediate it, and other fun things

Zero Days

Merge Conflicts… is when Thunderbird or When a patch is written
Firefox team identifies an on top of a particularbranch of Thunderbird, it
active threat - “in themay not merge cleanly
wild”, highly rated
with other branches. In
security issue - whichneeds immediate fixing. In simple cases, the merge
such cases we attempt to conflict is resolved byhand. For more complex
develop a patch, test,
merge issues, the patch
build and ship within 24
must be revised by thehours.
author.

Build Failures

Canceling Builds

If a build fails because ofa simple server issue
when running a task
(HTTP 500), the task issimply rerun. Otherwise,
the issue is assessed anda patch may need to bemodified then re-uplifted.
This requires a new build.

If a build candidate turns
out to be non-viable forrelease, the releaseprocess must becanceled. In this case, thecurrent phase of releaseis halted (if needed), andthe release is deleted
from Ship-It.

Common Complications (cont.)More of what can go wrong and how we remediate it, and other fun things

Emergency stopof updates
Balrog “Disable Updates
may be used quickly stop
all updates (both manualand automatic), no sign
off required.

Adjust update
rate for special
conditions
Balrog is used to adjust
update rate for desired
objectives, or to evencompletely disable
automatic updates.
We also prevent updates
for OS versions which areEOL.

1 (relman) Prep: ensure security advisories are prepared,
assess channel’s current quality and future needs, seek
needed patches, evaluate and approve appropriate patch
uplift requests, (releng) build/CI fixes and improvements

SummaryTimeline

2 (releng) Apply approved patch uplifts, or do a merge(close trees, perform merge, open trees),
3 (releng) Pin to newest corresponding version of Firefox,
set Thunderbird version number, update bugs with patch
uplift information, write draft release notes
4 (relman) Use Ship-it to create new version for Taskcluster
and promote the build, (releng) fixes anomalies as needed
5 (relman) Inform smoke testers of available candidate
build, (testers) exercise build, (relman) assess feedback
6 (relman) Push build to CDN, (releng) fix any anomalies
7 (relman) Ship release, set Balrog update rate, (security)
make advisories live, (releng) make release notes live
8 (relman) Monitor release feedback/health, seekcorrections in case of poor health, adjust update rate
releng = Release Engineering

relman = Release Manager

Links and Further Reading●

Download Channels: ESR, Beta, Daily, default release●

Thunderbird’s release and events calendar (closely follows
the Firefox release schedule)●

Thunderbird Release Notes●Thunderbird CI Docs

●Treeherder●

Thunderbird statistics●

Thunderbird crash statistics

Thank you!