Technical Debt
and Open Source
Development
A discussion towards a better understanding of technicaldebt and how open source development helps mitigate it

By Ibrahim Haddad, Ph.D. & Cedric Bail, M.Sc.
A Publication of The Linux Foundation | July 2020

www.linuxfoundation.org

Abstract
This paper grew out of a phone conversation theauthors had on a Sunday morning mid-March 2020
while being confined in their homes. Both authors
worked together within the Open Source Group at
Samsung Research and directly experienced minimizing
internally carried technical debt via working with
upstream open source projects. That experience
covered dozens of open source projects used acrossmultiple products and business units with varying
degrees of involvement and experience with upstream
development. The paper provides an overview of the
problem of technical debt at large scale. It includes
discussions on identifying technical debt, how tominimize it, the role of open source development, andstrategies to address the issue.

Disclaimer
The opinions expressed in this paper are solely theauthors’ and do not necessarily represent their current
or past employers’ views. The authors would like toapologize in advance for any error or omission and areopen to feedback and updates.

Technical Debt and Open Source Development

The Linux Foundation 2

Table of Contents
Technical Debt

4

Definition

10

4

Example 1: The Role of Custom Build
Systems

Symptoms

4

Example 2: User Interface Framework

11

Types of Technical Debt

4

The Role of Upstream Development

12

Temporary Technical Debt

5

Upstream - Unifier of Efforts

13

Unknown Technical Debt

5

Addressing Technical Debt at Scale

14

Purposely Created Technical Debt

5

At the Policy and Process Level

14

Obsolete Technical Debt

5

At the Development Level

14

Organizational Technical Debt

5

15

The Many Causes of Technical Debt

5

Too late! Technical debt is already here.
What should we do?

Consequences

6

Recommended Practices

16

How does technical debt accumulate?

7

Conclusion

17

Working with Technical Debt

8

Feedback

18

Identifying Technical Debt

8

About the authors

18

Minimizing Technical Debt

8

Technical Debt and Open Source Development

Choice of programming language
Choice of ecosystem

89Choice of dependencies

9

The Linux Foundation 3

Technical Debt
Definition
Technical debt, a term used in software development,
refers to the cost of maintaining source code that wascaused by a deviation from the main branch where joint
development happens. A wider interpretation of what
constitutes technical debt is proprietary code by itself:• It has been developed by a single organization.
• It is source code that the organization alone needs
to carry and maintain.

• In some cases, the organization depends on apartner’s ability to maintain the code and carry thatsaid debt.

A noteworthy clarification is that upstream code is
not without technical debt when the upstream project
doesn’t have resources/time to maintain itself via its
developers’ community. An example of this scenario isthe various companies that depended on the OpenSSL
project without contributing to the project and realizingthat the project was maintained only by a single person
during their spare time. This was the specific casescenario that motivated the Linux Foundation to launch
its Core Infrastructure Initiative to support open source
projects critical to our modern infrastructure.

Symptoms
How would you identify the symptoms pointing to
the existence of technical debt? And, what are these
symptoms? In this section, based on our experiences,
we list several such symptoms with a brief description

Technical Debt and Open Source Developmentof each one of them. This is not intended as an exhaustiveand comprehensive list but rather a list of the mostcommon and widely observed technical debt symptoms.

• Slower release cadence Time increases between
the delivery of new features• Increased onboarding time for new developers
Onboarding new developers becomes highly
involved due to code complexity where only
insider developers are familiar with the codebase.
The second manifestation of this symptom is thedifficulty in retaining developers or hiring new
developers.

• Increased security issues At least, experiencing

more security issues than the main upstream branch.

• Increased efforts to maintain the code base

Maintenance tasks become more time consuming
as the body of code to maintain becomes larger andmore complex.

• Misalignment with the upstream development
cycle illustrated in the inability to maintain pace,
be aligned, with the upstream development andrelease cycles.

Types of Technical Debt
There are several types of technical debt. We are
not aware of a standardized or commonly agreed-upondefinition to describe these various technical debt
types. Therefore, in this section, we present ourinterpretation.

The Linux Foundation 4

Temporary Technical Debt

A team may be working on a complex feature that possibly
touches several components, systems, or subsystems.
The need arises to carry out certain technical debt fora temporary period as things get developed and integrated
with the upstream branch. We are aware of the fact that
we’re creating technical debt. However, our purpose isto accelerate product development, and the end goal
is to repurpose and merge the fork with the upstream
branch at a later point.

and created a defacto solution (or a standard) thatis now incompatible with what the organization has
developed. This situation makes that development asource of technical debt by obsolescence.

Organizational Technical Debt

Unknown Technical Debt

It is widely discussed how the source code that anenterprise creates often matches that enterprise’sorganization -- a very interesting theory. In somecases, it so happens where code should be developed
and where it ends up being developed do not match.
When developers cannot push back on managers andhave the code written by the right people, or they can’t
contribute to the right piece of code, the result is oftena piece of duct tape on code that shouldn’t be there
and that nobody wants to deploy. This is technical debt.

Purposely Created Technical Debt

The Many Causes ofTechnical Debt

Unknowingly creating technical debt as a result of badengineering practices. An example of this scenario
is poorly-written code that is not accepted into the
upstream branch and is not even a candidate forreusing somewhere else. We’re stuck with this code.

This unusual type of technical debt is being createdon purpose. An example of such a case would be anorganization that wants to maintain certain features
exclusive to them without sharing them with thebroader community. As a result, such organizations findthemselves creating this fork to keep it independent
without merging it with the upstream branch. Overtime, such forks grow, leading to larger technical debt
and increased associated maintenance costs.

Obsolete Technical Debt

Obsolete technical debt is a unique use case of technicaldebt resulting from the “not invented here” syndromeor the result of isolated development of new componentsthat could have benefited the broader community.
However, due to a lack of technical oversight orleadership, one organization only uses that development.
Simultaneously, the world moved on to solve the problem

Technical Debt and Open Source Development

A large number of factors contribute to the creationand growth of technical debt. In this section, we
explore the most common causes and provide a brief
description of each one.

• Low-quality code that can’t be upstreamed forsome reason, such as it doesn’t meet the code
quality criteria set by the target open source
project. Another manifestation of this factor is what
is commonly referred to as “spaghetti code.”• Self-serving code that is only useful to the specificcompany contributing the code without much use to
the general community. Such code (or functionality)
is usually not accepted upstream, and the recomm­endation is often to adjust it in such a way to benefit
general use cases and the wider number of users.

The Linux Foundation 5

• Fragmented development leading to duplicatedefforts and competing implementations• Lack of effort to drive code upstream -- in manycases, the team’s organization creating the code
does not have enough bandwidth to work withthe community and drive the code into being
accepted in the upstream branch. This may improve
effectiveness in the short term but creates technicaldebt and long term negative consequences in code
maintenance and upkeep.

• Intrusive code that requires additional coordination
across multiple components or various systems/
subsystems.

• Time to get code accepted upstream that causes

temporary technical debt until the code is accepted
and merged in the upstream branch. It has noadverse effect on the long term.

• Lack of testing preventing complete test coverageand causing failure in submission.

• Lack of documentation or documentation that is
not up-to-date.

• Poor technical leadership and inadequate

engagement with the technical community lead tobeing sidelined and having to, later on, play catch
up with the rest of the world as they move on.

• Ongoing change in requirements within the internal
development efforts in any given organization.

• Organizational obliviousness that combines

the aspects of poor technical leadership andunawareness of the technical direction of upstream
development. This scenario is increasingly
common in non-digital-native companies who areincreasingly forced into development work.

Consequences
Creating and carrying technical debt will have several
negative effects on development efforts, including:• The higher cost of code maintenance.
• Slower innovation and development cycles.
• Paying interest on the debt -- payment of technicaldebt is in the form of additional development
needed to keep up with the main branch, thecompetition, and the rest of the world.

• Possibly missing on new features in the mainbranch or having to backport all new development
into the forked branch internally.

• Duplicate work with the main branch arising due tothe delta between the internal and public branches
being too large.

The worst possible consequence is the effect on thelong term maintainability of the code base whereorganizations often find themselves maintaining theirown fork.

• Non-standard technology or lack of alignment witha given standard.

Technical Debt and Open Source Development

The Linux Foundation 6

How does technical debt accumulate?
Arthur Bloch is an American writer, author of Murphy’s Law books. He is quoted, “Friends come and go, but
enemies accumulate.” This is a great quote to reference when discussing technical debt as just like enemies,
technical debt accumulates. How does it happen? Figure 1 offers an illustration of a basic scenario that explains
how technical debt gets created and carried over.

Open Source
Project Tree
V 1.1
of upstream

Download

Product
SourceCode Tree

Apply
custompatchesV 1.2
of upstream

Fix custom codethat won’t work withnew version of
upstream code
New code updates
happening to producttree while adapting
latest upstream code

Test
Merge withproduct tree

Re-do
Everythingfor V 1.2 of
upstream

Figure 1: Development cycles without upstream integration

Technical Debt and Open Source Development

The Linux Foundation 7

Working with Technical Debt
Identifying Technical Debt
Every line of code is potentially technical debt that
consumes engineering time best put on specificbusiness needs. Figuring this out is mostly about
figuring out your business goals and what your teamis putting time on. Answering a few questions can help
with this thought exercise:• What is your team working on?
• What do you need to tell a new hire regarding yoursoftware stack?

• How often can you update your full software stack?
• Do you know all the software you depend on?
• What usually breaks?
• What dependencies are the most painful to deal with?
• What would have been the alternative for every
single component of the software stack used?

• How active are the communities of each of thesealternatives?

• How much work is needed to maintain that component
if the community disappears or gets stagnant?

• How much effort to switch to this component?
• How hard is it to debug problems reported by yourcustomers or engineers?

Technical Debt and Open Source Development

This series of questions will help you start discussing
technical debt and where your blind spots are.

Minimizing Technical Debt
The core question to address now is how to minimizetech­nical debt and minimize its impact on development
efforts.

Choice of programming language

The programming language or development frameworkused to build the product poses certain restrictions
on your developers. The higher the complexity of alanguage or a framework, the harder it is to maintain aworkforce that can work with it. It becomes a balance
between the constraint of the system you are building
your software for and the access to the necessary
developers base to get the job done. In general, formost cases, you are better off with a higher-level1
language as it will:• Facilitate hiring developers that provide good results.
• Help third parties identify and solve problems.
• Be portable and easily maintained due tocommunity involvement.

• Be more tolerant of mistakes and allow for a
simpler solution.

1 The choice of a high-level programming language is context
specific, however, as examples of such languages, we’d like tosuggest Go, Python, C#, and Typescript.

The Linux Foundation 8

• Important online source of documentation, tutorial,and pre-made solutions.

Choice of ecosystem

Your application is always built on top of a software
stack of language, framework, and operating system.
This is typical of the structure or composition of
the ecosystem the application is going to live in.
This ecosystem will shape the technical debt, anddevelopers should know how it aligns with their own
goals. For instance:• Linux distributions have different terms of support
and guarantee various levels of API/ABI/security
over time.

• The choice of programming language impacts the

Choice of dependencies

As the world moves forward, more open source
software of high quality is made available. This is acontinuous process, and it is essential to evaluate eachpiece of software that your organization is developing.
These dependencies simplify the necessary workand enable them to build more complex features andsolutions. Still, they can also become technical debt if
their communities are not healthy enough. Choosing
the right dependencies and contributing to theimportant ones to you reduces your technical debt by
sharing it with the world.
Also, what was true for the value of dependency a
few years ago might not be anymore. This meansthat like dealing with technical debt, evaluating yourdependencies needs to be done continuously.

ability to run software over time. It affects theability to execute code as the runtime and build
environment it runs within evolve. Also, it mightpose some limitations on your ability in finding new
developers to manage those changes.

• Modern language and framework also tend to

bypass Linux distributions to package software.
This factor might have a long term impact on yoursoftware in the same way the Linux distribution will,
so it should also follow your API/ABI/security needs.

Technical Debt and Open Source Development

The Linux Foundation 9

Example 1: The Role of Custom
Build Systems
Maintaining an operating system is more than a fulltime job. It requires making sure that

• It works reliably and securely over time,• Building each component is reproducible over time,• Each component is properly evaluated and tested,• Licenses are respected, and,• That you have a robust and secure update
mechanism.

This type of work is undervalued as putting together
a Linux environment can now be done in a matter ofhours; however, this is just the first step of maintaining
an operating system over a long period. Today, with
the ARM entrance in the server market, it is possible
to have a more easily standardized distribution forembedded devices. Debian, Ubuntu, Redhat, and SuSE,
to name a few, provide ARM server distributions thatrun just fine, or with a small tweak, on any embedded

Technical Debt and Open Source Development

device and remove the need for maintaining a custom
operating system and associated package build. It also
provides developers with standardized tools to transfer
knowledge from one environment, the cloud, to theembedded market. Finding developers that can now
work on embedded systems becomes easier as hiring
managers can tap into the larger market -- the Linuxserver market.
It is most likely that an important trend in embedded
device development is going to be in picking an ARM
server distribution with some kind of Long Term
Support, along with a small service written in a higherlevel language, much in the same way that the cloudindustry writes them. Python, Node.js, and Go all havea bright future in the embedded systems industry.
The next time you have an embedded Linux project,consider going with a standard Linux distribution thatprovides some form of Long Term Support and reduces
your future technical debt.

The Linux Foundation 10

Example 2: User Interface
Framework
It is easy to gather a few open source components andrun them on a Linux Kernel to name this assemblagea Linux Distribution. It is not very difficult to display
a picture on the screen with some text and call ita small UI framework. And in the same way thatmaintaining a Linux distribution is a never-endingtask, writing and maintaining your UI framework will
also be a never-ending task. Consider the need todisplay language from around the world correctly,
the need for accessibility, the need to scale up andfit a more constrained environment, and support
different rendering systems as the world moves tobetter technologies. There is no end to maintaining
any UI framework. You can easily observe these in thedevelopment of existing UI frameworks. Qt, GTK, andEFL are more than 20 years old today. They required
hundreds of developers to get where they are, and weshould expect them to require the same level of effort
for the next 20 years. React, and ReactNative require
hundreds of developers as well, a language change
isn’t changing the need to address all this external
constraint. When you pick a UI framework, understandTechnical Debt and Open Source Developmentthat you choose a community, and rely on them to carryits technical debt. Being able to step in and help might
be necessary to ensure that this community stays
healthy and keeps moving that debt off your shoulder.
It is also recommended to pay attention to the licenses
in effect and generally who owns any given project’s
IP assets. Depending on a UI Framework and without
being involved in its development nor paying any
licensing fees, you are inherently weakening one ofyour core dependencies if you are making a visualapplication and so increasing your technical debt. It
is usually hard to switch frameworks, and the more
complex your application, the harder it becomes tochange frameworks. As for Linux distributions, you’dwant to be involved in some form with your upstream
dependencies to align with your own needs in thelong run. Contributing to upstream can take manydifferent forms, and you should choose the one thatmatches your business the best (code, monetary,
documentation, marketing, etc.)

The Linux Foundation 11

The Role of Upstream Development
It is expected to branch out or fork and do your development as long as the end goal is to contribute back to the
upstream branch. Figure 2 illustrates this process.

Open Source
Project Tree
V 1.2
of upstream

V 1.1
of upstream

Downloadfrom Upstream
project

Product
SourceCode Tree

Apply
custompatches

Contribute Download fromchanges
upstream
to upstream
project
project
Test

Integrate inproduct tree

Apply fewer
custompatchesV 1.3
of upstream

Contribute Download fromchanges
upstream
to upstream
project
project
Test

Integrate inproduct tree

Apply fewer
custom patches
to latest
upstream code

Figure 2: Development cycles with upstream integration• Continuous integration and continuous delivery/

deployment: Features are available more quickly todevelopers; new code appears in developer trees
sooner. At a higher level of quality, each code commit
requires testing, regressions, and bugs are more visible.

Technical Debt and Open Source Development• Release early and often: The release early andoften practice has numerous proven advantages.
The “release early” allows others to provide
feedback and participate in the development,
welcoming new ideas that can be incorporated.

The Linux Foundation 12

Simultaneously, code is still flexible and offers
time for problems to be flagged by others before
development gets too far. On the other hand, the“release often” makes it possible for the codebasechanges to be easier to understand, debug, anddrive to maturity, and at the time, facilitates theability to maintain the rapid pace of development
and innovation.

• Peer review: Share as early as possible, evenduring the design phase of your code; Requests forcomments are expected; subsystem/maintainer
model with multi-layer hierarchy; by the timecode is released, it has typically been reviewed
many times. Code is always reviewed before beingcommitted. Enables projects to accept code froma much wider range of contributors (establishes a
web of trust).

• Ongoing or continuous testing:• Early discovery means faster triage and fixes.
• Smaller changes make troubleshooting easier.
• Regressions are noticed earlier.
• Helps subsystem maintainers determine which• Easier Maintenance (changing technology, security
fixes)• Better testing, bug report gathering, and analysis• Focus on modular designs and architecture:

Modularity has several benefits: it allows projects
to scale, minimizes contention over common codewith a smaller core and features implemented
as plugins reduce collisions, creates a natural
separation of tasks and scope, allows features
to be available more quickly to developers, andif well done modular designs often have fewer
interdependencies with clear interfaces.

Upstream - Unifier of Efforts
Done well, developing in alignment with upstream isa guarantee that you will have little to zero technicaldebt. However, that requires involvement and active
participation in upstream open source projects.
You can not enforce that upstream is going in the
direction that benefits you. Still, you can influence
it by contributing to it and sharing your goal/willwith upstream and influencing other contributors tounderstand your needs.

code submissions to accept.

• Projects may have multiple build and test cycles.
• Build service tools can automate the process.
• Typically tightly aligned with feature freezes.

Technical Debt and Open Source Development

The Linux Foundation 13

Addressing Technical Debt at Scale
If your organization uses hundreds or thousands ofopen source packages and makes modifications tomany of them, you will need to consider a contribution
strategy that will allow you to contribute code back to
the core or essential components to minimize technicaldebt towards such strategic components. What are someof the ways to address the technical debt on a large scale?
We explore two sets of practices that help that at policy
and process level and then at the development level.

At the Policy and Process Level• Blanket approval to contribute for dedicated open
source developers.

• Faster approval path that allows contributions to
upstream projects to flow much faster.

At the Development Level• Explain your business goals to your engineers by

sharing your vision and goals. In the end, they are
the ones implementing it, so they should know what
you have in mind.

• Ensure that your developers and new members

joining your team understand what technical debts
are and why you choose to maintain the technicaldebt you already have.

• Have realistic expectations in terms of merging yourcontributions with the upstream branch.

• Embrace the review/feedback cycle. There will be

multiple back-and-forth cycles as part of the review
process with the upstream development.

• Flexible IT support enables developers with

• Don’t be too selfish with your contributions. Get• Structure performance reviews that reward• Encourage your developers to explicitly leave• Guaranteed time for your developers to work with• Work for the short term but plan for the long term.

needed tooling (now facilitated with the Windows
Subsystem for Linux and virtual machines if you
can’t afford a dedicated Linux environment for yourdevelopers).
developers who follow the set processes/policies
and work towards minimizing technical debt
the upstream projectTechnical Debt and Open Source Development

involved with upstream on tasks that are notnecessarily directly needed in the short term by
your organization, but improve upstream viability
and health.
comments in the code when they are adding
technical debt.

The Linux Foundation 14

Too late! Technical debt is alreadyhere. What should we do?
The organization you work with has accumulated a
lot of technical debt, and all the symptoms are there.
Now, what should you do about it? We wish there wasa silver bullet, but there isn’t. The approach you takewill depend on several factors; however, you can start
examining these different options:• Choose what feature/functionality needs to be saved.
• Identify the code that is still useful.
• Remove code that should not be maintained orused anymore.

• Reduce the need for branches/fork.
• Refactor, clean and upstream the code that can be
upstreamed.

These activities are time-consuming and will slow down
your development cycle as you dedicated developers tofocus on this effort. It will also be hard for the organization
to add any features during that time and might be very
controversial or counter to the ongoing efforts.

Technical Debt and Open Source Development

It is also possible that there exists now an open source
project that does or could provide the feature or someof the features you need. Migrating to it might also
make more sense than trying to deal with the current
codebase. It is something not to forget, the world
moves even when you are not looking, and there might
be something out there now that does what you need.
Do not let your organization pride prevent you fromlooking at other solutions and use the one that makes
most business sense.
Another more radical approach: Give up on yourtechnical debt and drop the code altogether. This canbe done in multiple ways. If your business can notafford to maintain the code still and that there aren’t
enough clients to justify its existence, just drop it. For
that, be explicit with your clients and tell them that
this code is going to be deprecated and sunsetted.
This is the equivalent of declaring bankruptcy on yourtechnical debt and ceasing payment on it.

The Linux Foundation 15

Recommended Practices
In this section, we provide a list of recommended
practices listed in random order. Please keep in mind
that they may not all apply to your specific situationand organization as you read this section. Some may
apply, some may not. Therefore, it is essential to doa self-evaluation of an appropriate practice to adopt
within your company and possibly how such presented
practices can be adjusted to provide the best possible
outcome for you.

• Adopt an upstream first philosophy.
• Careful evaluation for any custom code that is notgoing upstream.

• Always plan to merge back with the main branch.

Forks and side branches are ok as long as there is aplan to merge with upstream.

• Align internal development effort with the
upstream branch release cadence• Allow fast approvals for upstream contributions.

This is done via a clear and lightweight policy andprocess to facilitate interactions with upstream
developers.

Technical Debt and Open Source Development• Update your performance metrics to incorporate

metrics related to technical debt as part ofthe overall performance goals to ensure thatdevelopment goals are not achieved due to a highor unacceptable technical debt cost.

• Train developers/managers to identify and mitigate
technical debt scenarios.

• Require all code to be properly documented to bebetter understood by upstream reviewers andmost likely will contribute to a faster acceptancecycle (Document also why code is not upstreamed)• Follow the release early and often practice.

Don’t build a huge code base and then decide to
upstream it. Share design, early code and submit
large contributions in smaller, independent patches
that build on each other.

• Track the code you choose to not upstream: Re-

evaluate at every opportunity. If you spot an
upward trend in the internal maintained code’s size,
this calls for a discussion and reevaluation of priordecisions.

The Linux Foundation 16

Conclusion
Open source has a significant role, and aligning yourdevelopment efforts with upstream open source
projects can result in a direct positive impact on theamount of the technical debt an organization carries.
Just as financial debt involves paying interest, technicaldebt has a different kind of interest that needs to be
carried: It’s not interest-free!
In many cases, technical debt is unavoidable shortterm. Carrying technical debt is mostly a decision thatdevelopers need to make all the time. The long term
goals of any engineering effort should be to minimize
and eliminate technical debt resulting from any
development effort. With proper policies, processes,
training, and tooling, organizations can help mitigate
and guide the engineering efforts towards lowering
technical debt.

Technical Debt and Open Source Development

The Linux Foundation 17

FeedbackSuggestions for improvement will be appreciated. Please send comments to the authors directly.

About the authors
Cedric Bail (M.Sc.) is a senior software engineer currently contracting with EASi specializing in embedded Linux.
Cedric has worked as a software engineer for various embedded Linux platforms at a mobile operator, an Internet
service provider, and, most recently, one of the largest consumer electronics companies. His focus has always
been to optimize software for constrained environments, improving their API, and introducing new components asneeded in upstream software to match business requirements.
Twitter: @cedric_bail
LinkedIn: linkedin.com/in/cedricbail/Email: cedric.bail@free.fr
Ibrahim Haddad (Ph.D.) is the Executive Director of the LF AI Foundation. Haddad has held technology andportfolio management roles at Ericsson Research, the Open Source Development Labs, Motorola, Palm, HewlettPackard, the Linux Foundation, and Samsung Research.
Twitter: @IbrahimAtLinux
Web: IbrahimAtLinux.com
LinkedIn: linkedin.com/in/ibrahimhaddad
Email: ibrahim@linuxfoundation.org

Technical Debt and Open Source Development

The Linux Foundation 18

The Linux Foundation promotes, protects andstandardizes Linux by providing unified resources
and services needed for open source to successfullycompete with closed platforms.
To learn more about The Linux Foundation or our otherinitiatives please visit us at www.linuxfoundation.org