2017
IEEEIEEE
SecureCybersecurity
DevelopmentDevelopment
Conference

Developers Need Support, Too: A Survey of Security Advice for Software Developers
Yasemin Acar, Christian Stransky,∗ Dominik Wermke, Charles Weir,† Michelle L. Mazurek,‡ and Sascha Fahl
Leibniz University Hannover, ∗ CISPA, Saarland University, † Security Lancaster, ‡ University of Maryland{acar,wermke,fahl}@sec.uni-hannover.de; stransky@cs.uni-saarland.de; c.weir1@lancaster.ac.uk; mmazurek@umd.edu

Abstract—Increasingly developers are becoming aware of theimportance of software security, as frequent high-proﬁle security incidents emphasize the need for secure code. Facedwith this new problem, most developers will use their normal
approach: web search. But are the resulting web resources
useful and effective at promoting security in practice? Recent
research has identiﬁed security problems arising from Q&A resources that help with speciﬁc secure-programming problems,
but the web also contains many general resources that discuss
security and secure programming more broadly, and to ourknowledge few if any of these have been empirically evaluated.
The continuing prevalence of security bugs suggests that thisguidance ecosystem is not currently working well enough:either effective guidance is not available, or it is not reaching
the developers who need it. This paper takes a ﬁrst step
toward understanding and improving this guidance ecosystem
by identifying and analyzing 19 general advice resources. The
results identify important gaps in the current ecosystem andprovide a basis for future work evaluating existing resourcesand developing new ones to ﬁll these gaps.

issues using web search, or by consulting peers [3]. A survey
by many of the current authors concluded the same, and also
determined experimentally the surprising result that programmers using digital books achieved better security thanthose using web search [4]. Nadi et al. found that Java developers perceived cryptography APIs as too low-level andpreferred more task-based solutions in documentations [5].
Further work from several of the current authors concluded
that documentation is critical to security outcomes whendevelopers use unfamiliar cryptography APIs [6].
When developers search the internet for guidance, someof the most popular results will often point to Stack Overﬂow.2 Oftentimes, the developer’s search will lead to a codesnippet on Stack Overﬂow, and the developer can be tempted
to copy and paste the snippet into their own code. Thisbehaviour has been shown to often lead to operational but
insecure code, widespread across hundreds of thousands ofapps [4], [6], [7], [8], [9], [10]. As Stack Overﬂow’s effecton code security has been investigated in depth, we focus
our analysis elsewhere. Similarly, code completion systems
found in IDEs often are not evaluated for the evolving
context found in real-world situations [11].
Beyond crowdsourcing application-speciﬁc problems
and documenting particular APIs, the web also contains
many general resources about security and secure programming. While we would hope that these resources are helpful,to our knowledge few if any have been empirically tested
for effectiveness. Moreover, the continuing prevalence of
security bugs suggests that this guidance ecosystem is fundamentally broken: either effective guidance is not available(if it is even possible), or it is not reaching the developers
who need it. We speculate that this web-based guidanceis particularly important for developers working outside oflarge mainstream corporations, who do not have access to
professional security teams or well-developed toolchains andframeworks supporting secure programming (e.g., Google’s
Tricorder [12]).
In this paper, we take a ﬁrst step toward understanding
and improving this guidance ecosystem. We have identiﬁed
and analyzed 19 guidance websites, to understand whatcurrently available guidance says; what it does and does notcover; and how it is structured. We found that, overall, thisgeneral-purpose guidance does not often provide concrete
examples, tutorials, or exercises to help developers practicethe concepts being described. We also found that while some

1. Introduction
The last two decades have seen explosive growth in thecreation and usage of software, as we now use computersand digital devices to manage almost every aspect of ourlives: to communicate, to plan, to manage our ﬁnances, to do
our shopping, and to remember all our security information.
Software holds sensitive information about us, controls our
ﬁnancial transactions, enables our personal communicationand social networking, and holds the intimate details of ourlives. This growth has led to a commensurate rise in thenumber of people working as software developers. In 1997,
the Bureau of Labor Statistics estimated just over 500,000
computer programmers in the United States; by 2016, thiscategory had been revised into four sub-categories and morethan tripled, to 1.6 million employees [1].
The increasing ubiquity of software has also led tothe increasing ubiquity of security bugs and associatedattacks [2]. An important question, therefore, is how tohelp the increasing population of developers adopt effective
security practices and write more secure code.1
Balebako et al. surveyed and interviewed more than 200
app developers and concluded most approached security
1. For simplicity, throughout this paper we refer to security and privacy
as security.

978-1-5386-3467-7/17 $31.00 © 2017 IEEE
DOI 10.1109/SecDev.2017.17

2. https://stackoverﬂow.com/

22

critical topics like secure networking, user input validation,and software testing are well represented, other importantconcepts like program analysis tools, data minimization,and social engineering are rarely mentioned. These results
provide a foundation for future research to ﬁll gaps as wellas to empirically evaluate existing guidance.

3.2. Content Organization
This category distinguishes different features that describe the content organization of a guide. We distinguish
two different types of guides: brief, single-section articles and more detailed, multi-section handbooks. We noted
whether a guide contained a tutorial, deﬁned as a stepby-step walkthrough of a speciﬁc real-world example, orany exercises to help developers become familiar with acertain security mechanism in a risk-free way. Whenever
we found code examples, we checked if they were readyto use to solve certain small tasks, e.g., encrypting a ﬁleor establishing a secure HTTPS connection. We thought
this to be important as ready-to-use code snippets can becopied directly into the programmer’s source code withoutmajor changes. We distinguish these ready-to-use snippetsfrom examples that present or describe speciﬁc API calls
one at a time rather than in usage context. In addition,
we checked whether a guide contained references to code
repositories such as GitHub or BitBucket. We noted when aguide referenced external articles, blog or Wikipedia posts,
or other external information sources. We noted the lasttime that the guide was updated as a measure of whether
it was outdated or maintained. Finally, we analyzed how
easy it was to ﬁnd speciﬁc information in a guide: thisincluded checking whether a guide had multiple hierarchicalsections to avoid extensive scrolling, whether multiple levels
of information granularity were provided (e.g., for novices,
experienced programmers, and experts) and whether theguide was easily searchable.

2. Selecting Online Resources
We collected online developer resources relevant to security by searching for variations of “developer,” “security”
and “guide” on various online search engines. This gives
us a collection of links to online resources, some of themcollections of other online resources, which we exclude from
our analysis in favour of directly investigating the linked
resources themselves.
We found 19 sites designed to help developers withsecurity related questions. We excluded forum posts and
Q&A platforms such as Stack Overﬂow, and included guides
such as blog posts from authoritative sources and ofﬁcial
guidelines from software providers and non-proﬁt organizations such as OWASP.3
We focused on those types of guidelines since they
carry authority for developers who feel the need to lookup security-related questions and concepts. The guidelines
we found covered mobile application security (Android, iOS
and Windows Mobile), web security, and general secure
programming. We focused on these three areas since they
cover the vast majority of today’s software development;
additionally, mobile and web security issues have received
a lot of attention from the security and privacy researchcommunity in the past.
Table 1 shows the resources we analyzed. We use theterm ‘handbook’ to describe structured websites containing
relatively large amounts of information; ‘article’ to describe
smaller sites, mainly arranged linearly.

3.3. Covered Topics
We next analyzed the different topics that were coveredin the guides we examined. We started from an initial listof topics we thought were important, and added others as
we encountered them in different guides. The ﬁnal list of
topics included:• Obfuscation: Motivation for code obfuscation obfuscation techniques and tools.
• Cryptography: Encrypted data storage, secure key
generation, etc.
• Secure networking: TLS/SSL, HTTPS, etc.
• Storage management: Backups, secure deletion, andproper management of shared storage.
• Privileges: Responsible use of privileges, principle of
least privilege.
• User input: Proper validation of user input to avoid,e.g., SQL injection, cross-site scripting, and memory
errors.
• Use of tools: Using of automated security-test tools including linters and other program-analysis techniques.
• Mobile security: Mobile-speciﬁc topics including privacy implications of mobile device tracking and geolocation.
• Library use: Using trusted libraries to avoid reinventing the security wheel, validating that selected libraries
do not introduce malicious behavior.

3. Evaluating Resources
We evaluated the 19 guides using a content-analysisbased manual coding approach [13]. We ﬁrst deﬁned a codebook that listed desirable features that might be present ina secure-development guide. These features are listed as theheading to Table 4. We identiﬁed several main categories:

3.1. Source
The source category describes the author of a guide:
we distinguish between ofﬁcial guidelines written by aframework/platform company, e.g., Google for Android orApple for iOS, and a third-party organization (for-proﬁt or
non-proﬁt, e.g., OWASP) providing a guide as a community
service.
3. cf. https://www.owasp.org

23

ID
1.

Title
Safeguard your code: 17 security tips
for developers
Best Practices for Security & Privacy
Secure Coding Practice Guidelines

Organization
InfoWorld, an online magazine for IT
and business professionals
GoogleUC Berkeley

7.
8.

Start with Security: A Guide for
Business
Android Secure Coding Standard
Mobile Application Security: 15 Best
Practices for App Developers
Top 10 Secure Coding Practices
Secure Coding Guide

US Government Federal Trade
Commission
Software Engineering Institute, CMU
Checkmarx, a development tool
vendorSoftware Engineering Institute, CMU
Apple

9.

Secure Mobile Development Guide

10.
11.

Secure Coding Practices Quick
Reference Guide Project
Secure Coding Cheat Sheet

NowSecure, company specializing inapp security testing and support
OWASP, a not-for-profit dedicated toapplication security
OWASP

12.

Security Guidance for Applications

Microsoft

13.

Security Checklist for Software
Developers
Website security
Web Fundamentals: Security and
Identity
Developer How To Guide

CERN, a research organization

2.
3.

4.
5.
6.

14.
15.
16.
17.
18.
19.

8 Tips for Better Mobile Application
Security
TOP 25 Most Dangerous Software
ErrorsIntro to secure Windows app
development

MozillaGoogleSANS, a not-for-profit specializing insoftware security training
UpWork, a developer recruitment site
SANS with MITRE, a non-profit
research company
Microsoft

Description
Article dated 2013
Online Android training materials from Google.
Guidance on ways to satisfy application software security
requirements, mainly in the form of links to otherresources.
Paper with guidelines on corporate IS security
Wiki with extensive guidelines.
Short blog article
Short summary of 10 general secure coding principles
Extensive online handbook covering technical aspects ofiOS security
Online handbook covering many aspects of mobile app
security.
Checklist of around 100 short bullet-point entries aroundgeneral coding
More detailed handbook describing principles of generalcoding.
Handbook with security guidance for web applications(outdated)
Site with general guidelines and tips for specific languages.
Extensive training article on web application security.
Several tips on web app security, with emphasis on Google
tools.
Article on how to avoid three common web security
vulnerabilities.
Blog article
Handbook, exploring types of security errors
Extensive handbook to secure programming in the MS
Windows environment.

TABLE 1. T HE 19 SECURITY GUIDES THAT WE IDENTIFIED AND ANALYZED .

Sources and Organization. We found that the majority(>55%) of the resources we analyzed were written bycompanies. All of these companies are in some way involved
in secure software development or beneﬁt from it. In 16% ofcases, the guides were published as part of the ofﬁcial developer documentation written by the vendor of a development
framework such as Apple for iOS, Google for Android, andMicrosoft for Windows Mobile. Interestingly, about a thirdof all guides were written by non-proﬁt organizations like
OWASP.
Most of the guides for which we could identify update
times (10 of 15) were last updated within the last two years;
one (Microsoft web-application guidance, ID 12) could beconsidered entirely obsolete, as it dated from 2003. Theseﬁndings suggest that when developers search for secureprogramming resources, they will frequently but not always
encounter up-to-date ones.
Overall, these resources did not tend to contain concrete,
low-level guidance. We found only ﬁve guides (IDs 5, 12,
15, 16, 19) that contained ready-to-use code snippets; most

Testing: Measures for examining ﬁnished or deployedsystems, including code review, fuzzing and penetration
testing.
• Data minimization: Limiting the collection, storage,
and transmission of personal information to protect
users’ privacy.
• Regulations: Security and privacy laws and regulations
in various jurisdictions.
• Threat modeling: Design-level analysis of security
requirements.
• Logging: Keeping records to enable post-hoc auditing.
• Password advice: Guidelines for the secure creationand storage of passwords (e.g., salting and hashingstored passwords).
• Social engineering: Tricking people into making security errors, such as giving away secret data improperly.
•

4. Results and Discussion
Table 4 provides an overview of our results.

24

ID 1 (InfoWorld)
ID 2 (Google Android)
ID 3 (UC Berkeley)
ID 4 (US FTC)
ID 5 (SEI, CMU)
ID 6 (Checkmarx)
ID 7 (SEI, CMU, Top 10)
ID 8 (Apple)
ID 9 (NowSecure)ID 10 (OWASP, quick reference)
ID 11 (OWASP, cheat sheet)
ID 12 (Microsoft)
ID 13 (CERN)
ID 14 (Mozilla)
ID 15 (Google)
ID 16 (SANS)
ID 17 (UpWork)ID 18 (SANS, MITRE)
ID 19 (Microsoft)

third-party
vendororganization
organization
organization
third-party
organization
vendorthird-party
non-proﬁt
non-proﬁt
vendororganization
non-proﬁt
vendornon-proﬁt
third-party
non-proﬁt
vendor

2013-02-04?
?
2015-06
2016-07-11
2015-08-19
2011-03-01
2016-09-13
2017-03-05
2010-08-11
2017-04-18
2003-07-01?
2017-05-24
2017-05-22?
2017-01-15
2011-07-27
2017-02-08

a
h
a
a
h
a
a
h
h
h
h
h
a
h
h
h
a
h
h

Covered Topics
Obfuscation
Cryptography
Secure Networking
Storage Management
Privileges
User Input
Use of Tools
Mobile Security
Library Use
Testing
Data Minimization
Regulations
Threat Modelling
Logging
Password Advice
Social Engineering

Last Update1

Exercise
Tutorials
Repository
Citations
Layered Information
Searchable
Sectioned

SourceReady to use code
API examples

Resource ID

Type2Content Organization

12?: date not speciﬁed
a: short article, h: detailed handbook
TABLE 2. F EATURES OF THE GUIDES , AS DETERMINED BY CONTENT ANALYSIS . A SHADED CIRCLE ( ) INDICATES A GUIDE EXHIBITS THISFEATURE ; AN EMPTY CIRCLE ( ) INDICATES IT DOES NOT.

of these offered help on secure network connections using
TLS. An additional three guides included API examples
that do not rise to the level of snippets: Google’s Android
security and privacy guide (ID 2) exhibits the Android
cryptography API, Apple’s Secure Coding Guide (ID 8)
gives examples for platform-speciﬁc security mechanisms,
and NowSecure (ID 9) contains negative examples that
demonstrate how not to use their SSL API. Interestingly, allthese API examples come from mobile application guides.
Sadly, none of the guidelines we checked contained
exercises that would have helped developers to learn security
APIs or frameworks. Similarly, only the Microsoft Windows
handbook (ID 19) contained a tutorial introducing developers to secure web programming techniques or referenced an
external GitHub code repository with example code demonstrating speciﬁc security features. Only six guides referenced
external information sources; a lack of such citations potentially undermines a reader’s conﬁdence in the guide’s
accuracy and inhibits further reading and learning. Overall,these results provide evidence of an important guidance gap
noted in our prior work [4]: ofﬁcial documents and corporate
guidelines do not provide the same level of detail and focus
on utility as, for example, Q&A sites.
On the positive side, we were heartened to observe thatmost guides provide layers of advice that can target readerswith different skill levels and are easily searchable.
Coverage of Topics. We found signiﬁcant variationin coverage across topics. Some important topics are wellcovered: cryptography, secure networking, privilege man-agement, and user input validation were all covered in atleast 14 guides, and testing was covered in 10. On the other
hand, there are several potentially critical topics with little tono coverage. Social engineering, which is a source of many
security horror stories from phishing to recklessness withUSB drives, is covered by only four guides. Logging, whichis critical for being able to audit a system and understand any
potential problems, is covered by only seven. Despite years
of intensive research and commercial development creating
and improving program-analysis tools, these tools are also
only mentioned in seven guides. Another topic mentioned
in only seven guides is data minimization, a critical modern
concept in a world of exploding data mining. Some of
the guides we analyzed may be too old to recognize thecritical importance of data minimization; nonetheless it isconcerning that developers searching for security help may
not encounter it. Using and validating trusted libraries is an
important and well-regarded practice, but it is mentioned byonly six guides, perhaps because it is assumed to be implied.
Finally, we note that laws and regulations are mentioned
in only two guides. One might assume this is an issue forlawyers and executives rather than for software developers,
but independent developers and small companies (the kindof developers most likely to be searching for guidance on theinternet rather than talking to a company’s dedicated securityteam) may not have separate compliance departments either.
Further, developers who have some knowledge of legal
requirements are less likely to make accidental errors that25

violate regulations.
We speculate that these coverage limitations arise fromthe nature of the authors creating these guides. The coveragesuggests a predominantly traditional approach to security,
based around technical support for developers. Looking at
the results, we may conclude that in future better results may
come from having a cross discipline team create the documentation, with representation from test, support, productmanagement, and legal experts.

to empirically evaluate the existing guidance, to understandwhich approaches do and don’t prove to be effective andwhy. By understanding what current guidance is missing, where it succeeds, and where it fails, we can hopeto provide a framework for developing better guidance,
both now and as secure programming continues to evolve.

References

5. Conclusion
Our brief survey of general security guidance available
on the web provides some insight into what developers—
especially those without formal security training and/or
without corporate security support—may encounter when
they search for information about how to write secure
code. They will ﬁnd accessible information, appropriately
layered and searchable, with good coverage of cryptography,
secure networking and the handling of user input and privileges. However there are signiﬁcant areas of concern: somereadily available advice is outdated; most of this generalpurpose guidance does not provide concrete examples or
exercises; and some critical topics like program analysis
tools, logging/auditing, and data minimization are not well
represented. To remedy this would require a rather different
team of authors from traditional security writers: pedagogical experts to generate exercises and tutorials, and humancentred security experts and legal experts to deal with social
engineering and regulations.
Our prior work found that “ofﬁcial” guidance (from
Google and from books) could promote stronger security
outcomes than community-based guidance from Stack Overﬂow [4]. The results of this survey, however, underscoreanother conclusion from that work: these “ofﬁcial” documents may not necessarily provide the content and format
that developers want or need in practice.
In this work, we take the preliminary step of identifying
and classifying a multitude of information sources. Further
work is needed to assess their quality, and to understandhow developers use these guides as well as how to increase
their effectiveness. Therefore, the question of how best toorganize online security help – guides, crowdsourcing sites,
aggregators that combine various sources of security advice– as well as how to ensure that the quality of such advice
remains high – remains open.
Thus, our results suggest two paths for understanding
and improving the security-guidance ecosystem. First, wemust examine whether and which of the gaps we haveidentiﬁed here—both in content and in format—represent
serious omissions, and which are ﬁlled by other resources
outside the scope of this paper. Second, we must continue[1]

Bureau of Labor Statistics, “Occupational Employment Statistics,”
2016. [Online]. Available: https://www.bls.gov/oes/tables.htm[2]

S. Morgan, “Top 2016 Cybersecurity Reports,” 2016. [Online].
Available: https://www.forbes.com/sites/stevemorgan/2016/05/09/top2016-cybersecurity-reports-out-from-att-cisco-dell-google-ibmmcafee-symantec-and-verizon/#2c8abdb51caf[3]

R. Balebako and L. Cranor, “Improving App Privacy: Nudging App
Developers to Protect User Privacy,” IEEE Security & Privacy,
vol. 12, no. 4, pp. 55–58, 2014.

[4]

Y. Acar, M. Backes, S. Fahl, D. Kim, M. L. Mazurek, and C. Stransky,“You Get Where You’re Looking For: The Impact of Information
Sources on Code Security,” in Proc. 37th IEEE Symposium on Security and Privacy (SP’16). IEEE, 2016.

[5]

S. Nadi, S. Krüger, M. Mezini, and E. Bodden, ““Jumping ThroughHoops”: Why do Java Developers Struggle With Cryptography
APIs?” in Proc. 37th IEEE/ACM International Conference on Software Engineering (ICSE’15). IEEE, 2016.

[6]

Y. Acar, M. Backes, S. Fahl, S. Garﬁnkel, D. Kim, M. L. Mazurek,
and C. Stransky, “Comparing the Usability of Cryptographic APIs,”in Proc. 38th IEEE Symposium on Security and Privacy (SP’17).
IEEE, 2017.

[7]

J. Xie, H. R. Lipford, and B. Chu, “Why do programmers makesecurity errors?” in Proc. 2011 IEEE Symposium on Visual Languages
and Human-Centric Computing (VL/HCC’11). IEEE, 2011.

[8]

S. Fahl, M. Harbach, H. Perl, M. Koetter, and M. Smith, “Rethinking
SSL Development in an Appiﬁed World,” in Proc. 20th ACM Conference on Computer and Communication Security (CCS’13). ACM,
2013.

[9]

S. Fahl, M. Harbach, Y. Acar, and M. Smith, “On The Ecological
Validity of a Password Study,” in Proc. 9th Symposium on Usable
Privacy and Security (SOUPS’13). USENIX Association, 2013.

[10] F. Fischer, K. Böttinger, H. Xiao, C. Stransky, Y. Acar, M. Backes,
and S. Fahl, “Stack Overﬂow Considered Harmful? The Impact of
Copy&Paste on Android Application Security,” in Proc. 38th IEEE
Symposium on Security and Privacy (SP’17). IEEE, 2017.
[11] S. Proksch, S. Amann, S. Nadi, and M. Mezini, “Evaluating the
Evaluations of Code Recommender Systems: A Reality Check,” inProc. 31st IEEE/ACM International Conference on Automated Software Engineering (ASE’16). ACM, 2016.
[12] C. Sadowski, J. v. Gogh, C. Jaspan, E. Sderberg, and C. Winter, “Tricorder: Building a Program Analysis Ecosystem,” in Proc.
37th IEEE/ACM International Conference on Software Engineering(ICSE’15). IEEE, 2015.
[13] K. Krippendorff, Content Analysis: An Introduction to Its Methodology (2nd ed.). SAGE Publications, 2004.

26