Penetration Test Report (QuarksLab): 18 04 720 REP v1.2

OpenSSL Security Assessment
Technical report
Ref. 18-04-720-REP
Version 1.2
Date 2019
Prepared for OSTIF
By Quarkslab
Quarkslab SAS
13 rue Saint-Ambroise
75011 Paris
France
Contents
1 Project Information 1
2 Executive Summary 2
2.1 Security/Bug concerns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.1.1 Bug . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.1.2 Vulnerability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.2 Security overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.2.1 TLS 1.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.2.2 PRNG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.2.3 SRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2.4 CAPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2.5 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3 Introduction 4
3.1 Context and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.1.1 Past Security Vulnerabilities . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.1.2 OpenSSL Source code architecture . . . . . . . . . . . . . . . . . . . . . 5
4 TLS 1.3 implementation 7
4.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.1.1 TCP Proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.1.2 Code instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.1.3 Code review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.2 Internal Mechanisms of TLS 1.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.2.1 Handshake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.2.2 0-RTT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.2.3 Hello Retry Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.2.4 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.3 Security problem and bug identified in TLS 1.3 of OpenSSL . . . . . . . . . . . 10
4.3.1 Client Denial of Service Bug (Debug mode) . . . . . . . . . . . . . . . . 10
4.3.2 Client Denial of Service Issue . . . . . . . . . . . . . . . . . . . . . . . . 11
4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5 PRNG implementation 15
5.1 API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
5.2 Test Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
5.3 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.3.1 Description of the Internals . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.3.2 Update Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.3.3 Coding Style . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.4 Entropy Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.4.1 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.4.2 Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
6 SRP implementation 21
6.1 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
6.2 Test Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
6.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
6.3.1 Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
6.3.2 Quality of the code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
6.3.3 Security Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
6.3.4 Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
6.3.5 Fuzzing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
6.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
7 Appendix 26
7.1 OpenSSL Patch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
7.2 How to apply the patch and start fuzzing . . . . . . . . . . . . . . . . . . . . . . 31
Bibliography 32
Ref.: 18-04-720-REP Quarkslab SAS 3
1. Project Information
Version Date Status Author
August 31, 2018 Document Creation Quarkslab
Version 1.0 November 28, 2018 First Version
Version 1.1 January 11, 2019 Intermediate Version
Version 1.2 January 16, 2019 Final Version
Quarkslab Position Contact Information
Frédéric Raynal CEO Quarkslab fraynal@quarkslab.com
OSTIF Position Contact Information
Derek Zimmer CEO OSTIF derek@ostif.org
Ref.: 18-04-720-REP Quarkslab SAS 1
2. Executive Summary
ThisreportdescribestheresultsofthesecurityassessmentofOpenSSL1.1.1madebyQuarkslab,
and funded by OSTIF. Two Quarkslab engineers worked on this audit for a total of 60 man
days of study.
Scope of the audit :
• TLS 1.3
• PRNG
• SRP
• CAPI
2.1 Security/Bug concerns
Two client Denial of Service vulnerability and bug were found, allowing an attacker to crash
the client:
2.1.1 Bug
• Ensure we send an alert on error when processing a ticket. Cf. https://github.com/
openssl/openssl/pull/6852
2.1.2 Vulnerability
• Resolve some TLSv1.3 alert issues. Cf. https://github.com/openssl/openssl/pull/6887
All these bug and vulnerability were fixed before OpenSSL 1.1.1 was launched.
2.2 Security overview
2.2.1 TLS 1.3
• The implementation follows the RFC 8446.
• The quality of the source code is good.
2.2.2 PRNG
• The implementation follows the NIST standard SP800-90A (Rev. 1).
• The quality of the source code could be improved.
Ref.: 18-04-720-REP Quarkslab SAS 2
OpenSSL Security Assessment, Release 1.2
2.2.3 SRP
• The protocol seems to be correctly implemented.
• The quality of the source code could be improved.
• TheinaccurateSRPcomment(mentionedatthebeginningofsection6.3,andthenlaterin
6.3.4) was actually recently corrected (independently of this report) in commit 495a1e5c3
(master branch only).
2.2.4 CAPI
• The quality of the source code could be improved.
CAPI is the interface to the Microsoft Cryptographic API, it is used to get certificates and keys
from Windows certificate store, encrypt/decrypt/sign with RSA/DSA...
CAPI is not directly exposed to the end user so there is no way to compromise OpenSSL with
it.
Malicious users should have admin access to potentially add a corrupted certificate in order
to compromise OpenSSL CAPI. We didn’t find any solution to fuzz CAPI, so we manually
reviewed the code. We didn’t find any vulnerability during this review. The CAPI code (en-
gines/e_capi.c) clearly lacks of comment.
2.2.5 Recommendations
We recommend to the OpenSSL Security Team to integrate the fuzzing methodology we have
developed during this security assessment and to fuzz new features of TLS 1.3. This fuzzing
methodology could be used to fuzz older TLS versions as well.
The quality of the source code should be improved with more pointers verification.
2.2. Security overview 3
3. Introduction
3.1 Context and Scope
This report describes the security assessment made by Quarkslab on OpenSSL 1.1.1, an open
source library used to secure network communications.
This audit has been carried out at the request of the Open Source Technology Improvement
Fund. Its goal was to evaluate the security of OpenSSL 1.1.1.
Two engineers from Quarkslab worked on this audit, for a total of 60 man days of study.
The Internet Engineering Task Force has been in charge of defining the TLS protocol. The 21st
of March, 2018, IETF finalized TLS 1.3.
This study focuses on the source code of OpenSSL 1.1.1 especially TLS 1.3, PRNG, SRP and
CAPI.
The version of OpenSSL source code that we analyzed is available in OpenSSL’s Github reposi-
tory with the dc55e4f70f401c5869410d6a0c068c18c3fd53ec commi