Repository: abhinavsingh/proxy.py
Pull request #160 (2019-11-10 01:08:33)

Add private / public key generation utils which comply with new requirements on Mac OS 10.15
See https://support.apple.com/en-us/HT210176 for background.

codecov[bot]: # [Codecov](https://codecov.io/gh/abhinavsingh/proxy.py/pull/160?src=pr&el=h1) Report
> Merging [#160](https://codecov.io/gh/abhinavsingh/proxy.py/pull/160?src=pr&el=desc) into [develop](https://codecov.io/gh/abhinavsingh/proxy.py/commit/93d8a55c0e39a7d8065dc814e84523277e055cb9?src=pr&el=desc) will **decrease** coverage by `1.42%`.
> The diff coverage is `4.87%`.

[![Impacted file tree graph](https://codecov.io/gh/abhinavsingh/proxy.py/pull/160/graphs/tree.svg?width=650&token=Zh9J7b4la2&height=150&src=pr)](https://codecov.io/gh/abhinavsingh/proxy.py/pull/160?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           develop     #160      +/-   ##
===========================================
- Coverage    74.22%   72.79%   -1.43%     
===========================================
  Files           25       26       +1     
  Lines         1936     1974      +38     
===========================================
  Hits          1437     1437              
- Misses         499      537      +38
```


| [Impacted Files](https://codecov.io/gh/abhinavsingh/proxy.py/pull/160?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [proxy/http/handler.py](https://codecov.io/gh/abhinavsingh/proxy.py/pull/160/diff?src=pr&el=tree#diff-cHJveHkvaHR0cC9oYW5kbGVyLnB5) | `79.07% <0%> (ø)` | :arrow_up: |
| [proxy/common/pki.py](https://codecov.io/gh/abhinavsingh/proxy.py/pull/160/diff?src=pr&el=tree#diff-cHJveHkvY29tbW9uL3BraS5weQ==) | `0% <0%> (ø)` | |
| [proxy/common/flags.py](https://codecov.io/gh/abhinavsingh/proxy.py/pull/160/diff?src=pr&el=tree#diff-cHJveHkvY29tbW9uL2ZsYWdzLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [proxy/core/acceptor.py](https://codecov.io/gh/abhinavsingh/proxy.py/pull/160/diff?src=pr&el=tree#diff-cHJveHkvY29yZS9hY2NlcHRvci5weQ==) | `78.94% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/abhinavsingh/proxy.py/pull/160?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/abhinavsingh/proxy.py/pull/160?src=pr&el=footer). Last update [93d8a55...2049749](https://codecov.io/gh/abhinavsingh/proxy.py/pull/160?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/.gitignore b/.gitignore
index d84f547..c9aa49f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -10,12 +10,15 @@
 coverage.xml
 proxy.py.iml
 *.pyc
-ca-*.pem
-https-*.pem
+
+*.csr
+*.crt
+*.key
 
 venv
 cover
 htmlcov
 dist
 build
-proxy.py.egg-info
+
+*.egg-info
diff --git a/Makefile b/Makefile
index 0907755..a50e29b 100644
--- a/Makefile
+++ b/Makefile
@@ -17,6 +17,7 @@ CA_SIGNING_KEY_FILE_PATH := ca-signing-key.pem
 .PHONY: lib-clean lib-test lib-package lib-release-test lib-release lib-coverage lib-lint lib-profile
 .PHONY: container container-run container-release
 .PHONY: dashboard dashboard-clean dashboard-package
+.PHONY: plugin-package-clean plugin-package
 
 all: lib-clean lib-test
 
@@ -91,6 +92,12 @@ dashboard-package-clean:
 dashboard-package: dashboard-package-clean
 	pushd dashboard && npm test && PYTHONPATH=.. python setup.py sdist bdist_wheel && popd
 
+plugin-package-clean:
+	pushd plugin_examples && rm -rf build && rm -rf dist && popd
+
+plugin-package: plugin-package-clean
+	pushd plugin_examples && PYTHONPATH=.. python setup.py sdist bdist_wheel && popd
+
 container:
 	docker build -t $(LATEST_TAG) -t $(IMAGE_TAG) .
 
diff --git a/README.md b/README.md
index add732a..2917a92 100644
--- a/README.md
+++ b/README.md
@@ -77,7 +77,7 @@ Table of Contents
     * [Unable to connect with proxy.py from remote host](#unable-to-connect-with-proxypy-from-remote-host)
     * [Basic auth not working with a browser](#basic-auth-not-working-with-a-browser)
     * [Docker image not working on MacOS](#docker-image-not-working-on-macos)
-    * [Unable to load custom plugins](#unable-to-load-custom-plugins)
+    * [Unable to load plugins](#unable-to-load-plugins)
     * [ValueError: filedescriptor out of range in select](#valueerror-filedescriptor-out-of-range-in-select)
 * [Flags](#flags)
 * [Changelog](#changelog)
@@ -129,8 +129,9 @@ Features
     - Can decrypt TLS traffic between clients and upstream servers
     - See [TLS Interception](#tls-interception)
 - Supported proxy protocols
-    - `http`
-    - `https`
+    - `http(s)`
+        - `http1`
+        - `http1.1` pipeline
     - `http2`
     - `websockets`
 - Optimized for large file uploads and downloads
@@ -968,9 +969,9 @@ See [moby/vpnkit exhausts docker resources](https://github.com/abhinavsingh/prox
 and [Connection refused: The proxy could not connect](https://github.com/moby/vpnkit/issues/469)
 for some background.
 
-## Unable to load custom plugins
+## Unable to load plugins
 
-Make sure your plugin modules are discoverable by adding them to `PYTHONPATH`.  Example:
+Make sure plugin modules are discoverable by adding them to `PYTHONPATH`.  Example:
 
 `PYTHONPATH=/path/to/my/app proxy --plugins my_app.proxyPlugin`
 
@@ -983,6 +984,8 @@ or, make sure to pass fully-qualified path as parameter, e.g.
 
 `proxy --plugins /path/to/my/app/my_app.proxyPlugin`
 
+Note that `pip install proxy.py` don't ship [plugin_examples](https://github.com/abhinavsingh/proxy.py/blob/develop/plugin_examples).
+
 ## GCE log viewer integration for proxy.py
 
 A starter [fluentd.conf](https://github.com/abhinavsingh/proxy.py/blob/develop/fluentd.conf)
diff --git a/dashboard/.gitignore b/dashboard/.gitignore
index 14a996a..89b66a3 100644
--- a/dashboard/.gitignore
+++ b/dashboard/.gitignore
@@ -2,5 +2,3 @@ tsbuild
 node_modules
 build
 dist
-
-proxy.py_dashboard.egg-info
diff --git a/dashboard/README.MD b/dashboard/README.md
similarity index 100%
rename from dashboard/README.MD
rename to dashboard/README.md
diff --git a/dashboard/src/proxy.ts b/dashboard/src/proxy.ts
index b21eb0c..4f3990f 100644
--- a/dashboard/src/proxy.ts
+++ b/dashboard/src/proxy.ts
@@ -43,10 +43,10 @@ class ApiDevelopment {
 }
 
 class WebsocketApi {
-  private hostname: string = 'localhost';
-  private port: number = 8899;
+  private hostname: string = window.location.hostname ? window.location.hostname : 'localhost';
+  private port: number = window.location.port ? Number(window.location.port) : 8899;
   private wsPrefix: string = '/dashboard';
-  private wsScheme: string = 'ws';
+  private wsScheme: string = window.location.protocol === 'http:' ? 'ws' : 'wss';
   private ws: WebSocket;
   private wsPath: string = this.wsScheme + '://' + this.hostname + ':' + this.port + this.wsPrefix;
 
diff --git a/plugin_examples/README.md b/plugin_examples/README.md
new file mode 100644
index 0000000..34f725c
--- /dev/null
+++ b/plugin_examples/README.md
@@ -0,0 +1,1 @@
+# Proxy.py Plugins
diff --git a/plugin_examples/setup.py b/plugin_examples/setup.py
new file mode 100644
index 0000000..18899f2
--- /dev/null
+++ b/plugin_examples/setup.py
@@ -0,0 +1,84 @@
+# -*- coding: utf-8 -*-
+"""
+    proxy.py
+    ~~~~~~~~
+    ⚡⚡⚡ Fast, Lightweight, Programmable, TLS interception capable
+    proxy server for Application debugging, testing and development.
+
+    :copyright: (c) 2013-present by Abhinav Singh and contributors.
+    :license: BSD, see LICENSE for more details.
+"""
+from setuptools import setup, find_packages
+
+from proxy.common.constants import __author__, __author_email__
+from proxy.common.constants import __homepage__, __description__, __download_url__, __license__
+
+VERSION = (0, 1, 0)
+__version__ = '.'.join(map(str, VERSION[0:3]))
+
+setup(
+    name='proxy.py-plugins',
+    version=__version__,
+    author=__author__,
+    author_email=__author_email__,
+    url=__homepage__,
+    description=__description__,
+    long_description=open('README.md').read().strip(),
+    long_description_content_type='text/markdown',
+    download_url=__download_url__,
+    license=__license__,
+    packages=find_packages(),
+    install_requires=['proxy.py'],
+    classifiers=[
+        'Development Status :: 5 - Production/Stable',
+        'Environment :: Console',
+        'Environment :: No Input/Output (Daemon)',
+        'Environment :: Web Environment',
+        'Environment :: MacOS X',
+        'Environment :: Plugins',
+        'Environment :: Win32 (MS Windows)',
+        'Framework :: Robot Framework',
+        'Framework :: Robot Framework :: Library',
+        'Intended Audience :: Developers',
+        'Intended Audience :: Education',
+        'Intended Audience :: End Users/Desktop',
+        'Intended Audience :: System Administrators',
+        'Intended Audience :: Science/Research',
+        'License :: OSI Approved :: BSD License',
+        'Natural Language :: English',
+        'Operating System :: MacOS',
+        'Operating System :: MacOS :: MacOS 9',
+        'Operating System :: MacOS :: MacOS X',
+        'Operating System :: POSIX',
+        'Operating System :: POSIX :: Linux',
+        'Operating System :: Unix',
+        'Operating System :: Microsoft',
+        'Operating System :: Microsoft :: Windows',
+        'Operating System :: Microsoft :: Windows :: Windows 10',
+        'Operating System :: Android',
+        'Operating System :: OS Independent',
+        'Programming Language :: Python :: Implementation',
+        'Programming Language :: Python :: 3 :: Only',
+        'Programming Language :: Python :: 3',
+        'Programming Language :: Python :: 3.6',
+        'Programming Language :: Python :: 3.7',
+        'Programming Language :: Python :: 3.8',
+        'Programming Language :: Python :: 3.9',
+        'Topic :: Internet',
+        'Topic :: Internet :: Proxy Servers',
+        'Topic :: Internet :: WWW/HTTP',
+        'Topic :: Internet :: WWW/HTTP :: Browsers',
+        'Topic :: Internet :: WWW/HTTP :: Dynamic Content',
+        'Topic :: Internet :: WWW/HTTP :: Dynamic Content :: CGI Tools/Libraries',
+        'Topic :: Internet :: WWW/HTTP :: HTTP Servers',
+        'Topic :: Scientific/Engineering :: Information Analysis',
+        'Topic :: Software Development :: Debuggers',
+        'Topic :: Software Development :: Libraries :: Python Modules',
+        'Topic :: System :: Monitoring',
+        'Topic :: System :: Networking',
+        'Topic :: System :: Networking :: Firewalls',
+        'Topic :: System :: Networking :: Monitoring',
+        'Topic :: Utilities',
+        'Typing :: Typed',
+    ],
+)
diff --git a/proxy/common/flags.py b/proxy/common/flags.py
index a3e5b2e..a49ba39 100644
--- a/proxy/common/flags.py
+++ b/proxy/common/flags.py
@@ -137,10 +137,11 @@ def init_parser() -> argparse.ArgumentParser:
         action='store_true',
         default=DEFAULT_ENABLE_WEB_SERVER,
         help='Default: False.  Whether to enable proxy.HttpWebServerPlugin.')
-    parser.add_argument('--hostname',
-                        type=str,
-                        default=str(DEFAULT_IPV6_HOSTNAME),
-                        help='Default: ::1. Server IP address.')
+    parser.add_argument(
+        '--hostname',
+        type=str,
+        default=str(DEFAULT_IPV6_HOSTNAME),
+        help='Default: ::1. Server IP address.')
     parser.add_argument(
         '--key-file',
         type=str,
diff --git a/proxy/common/pki.py b/proxy/common/pki.py
new file mode 100644
index 0000000..40eb526
--- /dev/null
+++ b/proxy/common/pki.py
@@ -0,0 +1,213 @@
+# -*- coding: utf-8 -*-
+"""
+    proxy.py
+    ~~~~~~~~
+    ⚡⚡⚡ Fast, Lightweight, Programmable Proxy Server in a single Python file.
+
+    :copyright: (c) 2013-present by Abhinav Singh and contributors.
+    :license: BSD, see LICENSE for more details.
+"""
+import contextlib
+import os
+import uuid
+import subprocess
+import tempfile
+import logging
+from typing import List, Generator, Optional, Tuple
+
+from .utils import bytes_
+from .constants import COMMA
+
+
+logger = logging.getLogger(__name__)
+
+
+DEFAULT_CONFIG = b'''[ req ]
+#default_bits		= 2048
+#default_md		    = sha256
+#default_keyfile 	= privkey.pem
+distinguished_name	= req_distinguished_name
+attributes		    = req_attributes
+
+[ req_distinguished_name ]
+countryName			    = Country Name (2 letter code)
+countryName_min			= 2
+countryName_max			= 2
+stateOrProvinceName		= State or Province Name (full name)
+localityName			= Locality Name (eg, city)
+0.organizationName		= Organization Name (eg, company)
+organizationalUnitName	= Organizational Unit Name (eg, section)
+commonName			    = Common Name (eg, fully qualified host name)
+commonName_max			= 64
+emailAddress			= Email Address
+emailAddress_max		= 64
+
+[ req_attributes ]
+challengePassword		= A challenge password
+challengePassword_min	= 4
+challengePassword_max	= 20'''
+
+
+def remove_passphrase(
+        key_in_path: str,
+        password: str,
+        key_out_path: str,
+        timeout: int = 10) -> bool:
+    """Remove passphrase from a private key."""
+    command = [
+        'openssl', 'rsa',
+        '-passin', 'pass:%s' % password,
+        '-in', key_in_path,
+        '-out', key_out_path
+    ]
+    return run_openssl_command(command, timeout)
+
+
+def gen_private_key(
+        key_path: str,
+        password: str,
+        bits: int = 2048,
+        timeout: int = 10) -> bool:
+    """Generates a private key."""
+    command = [
+        'openssl', 'genrsa', '-aes256',
+        '-passout', 'pass:%s' % password,
+        '-out', key_path, str(bits)
+    ]
+    return run_openssl_command(command, timeout)
+
+
+def gen_public_key(
+        public_key_path: str,
+        private_key_path: str,
+        private_key_password: str,
+        subject: str,
+        alt_subj_names: Optional[List[str]] = None,
+        extended_key_usage: Optional[str] = None,
+        validity_in_days: int = 365,
+        timeout: int = 10) -> bool:
+    """For a given private key, generates a corresponding public key."""
+    with ssl_config(alt_subj_names, extended_key_usage) as (config_path, has_extension):
+        command = [
+            'openssl', 'req', '-new', '-x509', '-sha256',
+            '-days', str(validity_in_days), '-subj', subject,
+            '-passin', 'pass:%s' % private_key_password,
+            '-config', config_path,
+            '-key', private_key_path, '-out', public_key_path
+        ]
+        if has_extension:
+            command.extend([
+                '-extensions', 'PROXY',
+            ])
+        return run_openssl_command(command, timeout)
+
+
+def gen_csr(
+        csr_path: str,
+        key_path: str,
+        password: str,
+        crt_path: str,
+        timeout: int = 10) -> bool:
+    """Generates a CSR based upon existing certificate and key file."""
+    command = [
+        'openssl', 'x509', '-x509toreq',
+        '-passin', 'pass:%s' % password,
+        '-in', crt_path, '-signkey', key_path,
+        '-out', csr_path
+    ]
+    return run_openssl_command(command, timeout)
+
+
+def sign_csr(
+        csr_path: str,
+        crt_path: str,
+        ca_key_path: str,
+        ca_key_password: str,
+        ca_crt_path: str,
+        serial: str,
+        alt_subj_names: Optional[List[str]] = None,
+        extended_key_usage: Optional[str] = None,
+        validity_in_days: int = 365,
+        timeout: int = 10) -> bool:
+    """Sign a CSR using CA key and certificate."""
+    with ext_file(alt_subj_names, extended_key_usage) as extension_path:
+        command = [
+            'openssl', 'x509', '-req', '-sha256',
+            '-CA', ca_crt_path,
+            '-CAkey', ca_key_path,
+            '-passin', 'pass:%s' % ca_key_password,
+            '-set_serial', serial,
+            '-days', str(validity_in_days),
+            '-extfile', extension_path,
+            '-in', csr_path,
+            '-out', crt_path,
+        ]
+        return run_openssl_command(command, timeout)
+
+
+def get_ext_config(
+        alt_subj_names: Optional[List[str]] = None,
+        extended_key_usage: Optional[str] = None) -> bytes:
+    config = b''
+    # Add SAN extension
+    if alt_subj_names is not None and len(alt_subj_names) > 0:
+        alt_names = []
+        for cname in alt_subj_names:
+            alt_names.append(b'DNS:%s' % bytes_(cname))
+        config += b'\nsubjectAltName=' + COMMA.join(alt_names)
+    # Add extendedKeyUsage section
+    if extended_key_usage is not None:
+        config += b'\nextendedKeyUsage=' + bytes_(extended_key_usage)
+    return config
+
+
+@contextlib.contextmanager
+def ext_file(
+        alt_subj_names: Optional[List[str]] = None,
+        extended_key_usage: Optional[str] = None) -> Generator[str, None, None]:
+    # Write config to temp file
+    config_path = os.path.join(tempfile.gettempdir(), uuid.uuid4().hex)
+    with open(config_path, 'wb') as cnf:
+        cnf.write(
+            get_ext_config(alt_subj_names, extended_key_usage))
+
+    yield config_path
+
+    # Delete temp file
+    os.remove(config_path)
+
+
+@contextlib.contextmanager
+def ssl_config(
+        alt_subj_names: Optional[List[str]] = None,
+        extended_key_usage: Optional[str] = None) -> Generator[Tuple[str, bool], None, None]:
+    config = DEFAULT_CONFIG
+
+    has_extension = False
+    if (alt_subj_names is not None and len(alt_subj_names) > 0) or \
+            extended_key_usage is not None:
+        has_extension = True
+        config += b'\n[PROXY]'
+
+    # Add custom extensions
+    config += get_ext_config(alt_subj_names, extended_key_usage)
+
+    # Write config to temp file
+    config_path = os.path.join(tempfile.gettempdir(), uuid.uuid4().hex)
+    with open(config_path, 'wb') as cnf:
+        cnf.write(config)
+
+    yield config_path, has_extension
+
+    # Delete temp file
+    os.remove(config_path)
+
+
+def run_openssl_command(command: List[str], timeout: int) -> bool:
+    cmd = subprocess.Popen(
+        command,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE
+    )
+    cmd.communicate(timeout=timeout)
+    return cmd.returncode == 0
diff --git a/proxy/core/acceptor.py b/proxy/core/acceptor.py
index d85ac44..44eed41 100644
--- a/proxy/core/acceptor.py
+++ b/proxy/core/acceptor.py
@@ -68,7 +68,10 @@ class AcceptorPool:
                 event_queue=self.event_queue
             )
             acceptor.start()
-            logger.debug('Started acceptor#%d process %d', acceptor_id, acceptor.pid)
+            logger.debug(
+                'Started acceptor#%d process %d',
+                acceptor_id,
+                acceptor.pid)
             self.acceptors.append(acceptor)
             self.work_queues.append(work_queue[0])
         logger.info('Started %d workers' % self.flags.num_workers)
diff --git a/proxy/http/handler.py b/proxy/http/handler.py
index b954893..d14165c 100644
--- a/proxy/http/handler.py
+++ b/proxy/http/handler.py
@@ -249,7 +249,10 @@ class HttpProtocolHandler(ThreadlessWork):
             ctx.load_cert_chain(
                 certfile=self.flags.certfile,
                 keyfile=self.flags.keyfile)
-            conn = ctx.wrap_socket(conn, server_side=True)
+            conn = ctx.wrap_socket(
+                conn,
+                server_side=True,
+            )
         return conn
 
     def connection_inactive_for(self) -> float:


Test patch:
diff --git a/tests/common/test_pki.py b/tests/common/test_pki.py
new file mode 100644
index 0000000..7a2b7d3
--- /dev/null
+++ b/tests/common/test_pki.py
@@ -0,0 +1,68 @@
+# -*- coding: utf-8 -*-
+"""
+    proxy.py
+    ~~~~~~~~
+    ⚡⚡⚡ Fast, Lightweight, Programmable Proxy Server in a single Python file.
+
+    :copyright: (c) 2013-present by Abhinav Singh and contributors.
+    :license: BSD, see LICENSE for more details.
+"""
+import unittest
+import subprocess
+from unittest import mock
+
+from proxy.common import pki
+
+
+class TestPki(unittest.TestCase):
+
+    @mock.patch('subprocess.Popen')
+    def test_run_openssl_command(self, mock_popen: mock.Mock) -> None:
+        command = ['my', 'custom', 'command']
+        mock_popen.return_value.returncode = 0
+        self.assertTrue(pki.run_openssl_command(command, 10))
+        mock_popen.assert_called_with(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+    def test_get_ext_config(self) -> None:
+        self.assertEqual(pki.get_ext_config(None, None), b'')
+        self.assertEqual(pki.get_ext_config([], None), b'')
+        self.assertEqual(pki.get_ext_config(['proxy.py'], None), b'\nsubjectAltName=DNS:proxy.py')
+        self.assertEqual(pki.get_ext_config(None, 'serverAuth'), b'\nextendedKeyUsage=serverAuth')
+        self.assertEqual(pki.get_ext_config(['proxy.py'], 'serverAuth'),
+                         b'\nsubjectAltName=DNS:proxy.py\nextendedKeyUsage=serverAuth')
+        self.assertEqual(pki.get_ext_config(['proxy.py', 'www.proxy.py'], 'serverAuth'),
+                         b'\nsubjectAltName=DNS:proxy.py,DNS:www.proxy.py\nextendedKeyUsage=serverAuth')
+
+    def test_ssl_config_no_ext(self) -> None:
+        with pki.ssl_config() as (config_path, has_extension):
+            self.assertFalse(has_extension)
+            with open(config_path, 'rb') as config:
+                self.assertEqual(config.read(), pki.DEFAULT_CONFIG)
+
+    def test_ssl_config(self) -> None:
+        with pki.ssl_config(['proxy.py']) as (config_path, has_extension):
+            self.assertTrue(has_extension)
+            with open(config_path, 'rb') as config:
+                self.assertEqual(config.read(), pki.DEFAULT_CONFIG + b'\n[PROXY]\nsubjectAltName=DNS:proxy.py')
+
+    def test_extfile_no_ext(self) -> None:
+        with pki.ext_file() as config_path:
+            with open(config_path, 'rb') as config:
+                self.assertEqual(config.read(), b'')
+
+    def test_extfile(self) -> None:
+        with pki.ext_file(['proxy.py']) as config_path:
+            with open(config_path, 'rb') as config:
+                self.assertEqual(config.read(), b'\nsubjectAltName=DNS:proxy.py')
+
+    def test_gen_private_key(self) -> None:
+        pass
+
+    def test_gen_public_key(self) -> None:
+        pass
+
+    def test_gen_csr(self) -> None:
+        pass
+
+    def test_sign_csr(self) -> None:
+        pass