Repository: developmentseed/starlette-cramjam (language: python)

Issue:
Use cramjam Compressor
Let me know if you find anything else amiss! :pray: 

For some reason tests failed locally via `tox` but are fine with `pytest` directly, I must be doing something wrong with tox.


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
new file mode 100644
index 0000000..8dd3e2e
--- /dev/null
+++ b/.github/workflows/ci.yml
@@ -0,0 +1,81 @@
+name: CI
+
+# On every pull request, but only on push to master
+on:
+  push:
+    branches:
+    - master
+    tags:
+    - '*'
+  pull_request:
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: [3.6, 3.7, 3.8, 3.9]
+
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install tox codecov pre-commit
+
+      # Run tox using the version of Python in `PATH`
+      - name: Run Tox
+        run: tox -e py
+
+      # Run pre-commit (only for python-3.8)
+      - name: run pre-commit
+        if: matrix.python-version == 3.8
+        run: pre-commit run --all-files
+
+      - name: Upload Results
+        if: success()
+        uses: codecov/codecov-action@v1
+        with:
+          file: ./coverage.xml
+          flags: unittests
+          name: ${{ matrix.platform }}-${{ matrix.tox-env }}
+          fail_ci_if_error: false
+
+  publish:
+    needs: [tests]
+    runs-on: ubuntu-latest
+    if: startsWith(github.event.ref, 'refs/tags') || github.event_name == 'release'
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python
+        uses: actions/setup-python@v1
+        with:
+          python-version: "3.x"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install tox
+
+      - name: Set tag version
+        id: tag
+        # https://stackoverflow.com/questions/58177786/get-the-current-pushed-tag-in-github-actions
+        run: echo ::set-output name=tag::${GITHUB_REF#refs/*/}
+
+      - name: Set module version
+        id: module
+        # https://stackoverflow.com/questions/58177786/get-the-current-pushed-tag-in-github-actions
+        run: echo ::set-output name=version::$(python setup.py --version)
+
+      - name: Build and publish
+        if: steps.tag.outputs.tag == steps.module.outputs.version
+        env:
+          TOXENV: release
+          TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
+          TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+        run: tox
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
new file mode 100644
index 0000000..bb30fa2
--- /dev/null
+++ b/.pre-commit-config.yaml
@@ -0,0 +1,24 @@
+repos:
+  - repo: https://github.com/psf/black
+    rev: 19.10b0
+    hooks:
+      - id: black
+        language_version: python
+
+  - repo: https://github.com/PyCQA/isort
+    rev: 5.4.2
+    hooks:
+      - id: isort
+        language_version: python
+
+  - repo: https://github.com/PyCQA/flake8
+    rev: 3.8.3
+    hooks:
+      - id: flake8
+        language_version: python
+
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v0.812
+    hooks:
+      - id: mypy
+        language_version: python
diff --git a/CHANGES.md b/CHANGES.md
new file mode 100644
index 0000000..1c3f846
--- /dev/null
+++ b/CHANGES.md
@@ -0,0 +1,3 @@
+## 0.1.0 (TBD)
+
+- Initial release.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 0000000..4b3d7dd
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,19 @@
+# Contributing
+
+Issues and pull requests are more than welcome.
+
+### dev install
+
+```bash
+$ git clone https://github.com/developmentseed/starlette-cramjam.git
+$ cd starlette-cramjam
+$ pip install -e .[dev]
+```
+
+### pre-commit
+
+This repo is set to use `pre-commit` to run *isort*, *flake8*, *pydocstring*, *black* ("uncompromising Python code formatter") and mypy when committing new code.
+
+```bash
+$ pre-commit install
+```
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..c7584c0
--- /dev/null
+++ b/README.md
@@ -0,0 +1,123 @@
+# starlette-cramjam
+
+<p align="center">
+  <em>Cramjam integration for Starlette ASGI framework.</em>
+</p>
+<p align="center">
+  <a href="https://github.com/developmentseed/starlette-cramjam/actions?query=workflow%3ACI" target="_blank">
+      <img src="https://github.com/developmentseed/starlette-cramjam/workflows/CI/badge.svg" alt="Test">
+  </a>
+  <a href="https://codecov.io/gh/developmentseed/starlette-cramjam" target="_blank">
+      <img src="https://codecov.io/gh/developmentseed/starlette-cramjam/branch/master/graph/badge.svg" alt="Coverage">
+  </a>
+  <a href="https://pypi.org/project/starlette-cramjam" target="_blank">
+      <img src="https://img.shields.io/pypi/v/starlette-cramjam?color=%2334D058&label=pypi%20package" alt="Package version">
+  </a>
+  <a href="https://pypistats.org/packages/starlette-cramjam" target="_blank">
+      <img src="https://img.shields.io/pypi/dm/starlette-cramjam.svg" alt="Downloads">
+  </a>
+  <a href="https://github.com/developmentseed/starlette-cramjam/blob/master/LICENSE" target="_blank">
+      <img src="https://img.shields.io/github/license/developmentseed/starlette-cramjam.svg" alt="Downloads">
+  </a>
+</p>
+
+---
+
+**Source Code**: <a href="https://github.com/developmentseed/starlette-cramjam" target="_blank">https://github.com/developmentseed/starlette-cramjam</a>
+
+---
+
+The `starlette-cramjam` middleware aims to provide a unique Compression middleware to support **Brotli**, **GZip** and **Deflate** compression algorithms with a minimal requirement.
+
+The middleware will compress responses for any request that includes "br", "gzip" or "deflate" in the Accept-Encoding header.
+
+As for the official `Starlette` middleware, the one provided by `starlette-cramjam will handle both standard and streaming responses.
+
+## Installation
+
+You can install `starlette-cramjam` from pypi
+
+```python
+$ pip install -U pip
+$ pip install starlette-cramjam
+```
+
+or install from source:
+
+```bash
+$ pip install -U pip
+$ pip install https://github.com/developmentseed/starlette-cramjam.git
+```
+
+## Usage
+
+The following arguments are supported:
+
+- **minimum_size** (Integer) - Do not compress responses that are smaller than this minimum size in bytes. Defaults to `500`.
+- **exclude_path** (Set of string) - Do not compress responses in response to specific `path` requests. Entries have to be valid regex expressions. Defaults to `{}`.
+- **exclude_mediatype** (Set of string) - Do not compress responses of specific media type (e.g `image/png`). Defaults to `{}`.
+
+#### Minimal (defaults) example
+
+```python
+import uvicorn
+
+from starlette.applications import Starlette
+from starlette.responses import PlainTextResponse
+
+from starlette_cramjam.middleware import CompressionMiddleware
+
+# create application
+app = Starlette()
+
+# register the CompressionMiddleware
+app.add_middleware(CompressionMiddleware)
+
+
+@app.route("/")
+def index(request):
+    return PlainTextResponse("Hello World")
+
+
+if __name__ == "__main__":
+    uvicorn.run(app, host="0.0.0.0", port=8000)
+```
+
+#### Using options
+
+```python
+import uvicorn
+
+from starlette.applications import Starlette
+from starlette.responses import PlainTextResponse, Response
+
+from starlette_cramjam.middleware import CompressionMiddleware
+
+# create application
+app = Starlette()
+
+# register the CompressionMiddleware
+app.add_middleware(
+    CompressionMiddleware,
+    minimum_size=0,  # should compress everything
+    exclude_path={"^/foo$"},  # do not compress response for the `/foo` request
+    exclude_mediatype={"image/jpeg"},  # do not compress jpeg
+)
+
+
+@app.route("/")
+def index(request):
+    return PlainTextResponse("Hello World")
+
+@app.route("/image")
+def foo(request):
+    return Response(b"This is a fake body", status_code=200, media_type="image/jpeg")
+
+@app.route("/foo")
+def foo(request):
+    return PlainTextResponse("Do not compress me.")
+
+
+if __name__ == "__main__":
+    uvicorn.run(app, host="0.0.0.0", port=8000)
+```
diff --git a/codecov.yml b/codecov.yml
new file mode 100644
index 0000000..c61977e
--- /dev/null
+++ b/codecov.yml
@@ -0,0 +1,8 @@
+comment: off
+
+coverage:
+    status:
+        project:
+            default:
+                target: auto
+                threshold: 5
diff --git a/requirements-dev.txt b/requirements-dev.txt
new file mode 100644
index 0000000..eaf5907
--- /dev/null
+++ b/requirements-dev.txt
@@ -0,0 +1,4 @@
+pytest
+pytest-cov
+pre-commit
+brotlipy
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..7fc48b3
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,2 @@
+starlette
+cramjam==2.4.0-rc2
diff --git a/setup.cfg b/setup.cfg
new file mode 100644
index 0000000..7de992b
--- /dev/null
+++ b/setup.cfg
@@ -0,0 +1,28 @@
+[metadata]
+version = attr:starlette_cramjam.version.__version__
+
+[bumpversion]
+current_version = 0.1.0
+commit = True
+tag = True
+tag_name = {new_version}
+
+[bumpversion:file:starlette_cramjam/version.py]
+search = __version__ = "{current_version}"
+replace = __version__ = "{new_version}"
+
+[isort]
+profile = black
+known_first_party = starlette_cramjam
+forced_separate = starlette
+
+[flake8]
+ignore = E501,W503,E203
+exclude = .git,__pycache__,docs/source/conf.py,old,build,dist
+max-complexity = 12
+max-line-length = 90
+
+[mypy]
+no_strict_optional = True
+ignore_missing_imports = True
+
diff --git a/setup.py b/setup.py
new file mode 100644
index 0000000..f7b07d5
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,42 @@
+"""starlette_cramjam module."""
+
+from setuptools import find_packages, setup
+
+with open("README.md") as f:
+    readme = f.read()
+
+# Runtime Requirements.
+inst_reqs = ["starlette", "cramjam>=2.4.0-rc2,<2.5"]
+
+# Extra Requirements
+extra_reqs = {
+    "test": ["pytest", "pytest-cov", "requests", "brotlipy"],
+    "dev": ["pytest", "pytest-cov", "requests", "brotlipy", "pre-commit"],
+}
+
+
+setup(
+    name="starlette_cramjam",
+    description=u"Cramjam integration for Starlette ASGI framework.",
+    long_description=readme,
+    long_description_content_type="text/markdown",
+    python_requires=">=3",
+    classifiers=[
+        "Intended Audience :: Information Technology",
+        "Intended Audience :: Science/Research",
+        "License :: OSI Approved :: BSD License",
+        "Programming Language :: Python :: 3.6",
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
+        "Programming Language :: Python :: 3.9",
+    ],
+    keywords="Cramjam compression ASGI Starlette",
+    author="Vincent Sarago",
+    author_email="vincent@developmentseed.org",
+    url="https://github.com/developmentseed/starlette-cramjam",
+    packages=find_packages(exclude=["tests"]),
+    include_package_data=True,
+    zip_safe=False,
+    install_requires=inst_reqs,
+    extras_require=extra_reqs,
+)
diff --git a/starlette_cramjam/__init__.py b/starlette_cramjam/__init__.py
new file mode 100644
index 0000000..80df3cf
--- /dev/null
+++ b/starlette_cramjam/__init__.py
@@ -0,0 +1,1 @@
+"""starlette_cramjam."""
diff --git a/starlette_cramjam/middleware.py b/starlette_cramjam/middleware.py
new file mode 100644
index 0000000..d27d3c3
--- /dev/null
+++ b/starlette_cramjam/middleware.py
@@ -0,0 +1,172 @@
+"""starlette_cramjam.middleware."""
+
+import re
+from typing import Any, Optional, Set
+
+import cramjam
+
+from starlette.datastructures import Headers, MutableHeaders
+from starlette.types import ASGIApp, Message, Receive, Scope, Send
+
+
+class CompressionMiddleware:
+    def __init__(
+        self,
+        app: ASGIApp,
+        minimum_size: int = 500,
+        exclude_path: Optional[Set[str]] = None,
+        exclude_mediatype: Optional[Set[str]] = None,
+    ) -> None:
+        """Init CompressionMiddleware.
+
+        Args:
+            app (ASGIApp): starlette/FastAPI application.
+            minimum_size: Minimal size, in bytes, for appliying compression. Defaults to 500.
+            exclude_path (set): Set of regex expression to use to exclude compression for request path. Defaults to {}.
+            exclude_mediatype (set): Set of media-type for which to exclude compression. Defaults to {}.
+
+        """
+        self.app = app
+        self.minimum_size = minimum_size
+        self.exclude_path = {re.compile(p) for p in exclude_path or set()}
+        self.exclude_mediatype = exclude_mediatype or set()
+
+    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+        if scope["type"] == "http":
+            headers = Headers(scope=scope)
+            accepted_encoding = headers.get("Accept-Encoding", "")
+
+            if self.exclude_path:
+                skip = any([x.fullmatch(scope["path"]) for x in self.exclude_path])
+            else:
+                skip = False
+
+            if not skip and "br" in accepted_encoding:
+                responder = CompressionResponder(
+                    self.app,
+                    cramjam.brotli.Compressor(),
+                    "br",
+                    self.minimum_size,
+                    self.exclude_mediatype,
+                )
+                await responder(scope, receive, send)
+                return
+
+            elif not skip and "gzip" in accepted_encoding:
+                responder = CompressionResponder(
+                    self.app,
+                    cramjam.gzip.Compressor(),
+                    "gzip",
+                    self.minimum_size,
+                    self.exclude_mediatype,
+                )
+                await responder(scope, receive, send)
+                return
+
+            elif not skip and "deflate" in accepted_encoding:
+                responder = CompressionResponder(
+                    self.app,
+                    cramjam.deflate.Compressor(),
+                    "deflate",
+                    self.minimum_size,
+                    self.exclude_mediatype,
+                )
+                await responder(scope, receive, send)
+                return
+
+        await self.app(scope, receive, send)
+
+
+class CompressionResponder:
+    def __init__(
+        self,
+        app: ASGIApp,
+        compressor: Any,
+        encoding_name: str,
+        minimum_size: int,
+        exclude_mediatype: Set[str],
+    ) -> None:
+
+        self.app = app
+        self.compressor = compressor
+        self.encoding_name = encoding_name
+        self.minimum_size = minimum_size
+        self.exclude_mediatype = exclude_mediatype
+        self.send = unattached_send  # type: Send
+        self.initial_message = {}  # type: Message
+        self.started = False
+
+    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+        self.send = send
+        await self.app(scope, receive, self.send_with_compression)
+
+    async def send_with_compression(self, message: Message) -> None:
+
+        message_type = message["type"]
+
+        if message_type == "http.response.start":
+            # Don't send the initial message until we've determined how to
+            # modify the outgoing headers correctly.
+            self.initial_message = message
+
+        elif message_type == "http.response.body" and not self.started:
+            self.started = True
+            body = message.get("body", b"")
+            more_body = message.get("more_body", False)
+
+            headers = MutableHeaders(raw=self.initial_message["headers"])
+
+            if headers.get("Content-Type") in self.exclude_mediatype:
+                # Don't apply compression if mediatype should be excluded
+                await self.send(self.initial_message)
+                await self.send(message)
+
+            elif len(body) < self.minimum_size and not more_body:
+                # Don't apply compression to small outgoing responses.
+                await self.send(self.initial_message)
+                await self.send(message)
+
+            elif not more_body:
+                # Standard compressed response.
+                self.compressor.compress(body)
+                body = bytes(self.compressor.finish())
+
+                headers["Content-Encoding"] = self.encoding_name
+                headers["Content-Length"] = str(len(body))
+                headers.add_vary_header("Accept-Encoding")
+                message["body"] = body
+
+                await self.send(self.initial_message)
+                await self.send(message)
+
+            else:
+                # Initial body in streaming compressed response.
+                headers["Content-Encoding"] = self.encoding_name
+                headers.add_vary_header("Accept-Encoding")
+
+                # https://gist.github.com/CMCDragonkai/6bfade6431e9ffb7fe88#content-length
+                # Content-Length header will not allow streaming
+                del headers["Content-Length"]
+                self.compressor.compress(body)
+                message["body"] = bytes(self.compressor.flush())
+
+                await self.send(self.initial_message)
+                await self.send(message)
+
+        elif message_type == "http.response.body":
+            # Remaining body in streaming compressed response.
+            body = message.get("body", b"")
+            more_body = message.get("more_body", False)
+
+            self.compressor.compress(body)
+            if not more_body:
+                message["body"] = bytes(self.compressor.finish())
+                await self.send(message)
+                return
+            message["body"] = bytes(self.compressor.flush())
+
+            await self.send(message)
+
+
+async def unattached_send(message: Message) -> None:
+    raise RuntimeError("send awaitable not set")  # pragma: no cover
diff --git a/starlette_cramjam/version.py b/starlette_cramjam/version.py
new file mode 100644
index 0000000..4e54c27
--- /dev/null
+++ b/starlette_cramjam/version.py
@@ -0,0 +1,3 @@
+"""starlette-cramjam version."""
+
+__version__ = "0.1.0"
diff --git a/tox.ini b/tox.ini
new file mode 100644
index 0000000..6f8acc2
--- /dev/null
+++ b/tox.ini
@@ -0,0 +1,30 @@
+[tox]
+envlist = py36,py37,py38,py39
+
+[testenv]
+extras = test
+commands=
+    python -m pytest --cov starlette_cramjam --cov-report xml --cov-report term-missing --ignore=venv
+
+# Release tooling
+[testenv:build]
+basepython = python3
+skip_install = true
+deps =
+    wheel
+    setuptools
+commands =
+    python setup.py sdist
+
+[testenv:release]
+basepython = python3
+skip_install = true
+setenv =
+    TWINE_USERNAME = {env:TWINE_USERNAME}
+    TWINE_PASSWORD = {env:TWINE_PASSWORD}
+deps =
+    {[testenv:build]deps}
+    twine >= 1.5.0
+commands =
+    {[testenv:build]commands}
+    twine upload --skip-existing dist/*


Test patch:
diff --git a/tests/__init__.py b/tests/__init__.py
new file mode 100644
index 0000000..17f4d06
--- /dev/null
+++ b/tests/__init__.py
@@ -0,0 +1,1 @@
+"""starlette_cramjam tests suite."""
diff --git a/tests/test_middlewares.py b/tests/test_middlewares.py
new file mode 100644
index 0000000..7220222
--- /dev/null
+++ b/tests/test_middlewares.py
@@ -0,0 +1,142 @@
+"""Main test for cramjam middleware.
+
+This tests are the same as the ones from starlette.tests.middleware.test_gzip but using multiple encoding.
+
+"""
+import pytest
+
+from starlette.applications import Starlette
+from starlette.responses import PlainTextResponse, Response, StreamingResponse
+from starlette.testclient import TestClient
+from starlette_cramjam.middleware import CompressionMiddleware
+
+
+@pytest.mark.parametrize("method", ["br", "gzip", "deflate"])
+def test_compressed_responses(method):
+    app = Starlette()
+
+    app.add_middleware(CompressionMiddleware)
+
+    @app.route("/")
+    def homepage(request):
+        return PlainTextResponse("x" * 4000, status_code=200)
+
+    client = TestClient(app)
+    response = client.get("/", headers={"accept-encoding": method})
+    assert response.status_code == 200
+    assert response.text == "x" * 4000
+    assert response.headers["Content-Encoding"] == method
+    assert int(response.headers["Content-Length"]) < 4000
+
+
+@pytest.mark.parametrize("method", ["br", "gzip", "deflate"])
+def test_streaming_response(method):
+    app = Starlette()
+    app.add_middleware(CompressionMiddleware, minimum_size=1)
+
+    @app.route("/")
+    def homepage(request):
+        async def generator(bytes, count):
+            for _ in range(count):
+                yield bytes
+
+        streaming = generator(bytes=b"x" * 400, count=10)
+        return StreamingResponse(streaming, status_code=200)
+
+    client = TestClient(app)
+
+    response = client.get("/", headers={"accept-encoding": method})
+    assert response.status_code == 200
+    assert response.headers["Content-Encoding"] == method
+    assert "Content-Length" not in response.headers
+    assert response.text == "x" * 4000
+
+
+def test_not_in_accept_encoding():
+    app = Starlette()
+
+    app.add_middleware(CompressionMiddleware)
+
+    @app.route("/")
+    def homepage(request):
+        return PlainTextResponse("x" * 4000, status_code=200)
+
+    client = TestClient(app)
+    response = client.get("/", headers={"accept-encoding": "identity"})
+    assert response.status_code == 200
+    assert response.text == "x" * 4000
+    assert "Content-Encoding" not in response.headers
+    assert int(response.headers["Content-Length"]) == 4000
+
+
+def test_ignored_for_small_responses():
+    app = Starlette()
+
+    app.add_middleware(CompressionMiddleware)
+
+    @app.route("/")
+    def homepage(request):
+        return PlainTextResponse("OK", status_code=200)
+
+    client = TestClient(app)
+    response = client.get("/", headers={"accept-encoding": "gzip"})
+    assert response.status_code == 200
+    assert response.text == "OK"
+    assert "Content-Encoding" not in response.headers
+    assert int(response.headers["Content-Length"]) == 2
+
+
+@pytest.mark.parametrize("method", ["br", "gzip", "deflate"])
+def test_compressed_skip_on_content_type(method):
+    app = Starlette()
+
+    app.add_middleware(CompressionMiddleware, exclude_mediatype={"image/png"})
+
+    @app.route("/")
+    def homepage(request):
+        return Response(b"foo" * 1000, status_code=200, media_type="image/png")
+
+    @app.route("/foo")
+    def foo(request):
+        return Response(b"foo" * 1000, status_code=200, media_type="image/jpeg")
+
+    client = TestClient(app)
+    response = client.get("/", headers={"accept-encoding": method})
+    assert response.status_code == 200
+    assert "Content-Encoding" not in response.headers
+
+    response = client.get("/foo", headers={"accept-encoding": method})
+    assert response.status_code == 200
+    assert response.headers["Content-Encoding"] == method
+
+
+@pytest.mark.parametrize("method", ["br", "gzip", "deflate"])
+def test_compressed_skip_on_path(method):
+    app = Starlette()
+
+    app.add_middleware(CompressionMiddleware, exclude_path={"^/f.+"}, minimum_size=0)
+
+    @app.route("/")
+    def homepage(request):
+        return PlainTextResponse("yep", status_code=200)
+
+    @app.route("/foo")
+    def foo(request):
+        return Response("also yep but with /foo", status_code=200)
+
+    @app.route("/dontskip/foo")
+    def foo2(request):
+        return Response("also yep but with /dontskip/foo", status_code=200)
+
+    client = TestClient(app)
+    response = client.get("/", headers={"accept-encoding": method})
+    assert response.status_code == 200
+    assert response.headers["Content-Encoding"] == method
+
+    response = client.get("/dontskip/foo", headers={"accept-encoding": method})
+    assert response.status_code == 200
+    assert response.headers["Content-Encoding"] == method
+
+    response = client.get("/foo", headers={"accept-encoding": method})
+    assert response.status_code == 200
+    assert "Content-Encoding" not in response.headers