Package: securedrop-sdk
By contributing to this project, you agree to abide by ourCode of Conduct.Python SDK for SecureDropThis SDK provides a convenient Python interface to theSecureDrop Journalist Interface API. The development of the SDK was primarily motivated by the creation of theSecureDrop Workstationbased on Qubes OS.The SDK is currently used by theSecureDrop Clientthat is a component of the SecureDrop Workstation. When used in Qubes OS, the SDK uses thesecuredrop-proxyservice, as the VM which runs the client does not have network access by design.Quick Startmakevenvsource.venv/bin/activate
maketestWe cover all the API calls supported by the SecureDrop Journalist Interface API.To install the SDK into yourvirtualenvfor testing purposes:pip uninstall securedrop-sdk
pip install git+https://github.com/freedomofpress/securedrop-sdk@my_branch#egg=securedrop-sdkUpgrading one single dependencyTo upgrade one single Python dependency, sayrequests, run the following:PACKAGE=requestsmakeupgrade-pipThis will upgrade bothdev-requirementsand primaryrequirements.Running testsTo run all tests and checks, run:makecheckTo run all tests, run:maketestTo run all tests that make API calls over HTTP, run:maketestTESTS=tests/test_api.pyTo run all tests that make API calls over qrexec, run:maketestTESTS=tests/test_apiproxy.pyTo run a single test, specify file name, class name, and test name, e.g.:maketestTESTS=tests/test_api.py::TestAPI::test_get_sourcesCreating and updating testsWhen tests are run, they replay recorded API request and response data instead of making actual API calls to a server. This is why tests can pass even when there is no server running. If the server ever changes its API or you want to add new tests that make API calls, then you'll need to record new request and response data by following the steps outlined below.Note:We have a CI test that does not use the recorded API request and response data in order to make sure we are testing the latest changes to the SDK against the latest server API (seetest-against-latest-apiinhttps://github.com/freedomofpress/securedrop-sdk/blob/main/.circleci/config.yml).We usevcrpyto record and replay API calls made over HTTP and a decorator called@qrexec_datasaverto record and replay API calls made over qrexec. Each request made from a test and its response from the server is stored in a "cassette" in thedatadirectory. Tests replay these cassettes instead of making actual API calls to a server.If you run the tests and see the following vcrpy warning, then you'll need to re-record cassettes because none of the existing cassettes contain the expected API call and we don't allow existing cassettes to be overwritten:Can't overwrite existing cassette ('<path-to-cassette-for-a-functional-test>') in your current record mode ('once').The steps to generate new cassettes are split into two sections based on communication protocol:Generating cassettes for API calls over HTTPandGenerating cassettes for API calls over qrexec.Generating cassettes for API calls over HTTPStart the server in a docker container by running:NUM_SOURCES=5makedevDelete the cassettes you wish to regenerate or just delete all yaml files by running:rmdata/*.ymlIf you are only adding a new test and not modifying existing ones, you can
skip this step, but you still need to remove the authentication setup during
cassette generation. Otherwise you will get 403 errors for API endpoints that
require a valid token. Remove the setup cassette by running:rmdata/test-setup.yml(You can reinstate the unmodified version later.)Generate new cassettes that make API calls over HTTP by running:maketestTESTS=tests/test_api.pyNote: Some tests alter source and conversation data on the server so you may need to restart the server in between test runs.Generating cassettes for API calls over qrexecIn order to generate cassettes for tests that make API calls over qrexec, you'll need to run the server and proxy on a separate VM. If this is the first time you are generating cassettes, first follow the steps outlined in theTest setup for qrexec communicationsection, which will help you set up a new VM calledsd-dev-proxy.Once your proxy VM is set up, follow these steps:Start the server in a docker container onsd-dev-proxyby running:NUM_SOURCES=5makedevDelete the cassettes you wish to regenerate or just delete all JSON files by running:rmdata/*.jsonIf you are only adding a new test and not modifying existing ones, you can
skip this step, but you still need to remove the authentication setup during
cassette generation. Otherwise you will get 403 errors for API endpoints that
require a valid token. Remove the setup cassette by running:rmdata/setup_method.json(You can reinstate the unmodified version later.)Make qrexec calls to the server and collect response data. To run all proxy
tests:maketestTESTS=tests/test_apiproxy.pyNote:If you get a 403 error it may also be because the test is trying to
reuse an old TOTP code, so wait for 60 seconds and try again. Some tests alter
source and conversation data on the server so you should restart the server in
between test runs.Test setup for qrexec communicationIf this is the first time you are generating new cassettes that make API calls over qrexec, then you'll need to set up a new VM for running the server and proxy following these steps:Create a new AppVM based on thedebian-10template calledsd-dev-proxy.Install the lastest proxy package:wgethttps://apt.freedom.press/pool/main/s/securedrop-proxy/<latest-package>.deb
dpkg-i<latest-package>.debCreate/home/user/.securedrop_proxy/sd-proxy.yamlwith the following contents (assuming the VM you'll be running the SDK tests from is calledsd-dev):host: 127.0.0.1
scheme: http
port: 8081
target_vm: sd-dev
dev: FalseInstall Docker.Clonesecuredroponsd-dev-proxyand run the server in a Docker container:gitclonehttps://github.com/freedomofpress/securedropcdsecuredrop
virtualenv.venv--python=python3source.venv/bin/activate
pipinstall-rsecuredrop/requirements/python3/develop-requirements.txtNUM_SOURCES=5makedevOpen a terminal insd-devand create/etc/sd-sdk.confwith the following contents:[proxy]
name=sd-dev-proxyModify/etc/qubes-rpc/policy/securedrop.Proxyindom0by adding the following line to the top of the file so that the sdk tests can make calls to the proxy:sd-dev sd-dev-proxy allowNOTE:You may want to switch back to the RPC configuration files in their as-provisioned state before amake testrun indom0, as this and the following change to the RPC policies will break the strict validation of the RPC policies that is one of those tests.Modify/etc/qubes-rpc/policy/qubes.Filecopyindom0by adding the following line to the top of the file so that the proxy can send files over qrexec to the sdk:sd-dev-proxy sd-dev allowVerify qrexec communication betweensd-dev-proxyandsd-devis set up properly.a. Run the server onsd-dev-proxyif it isn't already running:NUM_SOURCES=5makedevb. With the main branch of this repo checked out onsd-dev, comment out the@qrexec_datasaverdecorator above thetest_apiproxy.py::TestAPIProxy::setUpmethod so thattest_api_authmakes an actual API call over qrexec.
c. Runtest_api_auth:maketestTESTS=tests/test_apiproxy.py::TestAPIProxy::test_api_authNote:If the test fails, runjournalctl -findom0before trying again to see if communication betweensd-devandsd-dev-proxyis being denied. A successful log looks like this:Aug 28 15:45:13 dom0 qrexec[1474]: securedrop.Proxy: sd-dev -> sd-dev-proxy: allowed to sd-dev-proxyReleasingTo make a release, you should:Create a branch namedrelease/$new_version_numberUpdateCHANGELOG.mdandsetup.pyCommit the changes.Create a PR and get the PR reviewed and merged intomain.git tag $new_version_numberand push the new tag.Checkout the new tag locally.Push the new release source tarball to the PSF's PyPIfollowing this documentation. Do not upload the wheel (by deleting it from yourdist/directory prior to upload).If you want to publish the new SDK release to the FPF PyPI mirror, Hop over to the thesecuredrop-debian-packagingrepo and follow thebuild-a-packageinstructions to push the package up to our PyPI mirror:https://pypi.org/simpleContributingPlease readCONTRIBUTING.mdfor details on our code of conduct, and the process for submitting pull requests to us.VersioningWe useSemVerfor versioning. For the versions available, see thetags on this repository.LicenseThe Python SecureDrop SDK is licensed in the GPLv3. SeeLICENSEfor more details.