Project: userver (App Definition and Development / Streaming & Messaging)
File: chaos_testing.md

A system could not achieve high availability if it does not tolerate failures. In microservices and other distributed network architectures the hardware and network itself is the source of failures. Luckily userver provides a network proxy to simulate network errors and whose behavior could be controlled from @ref scripts/docs/en/userver/functional_testing.md "functional tests". In other words, it is a tool to do chaos testing. To implement the tests, the pytestuserver.chaos.TcpGate and pytestuserver.chaos.UdpGate are placed between client and server. After that, the proxy could be used to test the client and the server fault tolerance. The following network conditions could be simulated: | Network condition | Possible reasons for the behavior in production | |--|--| | Server doesn't accept new connections | Server is overloaded; network misconfigured | | Client/server does not read data from socket | Client/server is overloaded; deadlock in client/server thread; OS lost the data on socket due to overload | | Client/server reads data, but does not respond | Client/server is overloaded; deadlock in client/server thread | | Client/server closes connections | Client/server restart; client/server overloaded and shutdowns connections | | Client/server closes the socket when receives data | Client/server congestion control in action | | Client/server corrupts the response data | Stack overflow or other memory corruption at client/server side | | Network works with send/receive delays | Poorly performing network; interference in the network | | Limited bandwidth | Poorly performing network; | | Connections close on timeout | Timeout on client/server/network interactions from client/server provider | | Data slicing | Interference in the network; poor routers configuration | Keep in mind the following picture: For client requests: ``` Client -> chaos.TcpGate -> Server ``` For server responses: ``` Client <- chaos.TcpGate <- Server ``` Gates allows you to simulate various network conditions on `client` and `server` parts

For example, if you wish to corrupt data from Client to Server, you should use `gate.toservercorrupt_data()`. If you wish to corrupt data that goes from Server to Client, you should use `gate.toclientcorrupt_data()`. To restore the non failure behavior use `gate.toclientpass()` and `gate.toserverpass()` respectively. Use @ref pytest_userver.chaos.TcpGate "chaos.TcpGate" when you need to deal with messages over TCP protocol (for example HTTP). This gate supports multiple connections from different clients to one server. TcpGate accepts incoming connections which trigger swapning of new clients connected to server. Use @ref pytest_userver.chaos.UdpGate "chaos.UdpGate" when you work with UDP protocol (for example DNS). UdpGate supports data providing between only one client and one server. If you wish to connect more than one Client to Server, please use different instance of UdpGate for every client. Import `chaos` to use @ref pytest_userver.chaos.TcpGate "chaos.TcpGate": @code{.py} from pytest_userver import chaos @endcode The proxy usually should be started before the service and should keep working between the service tests runs. To achieve that make a fixture with the session scope and add it to the service fixture dependencies. An example how to setup the proxy to stand between the service and PostgreSQL database: @snippet postgresql/functionaltests/basicchaos/conftest.py gate start The proxy should be reset to the default state between the test runs, to the state when connections are accepted and the data pass from client to server and from server to client: @snippet postgresql/functionaltests/basicchaos/conftest.py gate fixture After that, everything is ready to write chaos tests: @snippet postgresql/functionaltests/basicchaos/tests/test_postgres.py test cc Note that the most important part of the test is to make sure that the service restores functionality after the failures. Touch the dead connections and make sure that the service restores: @snippet postgresql/functionaltests/basicchaos/utils.py consumedeaddb_connections @snippet postgresql/functionaltests/basicchaos/tests/test_postgres.py restore @ref postgresql/functionaltests/basicchaos/tests/test_postgres.py @ref postgresql/functionaltests/basicchaos/conftest.py @ref postgresql/functionaltests/basicchaos/utils.py @ref postgresql/functionaltests/basicchaos/postgres_service.cpp @ref postgresql/functionaltests/basicchaos/static_config.yaml @ref postgresql/functionaltests/basicchaos/schemas/postgresql/key_value.sql @ref postgresql/functionaltests/basicchaos/CMakeLists.txt @ref testsuite/pytestplugins/pytestuserver/chaos.py @htmlonly <div class="bottom-nav"> @endhtmlonly @ref scripts/docs/en/userver/functionaltesting.md | @ref scripts/docs/en/userver/profilecontext_switches.md @htmlonly </div> @endhtmlonly @example postgresql/functionaltests/basicchaos/conftest.py @example postgresql/functionaltests/basicchaos/utils.py @example postgresql/functionaltests/basicchaos/tests/test_postgres.py @example postgresql/functionaltests/basicchaos/postgres_service.cpp @example postgresql/functionaltests/basicchaos/static_config.yaml @example postgresql/functionaltests/basicchaos/schemas/postgresql/key_value.sql @example postgresql/functionaltests/basicchaos/CMakeLists.txt @example testsuite/pytestplugins/pytestuserver/chaos.py