Project: APISIX (Orchestration & Management / API Gateway)
File: testing-framework.md

title: Introducing APISIX's testing framework <!-- --> APISIX uses a testing framework based on test-nginx: https://github.com/openresty/test-nginx. For details, you can check the of this project. If you want to test the CLI behavior of APISIX (`./bin/apisix`), you need to write a shell script in the t/cli directory to test it. You can refer to the existing test scripts for more details. If you want to test the others, you need to write test code based on the framework. Here, we briefly describe how to do simple testing based on this framework. you need to write test cases in the t/ directory, in a corresponding `.t` file. Note that a single test file should not exceed `800` lines, and if it is too long, it needs to be divided by a suffix. For example: ``` t/ admin consumers.t consumers2.t ``` Both `consumers.t` and `consumers2.t` contain tests for consumers in the Admin API. Some of the test files start with this paragraph: ``` addblockpreprocessor(sub { my ($block) = @_; if (! $block->request) { $block->set_value("request", "GET /t"); } if (! $block->noerrorlog && ! $block->error_log) { $block->setvalue("noerror_log", "[error]\n[alert]"); } }); ``` It means that all tests in this test file that do not define `request` are set to `GET /t`. The same is true for error_log. When testing a behavior, we need to prepare the configuration. If the configuration is from etcd: We can set up specific configurations through the Admin API. ``` === TEST 7: refer to empty nodes upstream config location /t { contentbylua_block { local core = require("apisix.core") local t = require("lib.test_admin").test local code, message = t('/apisix/admin/routes/1', ngx.HTTP_PUT, [[{ "methods": ["GET"], "upstream_id": "1", "uri": "/index.html" }]] ) if code >= 300 then ngx.status = code ngx.print(message) return end ngx.say(message) } } request GET /t response_body passed ``` Then trigger it in a later test: ``` === TEST 8: hit empty nodes upstream request GET /index.html error_code: 503 error_log no valid upstream node ``` To test the code, we need to provide a mock upstream. For HTTP request, the upstream code is put in `t/lib/server.lua`. HTTP request with a given `path` will trigger the method in the same name. For example, a call to `/server_port` will call the `M.serverport`. For TCP request, a dummy upstream is used: ``` local sock = ngx.req.socket() local data = sock:receive("1") ngx.say("hello world") ``` If you want to custom the TCP upstream logic, you can use: ``` streamupstreamcode local sock = ngx.req.socket() local data = sock:receive("1") ngx.sleep(0.2) ngx.say("hello world") ``` We can initiate a request with `request` and set the request headers with `more_headers`. For example. ``` request PUT /hello?xx=y&xx=z&&y=&&z body part of the request more_headers X-Req: foo X-Req: bar X-Resp: cat ``` Lua code can be used to send multiple requests. One request after another: ``` config location /t { contentbylua_block { local http = require "resty.http" local uri = "http://127.0.0.1:" .. ngx.var.server_port .. "/server_port" local ports_count = {} for i = 1, 12 do local httpc = http.new() local res, err = httpc:request_uri(uri, {method = "GET"}) if not res then ngx.say(err) return end portscount[res.body] = (portscount[res.body] or 0) + 1 end } } ``` Sending multiple requests concurrently: ``` config location /t { contentbylua_block { local http = require "resty.http" local uri = "http://127.0.0.1:" .. ngx.var.server_port .. "/server_port?var=2&var2=" local t = {} local ports_count = {} for i = 1, 180 do local th = assert(ngx.thread.spawn(function(i) local httpc = http.new() local res, err = httpc:request_uri(uri..i, {method = "GET"}) if not res then ngx.log(ngx.ERR, err) return end portscount[res.body] = (portscount[res.body] or 0) + 1 end, i))

th) end for i, th in ipairs(t) do ngx.thread.wait(th) end } } ``` We can use `stream_request` to send a TCP request, for example: ``` stream_request hello ``` To send a TLS over TCP request, we can combine `streamtlsrequest` with `stream_sni`: ``` streamtlsrequest mmm stream_sni: xx.com ``` The following assertions are commonly used. Check status (if not set, the framework will check if the request has 200 status code). ``` error_code: 405 ``` Check response headers. ``` response_headers X-Resp: foo X-Req: foo, bar ``` Check response body. ``` response_body [{"count":12, "port": "1982"}] ``` Check the TCP response. When the request is sent via `stream_request`: ``` stream_response receive stream response error: connection reset by peer ``` When the request is sent via `streamtlsrequest`: ``` response_body receive stream response error: connection reset by peer ``` Checking the error log (via grep error log with regular expression). ``` greperrorlog eval qr/hashon: header|chashkey: "custom-one"/ greperrorlog_out hash_on: header chash_key: "custom-one" hash_on: header chash_key: "custom-one" hash_on: header chash_key: "custom-one" hash_on: header chash_key: "custom-one" ``` The default log level is `info`, but you can get the debug level log with ` log_level: debug`. The test framework listens to multiple ports when it is started. 1980/1981/1982/5044: HTTP upstream port. We provide a mock upstream server for testing. See below for more information. 1983: HTTPS upstream port 1984: APISIX HTTP port. Can be used to verify HTTP related gateway logic, such as concurrent access to an API. 1985: APISIX TCP port. Can be used to verify TCP related gateway logic, such as concurrent access to an API. 1994: APISIX HTTPS port. Can be used to verify HTTPS related gateway logic, such as testing certificate matching logic. 1995: TCP upstream port 2005: APISIX TLS over TCP port. Can be used to verify TLS over TCP related gateway logic, such as concurrent access to an API. The methods in `t/lib/server.lua` are executed when accessing the upstream port. `_M.go` is the entry point for this file. When the request accesses the upstream `/xxx`, the `M.xxx` method is executed. For example, a request for `/hello` will execute `M.hello`. This allows us to write methods inside `t/lib/server.lua` to emulate specific upstream logic, such as sending special responses. Note that before adding new methods to `t/lib/server.lua`, make sure that you can reuse existing methods. Assume your current work directory is the root of the apisix source code. Git clone the latest to `../test-nginx`. Run the test: `prove -I. -I../test-nginx/inc -I../test-nginx/lib -r t/path/to/file.t`. The Nginx configuration and logs generated by the test cases are located in the t/servroot directory. The Nginx configuration template for testing is located in t/APISIX.pm. Three notes can be used to control which parts of the tests are executed. FIRST & LAST: ``` === TEST 1: vars rule with ! (set) FIRST config ... response_body passed === TEST 2: vars rule with ! (hit) request GET /hello?name=jack&age=17 LAST error_code: 403 response_body Fault Injection! ``` ONLY: ``` === TEST 1: list empty resources ONLY config ... response_body {"count":0,"node":{"dir":true,"key":"/apisix/upstreams","nodes":[]}} ``` It is possible to execute shell commands while writing tests in test-nginx for APISIX. We expose this feature via `exec` code block. The `stdout` of the executed process can be captured via `responsebody` code block and `stderr` (if any) can be captured by filtering error.log through `greperror_log`. Here is an example: ``` === TEST 1: check exec stdout exec echo hello world response_body hello world === TEST 2: when exec returns an error exec echxo hello world greperrorlog eval qr/failed to execute the script [ -~]*/ greperrorlog_out failed to execute the script with status: 127, reason: exit, stderr: /bin/sh: 1: echxo: not found ```