apache/trafficserver — pull request #141 by duderino (2014-11-11 20:34:54+00:00)
Title: Add initial python-based functional test framework with example test

# Apache Traffic Server Functional Test Framework

The goal of the functional test framework is to allow traffic server contributors to develop and exercise a reasonable functional test framework from the comfort of their development environments. This encourages traffic server developers to write their own functional test cases (it's not the province of a detached QA team). This also gives traffic server developers critical test feedback as early in the development process as possible.

The framework consists of a 'process manager' responsible for configuring and spawning 1+ traffic server processes, a configuration-driven reusable origin process so you don't have to write your own for every test case (though sometimes you will), and a small number of utilities for the test cases themselves. In this context the test cases are traffic server HTTP/SPDY clients. That is:

* test cases/clients --> traffic server --> (configurable/reusable origin | ad hoc origin)

## Framework Prerequisites

* Python 2.6+ (the default python on RHEL, CentOS, etc. 6.5+)
* [Twisted](https://twistedmatrix.com/trac/) 14.0.2+
1. [Install pip](https://pip.pypa.io/en/latest/installing.html)
2. Install twisted and its dependencies using pip:
```
$ sudo pip install --upgrade pyOpenSSL zope.interface twisted
```

Individual test cases may add additional dependencies.

## The ProcessManager class:

The ProcessManager class can be found in tests/framework/atstf.py. It is responsible for configuring, starting, and stopping the processes used by a test suite.

### ProcessManager Configuration

The ProcessManager is driven by a JSON config file that describes all managed processes. Individual test cases can also read this config file to discover server ports, hostnames, etc. As long as test cases do not hardcode these values they can later be turned against an external functional test environment.

#### config.json path

By default the ProcessManager looks for a 'config.json' file in the test suite's current working directory. An alternate location can be passed to its constructor:

```python
tm = atstf.ProcessManager(config_path="/tmp/config.json")
```

#### trafficserver and other paths

Test suites inside the trafficserver source repo only have to tell the ProcessManager the path to the source repo root and it will infer the rest:
```python
tm = atstf.ProcessManager(root_path="../..")
```

Test suites outside the trafficserver source repo have to explicitly set each of the following paths:
```python
tm = atstf.ProcessManager(ats_path="/usr/local/bin/trafficserver",
default_config_path="~/trafficserver/proxy/config",
origin_path="~/trafficserver/tests/framework/origin.py")
```

The last two paths (default_config_path and origin_path) are paths into a local trafficserver source repo, but we can potentially incorporate those into a 'devel' package in the future.

#### logging:

You can tell the ProcessManager to log stuff including the stdout and stderr of the processes it manages. The ProcessManager uses python's [logging module](https://docs.python.org/2/library/logging.html). If you configure python's logging module outside of ProcessManager, the ProcessManager will just use that config. If you don't want to configure the logging module you can tell the ProcessManager to do it for you by passing it a log_level (e.g., logging.INFO):

```python
import logging

tm = atstf.ProcessManager(log_level=logging.DEBUG)
```

#### max start time

By default the ProcessManager will wait for up to 5 seconds for the processes it manages to start. If any managed process does not start in time, the ProcessManager will throw an exception which should fail the test.

The ProcessManager considers a managed process to have started when it can successfully connect to all ports listed in the process's 'interfaces' section. More on this later.

To change the max start time:
```python
tm = atstf.ProcessManager(max_start_sec=3)
```

#### The config.json

The JSON configuration file used by the ProcessManager has the following common format:

{
"processes": {
: {
"type": <"ats"|"origin"|"python"|"other">,
"spawn":
"interfaces": {
: {
"hostname": ,
"port":
}
},

}
},

}

Because this must be valid JSON, process names must be unique. They will be incorporated into log messages and into exceptions thrown by the ProcessManager.

The process types supported by the ProcessManager are described below. Different process types may support additional configuration options.

The 'spawn' attribute defaults to true. If set to false the ProcessManager will not manage the process. This is intended to support turning a functional test case against an externally managed set of processes. The test case code can still discover hostnames and ports from the config.json without a code change.

The optional 'interfaces' attribute details which ports will be opened by a managed process. If specified the ProcessManager will block caller until either all ports are up or the 'max start time' is exceeded.

#### The config.json Process Types

The ProcessManager currently supports the following process types:

##### Process Type 'ats'

'ats' processes support additional config.json options.

Example:
```json
{
"servers": {
"ats1": {
"type": "ats",
"root": "ats1",
"interfaces": {
"http": {
"type": "http",
"hostname": "localhost",
"port": 8080
}
},
"config": {
"records.config": [
{
"match": "^CONFIG proxy.config.diags.debug.enabled",
"substitute": "CONFIG proxy.config.diags.debug.enabled INT 1"
}
],
"remap.config": [
{
"append": "map /foo/bar http://localhost:8082/foo/bar"
}
]
}
}
}
}
```

LIMITATION: Only the 'http' interface type is presently supported. Need to add HTTPS and SPDY ASAP.

The ProcessManager will run the trafficserver in the 'root' dir. This is the same as the trafficserver 'TS_ROOT' environment variable. If you configure multiple 'ats' processes, be sure to assign each of them a distinct root dir. The root dir should be relative to the test case dir and MUST NOT CONTAIN ANYTHING IMPORTANT! If ats crashes and the system is configured to capture cores, they should appear in the root dir.

Before trafficserver is started:

* The ProcessManager will rm -rf the 'root' dir if it already exists, then recreate it with a minimal directory structure (a few var dirs and an etc dir)
* The ProcessManager will copy over all the default config files and body factory files from proxy/config
* If the config.json has a 'config' section, the ProcessManager will transform the copied default config files accordingly. On a per config file basis:
* match/substitue directives will replace any line that matches a regex (match) with a substitution line
* append directives will simply add lines to the end of the config file.

After trafficserver is started:

* trafficserver will create transient files in the root/var dirs, including log files

##### Process Type 'origin'

A configurable origin server. Multiple types of 'origins' can be specified on a per-path basis, but currently only a 'chunkedresponse' type has been developed. The 'chunkedresponse' origin type supports the following configuration:

1. response status code
1. response headers
1. number of chunks to send in the response
1. size of each chunk
1. byte value of each byte in chunk
1. seconds to wait before sending the first chunk
1. seconds to wait before sending all subsequent chunks

Example:
```json
{
"servers": {
"origin1": {
"type": "origin",
"interfaces": {
"http": {
"type": "http",
"hostname": "localhost",
"port": 8082
}
},
"actions": {
"GET": {
"/foo/bar": {
"type": "chunkedresponse",
"status_code": 200,
"headers": {
"content-type": "text/plain"
},
"delay_first_chunk_sec": 1,
"chunk_size_bytes": 1024,
"num_chunks": 10,
"delay_between_chunk_sec": 0,
"chunk_byte_value": 99
}
}
}
}
}
}
```

##### Process Type 'python'

The ProcessManager can manage ad hoc origins written in python providing they exit cleanly when sent a SIGTERM. If they specify an 'interfaces' section then the ProcessManager will also wait for them to start.

Example:
```json
{
"servers": {
"python1": {
"type": "python",
"script": "ad_hoc_origin.py",
"args": [
"python1",
"config.json"
],
"interfaces": {
"http": {
"type": "http",
"hostname": "localhost",
"port": 8082
}
}
}
}
}
```

##### Process Type 'other'

Finally the ProcessManager can manage any executable, again providing it exits cleanly when sent a SIGTERM. If it specifies an 'interface' secion then the ProcessManager will also wait for it to start.

Example:
```json
{
"servers": {
"other1": {
"type": "other",
"spawn": false,
"executable": "/usr/local/apache2/bin/httpd",
"args": [
"-f",
"/usr/local/apache2/conf/httpd.conf",
"-k",
"start",
"-x"
],
"interfaces": {
"http": {
"type": "http",
"hostname": "localhost",
"port": 8082
}
}
}
}
}
```

## Writing Test Cases

### Using the 'unittest' and 'twisted' modules

Test cases should use [the Python Standard Library's unittest module](https://docs.python.org/2/library/unittest.html). Test cases also need to send traffic to trafficserver. This particular example uses the [Python twisted framework](https://twistedmatrix.com/trac/), but other approaches are possible.

The main advantage of twisted is that it can be used for lightweight load testing. The main disadvantage is that Python's unittest module implements assertions on top of exceptions. Failed assertions in twisted callbacks will be eaten by the twisted framework and will not be incorporated into the test results.

This example works around this issue by saving the state of its interaction with trafficserver, then performing all assertions in the main thread against the saved state at the end of the test case:

```python
import sys
import unittest

from twisted.internet import reactor
from twisted.internet.defer import Deferred
from twisted.internet.protocol import Protocol
from twisted.web.client import Agent
from twisted.web.http_headers import Headers

sys.path = ['../framework'] + sys.path
import atstf

class ExampleTwistedTest(unittest.TestCase):
def test_ats1(self):
#
# Read test case configuration from the same config file used to start ATS and origin processes.
#

conf = atstf.parse_config()

ats1_conf = conf['processes']['ats1']

hostname = ats1_conf['interfaces']['http']['hostname']
port = ats1_conf['interfaces']['http']['port']

origin_conf = conf['processes']['origin1']

expected_status_code = origin_conf['actions']['GET']['/foo/bar']['status_code']
chunk_size_bytes = origin_conf['actions']['GET']['/foo/bar']['chunk_size_bytes']
num_chunks = origin_conf['actions']['GET']['/foo/bar']['num_chunks']
expected_bytes_received = chunk_size_bytes * num_chunks

#
# Create a twisted HTTP client that will interact with ATS and save all details of the interaction. We'll
# later assert that the saved details are correct. Ideally we'd just throw assertions into the client itself,
# but sadly assertions are implemented with exceptions and twisted eats any exception raised by our callbacks.
#
# See https://twistedmatrix.com/documents/current/web/howto/client.html for details on writing HTTP clients
#

agent = Agent(reactor)

request = agent.request('GET', "http://%s:%d/foo/bar" % (hostname.encode("utf-8"), port),
Headers({'User-Agent': ['ExampleTwistedTest']}), None)

response_handler = ResponseHandler(reactor)

request.addCallback(response_handler.handle_response)
request.addCallback(response_handler.handle_completion)
request.addErrback(response_handler.handle_error)

reactor.run()

#
# Now assert
#

self.assertEqual(expected_status_code, response_handler.get_status_code())
self.assertEqual(expected_bytes_received, response_handler.get_body_handler().get_bytes_received())
self.assertEqual("Response body fully received", \
response_handler.get_body_handler().get_reason().getErrorMessage())

class ResponseBodyHandler(Protocol):
def __init__(self, deferred):
self.__deferred = deferred
self.__bytes_received = 0
self.__reason = None

def dataReceived(self, data):
self.__bytes_received += len(data)

def connectionLost(self, reason):
#
# This function name is highly misleading. According to the web client docs:
#
# When the body has been completely delivered, the protocol's connectionLost method is called. It is important
# to inspect the Failure passed to connectionLost . If the response body has been completely received, the
# failure will wrap a twisted.web.client.ResponseDone exception. This indicates that it is known that all data
# has been received. It is also possible for the failure to wrap a twisted.web.http.PotentialDataLoss exception:
# this indicates that the server framed the response such that there is no way to know when the entire response
# body has been received. Only HTTP/1.0 servers should behave this way. Finally, it is possible for the
# exception to be of another type, indicating guaranteed data loss for some reason (a lost connection, a memory
# error, etc).
#
self.__reason = reason
self.__deferred.callback(None)

def get_bytes_received(self):
return self.__bytes_received

def get_reason(self):
return self.__reason

class ResponseHandler:
def __init__(self, reactor):
self.__reactor = reactor
self.__deferred = Deferred()
self.__body_handler = ResponseBodyHandler(self.__deferred)
self.__error = None
self.__status_code = None

def handle_response(self, response):
self.__status_code = response.code
response.deliverBody(self.__body_handler)
return self.__deferred

def handle_completion(self):
self.__reactor.stop()

def handle_error(self, error):
self.__error = error
self.__reactor.stop()

def get_body_handler(self):
return self.__body_handler

def get_error(self):
return self.__error

def get_status_code(self):
return self.__status_code
```

### Pulling it all together with a Test Suite

Multiple test cases should be run together in a test suite and the results of the entire suite should be summarized in a JUnit report suitable for Jenkins. This is also a good place to instantiate and start the ProcessManager (start it before any test case is invoked, stop it after all test cases finish).

Example:
```python
#!/bin/env python

import sys
import logging

sys.path = ['../framework'] + sys.path
import atstf

from twisted_test import ExampleTwistedTest

test_cases = [ExampleTwistedTest]

if __name__ == '__main__':
# Spawn ATS and origin processes

tm = atstf.ProcessManager(root_path="../..", config_path="config.json", log_level=logging.DEBUG, max_start_sec=5)

try:
tm.start()

# Run the test suite and generate a JUnit XML report

if atstf.run_tests(test_cases=test_cases, name='example', report='report.xml'):
sys.exit(0)
else:
sys.exit(1)
finally:
tm.stop()
```

Here the important bits are:

1. Initializing the ProcessManager and starting it before the running the test suite
1. Passing all the test case classes to run_tests(). This will auto discover all test cases in the test classes, run them, and spit the results out in a report.xml file in the cwd.
1. Exiting zero if the run_tests() method returns True, non-zero otherwise

Not so important is explicitly stopping the ProcessManager. It's good practice, but the ProcessManager will stop automatically on process exit.